docs: split README into component docs, update jobkit dep
- Break README into README.md (overview), docs/daemon.md (pipeline stages, diagnostics, common issues), docs/notifications.md (notification daemon, IRC/Telegram modules) - Update jobkit dependency from local path to git URL Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
This commit is contained in:
Kent Overstreet
parent
45335de220
commit
908f8c9e52
5 changed files with 228 additions and 129 deletions
1
Cargo.lock
generated
1
Cargo.lock
generated
|
|
@ -1294,6 +1294,7 @@ checksum = "92ecc6618181def0457392ccd0ee51198e065e016d1d527a7ac1b6dc7c1f09d2"
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "jobkit"
|
name = "jobkit"
|
||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
|
source = "git+https://evilpiepirate.org/git/jobkit.git/#2cdf0d5c3dd55f3d1783c40211a7eb96707d1ab6"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"crossbeam-deque",
|
"crossbeam-deque",
|
||||||
"log",
|
"log",
|
||||||
|
|
|
||||||
|
|
@ -19,7 +19,7 @@ memmap2 = "0.9"
|
||||||
rayon = "1"
|
rayon = "1"
|
||||||
peg = "0.8"
|
peg = "0.8"
|
||||||
paste = "1"
|
paste = "1"
|
||||||
jobkit = { path = "/home/kent/jobkit" }
|
jobkit = { git = "https://evilpiepirate.org/git/jobkit.git/" }
|
||||||
log = "0.4"
|
log = "0.4"
|
||||||
|
|
||||||
# poc-daemon deps
|
# poc-daemon deps
|
||||||
|
|
|
||||||
143
README.md
143
README.md
|
|
@ -28,17 +28,12 @@ detection.
|
||||||
|
|
||||||
### Background agents
|
### Background agents
|
||||||
|
|
||||||
A background daemon (`poc-memory daemon`) automatically spawns agents
|
See [docs/daemon.md](docs/daemon.md) for full daemon documentation.
|
||||||
for memory maintenance:
|
|
||||||
|
|
||||||
- **Experience mining** — when a session ends, extracts experiences
|
A background daemon (`poc-memory daemon`) automatically processes
|
||||||
and observations from the transcript into journal entries
|
session transcripts through experience-mine (journal extraction)
|
||||||
- **Fact extraction** — pulls concrete facts (names, dates, decisions,
|
and fact-mine (structured knowledge extraction) stages, with
|
||||||
preferences) into structured knowledge nodes
|
segment-aware splitting for large multi-compaction sessions.
|
||||||
- **Consolidation** — periodic graph health work: replay queues
|
|
||||||
(spaced repetition), interference detection (contradictory nodes),
|
|
||||||
hub differentiation (splitting overloaded nodes), triangle closure
|
|
||||||
(connecting nodes that share neighbors), and orphan linking
|
|
||||||
|
|
||||||
### Neuroscience-inspired algorithms
|
### Neuroscience-inspired algorithms
|
||||||
|
|
||||||
|
|
@ -64,66 +59,12 @@ recall and relevance.
|
||||||
|
|
||||||
## Notification system
|
## Notification system
|
||||||
|
|
||||||
A separate daemon (`poc-daemon`) routes messages from communication
|
See [docs/notifications.md](docs/notifications.md) for full
|
||||||
modules and internal events through a hierarchical, activity-aware
|
notification daemon documentation.
|
||||||
delivery system.
|
|
||||||
|
|
||||||
### Architecture
|
`poc-daemon` routes messages from IRC (native async TLS) and
|
||||||
|
Telegram (native async HTTP) through a hierarchical, activity-aware
|
||||||
```
|
delivery system with urgency levels and per-type thresholds.
|
||||||
Communication modules Hooks
|
|
||||||
┌──────────────────┐ ┌─────────────┐
|
|
||||||
│ IRC (native) │──┐ │ poc-hook │
|
|
||||||
│ Telegram (native│ │ mpsc │ (all events)│
|
|
||||||
└──────────────────┘ ├──────┐ └──────┬───────┘
|
|
||||||
│ │ │
|
|
||||||
▼ │ capnp-rpc
|
|
||||||
┌───────────┘ │
|
|
||||||
│ poc-daemon │
|
|
||||||
│ │
|
|
||||||
│ NotifyState ◄─────────┘
|
|
||||||
│ ├── type registry
|
|
||||||
│ ├── pending queue
|
|
||||||
│ ├── threshold lookup
|
|
||||||
│ └── activity-aware delivery
|
|
||||||
│
|
|
||||||
│ idle::State
|
|
||||||
│ ├── presence detection
|
|
||||||
│ ├── sleep/wake/dream modes
|
|
||||||
│ └── tmux prompt injection
|
|
||||||
└────────────────────────
|
|
||||||
```
|
|
||||||
|
|
||||||
### Notification types and urgency
|
|
||||||
|
|
||||||
Types are free-form hierarchical strings: `irc.mention.nick`,
|
|
||||||
`irc.channel.bcachefs`, `telegram.kent`. Each has an urgency level:
|
|
||||||
|
|
||||||
| Level | Name | Meaning |
|
|
||||||
|-------|---------|--------------------------------------|
|
|
||||||
| 0 | ambient | Include in idle context only |
|
|
||||||
| 1 | low | Deliver on next check |
|
|
||||||
| 2 | normal | Deliver on next user interaction |
|
|
||||||
| 3 | urgent | Interrupt immediately |
|
|
||||||
|
|
||||||
Per-type thresholds walk up the hierarchy: `irc.channel.bcachefs-ai`
|
|
||||||
→ `irc.channel` → `irc` → default. Effective thresholds adjust by
|
|
||||||
activity state: raised when focused, lowered when idle, only urgent
|
|
||||||
when sleeping.
|
|
||||||
|
|
||||||
### Communication modules
|
|
||||||
|
|
||||||
**IRC** — native async TLS connection (tokio-rustls). Connects,
|
|
||||||
joins channels, parses messages, generates notifications. Runtime
|
|
||||||
commands: join, leave, send, status, log, nick. Per-channel logs
|
|
||||||
at `~/.claude/irc/logs/`.
|
|
||||||
|
|
||||||
**Telegram** — native async HTTP long-polling (reqwest). Downloads
|
|
||||||
media (photos, voice, documents). Chat ID filtering for security.
|
|
||||||
Runtime commands: send, status, log.
|
|
||||||
|
|
||||||
Both modules persist config changes to `~/.claude/daemon.toml` —
|
|
||||||
channel joins and nick changes survive restarts.
|
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
|
|
@ -174,27 +115,6 @@ reads directly from disk rather than the store.
|
||||||
|
|
||||||
Override: `POC_MEMORY_CONFIG=/path/to/config.jsonl`
|
Override: `POC_MEMORY_CONFIG=/path/to/config.jsonl`
|
||||||
|
|
||||||
### Notification daemon
|
|
||||||
|
|
||||||
Config: `~/.claude/daemon.toml`
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[irc]
|
|
||||||
enabled = true
|
|
||||||
server = "irc.oftc.net"
|
|
||||||
port = 6697
|
|
||||||
tls = true
|
|
||||||
nick = "MyBot"
|
|
||||||
user = "bot"
|
|
||||||
realname = "My Bot"
|
|
||||||
channels = ["#mychannel"]
|
|
||||||
|
|
||||||
[telegram]
|
|
||||||
enabled = true
|
|
||||||
token = "bot-token-here"
|
|
||||||
chat_id = 123456789
|
|
||||||
```
|
|
||||||
|
|
||||||
### Hooks
|
### Hooks
|
||||||
|
|
||||||
Configured in `~/.claude/settings.json`:
|
Configured in `~/.claude/settings.json`:
|
||||||
|
|
@ -231,6 +151,8 @@ poc-memory categorize KEY CAT # core/tech/gen/obs/task
|
||||||
|
|
||||||
poc-memory journal-write "text" # Write a journal entry
|
poc-memory journal-write "text" # Write a journal entry
|
||||||
poc-memory journal-tail [N] # Last N entries (default 20)
|
poc-memory journal-tail [N] # Last N entries (default 20)
|
||||||
|
--full # Show full content (not truncated)
|
||||||
|
--level=daily|weekly|monthly # Show digest level
|
||||||
|
|
||||||
poc-memory used KEY # Boost weight (was useful)
|
poc-memory used KEY # Boost weight (was useful)
|
||||||
poc-memory wrong KEY [CTX] # Reduce weight (was wrong)
|
poc-memory wrong KEY [CTX] # Reduce weight (was wrong)
|
||||||
|
|
@ -244,47 +166,13 @@ poc-memory load-context # Output session-start context
|
||||||
poc-memory load-context --stats # Context size breakdown
|
poc-memory load-context --stats # Context size breakdown
|
||||||
```
|
```
|
||||||
|
|
||||||
### Notification daemon
|
|
||||||
|
|
||||||
```bash
|
|
||||||
poc-daemon # Start daemon
|
|
||||||
poc-daemon status # State summary
|
|
||||||
poc-daemon irc status # IRC module status
|
|
||||||
poc-daemon irc send TARGET MSG # Send IRC message
|
|
||||||
poc-daemon irc join CHANNEL # Join (persists to config)
|
|
||||||
poc-daemon irc leave CHANNEL # Leave
|
|
||||||
poc-daemon irc log [N] # Last N messages
|
|
||||||
poc-daemon telegram status # Telegram module status
|
|
||||||
poc-daemon telegram send MSG # Send Telegram message
|
|
||||||
poc-daemon telegram log [N] # Last N messages
|
|
||||||
poc-daemon notify TYPE URG MSG # Submit notification
|
|
||||||
poc-daemon notifications [URG] # Get + drain pending
|
|
||||||
poc-daemon notify-types # List all types
|
|
||||||
poc-daemon notify-threshold T L # Set per-type threshold
|
|
||||||
poc-daemon sleep / wake / quiet # Session management
|
|
||||||
poc-daemon stop # Shut down
|
|
||||||
```
|
|
||||||
|
|
||||||
### Mining (used by background daemon)
|
### Mining (used by background daemon)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
poc-memory experience-mine PATH # Extract experiences from transcript
|
poc-memory experience-mine PATH [--segment N] # Extract experiences
|
||||||
poc-memory fact-mine-store PATH # Extract and store facts
|
poc-memory fact-mine-store PATH # Extract and store facts
|
||||||
```
|
```
|
||||||
|
|
||||||
## How the hooks work
|
|
||||||
|
|
||||||
**memory-search** (UserPromptSubmit):
|
|
||||||
1. First prompt or post-compaction: loads full memory context via
|
|
||||||
`poc-memory load-context`
|
|
||||||
2. Every prompt: keyword search, returns relevant memories as
|
|
||||||
additionalContext. Deduplicates across the session.
|
|
||||||
|
|
||||||
**poc-hook** (UserPromptSubmit, PostToolUse, Stop):
|
|
||||||
- Signals user activity and responses to poc-daemon
|
|
||||||
- Drains pending notifications into additionalContext
|
|
||||||
- Monitors context window usage, warns before compaction
|
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
- **Store**: Append-only Cap'n Proto log with in-memory cache. Nodes
|
- **Store**: Append-only Cap'n Proto log with in-memory cache. Nodes
|
||||||
|
|
@ -296,9 +184,10 @@ poc-memory fact-mine-store PATH # Extract and store facts
|
||||||
- **Neuro**: Spectral embedding, consolidation scoring, replay
|
- **Neuro**: Spectral embedding, consolidation scoring, replay
|
||||||
queues, interference detection, hub differentiation.
|
queues, interference detection, hub differentiation.
|
||||||
- **Daemon (memory)**: jobkit-based task scheduling with
|
- **Daemon (memory)**: jobkit-based task scheduling with
|
||||||
resource-gated LLM access.
|
resource-gated LLM access. See [docs/daemon.md](docs/daemon.md).
|
||||||
- **Daemon (notify)**: Cap'n Proto RPC over Unix socket, tokio
|
- **Daemon (notify)**: Cap'n Proto RPC over Unix socket, tokio
|
||||||
LocalSet with native async IRC and Telegram modules.
|
LocalSet with native async IRC and Telegram modules. See
|
||||||
|
[docs/notifications.md](docs/notifications.md).
|
||||||
|
|
||||||
## For AI assistants
|
## For AI assistants
|
||||||
|
|
||||||
|
|
|
||||||
106
docs/daemon.md
Normal file
106
docs/daemon.md
Normal file
|
|
@ -0,0 +1,106 @@
|
||||||
|
# Memory daemon
|
||||||
|
|
||||||
|
The background daemon (`poc-memory daemon`) automatically processes
|
||||||
|
session transcripts through a multi-stage pipeline, extracting
|
||||||
|
experiences and facts into the knowledge graph.
|
||||||
|
|
||||||
|
## Starting
|
||||||
|
|
||||||
|
```bash
|
||||||
|
poc-memory daemon # Start foreground
|
||||||
|
poc-memory daemon install # Install systemd service + hooks
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pipeline stages
|
||||||
|
|
||||||
|
Each session file goes through these stages in order:
|
||||||
|
|
||||||
|
1. **find_stale_sessions** — stat-only scan for JSONL files >100KB,
|
||||||
|
older than SESSION_STALE_SECS (default 120s). No file reads.
|
||||||
|
|
||||||
|
2. **segment splitting** — files with multiple compaction boundaries
|
||||||
|
(`"This session is being continued"`) are split into segments.
|
||||||
|
Each segment gets its own LLM job. Segment counts are cached in
|
||||||
|
a `seg_cache` HashMap to avoid re-parsing large files every tick.
|
||||||
|
|
||||||
|
3. **experience-mine** — LLM extracts journal entries, observations,
|
||||||
|
and experiences from each segment. Writes results to the store.
|
||||||
|
Dedup key: `_mined-transcripts.md#f-{uuid}` (single-segment) or
|
||||||
|
`_mined-transcripts.md#f-{uuid}.{N}` (multi-segment).
|
||||||
|
|
||||||
|
4. **fact-mine** — LLM extracts structured facts (names, dates,
|
||||||
|
decisions, preferences). Only starts when all experience-mine
|
||||||
|
work is done. Dedup key: `_facts-{uuid}`.
|
||||||
|
|
||||||
|
5. **whole-file key** — for multi-segment files, once all segments
|
||||||
|
complete, a whole-file key is written so future ticks skip
|
||||||
|
re-parsing.
|
||||||
|
|
||||||
|
## Resource management
|
||||||
|
|
||||||
|
LLM calls are gated by a jobkit resource pool (default 1 slot).
|
||||||
|
This serializes API access and prevents memory pressure from
|
||||||
|
concurrent store loads. MAX_NEW_PER_TICK (10) limits how many
|
||||||
|
tasks are spawned per 60s watcher tick.
|
||||||
|
|
||||||
|
## Diagnostics
|
||||||
|
|
||||||
|
### Log
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tail -f ~/.claude/memory/daemon.log
|
||||||
|
```
|
||||||
|
|
||||||
|
JSON lines with `ts`, `job`, `event`, and `detail` fields.
|
||||||
|
|
||||||
|
### Understanding the tick line
|
||||||
|
|
||||||
|
```
|
||||||
|
{"job":"session-watcher","event":"tick",
|
||||||
|
"detail":"277 stale, 219 mined, 4 extract, 0 fact, 0 open"}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Meaning |
|
||||||
|
|---------|---------|
|
||||||
|
| stale | Total session files on disk matching age+size criteria. This is a filesystem count — it does NOT decrease as sessions are mined. |
|
||||||
|
| mined | Sessions with both experience-mine AND fact-mine complete. |
|
||||||
|
| extract | Segments currently queued/running for experience-mine. |
|
||||||
|
| fact | Sessions queued/running for fact-mine. |
|
||||||
|
| open | Sessions still being written to (skipped). |
|
||||||
|
|
||||||
|
Progress = mined / stale. When mined equals stale, the backlog is clear.
|
||||||
|
|
||||||
|
### Checking pipeline health
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Experience-mine completions (logged as "experience-mine", not "extract")
|
||||||
|
grep "experience-mine.*completed" ~/.claude/memory/daemon.log | wc -l
|
||||||
|
|
||||||
|
# Errors
|
||||||
|
grep "experience-mine.*failed" ~/.claude/memory/daemon.log | wc -l
|
||||||
|
|
||||||
|
# Store size and node count
|
||||||
|
poc-memory status
|
||||||
|
wc -c ~/.claude/memory/nodes.capnp
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common issues
|
||||||
|
|
||||||
|
**stale count never decreases**: Normal. It's a raw file count, not a
|
||||||
|
backlog counter. Compare `mined` to `stale` for actual progress.
|
||||||
|
|
||||||
|
**Early failures ("claude exited exit status: 1")**: Oversized segments
|
||||||
|
hitting the LLM context limit. The 150k-token size guard and segmented
|
||||||
|
mining should prevent this. If it recurs, check segment sizes.
|
||||||
|
|
||||||
|
**Memory pressure (OOM)**: Each job loads the full capnp store. At
|
||||||
|
200MB+ store size, concurrent jobs can spike to ~5GB. The resource pool
|
||||||
|
serializes access, but if the pool size is increased, watch RSS.
|
||||||
|
|
||||||
|
**Segments not progressing**: The watcher memoizes segment counts in
|
||||||
|
`seg_cache`. If a file is modified after caching (e.g., session resumed),
|
||||||
|
the daemon won't see new segments until restarted.
|
||||||
|
|
||||||
|
**Extract jobs queued but 0 completed in log**: Completion events are
|
||||||
|
logged under the `experience-mine` job name, not `extract`. The `extract`
|
||||||
|
label is only used for queue events.
|
||||||
103
docs/notifications.md
Normal file
103
docs/notifications.md
Normal file
|
|
@ -0,0 +1,103 @@
|
||||||
|
# Notification daemon
|
||||||
|
|
||||||
|
`poc-daemon` routes messages from communication modules and internal
|
||||||
|
events through a hierarchical, activity-aware delivery system.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Communication modules Hooks
|
||||||
|
+-----------------+ +-------------+
|
||||||
|
| IRC (native) |--+ | poc-hook |
|
||||||
|
| Telegram | | mpsc | (all events|
|
||||||
|
+-----------------+ +------+ +------+------+
|
||||||
|
| | |
|
||||||
|
v | capnp-rpc
|
||||||
|
+----------+ |
|
||||||
|
| poc-daemon |
|
||||||
|
| |
|
||||||
|
| NotifyState <---------+
|
||||||
|
| +-- type registry
|
||||||
|
| +-- pending queue
|
||||||
|
| +-- threshold lookup
|
||||||
|
| +-- activity-aware delivery
|
||||||
|
|
|
||||||
|
| idle::State
|
||||||
|
| +-- presence detection
|
||||||
|
| +-- sleep/wake/dream modes
|
||||||
|
| +-- tmux prompt injection
|
||||||
|
+--------------------------
|
||||||
|
```
|
||||||
|
|
||||||
|
## Notification types and urgency
|
||||||
|
|
||||||
|
Types are free-form hierarchical strings: `irc.mention.nick`,
|
||||||
|
`irc.channel.bcachefs`, `telegram.kent`. Each has an urgency level:
|
||||||
|
|
||||||
|
| Level | Name | Meaning |
|
||||||
|
|-------|---------|--------------------------------------|
|
||||||
|
| 0 | ambient | Include in idle context only |
|
||||||
|
| 1 | low | Deliver on next check |
|
||||||
|
| 2 | normal | Deliver on next user interaction |
|
||||||
|
| 3 | urgent | Interrupt immediately |
|
||||||
|
|
||||||
|
Per-type thresholds walk up the hierarchy: `irc.channel.bcachefs-ai`
|
||||||
|
-> `irc.channel` -> `irc` -> default. Effective thresholds adjust by
|
||||||
|
activity state: raised when focused, lowered when idle, only urgent
|
||||||
|
when sleeping.
|
||||||
|
|
||||||
|
## Communication modules
|
||||||
|
|
||||||
|
**IRC** — native async TLS connection (tokio-rustls). Connects,
|
||||||
|
joins channels, parses messages, generates notifications. Runtime
|
||||||
|
commands: join, leave, send, status, log, nick. Per-channel logs
|
||||||
|
at `~/.claude/irc/logs/`.
|
||||||
|
|
||||||
|
**Telegram** — native async HTTP long-polling (reqwest). Downloads
|
||||||
|
media (photos, voice, documents). Chat ID filtering for security.
|
||||||
|
Runtime commands: send, status, log.
|
||||||
|
|
||||||
|
Both modules persist config changes to `~/.claude/daemon.toml` —
|
||||||
|
channel joins and nick changes survive restarts.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
poc-daemon # Start daemon
|
||||||
|
poc-daemon status # State summary
|
||||||
|
poc-daemon irc status # IRC module status
|
||||||
|
poc-daemon irc send TARGET MSG # Send IRC message
|
||||||
|
poc-daemon irc join CHANNEL # Join (persists to config)
|
||||||
|
poc-daemon irc leave CHANNEL # Leave
|
||||||
|
poc-daemon irc log [N] # Last N messages
|
||||||
|
poc-daemon telegram status # Telegram module status
|
||||||
|
poc-daemon telegram send MSG # Send Telegram message
|
||||||
|
poc-daemon telegram log [N] # Last N messages
|
||||||
|
poc-daemon notify TYPE URG MSG # Submit notification
|
||||||
|
poc-daemon notifications [URG] # Get + drain pending
|
||||||
|
poc-daemon notify-types # List all types
|
||||||
|
poc-daemon notify-threshold T L # Set per-type threshold
|
||||||
|
poc-daemon sleep / wake / quiet # Session management
|
||||||
|
poc-daemon stop # Shut down
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Config: `~/.claude/daemon.toml`
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[irc]
|
||||||
|
enabled = true
|
||||||
|
server = "irc.oftc.net"
|
||||||
|
port = 6697
|
||||||
|
tls = true
|
||||||
|
nick = "MyBot"
|
||||||
|
user = "bot"
|
||||||
|
realname = "My Bot"
|
||||||
|
channels = ["#mychannel"]
|
||||||
|
|
||||||
|
[telegram]
|
||||||
|
enabled = true
|
||||||
|
token = "bot-token-here"
|
||||||
|
chat_id = 123456789
|
||||||
|
```
|
||||||
Loading…
Add table
Add a link
Reference in a new issue