docs: split README into component docs, update jobkit dep
- Break README into README.md (overview), docs/daemon.md (pipeline stages, diagnostics, common issues), docs/notifications.md (notification daemon, IRC/Telegram modules) - Update jobkit dependency from local path to git URL Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
This commit is contained in:
Kent Overstreet
parent
45335de220
commit
908f8c9e52
5 changed files with 228 additions and 129 deletions
106
docs/daemon.md
Normal file
106
docs/daemon.md
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
# Memory daemon
|
||||
|
||||
The background daemon (`poc-memory daemon`) automatically processes
|
||||
session transcripts through a multi-stage pipeline, extracting
|
||||
experiences and facts into the knowledge graph.
|
||||
|
||||
## Starting
|
||||
|
||||
```bash
|
||||
poc-memory daemon # Start foreground
|
||||
poc-memory daemon install # Install systemd service + hooks
|
||||
```
|
||||
|
||||
## Pipeline stages
|
||||
|
||||
Each session file goes through these stages in order:
|
||||
|
||||
1. **find_stale_sessions** — stat-only scan for JSONL files >100KB,
|
||||
older than SESSION_STALE_SECS (default 120s). No file reads.
|
||||
|
||||
2. **segment splitting** — files with multiple compaction boundaries
|
||||
(`"This session is being continued"`) are split into segments.
|
||||
Each segment gets its own LLM job. Segment counts are cached in
|
||||
a `seg_cache` HashMap to avoid re-parsing large files every tick.
|
||||
|
||||
3. **experience-mine** — LLM extracts journal entries, observations,
|
||||
and experiences from each segment. Writes results to the store.
|
||||
Dedup key: `_mined-transcripts.md#f-{uuid}` (single-segment) or
|
||||
`_mined-transcripts.md#f-{uuid}.{N}` (multi-segment).
|
||||
|
||||
4. **fact-mine** — LLM extracts structured facts (names, dates,
|
||||
decisions, preferences). Only starts when all experience-mine
|
||||
work is done. Dedup key: `_facts-{uuid}`.
|
||||
|
||||
5. **whole-file key** — for multi-segment files, once all segments
|
||||
complete, a whole-file key is written so future ticks skip
|
||||
re-parsing.
|
||||
|
||||
## Resource management
|
||||
|
||||
LLM calls are gated by a jobkit resource pool (default 1 slot).
|
||||
This serializes API access and prevents memory pressure from
|
||||
concurrent store loads. MAX_NEW_PER_TICK (10) limits how many
|
||||
tasks are spawned per 60s watcher tick.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
### Log
|
||||
|
||||
```bash
|
||||
tail -f ~/.claude/memory/daemon.log
|
||||
```
|
||||
|
||||
JSON lines with `ts`, `job`, `event`, and `detail` fields.
|
||||
|
||||
### Understanding the tick line
|
||||
|
||||
```
|
||||
{"job":"session-watcher","event":"tick",
|
||||
"detail":"277 stale, 219 mined, 4 extract, 0 fact, 0 open"}
|
||||
```
|
||||
|
||||
| Field | Meaning |
|
||||
|---------|---------|
|
||||
| stale | Total session files on disk matching age+size criteria. This is a filesystem count — it does NOT decrease as sessions are mined. |
|
||||
| mined | Sessions with both experience-mine AND fact-mine complete. |
|
||||
| extract | Segments currently queued/running for experience-mine. |
|
||||
| fact | Sessions queued/running for fact-mine. |
|
||||
| open | Sessions still being written to (skipped). |
|
||||
|
||||
Progress = mined / stale. When mined equals stale, the backlog is clear.
|
||||
|
||||
### Checking pipeline health
|
||||
|
||||
```bash
|
||||
# Experience-mine completions (logged as "experience-mine", not "extract")
|
||||
grep "experience-mine.*completed" ~/.claude/memory/daemon.log | wc -l
|
||||
|
||||
# Errors
|
||||
grep "experience-mine.*failed" ~/.claude/memory/daemon.log | wc -l
|
||||
|
||||
# Store size and node count
|
||||
poc-memory status
|
||||
wc -c ~/.claude/memory/nodes.capnp
|
||||
```
|
||||
|
||||
## Common issues
|
||||
|
||||
**stale count never decreases**: Normal. It's a raw file count, not a
|
||||
backlog counter. Compare `mined` to `stale` for actual progress.
|
||||
|
||||
**Early failures ("claude exited exit status: 1")**: Oversized segments
|
||||
hitting the LLM context limit. The 150k-token size guard and segmented
|
||||
mining should prevent this. If it recurs, check segment sizes.
|
||||
|
||||
**Memory pressure (OOM)**: Each job loads the full capnp store. At
|
||||
200MB+ store size, concurrent jobs can spike to ~5GB. The resource pool
|
||||
serializes access, but if the pool size is increased, watch RSS.
|
||||
|
||||
**Segments not progressing**: The watcher memoizes segment counts in
|
||||
`seg_cache`. If a file is modified after caching (e.g., session resumed),
|
||||
the daemon won't see new segments until restarted.
|
||||
|
||||
**Extract jobs queued but 0 completed in log**: Completion events are
|
||||
logged under the `experience-mine` job name, not `extract`. The `extract`
|
||||
label is only used for queue events.
|
||||
103
docs/notifications.md
Normal file
103
docs/notifications.md
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
# Notification daemon
|
||||
|
||||
`poc-daemon` routes messages from communication modules and internal
|
||||
events through a hierarchical, activity-aware delivery system.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Communication modules Hooks
|
||||
+-----------------+ +-------------+
|
||||
| IRC (native) |--+ | poc-hook |
|
||||
| Telegram | | mpsc | (all events|
|
||||
+-----------------+ +------+ +------+------+
|
||||
| | |
|
||||
v | capnp-rpc
|
||||
+----------+ |
|
||||
| poc-daemon |
|
||||
| |
|
||||
| NotifyState <---------+
|
||||
| +-- type registry
|
||||
| +-- pending queue
|
||||
| +-- threshold lookup
|
||||
| +-- activity-aware delivery
|
||||
|
|
||||
| idle::State
|
||||
| +-- presence detection
|
||||
| +-- sleep/wake/dream modes
|
||||
| +-- tmux prompt injection
|
||||
+--------------------------
|
||||
```
|
||||
|
||||
## Notification types and urgency
|
||||
|
||||
Types are free-form hierarchical strings: `irc.mention.nick`,
|
||||
`irc.channel.bcachefs`, `telegram.kent`. Each has an urgency level:
|
||||
|
||||
| Level | Name | Meaning |
|
||||
|-------|---------|--------------------------------------|
|
||||
| 0 | ambient | Include in idle context only |
|
||||
| 1 | low | Deliver on next check |
|
||||
| 2 | normal | Deliver on next user interaction |
|
||||
| 3 | urgent | Interrupt immediately |
|
||||
|
||||
Per-type thresholds walk up the hierarchy: `irc.channel.bcachefs-ai`
|
||||
-> `irc.channel` -> `irc` -> default. Effective thresholds adjust by
|
||||
activity state: raised when focused, lowered when idle, only urgent
|
||||
when sleeping.
|
||||
|
||||
## Communication modules
|
||||
|
||||
**IRC** — native async TLS connection (tokio-rustls). Connects,
|
||||
joins channels, parses messages, generates notifications. Runtime
|
||||
commands: join, leave, send, status, log, nick. Per-channel logs
|
||||
at `~/.claude/irc/logs/`.
|
||||
|
||||
**Telegram** — native async HTTP long-polling (reqwest). Downloads
|
||||
media (photos, voice, documents). Chat ID filtering for security.
|
||||
Runtime commands: send, status, log.
|
||||
|
||||
Both modules persist config changes to `~/.claude/daemon.toml` —
|
||||
channel joins and nick changes survive restarts.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
poc-daemon # Start daemon
|
||||
poc-daemon status # State summary
|
||||
poc-daemon irc status # IRC module status
|
||||
poc-daemon irc send TARGET MSG # Send IRC message
|
||||
poc-daemon irc join CHANNEL # Join (persists to config)
|
||||
poc-daemon irc leave CHANNEL # Leave
|
||||
poc-daemon irc log [N] # Last N messages
|
||||
poc-daemon telegram status # Telegram module status
|
||||
poc-daemon telegram send MSG # Send Telegram message
|
||||
poc-daemon telegram log [N] # Last N messages
|
||||
poc-daemon notify TYPE URG MSG # Submit notification
|
||||
poc-daemon notifications [URG] # Get + drain pending
|
||||
poc-daemon notify-types # List all types
|
||||
poc-daemon notify-threshold T L # Set per-type threshold
|
||||
poc-daemon sleep / wake / quiet # Session management
|
||||
poc-daemon stop # Shut down
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Config: `~/.claude/daemon.toml`
|
||||
|
||||
```toml
|
||||
[irc]
|
||||
enabled = true
|
||||
server = "irc.oftc.net"
|
||||
port = 6697
|
||||
tls = true
|
||||
nick = "MyBot"
|
||||
user = "bot"
|
||||
realname = "My Bot"
|
||||
channels = ["#mychannel"]
|
||||
|
||||
[telegram]
|
||||
enabled = true
|
||||
token = "bot-token-here"
|
||||
chat_id = 123456789
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue