New placeholder that expands query keys one hop through the graph,
giving agents visibility into what's already connected to the nodes
they're working on. Excludes the query keys themselves so there's
no duplication with {{nodes}}.
Added to transfer (sees existing semantic nodes linked to episodes,
so it REFINEs instead of duplicating) and challenger (sees neighbor
context to find real evidence for/against claims).
Also removes find_existing_observations — superseded by the
per-segment dedup fix and this general-purpose placeholder.
When building the {{conversations}} placeholder for the observation
agent, search for existing nodes relevant to each conversation
fragment and include them in the prompt. Uses seed matching + one-hop
graph expansion to find the neighborhood, so the extractor sees what
the graph already knows about these topics.
This helps prevent duplicate extractions, but the deeper bug is that
select_conversation_fragments doesn't track which conversations have
already been processed — that's next.
The observation agent was re-extracting the same conversations every
consolidation run because select_conversation_fragments had no tracking
of what had already been processed.
Extract shared helpers from the fact miner's dedup pattern:
- transcript_key(prefix, path): namespaced key from prefix + filename
- segment_key(base, idx): per-segment key
- keys_with_prefix(prefix): bulk lookup from store
- unmined_segments(path, prefix, known): find unprocessed segments
- mark_segment(...): mark a segment as processed
Rewrite select_conversation_fragments to use these with
_observed-transcripts prefix. Each compaction segment within a
transcript is now tracked independently — new segments from ongoing
sessions get picked up, already-processed segments are skipped.
When connectivity shows isolated nodes, print copy-pasteable
poc-memory graph link-add commands targeting the highest-degree
node in the largest cluster. Closes the diagnose→fix loop.
BFS-based connectivity analysis as a query pipeline stage. Shows
connected components, islands, and sample paths between result nodes
through the full graph (max 4 hops).
poc-memory query "content ~ 'made love' | connectivity"
poc-memory query "(content ~ 'A' OR content ~ 'B') | connectivity"
Also documented in query --help.
render now appends a links section showing up to 15 neighbors as
copy-pasteable `poc-memory render` commands, making the graph
naturally walkable without memorizing node keys.
query --help now documents the full language: expressions, fields,
operators, pipe stages, functions, and examples. Inline help in
cmd_query replaced with pointer to --help.
Load store from both cache (rkyv/bincode) and raw capnp logs,
then diff: missing nodes, phantom nodes, version mismatches.
Auto-rebuilds cache if inconsistencies found.
This would have caught the mysterious the-plan deletion — likely
caused by a stale/corrupt snapshot that silently dropped the node
while the capnp log still had it.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
The Provenance enum couldn't represent agents defined outside the
source code. Replace it with a Text field in the capnp schema so any
agent can write its own provenance label (e.g. "extractor:write",
"rename:tombstone") without a code change.
Schema: rename old enum fields to provenanceOld, add new Text
provenance fields. Old enum kept for reading legacy records.
Migration: from_capnp_migrate() falls back to old enum when the
new text field is empty.
Also adds `poc-memory tail` command for viewing recent store writes.
Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
Nodes actively found by search now show "Search hits: N ← actively
found by search, prefer to keep" in both the node section (seen by
extractor, linker, etc.) and rename candidate listings.
Extractor and rename prompts updated to respect this signal — merge
into high-hit nodes rather than demoting them, skip renaming nodes
that are working well in search.
memory-search now records which nodes it finds via the daemon's
record-hits RPC endpoint. The daemon owns the redb database
exclusively, avoiding file locking between processes.
The rename agent reads hit counts to deprioritize nodes that are
actively being found by search — renaming them would break working
queries. Daily check decays counters by 10% so stale hits fade.
Also switched RPC command reading from fixed 256-byte buffer to
read_to_string for unbounded command sizes.
First use case: search hit tracking for rename protection. Nodes
that memory-search actively finds shouldn't be renamed.
The counters module provides increment/read/decay operations backed
by redb (pure Rust, ACID, no C deps). Next step: wire into the
poc-memory daemon via RPC so the daemon owns the DB exclusively
and memory-search sends hits via RPC.
Also reverts the JSONL search-hits approach in favor of this.
Instead of a hard 7-day cutoff, sort rename candidates so the
least-recently visited come first. Naturally prioritizes unseen
nodes while allowing revisits once everything's been through.
The daemon was getting stuck when a claude subprocess hung — no
completion logged, job blocked forever, pending queue growing.
Use spawn() + watchdog thread instead of blocking output(). The
watchdog sleeps in 1s increments checking a cancel flag, sends
SIGTERM at 5 minutes, SIGKILL after 5s grace. Cancel flag ensures
the watchdog exits promptly when the child finishes normally.
Any time an agent creates a new node (WRITE_NODE) or the fact miner
stores extracted facts, a naming sub-agent now checks for conflicts
and ensures the key is meaningful:
- find_conflicts() searches existing nodes via component matching
- Haiku LLM decides: CREATE (good name), RENAME (better name),
or MERGE_INTO (fold into existing node)
- WriteNode actions may be converted to Refine on MERGE_INTO
Also updates the rename agent to handle _facts-<UUID> nodes —
these are no longer skipped, and the prompt explains how to name
them based on their domain/claim content.
Default search now uses exact key match only. Component matching
(--fuzzy) and content search (--content) are explicit flags. This
makes missing graph structure visible instead of silently falling
back to broad matching.
Shift from pattern abstraction (creating new nodes) to distillation
(refining existing nodes, demoting redundancies). Priority order:
merge redundancies > file into existing > improve existing > create new.
Query changed to neighborhood-aware: seed → spread → limit, so the
extractor works on related nodes rather than random high-priority ones.
New action type that halves a node's weight (min 0.05), enabling
extractors to mark redundant nodes for decay without deleting them.
Parser, apply logic, depth computation, and display all updated.
The daemon's compute_graph_health had a duplicated copy of the
consolidation planning thresholds that had drifted from the canonical
version (α<2.0 → +7 replay in daemon vs +10 in neuro).
Split consolidation_plan into _inner(store, detect_interference) so
the daemon can call consolidation_plan_quick (skips O(n²) interference)
while using the same threshold logic.
- Add run_and_apply() — combines run_one_agent + action application
into one call. Used by daemon job_consolidation_agent and
consolidate_full, which had identical run+apply loops.
- Port split_plan_prompt() to use split.agent via defs::resolve_placeholders
instead of loading the separate split-plan.md template. Make
resolve_placeholders public for this.
- Delete prompts/split-plan.md — superseded by agents/split.agent
which was already the canonical definition.
- Add compact_timestamp() to store — replaces 5 copies of
format_datetime(now_epoch()).replace([':', '-', 'T'], "")
Also fixes missing seconds (format_datetime only had HH:MM).
- Add ConsolidationPlan::to_agent_runs() — replaces identical
plan-to-runs-list expansion in consolidate.rs and daemon.rs.
- Port job_rename_agent to use run_one_agent — eliminates manual
prompt building, LLM call, report storage, and visit recording
that duplicated the shared pipeline.
- Rename Confidence::weight()/value() to delta_weight()/gate_value()
to clarify the distinction (delta metrics vs depth gating).
Three places duplicated the agent execution loop (build prompt → call
LLM → store output → parse actions → record visits): consolidate.rs,
knowledge.rs, and daemon.rs. Extract into run_one_agent() in
knowledge.rs that all three now call.
Also standardize consolidation agent prompts to use WRITE_NODE/LINK/REFINE
— the same commands the parser handles. Previously agents output
CATEGORIZE/NOTE/EXTRACT/DIGEST/DIFFERENTIATE/MERGE/COMPRESS which were
silently dropped after the second-LLM-call removal.
The consolidation pipeline previously made a second Sonnet call to
extract structured JSON actions from agent reports. This was both
wasteful (extra LLM call per consolidation) and lossy (only extracted
links and manual items, ignoring WRITE_NODE/REFINE).
Now actions are parsed and applied inline after each agent runs, using
the same parse_all_actions() parser as the knowledge loop. The daemon
scheduler's separate apply phase is also removed.
Also deletes 8 superseded/orphaned prompt .md files (784 lines) that
have been replaced by .agent files.
These prompts are now embedded in their .agent files or no longer
called from any code path.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
The four knowledge agents (observation, extractor, connector,
challenger) were hardcoded in knowledge.rs with their own node
selection logic that bypassed the query pipeline and visit tracking.
Now they're .agent files like the consolidation agents:
- extractor: not-visited:extractor,7d | sort:priority | limit:20
- observation: uses new {{CONVERSATIONS}} placeholder
- connector: type:semantic | not-visited:connector,7d
- challenger: type:semantic | not-visited:challenger,14d
The knowledge loop's run_cycle dispatches through defs::run_agent
instead of calling hardcoded functions, so all agents get visit
tracking automatically. This means the extractor now sees _facts-*
and _mined-transcripts nodes that it was previously blind to.
~200 lines of dead code removed (old runner functions, spectral
clustering for node selection, per-agent LLM dispatch).
New placeholders in defs.rs:
- {{CONVERSATIONS}} — raw transcript fragments for observation agent
- {{TARGETS}} — alias for {{NODES}} (challenger compatibility)
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
Adds `poc-memory daemon run-agent <type> <count>` CLI command that
sends an RPC to the daemon to queue agent runs, instead of spawning
separate processes.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
The parser was using split_once("\n\n") which broke when the prompt
started immediately after the JSON header (no blank line). Parse
the first line as JSON, treat the rest as the prompt body.
All agents now go through the config-driven path via .agent files.
agent_prompt() just delegates to defs::run_agent(). Remove the 100+
line hardcoded match block and the _pub wrapper functions — make the
formatters pub directly.
Replace the formatter dispatch with a generic {{placeholder}} lookup
system. Placeholders in prompt templates are resolved at runtime from
a table: topology, nodes, episodes, health, pairs, rename, split.
The query in the header selects what to operate on (keys for visit
tracking); placeholders pull in formatted context. Placeholders that
produce their own node selection (pairs, rename) contribute keys back.
Port health, separator, rename, and split agents to .agent files.
All 7 agents now use the config-driven path.
Each agent is a .agent file: JSON config on the first line, blank line,
then the raw prompt markdown. Fully self-contained, fully readable.
No separate template files needed.
Agents dir: checked into repo at poc-memory/agents/. Code looks there
first (via CARGO_MANIFEST_DIR), falls back to ~/.claude/memory/agents/.
Three agents migrated: replay, linker, transfer.
Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
New append-only visits.capnp log records which agent processed which
node and when. Only recorded on successful completion — transient
errors don't mark nodes as "seen."
Schema: AgentVisit{nodeUuid, nodeKey, agent, timestamp, outcome}
Storage: append_visits(), replay_visits(), in-memory VisitIndex
Recording: daemon records visits after successful LLM call
API: agent_prompt() returns AgentBatch{prompt, node_keys} so callers
know which nodes to mark as visited.
Groundwork for using visit recency in agent node selection — agents
will deprioritize recently-visited nodes.
New `poc-memory dedup` command (--apply for live run, dry-run by
default). Finds nodes sharing the same key but different UUIDs,
classifies them as identical or diverged, picks a survivor
(prefer most edges, then highest version), tombstones the rest,
and redirects all edges from doomed UUIDs to the survivor.
All write paths (upsert_node, upsert_provenance, delete_node,
rename_node, ingest_units) now hold StoreLock across the full
refresh→check→write cycle. This prevents the race where two
concurrent processes both see a key as "new" and create separate
UUIDs for it.
Adds append_nodes_unlocked() and append_relations_unlocked() for
callers already holding the lock. Adds refresh_nodes() to replay
log tail under lock before deciding create vs update.
Also adds find_duplicates() for detecting existing duplicates
in the log (replays full log, groups live nodes by key).
- Refactor split from serial batch to independent per-node tasks
(run-agent split N spawns N parallel tasks, gated by llm_concurrency)
- Replace cosine similarity edge inheritance with agent-assigned
neighbors in the plan JSON — the LLM already understands the
semantic relationships, no need to approximate with bag-of-words
- Add --strict-mcp-config to claude CLI calls to skip MCP server
startup (saves ~5s per call)
- Remove hardcoded 2000-char split threshold — let the agent decide
what's worth splitting
- Reload store before mutations to handle concurrent split races
Episodic nodes (journal entries, digests) are narratives that should
not be split even when large. Only semantic reference nodes that have
grown to cover multiple topics are candidates.
Phase 1 sends a large node with its neighbor communities to the LLM
and gets back a JSON split plan (child keys, descriptions, section
hints). Phase 2 fires one extraction call per child in parallel —
each gets the full parent content and extracts/reorganizes just its
portion.
This handles arbitrarily large nodes because output is always
proportional to one child, not the whole parent. Tested on the kent
node (19K chars → 3 children totaling 20K chars with clean topic
separation).
New files:
prompts/split-plan.md — phase 1 planning prompt
prompts/split-extract.md — phase 2 extraction prompt
prompts/split.md — original single-phase (kept for reference)
Modified:
agents/prompts.rs — split_candidates(), split_plan_prompt(),
split_extract_prompt(), agent_prompt "split" arm
agents/daemon.rs — job_split_agent() two-phase implementation,
RPC dispatch for "split" agent type
tui.rs — added "split" to AGENT_TYPES
New consolidation agent that reads node content and generates semantic
3-5 word kebab-case keys, replacing auto-generated slugs (5K+ journal
entries with truncated first-line slugs, 2.5K mined transcripts with
opaque UUIDs).
Implementation:
- prompts/rename.md: agent prompt template with naming conventions
- prompts.rs: format_rename_candidates() selects nodes with long
auto-generated keys, newest first
- daemon.rs: job_rename_agent() parses RENAME actions from LLM
output and applies them directly via store.rename_node()
- Wired into RPC handler (run-agent rename) and TUI agent types
- Fix epoch_to_local panic on invalid timestamps (fallback to UTC)
Rename dramatically improves search: key-component matching on
"journal#2026-02-28-violin-dream-room" makes the node findable by
"violin", "dream", or "room" — the auto-slug was unsearchable.
Per-agent-type tabs (health, replay, linker, separator, transfer,
apply, orphans, cap, digest, digest-links, knowledge) with dynamic
visibility — tabs only appear when tasks or log history exist.
Features:
- Overview tab: health gauges (α, gini, cc, episodic%), in-flight
tasks, and recent log entries
- Pipeline tab: table with phase ordering and status
- Per-agent tabs: active tasks, output logs, log history
- Log tab: auto-scrolling daemon.log tail
- Vim-style count prefix: e.g. 5r runs 5 iterations of the agent
- Flash messages for RPC feedback
- Tab/Shift-Tab navigation, number keys for tab selection
Also adds run-agent RPC to the daemon: accepts agent type and
iteration count, spawns chained tasks with LLM resource pool.
poc-memory status launches TUI when stdout is a terminal and daemon
is running, falls back to text output otherwise.
match_seeds() previously only found nodes whose keys exactly matched
search terms. This meant searches like "formal verification" or
"bcachefs plan" returned nothing — no nodes are keyed with those
exact strings.
Three-tier matching strategy:
1. Exact key match (full weight) — unchanged
2. Key component match (0.5× weight) — split keys on -/_/./#,
match individual words. "plan" now finds "the-plan", "verification"
finds "c-to-rust-verification-workflow", etc.
3. Content match (0.2× weight, capped at 50 hits) — search node
content for terms that didn't match any key. Catches nodes whose
keys are opaque but whose content is relevant.
Also adds prompt-based seeding to the hook pipeline: extract_query_terms
from the user's prompt and merge into the term set. Previously the hook
only seeded from transcript scanning (finding node keys as substrings
in conversation history), which meant fresh sessions or queries about
new topics produced no search results at all.
Wire up the PostToolUse handler to call memory-search --hook, passing
through the hook JSON on stdin. This drains pending context chunks
saved by the initial UserPromptSubmit load, delivering them one per
tool call until all chunks are delivered.
Claude Code's hook output limit (~10K chars) was truncating the full
context load. Split output into chunks at section boundaries, deliver
first chunk on UserPromptSubmit, save remaining chunks to disk for
drip-feeding on subsequent PostToolUse calls.
Two-pass algorithm: split at "--- KEY (group) ---" boundaries, then
merge adjacent small sections up to 9K per chunk. Separates session_id
guard (needed for chunk state) from prompt guard (needed only for
search), so PostToolUse events without a prompt can still pop chunks.
mark_returned() was append-only without checking if the key already
existed, causing duplicates to accumulate across hook invocations.
load_returned() then returned all entries including duplicates, which
made the returned count exceed the seen count, causing a u64 underflow
in the pre-seeded calculation.
Fix: check load_returned() before appending in mark_returned(), dedup
on read in load_returned(), and use saturating_sub for the pre-seeded
count as a safety net.
Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
Spawn memory-search --hook as a subprocess, piping the hook input
JSON through stdin and printing its stdout. This ensures memory
context injection goes through the same hook whose output Claude
Code reliably persists, fixing the issue where memory-search as a
separate hook had its output silently dropped.
Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
Move JsonlBackwardIter and find_last_compaction() from
parse-claude-conversation into a shared transcript module. Both
memory-search and parse-claude-conversation now use the same robust
compaction detection: mmap-based backward scan, JSON parsing to
verify user-type message, content prefix check.
Replaces memory-search's old detect_compaction() which did a forward
scan with raw string matching on "continued from a previous
conversation" — that could false-positive on the string appearing
in assistant output or tool results.
Add parse-claude-conversation as a new binary for debugging what's
in the context window post-compaction.
Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
chrono's timestamp_opt can return None during DST transitions.
Handle all three variants (Single, Ambiguous, None) instead of
unwrapping. For DST gaps, offset by one hour to land in valid
local time.
Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
Replace monolithic consolidate job with individual agent jobs
(replay, linker, separator, transfer, health) that run sequentially
and store reports. Multi-phase daily pipeline: agent runs → apply
actions → link orphans → cap degree → digest → digest links →
knowledge loop.
Add GraphHealth struct with graph metrics (alpha, gini, clustering
coefficient, episodic ratio) computed during health checks. Display
in `poc-memory daemon status`. Use cached metrics to build
consolidation plan without expensive O(n²) interference detection.
Add RPC consolidate command to trigger consolidation via socket.
Harden session watcher: skip transcripts with zero segments, improve
migration error handling.
Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
