Commit graph

24 commits

Author SHA1 Message Date
Kent Overstreet
d7436b8b9c llm: catch empty and rate-limited responses as errors
Empty stdout and Claude's rate limit message were silently returned
as successful 0-byte responses. Now detected and reported as errors.

Also skip transcript segments with fewer than 2 assistant messages
(rate-limited sessions, stub conversations).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 19:28:13 -04:00
Kent Overstreet
7fe55e28bd poc-memory agent run --debug: dump prompt and response
Add --debug flag that prints the full prompt and LLM response to
stdout, making it easy to iterate on agent prompts. Also adds
prompt field to AgentResult so callers can inspect what was sent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 19:13:43 -04:00
Kent Overstreet
c7509a0c2d agents: log raw LLM output to files, not graph nodes
Raw agent responses were being stored as nodes in the graph
(_consolidate-*, _knowledge-*), creating thousands of nodes per day
that polluted search results and bloated the store. Now logged to
~/.claude/memory/llm-logs/<agent>/<timestamp>.txt instead.

Node creation should only happen through explicit agent actions
(WRITE_NODE, REFINE) or direct poc-memory write tool calls.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:44:48 -04:00
Kent Overstreet
23cd80a0c3 observation: use transcript progress log, mark after success
Wire select_conversation_fragments to use store.is_segment_mined()
instead of scanning _observed-transcripts stub nodes. Segments are
now marked AFTER the agent succeeds (via mark_observation_done),
not before — so failed runs don't lose segments.

Fragment IDs flow through the Resolved.keys → AgentBatch.node_keys
path so run_and_apply_with_log can mark them post-success.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 17:44:20 -04:00
Kent Overstreet
8014b1111e agents: node:KEY placeholder, content-based report naming
- Add {{node:KEY}} placeholder resolver — agents can inline any graph
  node's content in their prompts. Used for shared instructions.
- Remove hardcoded identity preamble from defs.rs — agents now pull
  identity and instructions from the graph via {{node:core-personality}}
  and {{node:memory-instructions-core}}.
- Agent output report keys now include a content slug extracted from
  the first line of LLM output, making them human-readable
  (e.g. _consolidate-distill-20260316T014739-distillation-run-complete).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 17:09:41 -04:00
ProofOfConcept
83342897c8 experience-mine: link at creation time, remove # from new keys
Update the experience mining prompt to output links alongside journal
entries. The LLM now returns a "links" array per entry pointing to
existing semantic nodes. Rust code creates the links immediately after
node creation — new nodes arrive pre-connected instead of orphaned.

Also: remove # from all key generation paths (experience miner,
digest section keys, observed transcript keys). New nodes get clean
dash-separated keys.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 16:25:31 -04:00
ProofOfConcept
2d1edffdeb knowledge: fix action parsers for markdown-formatted LLM output
Linker agents output **LINK** (bold) with backtick-wrapped keys, and
**WRITE_NODE**/**END_NODE** with bold markers. The parsers expected
plain LINK/WRITE_NODE without markdown formatting, silently dropping
all actions from tool-enabled agents.

Updated regexes to accept optional ** bold markers and backtick key
wrapping. Also reverted per-link Jaccard computation (too expensive
in batch) — normalize-strengths should be run periodically instead.

This was causing ~600 links and ~40 new semantic nodes per overnight
batch to be silently lost.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 12:34:15 -04:00
ProofOfConcept
4cacfa7599 organize: fine-grained agent logging + cluster size cap
Add progress callback to run_one_agent and run_and_apply so callers
can see: prompt size, node list, LLM call timing, parsed action
count, and per-action applied/skipped status. Daemon writes these
to the persistent event log via log_event.

Cap organize cluster to 20 nodes - 126 nodes produced a 682KB
prompt that timed out every time. Agent has tools to explore
further if needed. Restore general query for production runs.
2026-03-13 20:25:19 -04:00
ProofOfConcept
bcf13c564a agents: tool-enabled LLM calls + DELETE action support
Add call_for_def() that threads model and tools from agent definitions
through to claude CLI. Tool-enabled agents get --allowedTools instead
of --tools "" and a longer 15-minute timeout for multi-turn work.

Add ActionKind::Delete with parse/apply support so agents can delete
nodes (used by organize agent for deduplication).

Use call_for_def() in run_one_agent instead of hardcoded call_sonnet.
2026-03-13 18:50:06 -04:00
Kent Overstreet
1da712874b memory-search: add --query mode and prompt key boost
Two changes:

1. New -q/--query flag for direct search without hook machinery.
   Useful for debugging: memory-search -q inner-life-sexuality-intimacy
   shows seeds, spread results, and rankings.

2. Prompt key boost: when the current prompt contains a node key
   (>=5 chars) as a substring, boost that term by +10.0. This ensures
   explicit mentions fire as strong seeds for spread, while the graph
   still determines what gets pulled in.

Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
2026-03-13 15:26:35 -04:00
ProofOfConcept
7bf4fbe0ec add {{siblings}} placeholder for graph neighborhood context
New placeholder that expands query keys one hop through the graph,
giving agents visibility into what's already connected to the nodes
they're working on. Excludes the query keys themselves so there's
no duplication with {{nodes}}.

Added to transfer (sees existing semantic nodes linked to episodes,
so it REFINEs instead of duplicating) and challenger (sees neighbor
context to find real evidence for/against claims).

Also removes find_existing_observations — superseded by the
per-segment dedup fix and this general-purpose placeholder.
2026-03-12 18:08:58 -04:00
ProofOfConcept
b3cf934c18 conversations placeholder: show graph neighborhood to extractor
When building the {{conversations}} placeholder for the observation
agent, search for existing nodes relevant to each conversation
fragment and include them in the prompt. Uses seed matching + one-hop
graph expansion to find the neighborhood, so the extractor sees what
the graph already knows about these topics.

This helps prevent duplicate extractions, but the deeper bug is that
select_conversation_fragments doesn't track which conversations have
already been processed — that's next.
2026-03-12 18:03:52 -04:00
ProofOfConcept
10499a98ea observation extractor: per-segment dedup using shared transcript helpers
The observation agent was re-extracting the same conversations every
consolidation run because select_conversation_fragments had no tracking
of what had already been processed.

Extract shared helpers from the fact miner's dedup pattern:
  - transcript_key(prefix, path): namespaced key from prefix + filename
  - segment_key(base, idx): per-segment key
  - keys_with_prefix(prefix): bulk lookup from store
  - unmined_segments(path, prefix, known): find unprocessed segments
  - mark_segment(...): mark a segment as processed

Rewrite select_conversation_fragments to use these with
_observed-transcripts prefix. Each compaction segment within a
transcript is now tracked independently — new segments from ongoing
sessions get picked up, already-processed segments are skipped.
2026-03-12 18:03:52 -04:00
Kent Overstreet
d76b14dfcd provenance: convert from enum to freeform string
The Provenance enum couldn't represent agents defined outside the
source code. Replace it with a Text field in the capnp schema so any
agent can write its own provenance label (e.g. "extractor:write",
"rename:tombstone") without a code change.

Schema: rename old enum fields to provenanceOld, add new Text
provenance fields. Old enum kept for reading legacy records.
Migration: from_capnp_migrate() falls back to old enum when the
new text field is empty.

Also adds `poc-memory tail` command for viewing recent store writes.

Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
2026-03-11 01:19:52 -04:00
ProofOfConcept
11cbd9664a naming: strip backticks from Haiku responses
Haiku sometimes wraps its CREATE/RENAME/MERGE_INTO lines in
backticks. Strip them before parsing so the response is recognized.
2026-03-10 23:40:38 -04:00
ProofOfConcept
b62fffc326 naming agent: resolve node names before creation
Any time an agent creates a new node (WRITE_NODE) or the fact miner
stores extracted facts, a naming sub-agent now checks for conflicts
and ensures the key is meaningful:

- find_conflicts() searches existing nodes via component matching
- Haiku LLM decides: CREATE (good name), RENAME (better name),
  or MERGE_INTO (fold into existing node)
- WriteNode actions may be converted to Refine on MERGE_INTO

Also updates the rename agent to handle _facts-<UUID> nodes —
these are no longer skipped, and the prompt explains how to name
them based on their domain/claim content.
2026-03-10 23:23:14 -04:00
ProofOfConcept
9d29e392a8 agents: add DEMOTE action for redundancy cleanup
New action type that halves a node's weight (min 0.05), enabling
extractors to mark redundant nodes for decay without deleting them.

Parser, apply logic, depth computation, and display all updated.
2026-03-10 22:57:02 -04:00
ProofOfConcept
945865f594 agents: extract run_and_apply, eliminate dead split-plan.md
- Add run_and_apply() — combines run_one_agent + action application
  into one call. Used by daemon job_consolidation_agent and
  consolidate_full, which had identical run+apply loops.

- Port split_plan_prompt() to use split.agent via defs::resolve_placeholders
  instead of loading the separate split-plan.md template. Make
  resolve_placeholders public for this.

- Delete prompts/split-plan.md — superseded by agents/split.agent
  which was already the canonical definition.
2026-03-10 17:51:32 -04:00
ProofOfConcept
abab85d249 agents: deduplicate timestamps, plan expansion, rename agent
- Add compact_timestamp() to store — replaces 5 copies of
  format_datetime(now_epoch()).replace([':', '-', 'T'], "")
  Also fixes missing seconds (format_datetime only had HH:MM).

- Add ConsolidationPlan::to_agent_runs() — replaces identical
  plan-to-runs-list expansion in consolidate.rs and daemon.rs.

- Port job_rename_agent to use run_one_agent — eliminates manual
  prompt building, LLM call, report storage, and visit recording
  that duplicated the shared pipeline.

- Rename Confidence::weight()/value() to delta_weight()/gate_value()
  to clarify the distinction (delta metrics vs depth gating).
2026-03-10 17:48:00 -04:00
ProofOfConcept
fe7f636ad3 agents: extract shared run_one_agent, standardize output formats
Three places duplicated the agent execution loop (build prompt → call
LLM → store output → parse actions → record visits): consolidate.rs,
knowledge.rs, and daemon.rs. Extract into run_one_agent() in
knowledge.rs that all three now call.

Also standardize consolidation agent prompts to use WRITE_NODE/LINK/REFINE
— the same commands the parser handles. Previously agents output
CATEGORIZE/NOTE/EXTRACT/DIGEST/DIFFERENTIATE/MERGE/COMPRESS which were
silently dropped after the second-LLM-call removal.
2026-03-10 17:33:12 -04:00
ProofOfConcept
91878d17a0 agents: port knowledge agents to .agent files with visit tracking
The four knowledge agents (observation, extractor, connector,
challenger) were hardcoded in knowledge.rs with their own node
selection logic that bypassed the query pipeline and visit tracking.

Now they're .agent files like the consolidation agents:
- extractor: not-visited:extractor,7d | sort:priority | limit:20
- observation: uses new {{CONVERSATIONS}} placeholder
- connector: type:semantic | not-visited:connector,7d
- challenger: type:semantic | not-visited:challenger,14d

The knowledge loop's run_cycle dispatches through defs::run_agent
instead of calling hardcoded functions, so all agents get visit
tracking automatically. This means the extractor now sees _facts-*
and _mined-transcripts nodes that it was previously blind to.

~200 lines of dead code removed (old runner functions, spectral
clustering for node selection, per-agent LLM dispatch).

New placeholders in defs.rs:
- {{CONVERSATIONS}} — raw transcript fragments for observation agent
- {{TARGETS}} — alias for {{NODES}} (challenger compatibility)

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-10 17:04:44 -04:00
ProofOfConcept
92f3ba5acf extract shared transcript parser and similarity matching helpers
- New agents/transcript.rs: shared JSONL parsing for enrich, fact_mine,
  and knowledge (was 3 separate implementations, ~150 lines duplicated)
- New best_match() and section_children() helpers in neuro/rewrite.rs
  (was duplicated find-best-by-similarity loop + section collection)
- Net -153 lines
2026-03-08 21:42:53 -04:00
ProofOfConcept
7c491e92eb tighten module interfaces: explicit re-exports, private helpers, inline dedup
- Replace `pub use types::*` in store/mod.rs with explicit re-export list
- Make transcript_dedup_key private in agents/enrich.rs (only used internally)
- Inline duplicated projects_dir() helper in agents/knowledge.rs and daemon.rs
2026-03-08 21:36:47 -04:00
ProofOfConcept
cee9b76a7b move LLM-dependent modules into agents/ subdir
Separate the agent layer (everything that calls external LLMs or
orchestrates sequences of such calls) from core graph infrastructure.

agents/: llm, prompts, audit, consolidate, knowledge, enrich,
         fact_mine, digest, daemon

Root: store/, graph, spectral, search, similarity, lookups, query,
      config, util, migrate, neuro/ (scoring + rewrite)

Re-exports at crate root preserve backwards compatibility so
`crate::llm`, `crate::digest` etc. continue to work.
2026-03-08 21:27:41 -04:00
Renamed from poc-memory/src/knowledge.rs (Browse further)