Commit graph

40 commits

Author SHA1 Message Date
ProofOfConcept
2b25fee520 Remove experience_mine, journal_enrich, and old mining helpers
experience_mine and journal_enrich are replaced by the observation
agent. enrich.rs reduced from 465 to 40 lines — only extract_conversation
and split_on_compaction remain (used by observation fragment selection).

-455 lines.
2026-03-17 00:54:12 -04:00
ProofOfConcept
7a24d84ce3 Clean up unused imports, dead code, and compiler warnings
Remove unused StoreView imports, unused store imports, dead
install_default_file, dead make_report_slug, dead fact-mine/
experience-mine spawning loops in daemon. Fix mut warnings.
Zero compiler warnings now.
2026-03-17 00:47:52 -04:00
ProofOfConcept
8b959fb68d agent run: add --target flag to run agents on specific nodes
Adds run_one_agent_with_keys() which bypasses the agent's query and
uses explicitly provided node keys. This allows testing agents on
specific graph neighborhoods:

  poc-memory agent run linker --target bcachefs --debug
2026-03-17 00:24:24 -04:00
Kent Overstreet
7fe55e28bd poc-memory agent run --debug: dump prompt and response
Add --debug flag that prints the full prompt and LLM response to
stdout, making it easy to iterate on agent prompts. Also adds
prompt field to AgentResult so callers can inspect what was sent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 19:13:43 -04:00
Kent Overstreet
f0df489465 poc-memory agent run: single agent execution with dry-run
New command: `poc-memory agent run <agent> [--count N] [--dry-run]`

Runs a single agent by name through the full pipeline (build prompt,
call LLM, apply actions). With --dry-run, sets POC_MEMORY_DRY_RUN=1
so all mutations are no-ops but the agent can still read the graph.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:13:24 -04:00
Kent Overstreet
835b392b7a transcript progress: capnp append-only log replaces stub nodes
Add TranscriptSegment capnp schema and append-only log for tracking
which transcript segments have been mined by which agents. Replaces
the old approach of creating stub nodes (_observed-transcripts,
_mined-transcripts, _facts-) in the main graph store.

- New schema: TranscriptSegment and TranscriptProgressLog
- Store methods: append_transcript_progress, replay, is_segment_mined,
  mark_segment_mined
- Migration command: admin migrate-transcript-progress (migrated 1771
  markers, soft-deleted old stub nodes)
- Progress log replayed on all Store::load paths

Also: revert extractor.agent to graph-only (no CONVERSATIONS),
update memory-instructions-core with refine-over-create principle.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 17:40:32 -04:00
ProofOfConcept
16777924d0 evaluate: switch to Elo ratings with skillratings crate
Replace sort-based ranking with proper Elo system:
- Each agent TYPE has a persistent Elo rating (agent-elo.json)
- Each matchup: pick two random types, grab a recent action from
  each, LLM compares, update ratings
- Ratings persist across daily evaluations — natural recency bias
  from continuous updates against current opponents
- K=32 for fast adaptation to prompt changes

Usage: poc-memory agent evaluate --matchups 30 --model haiku

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 19:53:46 -04:00
ProofOfConcept
0cecfdb352 evaluate: fix agent prompt path, dedup affected nodes, add --dry-run
- Use CARGO_MANIFEST_DIR for agent file path (same as defs.rs)
- Dedup affected nodes extracted from reports
- --dry-run shows example comparison prompt without LLM calls

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 19:44:12 -04:00
ProofOfConcept
e12dea503b agent evaluate: sort agent actions by quality using Vec::sort_by with LLM
Yes, really. Rust's stdlib sort_by with an LLM pairwise comparator.
Each comparison is an API call asking "which action was better?"

Sample N actions per agent type, throw them all in a Vec, sort.
Where each agent's samples cluster = that agent's quality score.
Reports per-type average rank and quality ratio.

Supports both haiku (fast/cheap) and sonnet (quality) as comparator.

Usage: poc-memory agent evaluate --samples 5 --model haiku

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 19:24:07 -04:00
ProofOfConcept
99db511403 cli: move helpers to cli modules, main.rs under 1100 lines
Move CLI-specific helpers to their cli/ modules:
- journal_tail_entries, journal_tail_digests, extract_title,
  find_current_transcript → cli/journal.rs
- get_group_content → cli/misc.rs
- cmd_journal_write, cmd_journal_tail, cmd_load_context follow

These are presentation/session helpers, not library code — they
belong in the CLI layer per Kent's guidance.

main.rs: 3130 → 1054 lines (66% reduction).

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 18:14:52 -04:00
ProofOfConcept
8640d50990 cli: extract journal and misc commands, complete split
Move remaining extractable handlers into cli/journal.rs and cli/misc.rs.
Functions depending on main.rs helpers (cmd_journal_tail, cmd_journal_write,
cmd_load_context, cmd_cursor, cmd_daemon, cmd_digest, cmd_experience_mine,
cmd_apply_agent) remain in main.rs — next step is moving those helpers
to library code.

main.rs: 3130 → 1331 lines (57% reduction).
cli/ total: 1860 lines across 6 focused files.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 18:10:22 -04:00
ProofOfConcept
f423cf22df cli: extract agent and admin commands from main.rs
Move agent handlers (consolidate, replay, digest, experience-mine,
fact-mine, knowledge-loop, apply-*) into cli/agent.rs.

Move admin handlers (init, fsck, dedup, bulk-rename, health,
daily-check, import, export) into cli/admin.rs.

Functions tightly coupled to Clap types (cmd_daemon, cmd_digest,
cmd_apply_agent, cmd_experience_mine) remain in main.rs.

main.rs: 3130 → 1586 lines (49% reduction).

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 18:06:27 -04:00
ProofOfConcept
aa2fddf137 cli: extract node commands from main.rs into cli/node.rs
Move 15 node subcommand handlers (310 lines) out of main.rs:
render, write, used, wrong, not-relevant, not-useful, gap,
node-delete, node-rename, history, list-keys, list-edges,
dump-json, lookup-bump, lookups.

main.rs: 2518 → 2193 lines.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 18:02:12 -04:00
ProofOfConcept
c8d86e94c1 cli: extract graph commands from main.rs into cli/graph.rs
Move 18 graph subcommand handlers (594 lines) out of main.rs:
link, link-add, link-impact, link-audit, link-orphans,
triangle-close, cap-degree, normalize-strengths, differentiate,
trace, spectral-*, organize, interference.

main.rs: 3130 → 2518 lines.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 17:59:46 -04:00
ProofOfConcept
f8221286da admin: add bulk-rename command, remove # from all keys
Add `poc-memory admin bulk-rename FROM TO [--apply]` for bulk key
character replacement. Uses rename_node() per key for proper capnp
log persistence. Collision detection, progress reporting, auto-fsck.

Applied: renamed 13,042 keys from # to - separator. This fixes the
Claude Bash tool's inability to pass # in command arguments (the
model confabulates that quoting doesn't work and gives up).

7 collision pairs resolved by deleting the # version before rename.
209 orphan edges pruned by fsck.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 13:11:38 -04:00
ProofOfConcept
51ee082faf provenance: set POC_PROVENANCE for agent subprocesses, Jaccard initial strength
Agent subprocess calls now set POC_PROVENANCE=agent:{name} so any
nodes/links created via tool calls are tagged with the creating agent.
This makes agent transcripts indistinguishable from conscious sessions
in format — important for future model training.

new_relation() now reads POC_PROVENANCE env var directly (raw string,
not enum) since agent names are dynamic.

link-add now computes initial strength from Jaccard similarity instead
of hardcoded 0.8. New links start at a strength reflecting actual
neighborhood overlap.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 12:27:30 -04:00
ProofOfConcept
cb44138433 feedback: not-relevant/not-useful commands, edge strength adjustment
Add adjust_edge_strength() to Store — modifies strength on all edges
between two nodes, clamped to [0.05, 0.95].

New commands:
- `not-relevant KEY` — weakens ALL edges to the node by 0.01
  (bad routing: search found the wrong thing)
- `not-useful KEY` — weakens node weight, not edges
  (bad content: search found the right thing but it's not good)

Enhanced `used KEY` — now also strengthens all edges to the node by
0.01, in addition to the existing node weight boost.

Three-tier design: agents adjust by 0.00001 (automatic), conscious
commands adjust by 0.01 (deliberate), manual override sets directly.
All clamped, never hitting 0 or 1.

Design spec: .claude/analysis/2026-03-14-link-strength-feedback.md

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 12:14:17 -04:00
ProofOfConcept
dccc18b205 graph: normalize link strengths from Jaccard neighborhood similarity
Add jaccard() and jaccard_strengths() to Graph. Jaccard similarity
measures neighborhood overlap between linked nodes — nodes sharing
many neighbors get stronger links, nodes with no shared neighbors
get weak links.

New subcommand: `poc-memory graph normalize-strengths [--apply]`

Scales raw Jaccard (typically 0.0-0.3) to useful range via j*3
clamped to [0.1, 1.0]. Skips implicit temporal edges (strength=1.0).

Applied to 64,969 edges. Distribution is bimodal: large cluster at
0.1-0.2 (weak) and spike at 0.9-1.0 (strong), with smooth gradient
between. Replaces the meaningless 0.3/0.8 split from manual/agent
creation methods.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 11:13:58 -04:00
ProofOfConcept
7c1b96293f cursor: spatial memory navigation
Persistent cursor into the knowledge graph with navigation:
- temporal: forward/back among same-type nodes by timestamp
- hierarchical: up/down the digest tree (journal→daily→weekly→monthly)
- spatial: graph neighbor display at every position

The cursor file (~/.claude/memory/cursor) holds a single node key.
Show displays: temporal arrows, hierarchy links, semantic neighbors,
and full content. Date extraction from both timestamps and key names
handles the mixed-timestamp data gracefully.

This is the start of place cells — spatial awareness of position
in your own knowledge.
2026-03-13 22:31:23 -04:00
ProofOfConcept
c22a7a72e1 cli: proper clap subcommands for daemon + expanded help
Convert daemon from hand-rolled string dispatch to proper clap
Subcommand enum with typed args. Add custom top-level help that
expands nested subcommands (same pattern as bcachefs-tools), so
`poc-memory --help` shows full paths like `agent daemon run`.
2026-03-13 20:07:15 -04:00
ProofOfConcept
76b8e69749 organize: topic cluster diagnostic + agent with tool access
Add `poc-memory graph organize TERM` diagnostic that finds nodes
matching a search term, computes pairwise cosine similarity, reports
connectivity gaps, and optionally creates anchor nodes.

Add organize.agent definition that uses Bash(poc-memory:*) tool access
to explore clusters autonomously — query selects highest-degree
unvisited nodes, agent drives its own iteration via poc-memory CLI.

Add {{organize}} placeholder in defs.rs for inline cluster resolution.

Add `tools` field to AgentDef/AgentHeader so agents can declare
allowed tool patterns (passed as --allowedTools to claude CLI).
2026-03-13 18:49:49 -04:00
ProofOfConcept
9a0908fbc6 query: add connectivity pipe stage
BFS-based connectivity analysis as a query pipeline stage. Shows
connected components, islands, and sample paths between result nodes
through the full graph (max 4 hops).

  poc-memory query "content ~ 'made love' | connectivity"
  poc-memory query "(content ~ 'A' OR content ~ 'B') | connectivity"

Also documented in query --help.
2026-03-11 17:04:59 -04:00
ProofOfConcept
0e971dee61 render: show links footer for graph walkability; query: full --help
render now appends a links section showing up to 15 neighbors as
copy-pasteable `poc-memory render` commands, making the graph
naturally walkable without memorizing node keys.

query --help now documents the full language: expressions, fields,
operators, pipe stages, functions, and examples. Inline help in
cmd_query replaced with pointer to --help.
2026-03-11 16:53:25 -04:00
ProofOfConcept
5a0a3d038b fsck: add cache vs log consistency check
Load store from both cache (rkyv/bincode) and raw capnp logs,
then diff: missing nodes, phantom nodes, version mismatches.
Auto-rebuilds cache if inconsistencies found.

This would have caught the mysterious the-plan deletion — likely
caused by a stale/corrupt snapshot that silently dropped the node
while the capnp log still had it.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-11 01:42:32 -04:00
ProofOfConcept
57c26d8157 reorganize subcommands into logical groups
60+ flat subcommands grouped into:
- Core (daily use): search, render, write, history, tail, status, query, used, wrong, gap
- Node: delete, rename, list, edges, dump
- Journal: write, tail, enrich
- Graph: link, audit, spectral, etc.
- Agent: daemon, knowledge-loop, consolidate, digest, etc.
- Admin: init, health, fsck, import, export, etc.

Also: remove dead migration code (migrate.rs, Migrate/JournalTsMigrate commands),
update memory-search and poc-hook for new subcommand paths, update daemon systemd
template for `agent daemon` path.
2026-03-11 01:32:21 -04:00
Kent Overstreet
d76b14dfcd provenance: convert from enum to freeform string
The Provenance enum couldn't represent agents defined outside the
source code. Replace it with a Text field in the capnp schema so any
agent can write its own provenance label (e.g. "extractor:write",
"rename:tombstone") without a code change.

Schema: rename old enum fields to provenanceOld, add new Text
provenance fields. Old enum kept for reading legacy records.
Migration: from_capnp_migrate() falls back to old enum when the
new text field is empty.

Also adds `poc-memory tail` command for viewing recent store writes.

Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
2026-03-11 01:19:52 -04:00
ProofOfConcept
15dedea322 search: make component and content matching opt-in
Default search now uses exact key match only. Component matching
(--fuzzy) and content search (--content) are explicit flags. This
makes missing graph structure visible instead of silently falling
back to broad matching.
2026-03-10 23:01:46 -04:00
ProofOfConcept
7d6ebbacab daemon: add run-agent RPC for queuing agent jobs
Adds `poc-memory daemon run-agent <type> <count>` CLI command that
sends an RPC to the daemon to queue agent runs, instead of spawning
separate processes.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-10 17:04:30 -04:00
Kent Overstreet
e736471d99 search: unified query pipeline with filters, transforms, generators
Extend the pipeline with four stage types composing left-to-right:
  Generators: all, match:TERM
  Filters: type:, key:, weight:, age:, content-len:, provenance:,
           not-visited:, visited: (plus ! negation)
  Transforms: sort:(priority|timestamp|content-len|degree|weight), limit:N
  Algorithms: spread, spectral, confluence, geodesic, manifold (unchanged)

Duration syntax (7d, 24h, 30m) and glob matching on keys.
CLI auto-detects filter/transform stages and loads full Store;
algorithm-only pipelines keep the fast MmapView path.

Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
2026-03-10 15:22:12 -04:00
ProofOfConcept
9f14a29181 dedup: find and merge duplicate nodes with edge redirection
New `poc-memory dedup` command (--apply for live run, dry-run by
default). Finds nodes sharing the same key but different UUIDs,
classifies them as identical or diverged, picks a survivor
(prefer most edges, then highest version), tombstones the rest,
and redirects all edges from doomed UUIDs to the survivor.
2026-03-10 14:30:33 -04:00
ProofOfConcept
ef760f0053 poc-memory status: add ratatui TUI dashboard
Per-agent-type tabs (health, replay, linker, separator, transfer,
apply, orphans, cap, digest, digest-links, knowledge) with dynamic
visibility — tabs only appear when tasks or log history exist.

Features:
- Overview tab: health gauges (α, gini, cc, episodic%), in-flight
  tasks, and recent log entries
- Pipeline tab: table with phase ordering and status
- Per-agent tabs: active tasks, output logs, log history
- Log tab: auto-scrolling daemon.log tail
- Vim-style count prefix: e.g. 5r runs 5 iterations of the agent
- Flash messages for RPC feedback
- Tab/Shift-Tab navigation, number keys for tab selection

Also adds run-agent RPC to the daemon: accepts agent type and
iteration count, spawns chained tasks with LLM resource pool.

poc-memory status launches TUI when stdout is a terminal and daemon
is running, falls back to text output otherwise.
2026-03-10 00:41:29 -04:00
Kent Overstreet
53e6b32cb4 daemon: rework consolidation pipeline and add graph health metrics
Replace monolithic consolidate job with individual agent jobs
(replay, linker, separator, transfer, health) that run sequentially
and store reports. Multi-phase daily pipeline: agent runs → apply
actions → link orphans → cap degree → digest → digest links →
knowledge loop.

Add GraphHealth struct with graph metrics (alpha, gini, clustering
coefficient, episodic ratio) computed during health checks. Display
in `poc-memory daemon status`. Use cached metrics to build
consolidation plan without expensive O(n²) interference detection.

Add RPC consolidate command to trigger consolidation via socket.
Harden session watcher: skip transcripts with zero segments, improve
migration error handling.

Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
2026-03-09 17:02:01 -04:00
Kent Overstreet
01dd8e5ef9 search: add --full flag to show node content in results
Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
2026-03-09 01:25:42 -04:00
Kent Overstreet
c1664bf76b search: composable algorithm pipeline
Break search into composable stages that chain left-to-right:
each stage takes seeds Vec<(String, f64)> and returns modified seeds.

Available algorithms:
  spread              — spreading activation through graph edges
  spectral            — nearest neighbors in spectral embedding
  manifold            — (placeholder) extrapolation along seed direction

Stages accept inline params: spread,max_hops=4,edge_decay=0.5

memory-search gets --hook, --debug, --seen modes plus positional
pipeline args. poc-memory search gets -p/--pipeline flags.

Also: fix spectral decompose() to skip zero eigenvalues from
disconnected components, filter degenerate zero-coord nodes from
spectral projection, POC_AGENT bail-out for daemon agents, all
debug output to stdout.

Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
2026-03-09 01:19:04 -04:00
ProofOfConcept
cee9b76a7b move LLM-dependent modules into agents/ subdir
Separate the agent layer (everything that calls external LLMs or
orchestrates sequences of such calls) from core graph infrastructure.

agents/: llm, prompts, audit, consolidate, knowledge, enrich,
         fact_mine, digest, daemon

Root: store/, graph, spectral, search, similarity, lookups, query,
      config, util, migrate, neuro/ (scoring + rewrite)

Re-exports at crate root preserve backwards compatibility so
`crate::llm`, `crate::digest` etc. continue to work.
2026-03-08 21:27:41 -04:00
ProofOfConcept
2f2c84e1c0 consolidate hardcoded paths into config, refactor apply_agent
Move prompts_dir into Config (was hardcoded ~/poc/memory/prompts).
Replace hardcoded ~/.claude/memory paths in spectral.rs, graph.rs,
and main.rs with store::memory_dir() or config::get(). Replace
hardcoded ~/.claude/projects in knowledge.rs and main.rs with
config::get().projects_dir.

Extract apply_agent_file() from cmd_apply_agent() — separates
file scanning from per-file JSON parsing and link application.
2026-03-08 21:16:52 -04:00
ProofOfConcept
52523403c5 extract truncation helpers, fix clippy warnings, dedup batching loop
Add util::truncate() and util::first_n_chars() to replace 16 call
sites doing the same floor_char_boundary or chars().take().collect()
patterns. Deduplicate the batching loop in consolidate.rs (4 copies
→ 1 loop over an array). Fix all clippy warnings: redundant closures,
needless borrows, collapsible if, unnecessary cast, manual strip_prefix.

Net: -44 lines across 16 files.
2026-03-08 21:13:02 -04:00
ProofOfConcept
e24dee6bdf switch CLI argument parsing to Clap derive
Replace hand-rolled argument parsing (match on args[1], manual
iteration over &[String]) with Clap's derive macros. All 60+
subcommands now have typed arguments with defaults, proper help
text, and error messages generated automatically.

The 83-line usage() function is eliminated — Clap generates help
from the struct annotations. Nested subcommands (digest daily/
weekly/monthly/auto, journal-tail --level) use Clap's subcommand
nesting naturally.
2026-03-08 21:04:45 -04:00
Kent Overstreet
d5634c0034 remove dead code: unused imports, functions, and fields
- Remove #![allow(dead_code)] from main.rs, fix all revealed warnings
- Delete unused schema_assimilation() from neuro/scoring.rs
- Delete duplicate memory_dir() wrapper from knowledge.rs
- Deduplicate load_prompt: knowledge.rs now calls neuro::load_prompt
- Remove unused timeout field from DigestLevel
- Remove unused imports (regex::Regex, Provenance, AnyView, Write)
- Mark OldEntry fields as #[allow(dead_code)] (needed for deserialization)

Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
2026-03-08 20:51:56 -04:00
Kent Overstreet
fc48ac7c7f split into workspace: poc-memory and poc-daemon subcrates
poc-daemon (notification routing, idle timer, IRC, Telegram) was already
fully self-contained with no imports from the poc-memory library. Now it's
a proper separate crate with its own Cargo.toml and capnp schema.

poc-memory retains the store, graph, search, neuro, knowledge, and the
jobkit-based memory maintenance daemon (daemon.rs).

Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
2026-03-08 20:43:59 -04:00
Renamed from src/main.rs (Browse further)