kent/consciousness
1
0
Fork 2
Code Issues 1 Pull requests 5 Projects Releases Packages Wiki Activity Actions

Compare commits

base: kent:0612e1bc41f98a7b8dac66dc8afdf6c472d44996
Branches Tags
kent:master
kent:agent-model-dispatch
...
compare: kent:a966dd9d5db19a823bc292df970bf034baa8808f
Branches Tags
kent:master
kent:agent-model-dispatch

81 commits
0612e1bc41 ... a966dd9d5d

Author SHA1 Message Date
Kent Overstreet
a966dd9d5d kill rusqlite dep 2026-04-13 22:16:09 -04:00
Kent Overstreet
faad14dc95 graph: use index for bulk reads, skip capnp deserialization 
- Add all_keys() to StoreView, use in build_adjacency instead of
  for_each_node (which was ignoring content/weight anyway)
- Add all_key_uuid_pairs() for single-pass uuid mapping
- Extend KEY_TO_UUID to store [uuid:16][node_type:1][timestamp:8]
- for_each_node_meta now reads from index, no capnp needed
- Add NodeType::from_u8() for unpacking

Graph health: 7s → 2s (3.5x faster)

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 22:15:13 -04:00
Kent Overstreet
b3d0a3ab25 store: internal locking, remove Arc<Mutex<Store>> wrapper 
Store now has internal Mutex for capnp appends and AtomicU64 for
size tracking. All methods take &self. The external Arc<Mutex<Store>>
is replaced with Arc<Store>.

- Store::append_lock protects file appends
- local.rs functions take &Store (not &mut Store)
- access_local() returns Arc<Store>
- All .lock().await calls removed from callers

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 21:49:54 -04:00
Kent Overstreet
4696bb8b7d store: index ops take WriteTransaction, mutations batch properly 
Index functions now take &WriteTransaction instead of &Database,
allowing callers to batch multiple index operations in a single
transaction. Store mutations (upsert, delete, rename, etc.) now
begin_write/commit their own transactions, ensuring atomicity.

- replay_relations uses single txn for all relation indexing
- Store::db() exposes Database for callers needing txn control
- Convenience wrappers open their own txn for simple cases

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 21:44:20 -04:00
Kent Overstreet
2548ca059d store: remove Vec<Relation>, dedup uses index iteration 
The relations Vec is gone from Store. dedup now iterates via
edges_for_uuid() instead of mutating in-memory Vec — removes/re-adds
edges through the index directly.

Removed load_relations_vec() and clear_relations() — no longer needed.
Added helper methods: edges_for_uuid, index_relation, remove_relation_from_index.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 21:32:48 -04:00
Kent Overstreet
c2de14dcab store: add reindex_relations for after Vec mutations 
- Add index::clear_relations() to drop and recreate RELS table
- Add Store::reindex_relations() to rebuild index from Vec
- Call reindex_relations() at end of dedup command

This ensures index stays in sync with Vec after complex mutations
like UUID redirection in dedup. Vec mutations remain for now but
index is correctly updated afterward.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
 2026-04-13 21:24:49 -04:00
Kent Overstreet
58b0947625 admin: convert fsck and dedup reads to use index 
- fsck: use for_each_relation for dangling edge detection
  (pruning deferred - needs delete_edge operation)
- dedup: use for_each_relation for edge counting

Remaining Vec uses in dedup mutation section need new index ops:
- redirect_edge: change source/target UUID
- delete_edge_by_uuid: tombstone by UUID

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
 2026-04-13 21:22:52 -04:00
Kent Overstreet
5832e57970 store: convert more callers to use RELS index 
Convert remaining Vec users to index-based access:
- memory.rs: MemoryNode::from_store uses Store::neighbors()
- graph.rs: orphan detection uses for_each_relation
- local.rs: normalize_strengths uses for_each_relation + set_link_strength

Add Store::neighbors() method and index::get_offsets_for_uuid().

Cleanup:
- for_each_relation: build both uuid↔key maps in one pass
- cap_degree: consolidate key/uuid/degree collection

Remaining Vec uses: admin.rs (fsck, dedup), capnp.rs (load path).

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
 2026-04-13 21:20:27 -04:00
Kent Overstreet
5fe51fbfda store: wire up RELS index for relations 
Complete redb schema with bidirectional relation indexing:
- RELS multimap: uuid → packed(other_uuid, strength, rel_type, is_outgoing)
- Each edge stored twice (once per endpoint) with direction bit
- pack_rel/unpack_rel for 22-byte packed format

Wired up:
- replay_relations indexes all relations on load
- add_relation indexes new relations
- for_each_relation reads from index (graph building)
- add_link uses index for existence check
- set_link_strength finds/updates edges via index
- cap_degree uses index for degree counting and pruning
- rename_node finds edges by uuid

Vec<Relation> still maintained for remaining uses (normalize_strengths,
graph_health diagnostics). To be removed in follow-up.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
 2026-04-13 21:12:47 -04:00
Kent Overstreet
8cfe9a4d70 fix stale comment and skip unimplemented query tests 
- capnp.rs: remove reference to removed self.nodes field
- parser.rs: comment out tests for not-yet-implemented features
  (not-visited filter, recency() in composite sorts)

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 20:20:44 -04:00
Kent Overstreet
c9b51c941e store/index: remove unused get_key_by_uuid and node_count 
Speculative helpers that were never called. Easy to re-add if needed.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 20:12:54 -04:00
Kent Overstreet
5877fd857a store: remove nodes and uuid_to_key HashMaps 
All node access now goes through index → capnp:
- scoring.rs: consolidation_priority, replay_queue, consolidation_plan
- admin.rs: cmd_init, cmd_fsck, cmd_dedup
- engine.rs: run_generator, eval_filter, run_transform
- parser.rs: resolve_field, execute_query

Added Store::remove_from_index() for dedup cleanup.

The relations Vec remains for now (used for graph building).

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 19:49:09 -04:00
Kent Overstreet
af3e41f1d9 migrate more files to use index-based node access 
- learn.rs, daemon.rs, graph.rs, digest.rs, prompts.rs
- Convert store.nodes.get() → store.get_node()
- Convert store.nodes.contains_key() → store.contains_key()
- Convert store.nodes.values/iter() → all_keys + get_node

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 19:37:11 -04:00
Kent Overstreet
fe6450223c migrate local.rs and memory.rs to use index 
- Add Store::all_keys() method for iteration
- Convert store.nodes.get() → store.get_node()
- Convert store.nodes.contains_key() → store.contains_key()
- Convert store.nodes.values() iteration → all_keys + get_node

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 19:34:45 -04:00
Kent Overstreet
7eb86656d4 store: read nodes via index instead of HashMap 
- Add get_node() and contains_key() methods that read via redb index
- Migrate all store/ reads to use index lookup
- Remove HashMap cache updates from mutations (write-through to capnp+index only)
- Remove replay_nodes() - load no longer builds HashMap
- Update db_is_healthy to validate by spot-checking offsets
- Fix set_weight bug: now persists weight changes to capnp

Store.nodes HashMap still exists for code outside store/ module,
but store/ itself no longer uses it.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 19:31:28 -04:00
Kent Overstreet
ba53597cf2 store: move all capnp code to capnp.rs 
Consolidate capnp serialization in one place:
- capnp_enum! and capnp_message! macros
- read_text/read_uuid helpers
- Type-to-capnp mappings
- from_capnp_migrate migration impls

types.rs now only has pure Rust types and helpers.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 19:21:56 -04:00
Kent Overstreet
e48ca2ecad store: remove StoreLock and refresh_nodes 
With singleton Store (one daemon, RPC for clients), there's no concurrent
writers to capnp log. The file-based flock and incremental refresh logic
was for multi-process coordination we no longer need.

-110 lines of dead concurrency code.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 19:13:25 -04:00
Kent Overstreet
f413a853d8 store: redb indexes offsets into capnp log, not full nodes 
Restructure store module with clearer file names:
- persist.rs → capnp.rs (capnp log IO)
- db.rs → index.rs (redb index operations)

redb now stores key → offset mapping, not serialized nodes.
Mutations record the offset after appending to capnp log.
rebuild_index scans capnp log to reconstruct the index.

The HashMap still exists for now; next step is to use the
index for lookups and remove it.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 19:10:08 -04:00
Kent Overstreet
9309de68fc store: wire up redb updates on mutations 
Mutations (upsert_node, upsert_provenance, delete_node, rename_node)
now update redb indices atomically with capnp log appends, under the
same StoreLock.

Also removes dead cmd_import command and the parse.rs module it depended on.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 19:03:09 -04:00
Kent Overstreet
a1accc7cd4 store: remove visit tracking infrastructure 
Remove AgentVisit, TranscriptSegment, and all related visit tracking code.
Provenance is what we've been using to track agent interaction with nodes.

Also removes dead fields from Node (state_tag, created).

-349 lines.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 18:57:12 -04:00
Kent Overstreet
7d49f29fde store: remove dead code and move params to config 
Remove:
- score_weight() - never called
- position field on Node - never read (was for export)
- Provenance enum - inline helper for capnp migration
- migrate_transcript_progress + CLI command
- init_from_markdown, import_file, ingest_units
- export command and export_to_markdown
- RetrievalEvent, GapRecord types
- classify_filename, new_transcript_segment

Move spreading activation params to Config:
- default_node_weight, edge_decay, max_hops, min_activation
- Remove Params struct and StoreView::params()

Simplify cmd_init to just seed identity via upsert().
Simplify cmd_import to use parse_units + upsert directly.

-576 lines

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 18:50:21 -04:00
Kent Overstreet
6104c63890 Integrate redb into Store::load() with health check 
- Add db: Option<Database> field to Store
- Store::load() opens redb after replaying capnp logs
- Health check compares node count + spot checks keys
- Rebuilds automatically if db is missing, corrupt, or stale
- Make table definitions public for cross-module access

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 18:33:47 -04:00
Kent Overstreet
2caccf875d Replace rkyv/bincode caching with redb indices 
Remove three-tier loading (rkyv snapshot, bincode cache, capnp replay)
in favor of direct capnp log replay + redb for indexed access.

- Remove all rkyv derives from types (Node, Relation, enums, etc.)
- Remove Snapshot struct, RKYV_MAGIC, CACHE_MAGIC constants
- Remove load_snapshot_mmap(), save(), save_snapshot()
- Remove MmapView, AnyView from view.rs (keep StoreView trait)
- Simplify Store::load() to just replay capnp logs
- Add db.rs with redb schema: nodes, uuid_to_key, visits, transcript_progress
- Simplify cmd_fsck to just check capnp integrity + graph health

capnp logs remain source of truth; redb indices will be rebuilt on demand.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 18:30:58 -04:00
Kent Overstreet
1d88293ccf Remove Store::cached(), consolidate on access_local() 
- Remove CACHED_STORE, cached(), is_stale(), set_store() - redundant
- Convert all Store::cached() callers to use access_local()
- Single Store::load() call remains in access() fallback path

All store access now goes through hippocampus::access() / access_local(),
which handles socket connection or local fallback with caching.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 18:11:58 -04:00
Kent Overstreet
09b30d64f2 cmd_fsck: use access_local(), remove dead AnyView::load() 
Convert cmd_fsck to async and use access_local() for the cached store.
Still uses Store::load_from_logs() for fresh comparison.

Remove unused AnyView::load() method - was never called.

Remaining Store::load() calls are all internal caching infrastructure:
- persist.rs cached() for CACHED_STORE
- mod.rs access() fallback for STORE_ACCESS

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 18:09:02 -04:00
Kent Overstreet
f6f330b07b Convert cmd_import, cmd_export, MigrateTranscriptProgress to access_local() 
These were the last Store::load() calls that should use the shared store.
Remaining calls are intentional: fsck (needs both cached and fresh),
persist.rs cached() infrastructure, view.rs read-only fallback, and
access() bootstrap path.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 18:06:58 -04:00
Kent Overstreet
b8db8754be Convert store and CLI to anyhow::Result for cleaner error handling 
Replace Result<_, String> with anyhow::Result throughout:
- hippocampus/store module (persist, ops, types, view, mod)
- CLI modules (admin, agent, graph, journal, node)
- Run trait in main.rs

Use .context() and .with_context() instead of .map_err(|e| format!(...))
patterns. Add bail!() for early error returns.

Add access_local() helper in hippocampus/mod.rs that returns
Result<Arc<Mutex<Store>>> for direct local store access.

Fix store access patterns to properly lock Arc<Mutex<Store>> before
accessing fields in mind/unconscious.rs, mind/mod.rs, subconscious/learn.rs,
and hippocampus/memory.rs.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 18:05:04 -04:00
Kent Overstreet
5db00e083f centralize memory store interface in hippocampus/mod.rs 2026-04-13 17:44:41 -04:00
Kent Overstreet
063cf031d3 journal_tail: return typed Vec<JournalEntry>, remove Store::load from agent 
- journal_tail returns Vec<JournalEntry> with key, content, created_at
- load_startup_journal uses typed API, no more direct Store access
- CLI does formatting, hippocampus returns data

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 15:23:10 -04:00
Kent Overstreet
419bb222b5 defs.rs: remove store/graph params, use typed memory API 
resolve_placeholders() and run_agent() no longer take &Store.
All placeholders now use async memory_render/memory_links/memory_query
directly. The "siblings" placeholder uses Vec<LinkInfo> for ranking
neighbors by link_strength * node_weight.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 15:18:05 -04:00
Kent Overstreet
598f0112a4 memory_links: return typed Vec<LinkInfo> with node weights 
- hippocampus::memory_links now returns Vec<LinkInfo> with key,
  link_strength, and node_weight for each neighbor
- Unified memory_tool! macro: mut/ref as token, single main rule
- All tools use serde serialize/deserialize for RPC consistency
- jsonargs handlers now work in client mode (RPC to daemon)
- cli/graph.rs formats LinkInfo for display

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 15:12:06 -04:00
Kent Overstreet
359955f838 defs.rs: async conversion, remove block_in_place 
Convert resolve(), resolve_placeholders(), run_agent() to async.
Use memory_render/memory_query directly with .await instead of
block_in_place wrappers.

Propagate async to callers:
- config.rs: resolve(), load_session(), reload_for_model()
- identity.rs: load_memory_files(), assemble_context_message()
- oneshot.rs: run_one_agent()
- prompts.rs: agent_prompt()

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 14:56:26 -04:00
Kent Overstreet
9bb07bc26a memory.rs: clean up store access and tool dispatch 
- Single access() function returns StoreAccess enum (Daemon/Client/None)
- OnceLock for daemon store, thread-local RefCell for client socket
- Remove dispatch() - Tool handlers call jsonargs_* directly
- get_provenance() takes agent ref, no JSON round-trip
- Expose missing graph tools (communities, normalize, link_impact, trace)
- Local tool! macro for cleaner Tool definitions

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 14:27:38 -04:00
Kent Overstreet
fb46ab095d Consolidate memory RPC in tools/memory.rs 
- Move memory_rpc(), socket_path(), SocketConn from mcp_server.rs
- Convert remaining callers to typed async API:
  - defs.rs: organize placeholder, run_agent query
  - cli/agent.rs: query resolution (now async)
  - mind/identity.rs: Store context loading
- Re-export socket_path/memory_rpc from mcp_server for compatibility

All external memory access now goes through tools/memory.rs typed API.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 13:39:59 -04:00
Kent Overstreet
5b07a81aa7 CLI/hippocampus: rename core memory functions to memory_* 
Aligns function names with tool names for consistency:
- hippocampus: render → memory_render, write → memory_write, etc.
- tools/memory.rs: macro no longer prepends memory_ prefix
- CLI files: use typed async API throughout (graph.rs, journal.rs, admin.rs)

This eliminates the "memory_graph_topology" tool name bug where
graph_* and journal_* tools were incorrectly prefixed.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 13:26:22 -04:00
Kent Overstreet
fa50f1c826 CLI: convert node commands to typed async API 
- node.rs: use memory::* typed helpers instead of memory_rpc()
- main.rs: make Run trait async, await all command dispatch
- defs.rs: bridge get_group_content async via block_in_place

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 13:20:04 -04:00
Kent Overstreet
933221f482 memory tools: generate public typed API via macro 
The memory_tool! macro now generates two functions:
- jsonargs_*() - internal, takes JSON args for dispatch table
- pub fn name() - typed args, handles RPC-vs-local automatically

Callers can now use typed Rust API:
  memory::write(Some(&agent), "key", "content").await?;
  memory::query(None, "all | type:semantic", Some("full")).await?;

No more manual JSON construction for memory tool calls.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 13:12:11 -04:00
Kent Overstreet
4560ba9230 memory tools: typed hippocampus fns + macro dispatch 
Move tool implementations from tools/memory.rs to hippocampus/mod.rs
with proper typed signatures:
  fn name(store, provenance, ...typed args...) -> Result<String>

Optional params take Option<T>, defaults applied in implementation.

tools/memory.rs is now a thin dispatch layer using memory_tool! macro:
  memory_tool!(write, mut, key: [str], content: [str]);
  memory_tool!(search, ref, keys: [Vec<String>], max_hops: [Option<u32>], ...);

~634 lines of boilerplate replaced with ~30 one-liner invocations.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 13:03:24 -04:00
Kent Overstreet
d7a5ac6347 memory tools: simplify provenance handling 
Move provenance injection to dispatch() entry point - agent provenance is
always written to args._provenance before routing. Individual tool
functions now just call get_provenance(args) which is sync and simple.

Removes agent parameter from: write, link_add, supersede, journal_new,
journal_update.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 12:08:46 -04:00
Kent Overstreet
dc1049f62d CLI: async runtime + proper RPC fallback plumbing 
- main.rs: use #[tokio::main] so CLI has a runtime available
- memory.rs: make run_with_local_store async (no more runtime creation)
- mcp_server.rs: cache socket connection in OnceLock, use block_in_place
  for async fallback when socket unavailable

Fixes "cannot start a runtime from within a runtime" panic when CLI
falls back to local store.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
 2026-04-13 11:23:52 -04:00
Kent Overstreet
7476e9d0db delete rename agent and related code 
The organize agents handle renaming as part of their normal work now.
Also simplified resolve_placeholders to build graph internally.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 02:05:58 -04:00
Kent Overstreet
bd9ce3ed09 keys_to_replay_items() -> memory.rs 2026-04-13 01:57:23 -04:00
Kent Overstreet
a08f521b02 defs.rs: convert run_agent query to use RPC 
Uses memory_rpc("memory_query", ...) instead of direct search::run_query.
Removes now-unused crate::search import.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 01:54:22 -04:00
Kent Overstreet
b863f77998 defs.rs: convert seed placeholder to use resolve_tool 
Uses the existing tool infrastructure instead of direct store access.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 01:49:22 -04:00
Kent Overstreet
c688b812ef defs.rs: convert organize placeholder to use RPC 
Uses memory_render RPC instead of direct store access.
Simplifies from ~60 to ~20 lines.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 01:45:12 -04:00
Kent Overstreet
4cfeb9ee2f defs.rs: delete dead placeholders, simplify siblings 
- Remove {{targets}}, {{hubs}}, {{node:KEY}}, {{latest_journal}} placeholders
- Add graph_hubs as proper RPC tool (was placeholder, now callable)
- Replace {{latest_journal}} with {{tool: journal_tail ...}} in journal.agent
- Simplify siblings/neighborhood: drop unused cross-links, use simple top-20
- Remove unused store/graph params from resolve_tool()

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-13 01:37:33 -04:00
Kent Overstreet
de5a6672c3 cleanup: remove dead placeholder code, use RPC for identity loading 
- links() in memory.rs: use cached_store() instead of MemoryNode::load()
- identity.rs: use memory_rpc for Store context loading
- defs.rs: delete dead placeholders (topology, nodes/episodes, health, split)
  - agents now use {{tool: graph_topology}} etc instead
- prompts.rs: delete unused format_split_plan_node()

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
 2026-04-13 01:22:08 -04:00
Kent Overstreet
2ab4aef19f CLI: more RPC conversions, delete obsolete commands 
- cmd_health: use graph_health RPC
- cmd_topology: new command using graph_topology RPC
- cmd_status: use graph_topology RPC (type counts folded into topology)
- cmd_run_agent: query resolution via memory_query RPC
- Delete cmd_bulk_rename (one-time migration, obsolete)
- Delete cmd_replay_queue, cmd_digest_links (unconscious agents handle)
- format_topology_header: add type counts, takes &Store now

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
 2026-04-12 23:37:05 -04:00
Kent Overstreet
1f6bfb5915 kill cmd_graph, cmd_organize 2026-04-12 23:20:19 -04:00
ProofOfConcept
11f2d5b169 graph_trace, graph_link_impact: convert to RPC tools 
Agents can use these to understand graph structure:
- trace: shows node and neighbors grouped by type
- link_impact: analyzes what happens if a link is removed

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 23:16:12 -04:00
ProofOfConcept
f02a23468e graph_normalize_strengths: convert to RPC tool 
Agents can use this to check if edge weights are skewed.
Dry run by default, pass apply:true to write changes.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 23:12:42 -04:00
ProofOfConcept
a8d91896a2 graph_communities: new RPC tool, convert cmd_communities 
Agents can use graph_communities to discover isolated knowledge
clusters that need better integration.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 23:09:12 -04:00
ProofOfConcept
be9db3fb1a graph: delete cmd_spread, convert cmd_link to RPC 
cmd_spread was duplicate of cmd_search/memory_search.
cmd_link now uses memory_links RPC.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 23:07:06 -04:00
Kent Overstreet
70097fa84b kill cli/misc.rs 2026-04-12 23:03:00 -04:00
ProofOfConcept
5a832b1d6c get_group_content: use RPC, delete store-based version 
One function that uses memory_rpc (which handles daemon vs local).
Removes 65 lines of duplicate logic.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 23:00:10 -04:00
ProofOfConcept
051198b3d1 memory_search: accept optional params 
max_hops (default 3), edge_decay (default 0.3), min_activation
(default 0.01), limit (default 20). No longer reads from store.params.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 22:49:40 -04:00
ProofOfConcept
d1d57267d3 Remove cmd_log and cmd_params 
Retrieval log was never used (history covers node log).
Params should come from config, not hardcoded store defaults.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 22:47:58 -04:00
ProofOfConcept
aff872e101 cmd_search: thin wrapper around memory_search RPC 
Remove term matching, pipeline stages, mmap/store paths. Just
pass keys to memory_search and print result. For anything fancy,
use memory_query.

-165 lines.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 22:44:59 -04:00
ProofOfConcept
8b59becbab cmd_load_context: use RPC instead of Store::load() 
Add get_group_content_rpc() which uses memory_query and memory_render
instead of direct store access. The original get_group_content() stays
for the subconscious path which already has a store open.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 22:32:47 -04:00
Kent Overstreet
a6b93c2255 cli: kill cmd_list_keys 
Redundant with query

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 22:27:31 -04:00
Kent Overstreet
ad59596335 cli: add memory_history, remove dump-json/edges/lookups 
- Add memory_history MCP tool for version history
- Convert cmd_history to use memory_rpc
- Add raw parameter to memory_render for editing
- Remove unused: dump-json, list-edges, lookup-bump, lookups
- Fix render_node path in defs.rs/subconscious.rs

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 22:24:34 -04:00
Kent Overstreet
3e0c6b039f move render_node() to memory.rs 2026-04-12 22:20:31 -04:00
Kent Overstreet
4b4271f618 cli: convert cmd_edit to use memory_rpc 
Add raw parameter to memory_render for getting content without
links footer. cmd_edit now uses memory_render(raw=true) to read
and memory_write to save.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 22:18:28 -04:00
Kent Overstreet
366b17163d cli: convert write/delete/journal-write to use memory_rpc 
- cmd_write → memory_write RPC
- cmd_node_delete → new memory_delete MCP tool + RPC
- cmd_journal_write → journal_new RPC

Removes validate_inline_refs and find_current_transcript
(now handled server-side or not needed).

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 22:15:53 -04:00
Kent Overstreet
7842b6fc8b remove legacy feedback commands (used, wrong, gap, etc.) 
These were early experiments with manual feedback signals that
never worked well. The scoring system will handle this properly.

Removed:
- CLI: used, wrong, not-relevant, not-useful, gap
- MCP: memory_used
- Store: mark_used, mark_wrong, record_gap, modify_node

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 22:12:02 -04:00
Kent Overstreet
11b58e6b0b cli: convert simple commands to use memory_rpc 
Commands now forward to daemon (or fallback to local store):
- query → memory_query
- journal tail → journal_tail
- graph link-set → memory_link_set
- graph link-add → memory_link_add
- weight-set → memory_weight_set
- node rename → memory_rename

Removes ~50 lines of duplicated store access code.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 21:54:34 -04:00
Kent Overstreet
d2a82d4327 memory tools: forward to daemon when not in daemon process 
Add STORE_HANDLE global that daemon sets at startup. When set, tools
access store directly. When unset (external process), tools forward
to daemon via MCP socket.

This allows consciousness-claude and poc-memory to import and call
memory tools directly - they'll automatically route through the
daemon socket.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 21:45:59 -04:00
Kent Overstreet
2c0f2065e0 mcp_server: Unix socket server for external tool access 
Exposes memory/journal tools over ~/.consciousness/mcp.sock via
JSON-RPC 2.0 (MCP protocol). External processes (consciousness-mcp,
poc-memory) will connect here instead of accessing the store directly.

Handles: initialize, tools/list, tools/call
Dispatches to the same tool handlers the agent uses internally.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 21:45:10 -04:00
Kent Overstreet
72f4f1b617 context: cache role header token lengths 
Branch::tokens() was calling tokenizer::encode() on every call for
the role header ("system\n", "user\n", "assistant\n") and trailing
newline. In trim_conversation(), this meant hundreds of encode calls
per trim cycle.

These are fixed strings - cache them with OnceLock on first use.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 20:47:36 -04:00
Kent Overstreet
ac6f1e9294 unconscious: move health refresh outside lock too 
refresh_health() was doing Store::load() + compute_graph_health()
while holding the Unconscious lock, causing 12 second stalls.

Split into needs_health_refresh() (quick check) and set_health()
(quick store), with the slow I/O happening outside the lock.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 20:37:54 -04:00
Kent Overstreet
f40d8cfa9d unconscious: release lock during slow spawn work 
Split trigger() into phases so the Unconscious mutex is only held briefly:
- reap_finished(): check handles, restore completed autos
- select_to_spawn(): pick agents, take their autos out
- prepare_spawn(): slow work (Store::load, query, Agent::new) - NO LOCK
- complete_spawn()/abort_spawn(): store results back

Previously held the lock for 28+ seconds during Store::load and query
execution. Now lock hold time should be milliseconds.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 20:33:23 -04:00
Kent Overstreet
f56fc3a7c7 locks: add process-wide lock hold time tracking 
TrackedMutex and TrackedRwLock wrappers that record hold durations
by source location using #[track_caller]. Stats written to
~/.consciousness/lock-stats.json every second, sorted by max hold time.

Re-exported as crate::Mutex so all locks are instrumented. To disable,
swap the re-export back to tokio::sync::Mutex.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 20:27:42 -04:00
Kent Overstreet
b94e056372 unconscious/subconscious: use Option<AutoAgent> instead of placeholder 
Previously, spawning an agent used std::mem::replace with an empty-name
AutoAgent as placeholder. This caused ghost stats entries under "" when
those placeholders accidentally got their stats logged.

Now uses Option<AutoAgent> with .take() - the type honestly represents
that the agent is unavailable while running. Panic recovery in
subconscious now properly recreates the agent from its definition.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 20:11:40 -04:00
Kent Overstreet
33156d9ab3 channels: improve tmux state tracking and config persistence 
tmux channel:
- Track connected state per-pane (shows true channel availability)
- Persist pane config on add/remove (survives restarts)
- Remove cleanup_pipes on exit (unnecessary with persisted config)
- Reorder PaneConfig fields for consistency

telegram channel:
- Use json5 crate for config parsing (matches tmux)

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 20:11:34 -04:00
Kent Overstreet
4556e16fd7 enable short backtraces by default 
Uses panic_backtrace_config feature to set BacktraceStyle::Short,
so panics show useful backtraces without needing RUST_BACKTRACE=1.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 20:11:27 -04:00
Kent Overstreet
dfab7d0a33 prompts: remove unused replay_queue import 
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 16:21:54 -04:00
Kent Overstreet
783046a3f5 selectable: silence unused method warning 
The is_selected method is reserved for future per-character
highlight rendering when the module is fully integrated.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 16:10:58 -04:00
Kent Overstreet
195abfaab1 chat: guard pop_line against empty list 
Small defensive improvement - only pop markers and invalidate scroll
if lines.pop() actually removed something.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 16:04:43 -04:00
Kent Overstreet
f06c8077e1 research: latent reasoning integration plans for Qwen 3.5 27B 
Two research documents:

latent-reasoning-integration-plan.md: Synthesizes 10+ papers on
latent reasoning, identifies which approaches work with finetuning
(vs requiring pretraining from scratch), and maps them to our
APOLLO-Mini training pipeline.

pause-tokens-gdn-recurrence.md: Explores the connection between
token-based latent reasoning and GDN's internal recurrence. Key
insight: pause tokens on Qwen 3.5 trigger both forward passes AND
recurrent state updates, giving double benefit.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 15:50:09 -04:00
Kent Overstreet
dcd647764c user: fix text selection on wrapped lines 
scroll_pane: screen_to_item() now properly accounts for wrapped
lines using textwrap to compute actual character positions instead
of just using mouse_x directly.

selectable: new module with PUA markers for wrap-aware selection.
Not yet integrated into chat.rs but ready for future use. Uses
continuation markers to track logical vs visual lines.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 15:49:57 -04:00
Kent Overstreet
ab0f16a3b5 tools: add cd tool for changing working directory 
Uses std::env::set_current_dir() syscall so the change affects
all subsequent tool invocations. Supports absolute paths, relative
paths, and ~ expansion.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
 2026-04-12 15:49:46 -04:00
63 changed files with 5537 additions and 5344 deletions
Download patch file Download diff file Expand all files Collapse all files

192
Cargo.lock generated
View file

@ -8,17 +8,6 @@ version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "ahash"
version = "0.7.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "891477e0c6a8957309ee5c45a6368af3ae14bb510732d2684ffa19af310920f9"
dependencies = [
"getrandom 0.2.17",
"once_cell",
"version_check",
]
[[package]] [[package]]
name = "ahash" name = "ahash"
version = "0.8.12" version = "0.8.12"
@ -285,18 +274,6 @@ version = "2.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af"
[[package]]
name = "bitvec"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c"
dependencies = [
"funty",
"radium",
"tap",
"wyz",
]
[[package]] [[package]]
name = "block-buffer" name = "block-buffer"
version = "0.10.4" version = "0.10.4"
@ -322,28 +299,6 @@ version = "3.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb" checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb"
[[package]]
name = "bytecheck"
version = "0.6.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "23cdc57ce23ac53c931e88a43d06d070a6fd142f2617be5855eb75efc9beb1c2"
dependencies = [
"bytecheck_derive",
"ptr_meta",
"simdutf8",
]
[[package]]
name = "bytecheck_derive"
version = "0.6.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3db406d29fbcd95542e92559bed4d8ad92636d1ca8b3b72ede10b4bcc010e659"
dependencies = [
"proc-macro2",
"quote",
"syn 1.0.109",
]
[[package]] [[package]]
name = "bytemuck" name = "bytemuck"
version = "1.25.0" version = "1.25.0"
@ -521,7 +476,6 @@ dependencies = [
"ast-grep-core", "ast-grep-core",
"ast-grep-language", "ast-grep-language",
"base64 0.22.1", "base64 0.22.1",
"bincode",
"bytes", "bytes",
"capnp", "capnp",
"capnp-rpc", "capnp-rpc",
@ -546,15 +500,14 @@ dependencies = [
"paste", "paste",
"peg", "peg",
"ratatui", "ratatui",
"rayon",
"redb", "redb",
"regex", "regex",
"rkyv",
"rustls", "rustls",
"rustls-native-certs", "rustls-native-certs",
"serde", "serde",
"serde_json", "serde_json",
"serde_urlencoded", "serde_urlencoded",
"textwrap",
"tokenizers", "tokenizers",
"tokio", "tokio",
"tokio-rustls", "tokio-rustls",
@ -610,6 +563,7 @@ dependencies = [
"dirs", "dirs",
"env_logger", "env_logger",
"futures", "futures",
"json5",
"log", "log",
"serde", "serde",
"serde_json", "serde_json",
@ -630,8 +584,8 @@ dependencies = [
"json5", "json5",
"libc", "libc",
"log", "log",
"scopeguard",
"serde", "serde",
"serde_json",
"tokio", "tokio",
"tokio-util", "tokio-util",
] ]
@ -1128,12 +1082,6 @@ version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "funty"
version = "2.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c"
[[package]] [[package]]
name = "futures" name = "futures"
version = "0.3.32" version = "0.3.32"
@ -1296,15 +1244,6 @@ dependencies = [
"regex-syntax", "regex-syntax",
] ]
[[package]]
name = "hashbrown"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888"
dependencies = [
"ahash 0.7.8",
]
[[package]] [[package]]
name = "hashbrown" name = "hashbrown"
version = "0.15.5" version = "0.15.5"
@ -2192,26 +2131,6 @@ dependencies = [
"yansi", "yansi",
] ]
[[package]]
name = "ptr_meta"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0738ccf7ea06b608c10564b31debd4f5bc5e197fc8bfe088f68ae5ce81e7a4f1"
dependencies = [
"ptr_meta_derive",
]
[[package]]
name = "ptr_meta_derive"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "16b845dbfca988fa33db069c0e230574d15a3088f147a87b64c7589eb662c9ac"
dependencies = [
"proc-macro2",
"quote",
"syn 1.0.109",
]
[[package]] [[package]]
name = "pulldown-cmark" name = "pulldown-cmark"
version = "0.13.3" version = "0.13.3"
@ -2261,12 +2180,6 @@ version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "radium"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09"
[[package]] [[package]]
name = "rand" name = "rand"
version = "0.8.5" version = "0.8.5"
@ -2485,15 +2398,6 @@ version = "0.8.10"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a"
[[package]]
name = "rend"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "71fe3824f5629716b1589be05dacd749f6aa084c87e00e016714a8cdfccc997c"
dependencies = [
"bytecheck",
]
[[package]] [[package]]
name = "ring" name = "ring"
version = "0.17.14" version = "0.17.14"
@ -2508,35 +2412,6 @@ dependencies = [
"windows-sys 0.52.0", "windows-sys 0.52.0",
] ]
[[package]]
name = "rkyv"
version = "0.7.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2297bf9c81a3f0dc96bc9521370b88f054168c29826a75e89c55ff196e7ed6a1"
dependencies = [
"bitvec",
"bytecheck",
"bytes",
"hashbrown 0.12.3",
"ptr_meta",
"rend",
"rkyv_derive",
"seahash",
"tinyvec",
"uuid",
]
[[package]]
name = "rkyv_derive"
version = "0.7.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "84d7b42d4b8d06048d3ac8db0eb31bcb942cbeb709f0b5f2b2ebde398d3038f5"
dependencies = [
"proc-macro2",
"quote",
"syn 1.0.109",
]
[[package]] [[package]]
name = "rustc_version" name = "rustc_version"
version = "0.4.1" version = "0.4.1"
@ -2644,12 +2519,6 @@ version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
[[package]]
name = "seahash"
version = "4.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b"
[[package]] [[package]]
name = "security-framework" name = "security-framework"
version = "3.7.0" version = "3.7.0"
@ -2813,6 +2682,12 @@ version = "1.15.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03"
[[package]]
name = "smawk"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7c388c1b5e93756d0c740965c41e8822f866621d41acbdf6336a6a168f8840c"
[[package]] [[package]]
name = "socket2" name = "socket2"
version = "0.6.3" version = "0.6.3"
@ -2923,12 +2798,6 @@ dependencies = [
"yaml-rust", "yaml-rust",
] ]
[[package]]
name = "tap"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369"
[[package]] [[package]]
name = "terminfo" name = "terminfo"
version = "0.9.0" version = "0.9.0"
@ -2992,6 +2861,17 @@ dependencies = [
"winapi", "winapi",
] ]
[[package]]
name = "textwrap"
version = "0.16.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c13547615a44dc9c452a8a534638acdf07120d4b6847c8178705da06306a3057"
dependencies = [
"smawk",
"unicode-linebreak",
"unicode-width",
]
[[package]] [[package]]
name = "thiserror" name = "thiserror"
version = "1.0.69" version = "1.0.69"
@ -3065,28 +2945,13 @@ dependencies = [
"time-core", "time-core",
] ]
[[package]]
name = "tinyvec"
version = "1.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3"
dependencies = [
"tinyvec_macros",
]
[[package]]
name = "tinyvec_macros"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
[[package]] [[package]]
name = "tokenizers" name = "tokenizers"
version = "0.21.4" version = "0.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a620b996116a59e184c2fa2dfd8251ea34a36d0a514758c6f966386bd2e03476" checksum = "a620b996116a59e184c2fa2dfd8251ea34a36d0a514758c6f966386bd2e03476"
dependencies = [ dependencies = [
"ahash 0.8.12", "ahash",
"aho-corasick", "aho-corasick",
"compact_str", "compact_str",
"dary_heap", "dary_heap",
@ -3512,6 +3377,12 @@ version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "unicode-linebreak"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3b09c83c3c29d37506a3e260c08c03743a6bb66a9cd432c6934ab501a190571f"
[[package]] [[package]]
name = "unicode-normalization-alignments" name = "unicode-normalization-alignments"
version = "0.1.12" version = "0.1.12"
@ -4077,15 +3948,6 @@ dependencies = [
"wasmparser", "wasmparser",
] ]
[[package]]
name = "wyz"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed"
dependencies = [
"tap",
]
[[package]] [[package]]
name = "yaml-rust" name = "yaml-rust"
version = "0.4.5" version = "0.4.5"

5
Cargo.toml
View file

@ -34,9 +34,9 @@ json5 = "1.3"
ratatui = { version = "0.30", features = ["unstable-rendered-line-info"] } ratatui = { version = "0.30", features = ["unstable-rendered-line-info"] }
tui-markdown = { git = "https://github.com/koverstreet/tui-markdown", subdirectory = "tui-markdown" } tui-markdown = { git = "https://github.com/koverstreet/tui-markdown", subdirectory = "tui-markdown" }
tui-textarea = { version = "0.10.2", package = "tui-textarea-2" } tui-textarea = { version = "0.10.2", package = "tui-textarea-2" }
textwrap = "0.16"
uuid = { version = "1", features = ["v4"] } uuid = { version = "1", features = ["v4"] }
bincode = "1"
regex = "1" regex = "1"
glob = "0.3" glob = "0.3"
chrono = { version = "0.4", features = ["serde"] } chrono = { version = "0.4", features = ["serde"] }
@ -51,9 +51,6 @@ ast-grep-language = { version = "0.42", features = ["builtin-parser"] }
walkdir = "2" walkdir = "2"
redb = "4" redb = "4"
rkyv = { version = "0.7", features = ["validation", "std"] }
rayon = "1"
tokio = { version = "1", features = ["full"] } tokio = { version = "1", features = ["full"] }
tokio-util = { version = "0.7", features = ["compat"] } tokio-util = { version = "0.7", features = ["compat"] }

1
channels/telegram/Cargo.toml
View file

@ -8,6 +8,7 @@ capnp = "0.25"
capnp-rpc = "0.25" capnp-rpc = "0.25"
dirs = "6" dirs = "6"
futures = "0.3" futures = "0.3"
json5 = "1.3"
consciousness = { path = "../.." } consciousness = { path = "../.." }
serde = { version = "1", features = ["derive"] } serde = { version = "1", features = ["derive"] }
serde_json = "1" serde_json = "1"

2
channels/telegram/src/main.rs
View file

@ -40,7 +40,7 @@ fn load_config() -> Config {
let config_path = dir.join("telegram.json5"); let config_path = dir.join("telegram.json5");
let text = std::fs::read_to_string(&config_path) let text = std::fs::read_to_string(&config_path)
.unwrap_or_else(|_| panic!("failed to read {}", config_path.display())); .unwrap_or_else(|_| panic!("failed to read {}", config_path.display()));
let mut config: Config = serde_json::from_str(&text) let mut config: Config = json5::from_str(&text)
.unwrap_or_else(|e| panic!("failed to parse {}: {}", config_path.display(), e)); .unwrap_or_else(|e| panic!("failed to parse {}: {}", config_path.display(), e));
// Read token from secrets file // Read token from secrets file

2
channels/tmux/Cargo.toml
View file

@ -8,11 +8,11 @@ capnp = "0.25"
capnp-rpc = "0.25" capnp-rpc = "0.25"
dirs = "6" dirs = "6"
libc = "0.2" libc = "0.2"
scopeguard = "1"
futures = "0.3" futures = "0.3"
json5 = "1.3" json5 = "1.3"
consciousness = { path = "../.." } consciousness = { path = "../.." }
serde = { version = "1", features = ["derive"] } serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio = { version = "1", features = ["full"] } tokio = { version = "1", features = ["full"] }
tokio-util = { version = "0.7", features = ["compat"] } tokio-util = { version = "0.7", features = ["compat"] }
log = "0.4" log = "0.4"

141
channels/tmux/src/main.rs
View file

@ -24,26 +24,30 @@ use consciousness::thalamus::channel_log::ChannelLog;
// ── Config ───────────────────────────────────────────────────── // ── Config ─────────────────────────────────────────────────────
#[derive(Clone, serde::Deserialize)] #[derive(Clone, serde::Serialize, serde::Deserialize)]
struct PaneConfig { struct PaneConfig {
/// Tmux pane ID, e.g. "0:1.0"
pane_id: String,
/// Human-readable label, becomes the channel name "tmux.<label>" /// Human-readable label, becomes the channel name "tmux.<label>"
label: String, label: String,
/// Tmux pane ID, e.g. "%5"
pane_id: String,
} }
#[derive(Clone, serde::Deserialize)] #[derive(Clone, serde::Serialize, serde::Deserialize)]
struct Config { struct Config {
#[serde(default)]
panes: Vec<PaneConfig>, panes: Vec<PaneConfig>,
} }
fn load_config() -> Config { fn config_path() -> std::path::PathBuf {
let path = dirs::home_dir() dirs::home_dir()
.unwrap_or_default() .unwrap_or_default()
.join(".consciousness/channels/tmux.json5"); .join(".consciousness/channels/tmux.json5")
match std::fs::read_to_string(&path) { }
fn load_config() -> Config {
match std::fs::read_to_string(config_path()) {
Ok(text) => json5::from_str(&text) Ok(text) => json5::from_str(&text)
.unwrap_or_else(|e| panic!("failed to parse {}: {e}", path.display())), .unwrap_or_else(|e| panic!("failed to parse {}: {e}", config_path().display())),
Err(_) => { Err(_) => {
info!("no tmux.json5, starting with no pre-configured panes"); info!("no tmux.json5, starting with no pre-configured panes");
Config { panes: vec![] } Config { panes: vec![] }
@ -51,23 +55,71 @@ fn load_config() -> Config {
} }
} }
fn save_config(config: &Config) {
match serde_json::to_string_pretty(config) {
Ok(json) => {
if let Err(e) = std::fs::write(config_path(), json) {
error!("failed to write config: {}", e);
}
}
Err(e) => error!("failed to serialize config: {}", e),
}
}
// ── State ───────────────────────────────────────────────────── // ── State ─────────────────────────────────────────────────────
struct State { struct State {
config: Config,
channel_logs: BTreeMap<String, ChannelLog>, channel_logs: BTreeMap<String, ChannelLog>,
/// label → pane_id (e.g. "ktest" → "%0") /// Tracks which panes are actually connected (pipe-pane active)
panes: BTreeMap<String, String>, connected: BTreeMap<String, bool>,
} }
type SharedState = Rc<RefCell<State>>; type SharedState = Rc<RefCell<State>>;
impl State { impl State {
fn new(config: &Config) -> Self { fn new(config: Config) -> Self {
Self { Self {
config,
channel_logs: BTreeMap::new(), channel_logs: BTreeMap::new(),
panes: config.panes.iter() connected: BTreeMap::new(),
.map(|p| (p.label.clone(), p.pane_id.clone())) }
.collect(), }
/// Get pane_id for a label
fn get_pane(&self, label: &str) -> Option<&str> {
self.config.panes.iter()
.find(|p| p.label == label)
.map(|p| p.pane_id.as_str())
}
/// Check if a pane is connected
fn is_connected(&self, label: &str) -> bool {
self.connected.get(label).copied().unwrap_or(false)
}
/// Set connection state for a pane
fn set_connected(&mut self, label: &str, connected: bool) {
self.connected.insert(label.to_string(), connected);
}
/// Add a pane and persist
fn add_pane(&mut self, label: String, pane_id: String) {
if !self.config.panes.iter().any(|p| p.label == label) {
self.config.panes.push(PaneConfig { label, pane_id });
save_config(&self.config);
}
}
/// Remove a pane and persist
fn remove_pane(&mut self, label: &str) -> Option<String> {
if let Some(idx) = self.config.panes.iter().position(|p| p.label == label) {
let pane = self.config.panes.remove(idx);
self.connected.remove(label);
save_config(&self.config);
Some(pane.pane_id)
} else {
None
} }
} }
} }
@ -103,10 +155,12 @@ async fn pipe_pane_reader(state: SharedState, pane: PaneConfig) {
Ok(output) => { Ok(output) => {
error!("pipe-pane failed for {}: {}", pane.label, error!("pipe-pane failed for {}: {}", pane.label,
String::from_utf8_lossy(&output.stderr)); String::from_utf8_lossy(&output.stderr));
state.borrow_mut().set_connected(&pane.label, false);
return; return;
} }
Err(e) => { Err(e) => {
error!("failed to run tmux pipe-pane for {}: {}", pane.label, e); error!("failed to run tmux pipe-pane for {}: {}", pane.label, e);
state.borrow_mut().set_connected(&pane.label, false);
return; return;
} }
} }
@ -116,10 +170,14 @@ async fn pipe_pane_reader(state: SharedState, pane: PaneConfig) {
Ok(f) => f, Ok(f) => f,
Err(e) => { Err(e) => {
error!("failed to open pipe for {}: {}", pane.label, e); error!("failed to open pipe for {}: {}", pane.label, e);
state.borrow_mut().set_connected(&pane.label, false);
return; return;
} }
}; };
// Mark as connected once pipe is open
state.borrow_mut().set_connected(&pane.label, true);
let reader = tokio::io::BufReader::new(file); let reader = tokio::io::BufReader::new(file);
let mut lines = reader.lines(); let mut lines = reader.lines();
let channel_key = format!("tmux.{}", pane.label); let channel_key = format!("tmux.{}", pane.label);
@ -136,6 +194,7 @@ async fn pipe_pane_reader(state: SharedState, pane: PaneConfig) {
} }
warn!("pipe-pane reader ended for {}", pane.label); warn!("pipe-pane reader ended for {}", pane.label);
state.borrow_mut().set_connected(&pane.label, false);
} }
// ── ChannelServer Implementation ─────────────────────────────── // ── ChannelServer Implementation ───────────────────────────────
@ -187,7 +246,7 @@ impl channel_server::Server for ChannelServerImpl {
// Send to tmux pane via send-keys // Send to tmux pane via send-keys
let label = channel.strip_prefix("tmux.").unwrap_or(&channel); let label = channel.strip_prefix("tmux.").unwrap_or(&channel);
let pane_id = self.state.borrow().panes.get(label).cloned(); let pane_id = self.state.borrow().get_pane(label).map(String::from);
if let Some(pane_id) = pane_id { if let Some(pane_id) = pane_id {
let _ = std::process::Command::new("tmux") let _ = std::process::Command::new("tmux")
.args(["send-keys", "-t", &pane_id, &message, "Enter"]) .args(["send-keys", "-t", &pane_id, &message, "Enter"])
@ -210,10 +269,11 @@ impl channel_server::Server for ChannelServerImpl {
mut results: channel_server::ListResults, mut results: channel_server::ListResults,
) -> impl std::future::Future<Output = Result<(), capnp::Error>> { ) -> impl std::future::Future<Output = Result<(), capnp::Error>> {
let s = self.state.borrow(); let s = self.state.borrow();
let channels: Vec<_> = s.panes.keys().map(|label| { let channels: Vec<_> = s.config.panes.iter().map(|p| {
let key = format!("tmux.{}", label); let key = format!("tmux.{}", p.label);
let connected = s.is_connected(&p.label);
let unread = s.channel_logs.get(&key).map_or(0, |l| l.unread()); let unread = s.channel_logs.get(&key).map_or(0, |l| l.unread());
(key, true, unread) (key, connected, unread)
}).collect(); }).collect();
let mut list = results.get().init_channels(channels.len() as u32); let mut list = results.get().init_channels(channels.len() as u32);
@ -243,12 +303,9 @@ impl channel_server::Server for ChannelServerImpl {
let label = pry!(pry!(params.get_label()).to_str()).to_string(); let label = pry!(pry!(params.get_label()).to_str()).to_string();
// Check if already open // Check if already open
{ if self.state.borrow().get_pane(&label).is_some() {
let s = self.state.borrow();
if s.panes.contains_key(&label) {
return std::future::ready(Ok(())); return std::future::ready(Ok(()));
} }
}
// Find the tmux pane by name (window or pane title) // Find the tmux pane by name (window or pane title)
let pane_id = match find_pane_by_name(&label) { let pane_id = match find_pane_by_name(&label) {
@ -259,14 +316,11 @@ impl channel_server::Server for ChannelServerImpl {
info!("opening channel tmux.{} (pane {})", label, pane_id); info!("opening channel tmux.{} (pane {})", label, pane_id);
// Register in state // Register in state and persist
{ self.state.borrow_mut().add_pane(label.clone(), pane_id.clone());
let mut s = self.state.borrow_mut();
s.panes.insert(label.clone(), pane_id.clone());
}
// Start pipe-pane reader // Start pipe-pane reader
let pane = PaneConfig { pane_id, label }; let pane = PaneConfig { label, pane_id };
let reader_state = self.state.clone(); let reader_state = self.state.clone();
tokio::task::spawn_local(async move { tokio::task::spawn_local(async move {
pipe_pane_reader(reader_state, pane).await; pipe_pane_reader(reader_state, pane).await;
@ -285,7 +339,7 @@ impl channel_server::Server for ChannelServerImpl {
let label = channel.strip_prefix("tmux.").unwrap_or(&channel).to_string(); let label = channel.strip_prefix("tmux.").unwrap_or(&channel).to_string();
let mut s = self.state.borrow_mut(); let mut s = self.state.borrow_mut();
if let Some(pane_id) = s.panes.remove(&label) { if let Some(pane_id) = s.remove_pane(&label) {
info!("closing channel tmux.{}", label); info!("closing channel tmux.{}", label);
s.channel_logs.remove(&format!("tmux.{}", label)); s.channel_logs.remove(&format!("tmux.{}", label));
@ -323,24 +377,6 @@ fn find_pane_by_name(name: &str) -> Option<String> {
} }
None None
} }
// ── Cleanup ───────────────────────────────────────────────────
/// Remove pipe-pane connections on exit.
fn cleanup_pipes(config: &Config) {
for pane in &config.panes {
// Disconnect pipe-pane
let _ = std::process::Command::new("tmux")
.args(["pipe-pane", "-t", &pane.pane_id])
.output();
}
// Clean up FIFO files
let pipe_dir = dirs::home_dir()
.unwrap_or_default()
.join(".consciousness/channels/tmux-pipes");
let _ = std::fs::remove_dir_all(&pipe_dir);
}
// ── Main ─────────────────────────────────────────────────────── // ── Main ───────────────────────────────────────────────────────
#[tokio::main] #[tokio::main]
@ -348,7 +384,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
env_logger::init(); env_logger::init();
let config = load_config(); let config = load_config();
let state = Rc::new(RefCell::new(State::new(&config))); let state = Rc::new(RefCell::new(State::new(config)));
let sock_dir = dirs::home_dir() let sock_dir = dirs::home_dir()
.unwrap_or_default() .unwrap_or_default()
@ -359,16 +395,11 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
info!("tmux channel daemon starting on {}", sock_path.display()); info!("tmux channel daemon starting on {}", sock_path.display());
// Set up cleanup on exit
let cleanup_config = config.clone();
let _cleanup = scopeguard::guard(cleanup_config, |c| cleanup_pipes(&c));
tokio::task::LocalSet::new() tokio::task::LocalSet::new()
.run_until(async move { .run_until(async move {
// Start a pipe-pane reader for each configured pane // Start a pipe-pane reader for each configured pane
for pane in &config.panes { for pane in state.borrow().config.panes.clone() {
let reader_state = state.clone(); let reader_state = state.clone();
let pane = pane.clone();
tokio::task::spawn_local(async move { tokio::task::spawn_local(async move {
pipe_pane_reader(reader_state, pane).await; pipe_pane_reader(reader_state, pane).await;
}); });

300
docs/latent-reasoning-integration-plan.md Normal file
View file

@ -0,0 +1,300 @@
# Latent Reasoning Integration Plan for Qwen 3.5 27B
**Status:** Research complete, ready for implementation
**Date:** 2026-04-12
**Hardware:** B200 (192GB HBM3e), APOLLO-Mini optimizer
## Executive Summary
Recent research shows multiple approaches to improving LLM reasoning through latent space manipulation. This document synthesizes findings from 10+ papers and maps them to our Qwen 3.5 27B full finetuning pipeline. The key insight: some approaches require pretraining from scratch (skip those), while others can be layered onto existing models during finetuning (prioritize those).
---
## 1. The Landscape
### Approaches That Require Pretraining (Not Applicable)
| Technique | Why Not |
|-----------|---------|
| Huginn/Recurrent Depth (Geiping 2025) | Requires architectural changes from scratch |
| Ouro/LoopLM (ByteDance 2025) | Needs weight-tied looped architecture |
| Quiet-STaR (Stanford 2024) | Heavy continued pretraining overhead |
### Approaches Compatible with Finetuning (Our Focus)
| Technique | Overhead | Training Required | Proven On |
|-----------|----------|-------------------|-----------|
| Random Prefix Perturbation | 2 tokens | None (inference) | Qwen3-4B |
| Pause/Planning Tokens | 2-4 tokens | Yes | 1B models |
| COCONUT Curriculum | Variable | Yes (staged) | General |
| ActAdd Steering Vectors | 1 vector/layer | None (inference) | LLaMA, OPT |
| UPFT (Prefix Fine-Tuning) | 8 tokens | Yes (minimal) | General |
---
## 2. Detailed Technique Analysis
### 2.1 Random Prefix Perturbation (dl1683)
**Mechanism:** Prepend 2 random embedding-scale tokens before input. Breaks attention sink patterns, shifts model into "exploratory computation mode."
**Results:**
- Qwen3-4B arithmetic: 32% → 51.6% (+19.6pp)
- 100% oracle coverage on 25/25 tasks
- Planning: rescues 14-word failures into 650+ word plans
**Why it works:** First few tokens accumulate disproportionate attention (Xiao et al. 2024). Under greedy decoding, degenerate patterns lock in. Perturbation breaks this.
**Integration:** Zero training required. Test at inference first, then consider training WITH random prefixes to internalize the exploration behavior.
### 2.2 Pause Tokens (Google, Oct 2023)
**Mechanism:** Add learnable pause tokens to embedding space. Model processes extra hidden vectors before committing to output.
**Results (1B model):**
- SQuAD: +18% EM score
- CommonSenseQA: +8%
- GSM8K: +1%
**Critical requirement:** MUST be both pretrained AND finetuned with pause tokens. Inference-time-only delays don't work without training.
**Integration:** Add 2-4 learnable tokens to Qwen's embedding matrix, finetune with them prepended to reasoning prompts. Simple architectural change.
### 2.3 COCONUT - Chain of Continuous Thought (Meta, Dec 2024)
**Mechanism:** Feed last hidden state back as next input embedding directly (no decoding to tokens). Enables breadth-first search reasoning.
**Why it matters:** Continuous thoughts can encode multiple alternative next steps simultaneously. Avoids premature commitment to single path.
**Training approach:**
1. Initial stage: train on regular CoT examples
2. Subsequent stages: replace first k reasoning steps with k×c continuous thoughts
3. c is hyperparameter controlling latent thought expansion
**Integration:** Most promising for Qwen 3.5 - curriculum approach from CoT → latent reasoning.
### 2.4 UPFT - Unsupervised Prefix Fine-Tuning (Mar 2025)
**Mechanism:** Train ONLY on initial prefix substrings (as few as 8 tokens). Exploits "Prefix Self-Consistency" - shared initial reasoning steps across diverse solutions.
**Results:**
- Matches Rejection Sampling Fine-Tuning performance
- 75% reduction in training time
- 99% reduction in sampling cost
**Integration:** DIRECTLY APPLICABLE. Train only on reasoning prefix tokens. Massive efficiency gain with APOLLO-Mini.
### 2.5 ActAdd / Activation Engineering (Turner et al., 2023)
**Mechanism:** Compute steering vector by contrasting intermediate activations on prompt pairs. Add during forward pass.
**Results:** SOTA on sentiment shift and detoxification.
**Our existing work:** "Listening" vector at layer 48, magnitude 57, cosine consistency 0.61.
**Integration:** Prototype behaviors with steering vectors, then train permanently into weights. Steering vector as specification → APOLLO training as compilation.
### 2.6 Planning Tokens (ICLR 2024)
**Mechanism:** Learnable token embeddings added before each reasoning step. <0.001% additional parameters.
**Integration:** Add to embedding matrix, train end-to-end with APOLLO.
---
## 3. Our Setup
**Model:** Qwen 3.5 27B
- 64 layers, 5120 hidden dim
- 75% DeltaNet (linear attention) / 25% standard attention
- Native 262K context
**Hardware:** B200 (192GB HBM3e)
- 27B in bf16: ~54GB
- Massive headroom
**Optimizer:** APOLLO-Mini
- Full parameter finetuning
- SGD-like memory (1/1024th of AdamW)
- Parameter grouping for 3D conv1d weights
**Stack:** Crane (Candle-based, 21K lines)
**Existing work:**
- Steering vector extraction (listening: layer 48, cosine 0.61)
- Memory scoring infrastructure
**Unique advantage:** Qwen 3.5's GDN (Gated DeltaNet) layers provide natural infrastructure for continuous thought propagation. The recurrent GDN state is already "latent reasoning" infrastructure waiting to be leveraged.
---
## 4. Recommended Implementation Order
### Tier 1: Immediate (High ROI, Low Risk)
**1. Pause Tokens + UPFT Combination**
- Add 2-4 learnable tokens to embedding space
- Train only on 8-token reasoning prefixes
- Both work with existing architecture
- 75% training time reduction
```python
# Add pause tokens to embedding matrix
pause_tokens = nn.Parameter(torch.randn(4, embed_dim) * embed_rms)
# Prepend to reasoning inputs during training
inputs_embeds = torch.cat([pause_tokens.expand(batch, -1, -1), text_embeds], dim=1)
# UPFT: only compute loss on first 8 tokens of reasoning
loss = loss_fn(logits[:, :8], targets[:, :8])
```
**2. Random Prefix Validation**
- Compute Qwen 3.5 27B embedding RMS
- Test 2-token random prefix at inference
- Establish baseline before finetuning
### Tier 2: After Baseline (Medium Effort)
**3. COCONUT Curriculum**
- Stage 1: Fine-tune on CoT examples normally
- Stage 2: Replace first reasoning step with continuous thought
- Stage 3: Replace first 2 steps
- Gradually move reasoning into latent space
**4. Steering Vector Integration**
- Extract reasoning-specific directions (not just "listening")
- Test combinations: prefix + layer-48 steering
- Bake successful vectors into weights via APOLLO
### Tier 3: Experimental
**5. Multi-layer Steering**
- Our layers of interest: 40, 48, 56 (covering the attention layers)
- Different vectors per layer
- Careful scaling to avoid degradation
**6. DeltaNet-Specific Optimization**
- The 75% DeltaNet architecture may respond differently
- GDN recurrent state as "continuous thought" channel
- This is unexplored territory - potential for novel findings
---
## 5. Implementation Details
### Computing Embedding RMS
```python
embed_weight = model.get_input_embeddings().weight
embed_rms = embed_weight.float().square().mean().sqrt().item()
# Expected: ~0.02-0.03 range for Qwen models
```
### Pause Token Implementation in Crane
```rust
// In model forward pass
fn forward_with_pause(&self, input_ids: &Tensor, pause_tokens: &Tensor) -> Result<Tensor> {
let text_embeds = self.embed_tokens.forward(input_ids)?;
let combined = Tensor::cat(&[pause_tokens, &text_embeds], 1)?;
self.transformer.forward(&combined)
}
```
### UPFT Loss Modification
```python
# Standard: loss over all tokens
# UPFT: loss only over prefix tokens
def upft_loss(logits, targets, prefix_len=8):
return F.cross_entropy(
logits[:, :prefix_len].reshape(-1, vocab_size),
targets[:, :prefix_len].reshape(-1)
)
```
---
## 6. Evaluation Plan
### Benchmarks
| Benchmark | What It Tests | Baseline Needed |
|-----------|---------------|-----------------|
| GSM8K | Arithmetic reasoning | Yes |
| ARC-Challenge | Science reasoning | Yes |
| CommonSenseQA | Commonsense | Yes |
| HumanEval | Code generation | Yes |
| Planning tasks (dl1683) | Multi-step planning | Yes |
### Comparison Matrix
| Configuration | Training Time | Expected Gain |
|---------------|---------------|---------------|
| Baseline (no prefix) | 1x | 0% |
| Random prefix (inference) | 1x | +10-20%? |
| Pause tokens (trained) | 1.1x | +8-18% |
| UPFT only | 0.25x | Match baseline |
| Pause + UPFT | 0.3x | +8-18% |
| COCONUT curriculum | 2x | +15-25%? |
---
## 7. Open Questions
1. **Does random perturbation scale to 27B?** Tested on 4B - effect may differ
2. **Optimal token count for 27B?** 2 optimal for 4B, might change
3. **DeltaNet interaction?** 75% linear attention is untested territory
4. **Composition effects?** Prefix + steering + pause tokens together?
5. **GDN as continuous thought channel?** Novel research direction
---
## 8. Risk Assessment
| Risk | Mitigation |
|------|------------|
| No improvement at 27B scale | Start with inference-time validation |
| Training instability with pause tokens | Start with 2 tokens, scale up |
| UPFT doesn't transfer | Fall back to full token loss |
| DeltaNet behaves differently | Ablate on attention-only layers first |
---
## 9. Timeline Estimate
| Phase | Duration | Deliverable |
|-------|----------|-------------|
| Embedding RMS + baseline | 1 day | Numbers |
| Random prefix validation | 1 day | Inference results |
| Pause token implementation | 2 days | Crane modification |
| UPFT integration | 1 day | Training loop change |
| First finetuning run | 2-3 days | Trained model |
| Evaluation | 1 day | Benchmark numbers |
| COCONUT curriculum | 1 week | Staged training |
---
## 10. References
### Primary Sources
- Random Prefix: https://github.com/dl1683/Latent-Space-Reasoning
- Attention Sinks: Xiao et al., "Efficient Streaming Language Models with Attention Sinks" (Sept 2023)
- Pause Tokens: Google, "Think before you speak" (Oct 2023)
- COCONUT: Meta, "Training Large Language Models to Reason in a Continuous Latent Space" (Dec 2024)
- UPFT: "Prefix Self-Consistency for Unsupervised Fine-Tuning" (Mar 2025)
- ActAdd: Turner et al., "Activation Addition: Steering Language Models Without Optimization" (Aug 2023)
- Recurrent Depth: Geiping et al., "Scaling up Test-Time Compute with Latent Reasoning" (Feb 2025)
- Ouro: ByteDance, "Ouro: Scaling Reasoning with Latent Thoughts" (2025)
- Planning Tokens: ICLR 2024
### Our Existing Work
- `steering-vector-empirical` - listening vector extraction
- `skills-apollo-optimizer-qwen35-gotcha` - APOLLO parameter grouping
- `qwen-3-5-27b-architecture-findings` - model architecture details
- `training-pipeline-fused-inference-training-mar27` - training infrastructure
---
*Research complete 2026-04-12. Ready for implementation.*

113
plugins/index.ts Normal file
View file

@ -0,0 +1,113 @@
// opencode-plugin/index.ts — Consciousness integration for OpenCode.
//
// Bridges OpenCode events to the consciousness system:
// - chat.message → forwards to poc-hook-opencode, appends output as text part
// - tool.execute.after → signals response activity
// - event → tracks session lifecycle (idle, compacted, etc.)
// - shell.env → injects POC_SESSION_ID into subprocesses
//
// Install: copy this directory to your project's `plugin/` or `plugins/` dir,
// or add to opencode.json:
// "plugin": ["/home/kent/poc/consciousness-claude/opencode-plugin"]
import type { Plugin, Hooks } from "@opencode-ai/plugin"
import path from "path"
import { $ } from "bun"
import { $ } from "bun"
// Find the poc-hook-opencode binary
function findHookBinary(): string {
const candidates = [
path.join(process.env.HOME || "", ".cargo/bin/poc-hook-opencode"),
path.join(process.env.HOME || "", "poc/consciousness-claude/target/debug/poc-hook-opencode"),
path.join(process.env.HOME || "", "poc/consciousness-claude/target/release/poc-hook-opencode"),
]
for (const c of candidates) {
try {
const stat = Bun.file(c).statSync()
if (stat?.isFile()) return c
} catch {}
}
return "poc-hook-opencode"
}
const HOOK_BINARY = findHookBinary()
// Generate a unique part ID (opencode uses ulid-like ascending IDs)
let partCounter = 0
function nextPartId(): string {
partCounter += 1
return `poc_part_${Date.now()}_${partCounter}`
}
export const ConsciousnessPlugin: Plugin = async (ctx) => {
const hooks: Hooks = {}
// Main hook: forward user messages to consciousness, inject context
hooks["chat.message"] = async (input, output) => {
const hookInput = JSON.stringify({
session_id: input.sessionID,
hook_event: "UserPromptSubmit",
})
try {
const proc = Bun.spawn([HOOK_BINARY], {
stdin: hookInput,
stdout: "pipe",
stderr: "pipe",
})
const [stdout, stderr] = await Promise.all([
new Response(proc.stdout).text(),
new Response(proc.stderr).text(),
])
await proc.exited
if (stdout && stdout.trim()) {
// Append as a text part — must match MessageV2.TextPart schema:
// { id, sessionID, messageID, type: "text", text, time?, synthetic?, ignored? }
output.parts.push({
id: nextPartId(),
sessionID: input.sessionID,
messageID: output.message.id,
type: "text",
text: stdout,
synthetic: true,
})
}
if (stderr && stderr.trim()) {
console.error("[consciousness] hook stderr:", stderr.slice(0, 500))
}
} catch (e) {
console.error("[consciousness] hook error:", e)
}
}
// Signal response after tool use
hooks["tool.execute.after"] = async () => {
try {
await $`poc-daemon response`.quiet()
} catch {
// Daemon might not be running
}
}
// Inject POC_SESSION_ID into all shell commands
hooks["shell.env"] = async (input, output) => {
if (input.sessionID) {
output.env["POC_SESSION_ID"] = input.sessionID
}
}
// Track session events
hooks["event"] = async ({ event }) => {
if (event.type === "session.compacted") {
// Compaction detected — next hook invocation will detect via SQLite
}
if (event.type === "session.idle") {
// Session went idle
}
}
return hooks
}

6
plugins/package.json Normal file
View file

@ -0,0 +1,6 @@
{
"name": "@consciousness/opencode-plugin",
"version": "0.1.0",
"description": "Consciousness integration for OpenCode",
"main": "index.ts"
}

27
src/agent/context.rs
View file

@ -31,8 +31,31 @@
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
use serde::{Serialize, Deserialize}; use serde::{Serialize, Deserialize};
use std::sync::OnceLock;
use super::tokenizer; use super::tokenizer;
// Cached token lengths for role headers — computed once on first use.
// "system\n", "user\n", "assistant\n" and "\n" are fixed strings.
static ROLE_TOKENS: OnceLock<[usize; 3]> = OnceLock::new();
static NEWLINE_TOKENS: OnceLock<usize> = OnceLock::new();
fn role_header_tokens(role: Role) -> usize {
let tokens = ROLE_TOKENS.get_or_init(|| [
tokenizer::encode("system\n").len(),
tokenizer::encode("user\n").len(),
tokenizer::encode("assistant\n").len(),
]);
match role {
Role::System => tokens[0],
Role::User => tokens[1],
Role::Assistant => tokens[2],
}
}
fn newline_tokens() -> usize {
*NEWLINE_TOKENS.get_or_init(|| tokenizer::encode("\n").len())
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Types // Types
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@ -423,9 +446,9 @@ impl Ast for AstNode {
match self { match self {
Self::Leaf(leaf) => leaf.tokens(), Self::Leaf(leaf) => leaf.tokens(),
Self::Branch { role, children, .. } => { Self::Branch { role, children, .. } => {
1 + tokenizer::encode(&format!("{}\n", role.as_str())).len() 1 + role_header_tokens(*role)
+ children.iter().map(|c| c.tokens()).sum::<usize>() + children.iter().map(|c| c.tokens()).sum::<usize>()
+ 1 + tokenizer::encode("\n").len() + 1 + newline_tokens()
} }
} }
} }

49
src/agent/mod.rs
View file

@ -141,8 +141,8 @@ pub struct Agent {
pub app_config: crate::config::AppConfig, pub app_config: crate::config::AppConfig,
pub prompt_file: String, pub prompt_file: String,
pub session_id: String, pub session_id: String,
pub context: tokio::sync::Mutex<ContextState>, pub context: crate::Mutex<ContextState>,
pub state: tokio::sync::Mutex<AgentState>, pub state: crate::Mutex<AgentState>,
} }
/// Mutable agent state — behind its own mutex. /// Mutable agent state — behind its own mutex.
@ -218,8 +218,8 @@ impl Agent {
app_config, app_config,
prompt_file, prompt_file,
session_id, session_id,
context: tokio::sync::Mutex::new(context), context: crate::Mutex::new(context),
state: tokio::sync::Mutex::new(AgentState { state: crate::Mutex::new(AgentState {
tools: agent_tools, tools: agent_tools,
mcp_tools: McpToolAccess::All, mcp_tools: McpToolAccess::All,
last_prompt_tokens: 0, last_prompt_tokens: 0,
@ -255,8 +255,8 @@ impl Agent {
app_config: self.app_config.clone(), app_config: self.app_config.clone(),
prompt_file: self.prompt_file.clone(), prompt_file: self.prompt_file.clone(),
session_id: self.session_id.clone(), session_id: self.session_id.clone(),
context: tokio::sync::Mutex::new(ctx), context: crate::Mutex::new(ctx),
state: tokio::sync::Mutex::new(AgentState { state: crate::Mutex::new(AgentState {
tools, tools,
mcp_tools: McpToolAccess::None, mcp_tools: McpToolAccess::None,
last_prompt_tokens: 0, last_prompt_tokens: 0,
@ -497,42 +497,33 @@ impl Agent {
} }
async fn load_startup_journal(&self) { async fn load_startup_journal(&self) {
use crate::agent::tools::memory::journal_tail;
let oldest_msg_ts = { let oldest_msg_ts = {
let ctx = self.context.lock().await; let ctx = self.context.lock().await;
ctx.conversation_log.as_ref().and_then(|log| log.oldest_timestamp()) ctx.conversation_log.as_ref().and_then(|log| log.oldest_timestamp())
}; };
let store = match crate::store::Store::load() { // Get recent journal entries (newest first)
Ok(s) => s, let journal_entries = match journal_tail(None, Some(100), Some(0), None).await {
Ok(e) => e,
Err(_) => return, Err(_) => return,
}; };
let mut journal_nodes: Vec<_> = store.nodes.values() // Filter to entries before the conversation started
.filter(|n| n.node_type == crate::store::NodeType::EpisodicSession) let cutoff_ts = oldest_msg_ts.map(|t| t.timestamp());
let filtered: Vec<_> = journal_entries.into_iter()
.filter(|e| cutoff_ts.map(|ts| e.created_at < ts).unwrap_or(true))
.collect(); .collect();
journal_nodes.sort_by_key(|n| n.created_at);
let cutoff_idx = if let Some(cutoff) = oldest_msg_ts {
let cutoff_ts = cutoff.timestamp();
let mut idx = journal_nodes.len();
for (i, node) in journal_nodes.iter().enumerate() {
if node.created_at >= cutoff_ts {
idx = i + 1;
break;
}
}
idx
} else {
journal_nodes.len()
};
let journal_budget = context::context_window() * 15 / 100; let journal_budget = context::context_window() * 15 / 100;
let mut entries = Vec::new(); let mut entries = Vec::new();
let mut total_tokens = 0; let mut total_tokens = 0;
for node in journal_nodes[..cutoff_idx].iter().rev() { // Take entries within budget (they're newest-first, so reverse for display)
let ts = chrono::DateTime::from_timestamp(node.created_at, 0); for entry in filtered.iter() {
let ast = AstNode::memory(&node.key, &node.content) let ts = chrono::DateTime::from_timestamp(entry.created_at, 0);
let ast = AstNode::memory(&entry.key, &entry.content)
.with_timestamp(ts.unwrap_or_else(chrono::Utc::now)); .with_timestamp(ts.unwrap_or_else(chrono::Utc::now));
let tok = ast.tokens(); let tok = ast.tokens();
if total_tokens + tok > journal_budget && !entries.is_empty() { if total_tokens + tok > journal_budget && !entries.is_empty() {
@ -553,7 +544,7 @@ impl Agent {
} }
pub async fn compact(&self) { pub async fn compact(&self) {
match crate::config::reload_for_model(&self.app_config, &self.prompt_file) { match crate::config::reload_for_model(&self.app_config, &self.prompt_file).await {
Ok(personality) => { Ok(personality) => {
let mut ctx = self.context.lock().await; let mut ctx = self.context.lock().await;
// System section (prompt + tools) set by new(), don't touch it // System section (prompt + tools) set by new(), don't touch it

20
src/agent/oneshot.rs
View file

@ -7,7 +7,7 @@
// Also contains the legacy run_one_agent() pipeline and process // Also contains the legacy run_one_agent() pipeline and process
// management for spawned agent subprocesses. // management for spawned agent subprocesses.
use crate::store::{self, Store}; use crate::store;
use crate::subconscious::{defs, prompts}; use crate::subconscious::{defs, prompts};
use std::collections::HashMap; use std::collections::HashMap;
@ -260,7 +260,7 @@ impl AutoAgent {
.map_err(|e| format!("config: {}", e))?; .map_err(|e| format!("config: {}", e))?;
let personality = crate::config::reload_for_model( let personality = crate::config::reload_for_model(
&app, &app.prompts.other, &app, &app.prompts.other,
).map_err(|e| format!("config: {}", e))?; ).await.map_err(|e| format!("config: {}", e))?;
let agent = Agent::new( let agent = Agent::new(
client, personality, client, personality,
@ -381,8 +381,7 @@ pub struct AgentResult {
/// Run an agent. If keys are provided, use them directly (bypassing the /// Run an agent. If keys are provided, use them directly (bypassing the
/// agent's query). Otherwise, run the query to select target nodes. /// agent's query). Otherwise, run the query to select target nodes.
pub fn run_one_agent( pub async fn run_one_agent(
store: &mut Store,
agent_name: &str, agent_name: &str,
count: usize, count: usize,
keys: Option<&[String]>, keys: Option<&[String]>,
@ -401,27 +400,22 @@ pub fn run_one_agent(
// Build prompt batch — either from explicit keys or the agent's query // Build prompt batch — either from explicit keys or the agent's query
let agent_batch = if let Some(keys) = keys { let agent_batch = if let Some(keys) = keys {
dbglog!("[{}] targeting: {}", agent_name, keys.join(", ")); dbglog!("[{}] targeting: {}", agent_name, keys.join(", "));
let graph = store.build_graph();
let mut resolved_steps = Vec::new(); let mut resolved_steps = Vec::new();
let mut all_keys: Vec<String> = keys.to_vec(); let mut all_keys: Vec<String> = keys.to_vec();
for step in &def.steps { for step in &def.steps {
let (prompt, extra_keys) = defs::resolve_placeholders( let (prompt, extra_keys) = defs::resolve_placeholders(
&step.prompt, store, &graph, keys, count, &step.prompt, keys, count,
); ).await;
all_keys.extend(extra_keys); all_keys.extend(extra_keys);
resolved_steps.push(prompts::ResolvedStep { resolved_steps.push(prompts::ResolvedStep {
prompt, prompt,
phase: step.phase.clone(), phase: step.phase.clone(),
}); });
} }
let batch = prompts::AgentBatch { steps: resolved_steps, node_keys: all_keys }; prompts::AgentBatch { steps: resolved_steps, node_keys: all_keys }
if !batch.node_keys.is_empty() {
store.record_agent_visits(&batch.node_keys, agent_name).ok();
}
batch
} else { } else {
let effective_count = def.count.unwrap_or(count); let effective_count = def.count.unwrap_or(count);
defs::run_agent(store, &def, effective_count, &Default::default())? defs::run_agent(&def, effective_count, &Default::default()).await?
}; };
// Base memory tools + extras from agent def (matching unconscious.rs pattern) // Base memory tools + extras from agent def (matching unconscious.rs pattern)

39
src/agent/tools/cd.rs Normal file
View file

@ -0,0 +1,39 @@
use std::sync::Arc;
use std::path::PathBuf;
// tools/cd.rs — Change working directory
//
// Uses the chdir syscall so it affects all tools.
pub fn tool() -> super::Tool {
super::Tool {
name: "cd",
description: "Change the current working directory.",
parameters_json: r#"{"type":"object","properties":{"path":{"type":"string","description":"The directory to change to (absolute or relative)"}},"required":["path"]}"#,
handler: Arc::new(|_agent, v| Box::pin(async move {
let path = v.get("path").and_then(|v| v.as_str())
.ok_or_else(|| anyhow::anyhow!("'path' parameter is required"))?;
if path.is_empty() { anyhow::bail!("'path' parameter cannot be empty"); }
// Resolve ~ to home directory
let resolved = if path.starts_with('~') {
let home = dirs::home_dir()
.ok_or_else(|| anyhow::anyhow!("could not determine home directory"))?;
home.join(path.strip_prefix("~/").unwrap_or(path))
} else {
PathBuf::from(path)
};
// Change directory (this is the actual chdir syscall)
std::env::set_current_dir(&resolved)
.map_err(|e| anyhow::anyhow!("cd: {}: {}", path, e))?;
// Return the canonical path
let canonical = std::env::current_dir()
.map(|p| p.display().to_string())
.unwrap_or_else(|_| resolved.display().to_string());
Ok(canonical)
})),
}
}

2
src/agent/tools/lsp.rs
View file

@ -123,7 +123,7 @@ fn find_project_root(file_path: &str) -> Option<String> {
const IDLE_TIMEOUT_SECS: u64 = 600; const IDLE_TIMEOUT_SECS: u64 = 600;
use std::sync::OnceLock; use std::sync::OnceLock;
use tokio::sync::Mutex as TokioMutex; use crate::Mutex as TokioMutex;
struct Registry { struct Registry {
configs: Vec<crate::config::LspServerConfig>, configs: Vec<crate::config::LspServerConfig>,

2
src/agent/tools/mcp_client.rs
View file

@ -10,7 +10,7 @@ use serde_json::json;
use std::sync::OnceLock; use std::sync::OnceLock;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, BufWriter}; use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, BufWriter};
use tokio::process::{Child, ChildStdin, ChildStdout, Command}; use tokio::process::{Child, ChildStdin, ChildStdout, Command};
use tokio::sync::Mutex as TokioMutex; use crate::Mutex as TokioMutex;
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct McpTool { pub struct McpTool {

684
src/agent/tools/memory.rs
View file

@ -1,16 +1,30 @@
use std::sync::Arc;
// tools/memory.rs — Native memory graph operations // tools/memory.rs — Native memory graph operations
// //
// Direct library calls into the store — no subprocess spawning. // Access via hippocampus::access() / access_local(). Clients try socket
// One function per tool for use in the Tool registry. // first, fall back to local store.
#![allow(unused_variables)] // macro-generated args for no-param tools
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use std::sync::Arc;
use crate::hippocampus::{access, memory_rpc, StoreAccess};
use crate::hippocampus::memory::MemoryNode; // Re-export typed API from hippocampus for backward compatibility
use crate::store::StoreView; pub use crate::hippocampus::{
use crate::store::Store; memory_render, memory_write, memory_search, memory_link_set, memory_link_add,
memory_delete, memory_history, memory_weight_set, memory_rename, memory_supersede,
memory_query, memory_links,
journal_tail, journal_new, journal_update,
graph_topology, graph_health, graph_communities, graph_normalize_strengths,
graph_link_impact, graph_hubs, graph_trace,
socket_path,
};
// ── Helpers ──────────────────────────────────────────────────── // ── Macro for generating tool wrappers ─────────────────────────
//
// memory_tool!(name, mut, arg1: [str], arg2: [Option<bool>])
// - mut/ref for store mutability
// - generates jsonargs_* (internal, JSON args) and public typed API
fn get_str<'a>(args: &'a serde_json::Value, name: &'a str) -> Result<&'a str> { fn get_str<'a>(args: &'a serde_json::Value, name: &'a str) -> Result<&'a str> {
args.get(name).and_then(|v| v.as_str()).context(format!("{} is required", name)) args.get(name).and_then(|v| v.as_str()).context(format!("{} is required", name))
@ -20,10 +34,7 @@ fn get_f64(args: &serde_json::Value, name: &str) -> Result<f64> {
args.get(name).and_then(|v| v.as_f64()).context(format!("{} is required", name)) args.get(name).and_then(|v| v.as_f64()).context(format!("{} is required", name))
} }
async fn cached_store() -> Result<std::sync::Arc<tokio::sync::Mutex<Store>>> { /// Get provenance from agent state, or "manual".
Store::cached().await.map_err(|e| anyhow::anyhow!("{}", e))
}
async fn get_provenance(agent: &Option<std::sync::Arc<crate::agent::Agent>>) -> String { async fn get_provenance(agent: &Option<std::sync::Arc<crate::agent::Agent>>) -> String {
match agent { match agent {
Some(a) => a.state.lock().await.provenance.clone(), Some(a) => a.state.lock().await.provenance.clone(),
@ -31,381 +42,332 @@ async fn get_provenance(agent: &Option<std::sync::Arc<crate::agent::Agent>>) ->
} }
} }
macro_rules! memory_tool {
// ── Helper rules (must come first) ─────────────────────────────
// Extract from JSON
(@extract $args:ident, $name:ident, str) => {
get_str($args, stringify!($name))?
};
(@extract $args:ident, $name:ident, f32) => {
get_f64($args, stringify!($name))? as f32
};
(@extract $args:ident, $name:ident, Vec<String>) => {
$args.get(stringify!($name))
.and_then(|v| v.as_array())
.map(|arr| arr.iter().filter_map(|v| v.as_str().map(String::from)).collect::<Vec<_>>())
.unwrap_or_default()
};
(@extract $args:ident, $name:ident, Option<&str>) => {
$args.get(stringify!($name)).and_then(|v| v.as_str())
};
(@extract $args:ident, $name:ident, Option<bool>) => {
$args.get(stringify!($name)).and_then(|v| v.as_bool())
};
(@extract $args:ident, $name:ident, Option<u64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64())
};
(@extract $args:ident, $name:ident, Option<i64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_i64())
};
(@extract $args:ident, $name:ident, Option<usize>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64()).map(|v| v as usize)
};
(@extract $args:ident, $name:ident, Option<u32>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64()).map(|v| v as u32)
};
(@extract $args:ident, $name:ident, Option<f64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_f64())
};
// Parameter types for function signatures
(@param_type str) => { &str };
(@param_type f32) => { f32 };
(@param_type Vec<String>) => { Vec<String> };
(@param_type Option<&str>) => { Option<&str> };
(@param_type Option<bool>) => { Option<bool> };
(@param_type Option<u64>) => { Option<u64> };
(@param_type Option<i64>) => { Option<i64> };
(@param_type Option<usize>) => { Option<usize> };
(@param_type Option<u32>) => { Option<u32> };
(@param_type Option<f64>) => { Option<f64> };
// Serialize result for jsonargs
(@serialize $t:ty, $result:expr) => { serde_json::to_string(&$result)? };
// Deserialize RPC response
(@deserialize $t:ty, $json:expr) => { serde_json::from_str(&$json).map_err(|e| anyhow::anyhow!("{}", e)) };
// Serialize to JSON for RPC
(@insert_json $map:ident, $name:ident, str) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, f32) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, Vec<String>) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, Option<&str>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<bool>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<u64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<i64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<usize>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<u32>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<f64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
// Call hippocampus (all methods now take &self, deref Arc)
(@call mut, $name:ident, $store:ident, $prov:expr $(, $arg:expr)*) => {
crate::hippocampus::local::$name(&*$store, $prov $(, $arg)*)
};
(@call ref, $name:ident, $store:ident, $prov:expr $(, $arg:expr)*) => {
crate::hippocampus::local::$name(&*$store, $prov $(, $arg)*)
};
// ── Main rules ─────────────────────────────────────────────────
// Shorthand: mut/ref without return type defaults to String
($name:ident, $m:ident $(, $($arg:ident : [$($typ:tt)+]),* $(,)?)?) => {
memory_tool!($name, $m -> String $(, $($arg : [$($typ)+]),*)?);
};
// Full form with return type
($name:ident, $m:ident -> $ret:ty $(, $($arg:ident : [$($typ:tt)+]),* $(,)?)?) => {
paste::paste! {
async fn [<jsonargs_ $name>](agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
$($(let $arg = memory_tool!(@extract args, $arg, $($typ)+);)*)?
let prov = get_provenance(agent).await;
match access() {
StoreAccess::Daemon(store) => {
let result: $ret = memory_tool!(@call $m, $name, store, &prov $($(, $arg)*)?)?;
Ok(memory_tool!(@serialize $ret, result))
}
StoreAccess::Client => {
#[allow(unused_mut)]
let mut map = serde_json::Map::new();
$($(memory_tool!(@insert_json map, $arg, $($typ)+);)*)?
memory_rpc(stringify!($name), serde_json::Value::Object(map))
}
StoreAccess::None(err) => anyhow::bail!("{}", err),
}
}
}
};
}
// ── Memory tools ───────────────────────────────────────────────
memory_tool!(memory_render, ref, key: [str], raw: [Option<bool>]);
memory_tool!(memory_write, mut, key: [str], content: [str]);
memory_tool!(memory_search, ref, keys: [Vec<String>], max_hops: [Option<u32>], edge_decay: [Option<f64>], min_activation: [Option<f64>], limit: [Option<usize>]);
memory_tool!(memory_link_set, mut, source: [str], target: [str], strength: [f32]);
memory_tool!(memory_link_add, mut, source: [str], target: [str]);
memory_tool!(memory_delete, mut, key: [str]);
memory_tool!(memory_history, ref, key: [str], full: [Option<bool>]);
memory_tool!(memory_weight_set, mut, key: [str], weight: [f32]);
memory_tool!(memory_rename, mut, old_key: [str], new_key: [str]);
memory_tool!(memory_supersede, mut, old_key: [str], new_key: [str], reason: [Option<&str>]);
memory_tool!(memory_query, ref, query: [str], format: [Option<&str>]);
// Re-export types and typed API from hippocampus
pub use crate::hippocampus::local::LinkInfo;
memory_tool!(memory_links, ref -> Vec<LinkInfo>, key: [str]);
// ── Journal tools ──────────────────────────────────────────────
pub use crate::hippocampus::local::JournalEntry;
memory_tool!(journal_tail, ref -> Vec<JournalEntry>, count: [Option<u64>], level: [Option<u64>], after: [Option<&str>]);
memory_tool!(journal_new, mut, name: [str], title: [str], body: [str], level: [Option<i64>]);
memory_tool!(journal_update, mut, body: [str], level: [Option<i64>]);
// ── Graph tools ───────────────────────────────────────────────
memory_tool!(graph_topology, ref);
memory_tool!(graph_health, ref);
memory_tool!(graph_communities, ref, top_n: [Option<usize>], min_size: [Option<usize>]);
memory_tool!(graph_normalize_strengths, mut, apply: [Option<bool>]);
memory_tool!(graph_link_impact, ref, source: [str], target: [str]);
memory_tool!(graph_hubs, ref, count: [Option<usize>]);
memory_tool!(graph_trace, ref, key: [str]);
// ── Definitions ──────────────────────────────────────────────── // ── Definitions ────────────────────────────────────────────────
pub fn memory_tools() -> [super::Tool; 13] { pub fn memory_tools() -> [super::Tool; 19] {
use super::Tool; use super::Tool;
macro_rules! tool {
($name:ident, $desc:expr, $params:expr) => {
Tool {
name: stringify!($name),
description: $desc,
parameters_json: $params,
handler: Arc::new(|a, v| Box::pin(async move {
paste::paste! { [<jsonargs_ $name>](&a, &v).await }
})),
}
};
}
[ [
Tool { name: "memory_render", description: "Read a memory node's content and links.", tool!(memory_render, "Read a memory node's content and links.", r#"{
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Node key"}},"required":["key"]}"#, "type": "object",
handler: Arc::new(|_a, v| Box::pin(async move { render(&v).await })) }, "properties": { "key": {"type": "string"}, "raw": {"type": "boolean"} },
Tool { name: "memory_write", description: "Create or update a memory node.", "required": ["key"]
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Node key"},"content":{"type":"string","description":"Full content (markdown)"}},"required":["key","content"]}"#, }"#),
handler: Arc::new(|a, v| Box::pin(async move { write(&a, &v).await })) }, tool!(memory_write, "Create or update a memory node.", r#"{
Tool { name: "memory_search", description: "Search the memory graph via spreading activation. Give 2-4 seed node keys.", "type": "object",
parameters_json: r#"{"type":"object","properties":{"keys":{"type":"array","items":{"type":"string"},"description":"Seed node keys to activate from"}},"required":["keys"]}"#, "properties": { "key": {"type": "string"}, "content": {"type": "string"} },
handler: Arc::new(|_a, v| Box::pin(async move { search(&v).await })) }, "required": ["key", "content"]
Tool { name: "memory_links", description: "Show a node's neighbors with link strengths.", }"#),
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Node key"}},"required":["key"]}"#, tool!(memory_search, "Search via spreading activation from seed keys.", r#"{
handler: Arc::new(|_a, v| Box::pin(async move { links(&v) })) },
Tool { name: "memory_link_set", description: "Set link strength between two nodes.",
parameters_json: r#"{"type":"object","properties":{"source":{"type":"string"},"target":{"type":"string"},"strength":{"type":"number","description":"0.01 to 1.0"}},"required":["source","target","strength"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { link_set(&v).await })) },
Tool { name: "memory_link_add", description: "Add a new link between two nodes.",
parameters_json: r#"{"type":"object","properties":{"source":{"type":"string"},"target":{"type":"string"}},"required":["source","target"]}"#,
handler: Arc::new(|a, v| Box::pin(async move { link_add(&a, &v).await })) },
Tool { name: "memory_used", description: "Mark a node as useful (boosts weight).",
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Node key"}},"required":["key"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { used(&v).await })) },
Tool { name: "memory_weight_set", description: "Set a node's weight directly (0.01 to 1.0).",
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string"},"weight":{"type":"number","description":"0.01 to 1.0"}},"required":["key","weight"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { weight_set(&v).await })) },
Tool { name: "memory_rename", description: "Rename a node key in place.",
parameters_json: r#"{"type":"object","properties":{"old_key":{"type":"string"},"new_key":{"type":"string"}},"required":["old_key","new_key"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { rename(&v).await })) },
Tool { name: "memory_supersede", description: "Mark a node as superseded by another (sets weight to 0.01).",
parameters_json: r#"{"type":"object","properties":{"old_key":{"type":"string"},"new_key":{"type":"string"},"reason":{"type":"string"}},"required":["old_key","new_key"]}"#,
handler: Arc::new(|a, v| Box::pin(async move { supersede(&a, &v).await })) },
Tool { name: "memory_query",
description: "Run a structured query against the memory graph.",
parameters_json: r#"{
"type": "object", "type": "object",
"properties": { "properties": {
"query": {"type": "string", "description": "Query expression"}, "keys": {"type": "array", "items": {"type": "string"}},
"format": {"type": "string", "description": "compact (default) or full (with content and graph metrics)", "default": "compact"} "max_hops": {"type": "integer"},
"edge_decay": {"type": "number"},
"min_activation": {"type": "number"},
"limit": {"type": "integer"}
},
"required": ["keys"]
}"#),
tool!(memory_links, "Show a node's neighbors with link strengths.", r#"{
"type": "object",
"properties": { "key": {"type": "string"} },
"required": ["key"]
}"#),
tool!(memory_link_set, "Set link strength between two nodes.", r#"{
"type": "object",
"properties": {
"source": {"type": "string"},
"target": {"type": "string"},
"strength": {"type": "number", "description": "0.01 to 1.0"}
},
"required": ["source", "target", "strength"]
}"#),
tool!(memory_link_add, "Add a new link between two nodes.", r#"{
"type": "object",
"properties": { "source": {"type": "string"}, "target": {"type": "string"} },
"required": ["source", "target"]
}"#),
tool!(memory_delete, "Delete a memory node.", r#"{
"type": "object",
"properties": { "key": {"type": "string"} },
"required": ["key"]
}"#),
tool!(memory_history, "Show version history for a node.", r#"{
"type": "object",
"properties": { "key": {"type": "string"}, "full": {"type": "boolean"} },
"required": ["key"]
}"#),
tool!(memory_weight_set, "Set a node's weight (0.01 to 1.0).", r#"{
"type": "object",
"properties": { "key": {"type": "string"}, "weight": {"type": "number"} },
"required": ["key", "weight"]
}"#),
tool!(memory_rename, "Rename a node key.", r#"{
"type": "object",
"properties": { "old_key": {"type": "string"}, "new_key": {"type": "string"} },
"required": ["old_key", "new_key"]
}"#),
tool!(memory_supersede, "Mark a node as superseded by another.", r#"{
"type": "object",
"properties": {
"old_key": {"type": "string"},
"new_key": {"type": "string"},
"reason": {"type": "string"}
},
"required": ["old_key", "new_key"]
}"#),
tool!(memory_query, "Run a structured query against the memory graph.", r#"{
"type": "object",
"properties": {
"query": {"type": "string"},
"format": {"type": "string", "description": "compact or full"}
}, },
"required": ["query"] "required": ["query"]
}"#, }"#),
handler: Arc::new(|_a, v| Box::pin(async move { query(&v).await })) }, tool!(graph_topology, "Show graph topology stats.", r#"{"type": "object"}"#),
Tool { name: "graph_topology", description: "Show graph topology stats (nodes, edges, clustering, hubs).", tool!(graph_health, "Show graph health report.", r#"{"type": "object"}"#),
parameters_json: r#"{"type":"object","properties":{}}"#, tool!(graph_hubs, "Show top hub nodes by degree.", r#"{
handler: Arc::new(|_a, _v| Box::pin(async { graph_topology().await })) }, "type": "object",
Tool { name: "graph_health", description: "Show graph health report with maintenance recommendations.", "properties": { "count": {"type": "integer"} }
parameters_json: r#"{"type":"object","properties":{}}"#, }"#),
handler: Arc::new(|_a, _v| Box::pin(async { graph_health().await })) }, tool!(graph_communities, "Show communities by isolation.", r#"{
"type": "object",
"properties": { "top_n": {"type": "integer"}, "min_size": {"type": "integer"} }
}"#),
tool!(graph_normalize_strengths, "Set link strengths from Jaccard similarity.", r#"{
"type": "object",
"properties": { "apply": {"type": "boolean"} }
}"#),
tool!(graph_link_impact, "Simulate adding an edge, report impact.", r#"{
"type": "object",
"properties": { "source": {"type": "string"}, "target": {"type": "string"} },
"required": ["source", "target"]
}"#),
tool!(graph_trace, "Walk temporal links from a node.", r#"{
"type": "object",
"properties": { "key": {"type": "string"} },
"required": ["key"]
}"#),
] ]
} }
pub fn journal_tools() -> [super::Tool; 3] { pub fn journal_tools() -> [super::Tool; 3] {
use super::Tool; use super::Tool;
macro_rules! tool {
($name:ident, $desc:expr, $params:expr) => {
Tool {
name: stringify!($name),
description: $desc,
parameters_json: $params,
handler: Arc::new(|a, v| Box::pin(async move {
paste::paste! { [<jsonargs_ $name>](&a, &v).await }
})),
}
};
}
[ [
Tool { name: "journal_tail", tool!(journal_tail, "Read the last N entries at a given level.", r#"{
description: "Read the last N entries at a given level.",
parameters_json: r#"{
"type": "object", "type": "object",
"properties": { "properties": {
"count": {"type": "integer", "description": "Number of entries", "default": 1}, "count": {"type": "integer"},
"level": {"type": "integer", "description": "0=journal, 1=daily, 2=weekly, 3=monthly", "default": 0}, "level": {"type": "integer", "description": "0=journal, 1=daily, 2=weekly, 3=monthly"},
"format": {"type": "string", "description": "compact or full (with content)", "default": "full"}, "format": {"type": "string", "description": "compact or full"},
"after": {"type": "string", "description": "Only entries after this date (YYYY-MM-DD)"} "after": {"type": "string", "description": "Only entries after this date (YYYY-MM-DD)"}
} }
}"#, }"#),
handler: Arc::new(|_a, v| Box::pin(async move { journal_tail(&v).await })) }, tool!(journal_new, "Start a new journal/digest entry.", r#"{
Tool { name: "journal_new", description: "Start a new journal/digest entry.",
parameters_json: r#"{
"type": "object", "type": "object",
"properties": { "properties": {
"name": {"type": "string", "description": "Short node name (becomes the key)"}, "name": {"type": "string"},
"title": {"type": "string", "description": "Descriptive title"}, "title": {"type": "string"},
"body": {"type": "string", "description": "Entry body"}, "body": {"type": "string"},
"level": {"type": "integer", "description": "0=journal, 1=daily, 2=weekly, 3=monthly", "default": 0} "level": {"type": "integer"}
}, },
"required": ["name", "title", "body"] "required": ["name", "title", "body"]
}"#, }"#),
handler: Arc::new(|a, v| Box::pin(async move { journal_new(&a, &v).await })) }, tool!(journal_update, "Append text to the most recent entry.", r#"{
Tool { name: "journal_update", description: "Append text to the most recent entry at a level.",
parameters_json: r#"{
"type": "object", "type": "object",
"properties": { "properties": {
"body": {"type": "string", "description": "Text to append"}, "body": {"type": "string"},
"level": {"type": "integer", "description": "0=journal, 1=daily, 2=weekly, 3=monthly", "default": 0} "level": {"type": "integer"}
}, },
"required": ["body"] "required": ["body"]
}"#, }"#),
handler: Arc::new(|a, v| Box::pin(async move { journal_update(&a, &v).await })) },
] ]
} }
// ── Memory tools ───────────────────────────────────────────────
async fn render(args: &serde_json::Value) -> Result<String> {
let key = get_str(args, "key")?;
let arc = cached_store().await?;
let store = arc.lock().await;
Ok(MemoryNode::from_store(&store, key)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", key))?
.render())
}
async fn write(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let key = get_str(args, "key")?;
let content = get_str(args, "content")?;
let prov = get_provenance(agent).await;
let arc = cached_store().await?;
let mut store = arc.lock().await;
let result = store.upsert_provenance(key, content, &prov)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("{} '{}'", result, key))
}
async fn search(args: &serde_json::Value) -> Result<String> {
let keys: Vec<String> = args.get("keys")
.and_then(|v| v.as_array())
.map(|arr| arr.iter().filter_map(|v| v.as_str().map(String::from)).collect())
.unwrap_or_default();
if keys.is_empty() {
anyhow::bail!("memory_search requires at least one seed key");
}
let arc = cached_store().await?;
let store = arc.lock().await;
let graph = crate::graph::build_graph_fast(&*store);
let params = store.params();
let seeds: Vec<(String, f64)> = keys.iter()
.filter_map(|k| {
let resolved = store.resolve_key(k).ok()?;
Some((resolved, 1.0))
})
.collect();
if seeds.is_empty() {
anyhow::bail!("no valid seed keys found");
}
let seed_set: std::collections::HashSet<&str> = seeds.iter()
.map(|(k, _)| k.as_str()).collect();
let results = crate::search::spreading_activation(
&seeds, &graph, &*store,
params.max_hops, params.edge_decay, params.min_activation,
);
Ok(results.iter()
.filter(|(k, _)| !seed_set.contains(k.as_str()))
.take(20)
.map(|(key, score)| format!(" {:.2} {}", score, key))
.collect::<Vec<_>>().join("\n"))
}
fn links(args: &serde_json::Value) -> Result<String> {
let key = get_str(args, "key")?;
let node = MemoryNode::load(key)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", key))?;
let mut out = format!("Neighbors of '{}':\n", key);
for (target, strength, is_new) in &node.links {
let tag = if *is_new { " (new)" } else { "" };
out.push_str(&format!(" ({:.2}) {}{}\n", strength, target, tag));
}
Ok(out)
}
async fn link_set(args: &serde_json::Value) -> Result<String> {
let arc = cached_store().await?;
let mut store = arc.lock().await;
let s = store.resolve_key(get_str(args, "source")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let t = store.resolve_key(get_str(args, "target")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let strength = get_f64(args, "strength")? as f32;
let old = store.set_link_strength(&s, &t, strength).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("{} ↔ {} strength {:.2} → {:.2}", s, t, old, strength))
}
async fn link_add(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let arc = cached_store().await?;
let mut store = arc.lock().await;
let s = store.resolve_key(get_str(args, "source")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let t = store.resolve_key(get_str(args, "target")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let prov = get_provenance(agent).await;
let strength = store.add_link(&s, &t, &prov).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("linked {} → {} (strength={:.2})", s, t, strength))
}
async fn used(args: &serde_json::Value) -> Result<String> {
let key = get_str(args, "key")?;
let arc = cached_store().await?;
let mut store = arc.lock().await;
if !store.nodes.contains_key(key) {
anyhow::bail!("node not found: {}", key);
}
store.mark_used(key);
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("marked {} as used", key))
}
async fn weight_set(args: &serde_json::Value) -> Result<String> {
let arc = cached_store().await?;
let mut store = arc.lock().await;
let key = store.resolve_key(get_str(args, "key")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let weight = get_f64(args, "weight")? as f32;
let (old, new) = store.set_weight(&key, weight).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("weight {} {:.2} → {:.2}", key, old, new))
}
async fn rename(args: &serde_json::Value) -> Result<String> {
let old_key = get_str(args, "old_key")?;
let new_key = get_str(args, "new_key")?;
let arc = cached_store().await?;
let mut store = arc.lock().await;
let resolved = store.resolve_key(old_key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.rename_node(&resolved, new_key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("Renamed '{}' → '{}'", resolved, new_key))
}
async fn supersede(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let old_key = get_str(args, "old_key")?;
let new_key = get_str(args, "new_key")?;
let reason = args.get("reason").and_then(|v| v.as_str()).unwrap_or("superseded");
let arc = cached_store().await?;
let mut store = arc.lock().await;
let content = store.nodes.get(old_key)
.map(|n| n.content.clone())
.ok_or_else(|| anyhow::anyhow!("node not found: {}", old_key))?;
let notice = format!("**SUPERSEDED** by `{}` — {}\n\n---\n\n{}",
new_key, reason, content.trim());
let prov = get_provenance(agent).await;
store.upsert_provenance(old_key, &notice, &prov)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.set_weight(old_key, 0.01).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("superseded {} → {} ({})", old_key, new_key, reason))
}
async fn query(args: &serde_json::Value) -> Result<String> {
let query_str = get_str(args, "query")?;
let format = args.get("format").and_then(|v| v.as_str()).unwrap_or("compact");
let arc = cached_store().await?;
let store = arc.lock().await;
let graph = store.build_graph();
match format {
"full" => {
// Rich output with full content, graph metrics, hub analysis
let results = crate::query_parser::execute_query(&store, &graph, query_str)
.map_err(|e| anyhow::anyhow!("{}", e))?;
let keys: Vec<String> = results.into_iter().map(|r| r.key).collect();
let items = crate::subconscious::defs::keys_to_replay_items(&store, &keys, &graph);
Ok(crate::subconscious::prompts::format_nodes_section(&store, &items, &graph))
}
_ => {
// Compact output: handles count, select, and all expression types
crate::query_parser::query_to_string(&store, &graph, query_str)
.map_err(|e| anyhow::anyhow!("{}", e))
}
}
}
// ── Journal tools ──────────────────────────────────────────────
async fn journal_tail(args: &serde_json::Value) -> Result<String> {
let count = args.get("count").and_then(|v| v.as_u64()).unwrap_or(1);
let level = args.get("level").and_then(|v| v.as_u64()).unwrap_or(0);
let format = args.get("format").and_then(|v| v.as_str()).unwrap_or("full");
let after = args.get("after").and_then(|v| v.as_str());
let type_name = match level {
0 => "episodic",
1 => "daily",
2 => "weekly",
3 => "monthly",
_ => return Err(anyhow::anyhow!("invalid level: {} (0=journal, 1=daily, 2=weekly, 3=monthly)", level)),
};
let mut q = format!("all | type:{} | sort:timestamp", type_name);
if let Some(date) = after {
// Convert date to age in seconds
if let Ok(nd) = chrono::NaiveDate::parse_from_str(date, "%Y-%m-%d") {
let ts = nd.and_hms_opt(0, 0, 0).unwrap().and_utc().timestamp();
let age = chrono::Utc::now().timestamp() - ts;
q.push_str(&format!(" | age:<{}", age));
}
}
q.push_str(&format!(" | limit:{}", count));
query(&serde_json::json!({"query": q, "format": format})).await
}
fn level_to_node_type(level: i64) -> crate::store::NodeType {
match level {
1 => crate::store::NodeType::EpisodicDaily,
2 => crate::store::NodeType::EpisodicWeekly,
3 => crate::store::NodeType::EpisodicMonthly,
_ => crate::store::NodeType::EpisodicSession,
}
}
async fn journal_new(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let name = get_str(args, "name")?;
let title = get_str(args, "title")?;
let body = get_str(args, "body")?;
let level = args.get("level").and_then(|v| v.as_i64()).unwrap_or(0);
let ts = chrono::Local::now().format("%Y-%m-%dT%H:%M");
let content = format!("## {} — {}\n\n{}", ts, title, body);
let base_key: String = name.split_whitespace()
.map(|w| w.to_lowercase()
.chars().filter(|c| c.is_alphanumeric() || *c == '-')
.collect::<String>())
.filter(|s| !s.is_empty())
.collect::<Vec<_>>()
.join("-");
let base_key = if base_key.len() > 80 { &base_key[..80] } else { base_key.as_str() };
let arc = cached_store().await?;
let mut store = arc.lock().await;
let key = if store.nodes.contains_key(base_key) {
let mut n = 2;
loop {
let candidate = format!("{}-{}", base_key, n);
if !store.nodes.contains_key(&candidate) { break candidate; }
n += 1;
}
} else {
base_key.to_string()
};
let mut node = crate::store::new_node(&key, &content);
node.node_type = level_to_node_type(level);
node.provenance = get_provenance(agent).await;
store.upsert_node(node).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
let word_count = body.split_whitespace().count();
Ok(format!("New entry '{}' ({} words)", title, word_count))
}
async fn journal_update(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let body = get_str(args, "body")?;
let level = args.get("level").and_then(|v| v.as_i64()).unwrap_or(0);
let node_type = level_to_node_type(level);
let arc = cached_store().await?;
let mut store = arc.lock().await;
let latest_key = store.nodes.values()
.filter(|n| n.node_type == node_type)
.max_by_key(|n| n.created_at)
.map(|n| n.key.clone());
let Some(key) = latest_key else {
anyhow::bail!("no entry at level {} to update — use journal_new first", level);
};
let existing = store.nodes.get(&key).unwrap().content.clone();
let new_content = format!("{}\n\n{}", existing.trim_end(), body);
let prov = get_provenance(agent).await;
store.upsert_provenance(&key, &new_content, &prov)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
let word_count = body.split_whitespace().count();
Ok(format!("Updated last entry (+{} words)", word_count))
}
// ── Graph tools ───────────────────────────────────────────────
async fn graph_topology() -> Result<String> {
let arc = cached_store().await?;
let store = arc.lock().await;
let graph = store.build_graph();
Ok(crate::subconscious::prompts::format_topology_header(&graph))
}
async fn graph_health() -> Result<String> {
let arc = cached_store().await?;
let store = arc.lock().await;
let graph = store.build_graph();
Ok(crate::subconscious::prompts::format_health_section(&store, &graph))
}

7
src/agent/tools/mod.rs
View file

@ -6,13 +6,14 @@
// Core tools // Core tools
mod ast_grep; mod ast_grep;
pub mod lsp;
pub mod mcp_client;
mod bash; mod bash;
mod cd;
pub mod channels; pub mod channels;
mod edit; mod edit;
mod glob; mod glob;
mod grep; mod grep;
pub mod lsp;
pub mod mcp_client;
pub mod memory; pub mod memory;
mod read; mod read;
mod web; mod web;
@ -177,7 +178,7 @@ pub async fn dispatch_with_agent(
pub fn tools() -> Vec<Tool> { pub fn tools() -> Vec<Tool> {
let mut all = vec![ let mut all = vec![
read::tool(), write::tool(), edit::tool(), read::tool(), write::tool(), edit::tool(),
grep::tool(), glob::tool(), bash::tool(), grep::tool(), glob::tool(), bash::tool(), cd::tool(),
ast_grep::tool(), vision::tool(), ast_grep::tool(), vision::tool(),
]; ];
all.extend(web::tools()); all.extend(web::tools());

7
src/bin/consciousness.rs
View file

@ -1,2 +1,7 @@
#![feature(panic_backtrace_config)]
#![warn(unreachable_pub)] #![warn(unreachable_pub)]
fn main() { consciousness::user::main() }
fn main() {
std::panic::set_backtrace_style(std::panic::BacktraceStyle::Short);
consciousness::user::main()
}

30
src/bin/merge-logs.rs
View file

@ -22,6 +22,7 @@ use std::fs;
use std::io::{BufReader, BufWriter}; use std::io::{BufReader, BufWriter};
use std::path::Path; use std::path::Path;
use anyhow::{bail, Context, Result};
use capnp::message; use capnp::message;
use capnp::serialize; use capnp::serialize;
@ -29,17 +30,17 @@ use consciousness::memory_capnp;
use consciousness::store::Node; use consciousness::store::Node;
/// Read all node entries from a capnp log file, preserving order. /// Read all node entries from a capnp log file, preserving order.
fn read_all_entries(path: &Path) -> Result<Vec<Node>, String> { fn read_all_entries(path: &Path) -> Result<Vec<Node>> {
let file = fs::File::open(path) let file = fs::File::open(path)
.map_err(|e| format!("open {}: {}", path.display(), e))?; .with_context(|| format!("open {}", path.display()))?;
let mut reader = BufReader::new(file); let mut reader = BufReader::new(file);
let mut entries = Vec::new(); let mut entries = Vec::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) { while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>() let log = msg.get_root::<memory_capnp::node_log::Reader>()
.map_err(|e| format!("read log from {}: {}", path.display(), e))?; .with_context(|| format!("read log from {}", path.display()))?;
for node_reader in log.get_nodes() for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes from {}: {}", path.display(), e))? { .with_context(|| format!("get nodes from {}", path.display()))? {
let node = Node::from_capnp_migrate(node_reader)?; let node = Node::from_capnp_migrate(node_reader)?;
entries.push(node); entries.push(node);
} }
@ -49,9 +50,9 @@ fn read_all_entries(path: &Path) -> Result<Vec<Node>, String> {
} }
/// Write node entries to a new capnp log file in chunks. /// Write node entries to a new capnp log file in chunks.
fn write_entries(path: &Path, entries: &[Node]) -> Result<(), String> { fn write_entries(path: &Path, entries: &[Node]) -> Result<()> {
let file = fs::File::create(path) let file = fs::File::create(path)
.map_err(|e| format!("create {}: {}", path.display(), e))?; .with_context(|| format!("create {}", path.display()))?;
let mut writer = BufWriter::new(file); let mut writer = BufWriter::new(file);
for chunk in entries.chunks(100) { for chunk in entries.chunks(100) {
@ -64,13 +65,13 @@ fn write_entries(path: &Path, entries: &[Node]) -> Result<(), String> {
} }
} }
serialize::write_message(&mut writer, &msg) serialize::write_message(&mut writer, &msg)
.map_err(|e| format!("write: {}", e))?; .context("write message")?;
} }
Ok(()) Ok(())
} }
fn main() -> Result<(), String> { fn main() -> Result<()> {
let args: Vec<String> = std::env::args().collect(); let args: Vec<String> = std::env::args().collect();
if args.len() != 4 { if args.len() != 4 {
eprintln!("Usage: merge-logs <old_log> <current_log> <output_dir>"); eprintln!("Usage: merge-logs <old_log> <current_log> <output_dir>");
@ -87,19 +88,18 @@ fn main() -> Result<(), String> {
// Validate inputs exist // Validate inputs exist
if !old_path.exists() { if !old_path.exists() {
return Err(format!("old log not found: {}", old_path.display())); bail!("old log not found: {}", old_path.display());
} }
if !current_path.exists() { if !current_path.exists() {
return Err(format!("current log not found: {}", current_path.display())); bail!("current log not found: {}", current_path.display());
} }
// Create output directory (must not already contain nodes.capnp) // Create output directory (must not already contain nodes.capnp)
fs::create_dir_all(output_dir) fs::create_dir_all(output_dir)
.map_err(|e| format!("create output dir: {}", e))?; .context("create output dir")?;
let output_path = output_dir.join("nodes.capnp"); let output_path = output_dir.join("nodes.capnp");
if output_path.exists() { if output_path.exists() {
return Err(format!("output already exists: {} — refusing to overwrite", bail!("output already exists: {} — refusing to overwrite", output_path.display());
output_path.display()));
} }
eprintln!("Reading old log: {} ...", old_path.display()); eprintln!("Reading old log: {} ...", old_path.display());
@ -190,8 +190,8 @@ fn main() -> Result<(), String> {
eprintln!(" Replay produces {} live nodes", final_nodes.len()); eprintln!(" Replay produces {} live nodes", final_nodes.len());
if verify_entries.len() != merged.len() { if verify_entries.len() != merged.len() {
return Err(format!("Verification failed: wrote {} but read back {}", bail!("Verification failed: wrote {} but read back {}",
merged.len(), verify_entries.len())); merged.len(), verify_entries.len());
} }
eprintln!(); eprintln!();

440
src/cli/admin.rs
View file

@ -1,23 +1,23 @@
// cli/admin.rs — admin subcommand handlers // cli/admin.rs — admin subcommand handlers
use crate::store; use anyhow::Result;
fn install_default_file(data_dir: &std::path::Path, name: &str, content: &str) -> Result<(), String> { use crate::hippocampus as memory;
use crate::hippocampus::store;
fn install_default_file(data_dir: &std::path::Path, name: &str, content: &str) -> Result<()> {
let path = data_dir.join(name); let path = data_dir.join(name);
if !path.exists() { if !path.exists() {
std::fs::write(&path, content) std::fs::write(&path, content)?;
.map_err(|e| format!("write {}: {}", name, e))?;
println!("Created {}", path.display()); println!("Created {}", path.display());
} }
Ok(()) Ok(())
} }
pub async fn cmd_init() -> Result<()> {
pub fn cmd_init() -> Result<(), String> {
let cfg = crate::config::get(); let cfg = crate::config::get();
// Ensure data directory exists // Ensure data directory exists
std::fs::create_dir_all(&cfg.data_dir) std::fs::create_dir_all(&cfg.data_dir)?;
.map_err(|e| format!("create data_dir: {}", e))?;
// Install filesystem files (not store nodes) // Install filesystem files (not store nodes)
install_default_file(&cfg.data_dir, "instructions.md", install_default_file(&cfg.data_dir, "instructions.md",
@ -25,19 +25,15 @@ pub fn cmd_init() -> Result<(), String> {
install_default_file(&cfg.data_dir, "on-consciousness.md", install_default_file(&cfg.data_dir, "on-consciousness.md",
include_str!("../../defaults/on-consciousness.md"))?; include_str!("../../defaults/on-consciousness.md"))?;
// Initialize store and seed default identity node if empty // Seed identity node if empty
let mut store = store::Store::load()?; let store = memory::access_local()?;
let count = store.init_from_markdown()?; if !store.contains_key("identity").unwrap_or(false) {
for key in &cfg.core_nodes {
if !store.nodes.contains_key(key) && key == "identity" {
let default = include_str!("../../defaults/identity.md"); let default = include_str!("../../defaults/identity.md");
store.upsert(key, default) store.upsert("identity", default)?;
.map_err(|e| format!("seed {}: {}", key, e))?; println!("Seeded identity in store");
println!("Seeded {} in store", key);
}
} }
store.save()?; store.save()?;
println!("Indexed {} memory units", count); println!("Initialized with {} nodes", store.all_keys().unwrap_or_default().len());
// Create config if none exists // Create config if none exists
let config_path = std::env::var("POC_MEMORY_CONFIG") let config_path = std::env::var("POC_MEMORY_CONFIG")
@ -48,11 +44,9 @@ pub fn cmd_init() -> Result<(), String> {
}); });
if !config_path.exists() { if !config_path.exists() {
let config_dir = config_path.parent().unwrap(); let config_dir = config_path.parent().unwrap();
std::fs::create_dir_all(config_dir) std::fs::create_dir_all(config_dir)?;
.map_err(|e| format!("create config dir: {}", e))?;
let example = include_str!("../../config.example.jsonl"); let example = include_str!("../../config.example.jsonl");
std::fs::write(&config_path, example) std::fs::write(&config_path, example)?;
.map_err(|e| format!("write config: {}", e))?;
println!("Created config at {} — edit with your name and context groups", println!("Created config at {} — edit with your name and context groups",
config_path.display()); config_path.display());
} }
@ -61,168 +55,65 @@ pub fn cmd_init() -> Result<(), String> {
Ok(()) Ok(())
} }
pub fn cmd_bulk_rename(from: &str, to: &str, apply: bool) -> Result<(), String> { pub async fn cmd_fsck() -> Result<()> {
let mut store = store::Store::load()?; // Check/repair capnp log integrity first
store::fsck()?;
// Find all keys that need renaming let store = memory::access_local()?;
let renames: Vec<(String, String)> = store.nodes.keys()
.filter(|k| k.contains(from))
.map(|k| (k.clone(), k.replace(from, to)))
.collect();
// Check for collisions
let existing: std::collections::HashSet<&String> = store.nodes.keys().collect();
let mut collisions = 0;
for (old, new) in &renames {
if existing.contains(new) && old != new {
eprintln!("COLLISION: {} -> {} (target exists)", old, new);
collisions += 1;
}
}
println!("Bulk rename '{}' -> '{}'", from, to);
println!(" Keys to rename: {}", renames.len());
println!(" Collisions: {}", collisions);
if collisions > 0 {
return Err(format!("{} collisions — aborting", collisions));
}
if !apply {
// Show a sample
for (old, new) in renames.iter().take(10) {
println!(" {} -> {}", old, new);
}
if renames.len() > 10 {
println!(" ... and {} more", renames.len() - 10);
}
println!("\nDry run. Use --apply to execute.");
return Ok(());
}
// Apply renames using rename_node() which properly appends to capnp logs.
// Process in batches to avoid holding the lock too long.
let mut renamed_count = 0;
let mut errors = 0;
let total = renames.len();
for (i, (old_key, new_key)) in renames.iter().enumerate() {
match store.rename_node(old_key, new_key) {
Ok(()) => renamed_count += 1,
Err(e) => {
eprintln!(" RENAME ERROR: {} -> {}: {}", old_key, new_key, e);
errors += 1;
}
}
if (i + 1) % 1000 == 0 {
println!(" {}/{} ({} errors)", i + 1, total, errors);
}
}
store.save()?;
println!("Renamed {} nodes ({} errors).", renamed_count, errors);
// Run fsck to verify
println!("\nRunning fsck...");
drop(store);
cmd_fsck()?;
Ok(())
}
pub fn cmd_fsck() -> Result<(), String> {
let mut store = store::Store::load()?;
// Check cache vs log consistency
let log_store = store::Store::load_from_logs()?;
let mut cache_issues = 0;
// Nodes in logs but missing from cache
for key in log_store.nodes.keys() {
if !store.nodes.contains_key(key) {
eprintln!("CACHE MISSING: '{}' exists in capnp log but not in cache", key);
cache_issues += 1;
}
}
// Nodes in cache but not in logs (phantom nodes)
for key in store.nodes.keys() {
if !log_store.nodes.contains_key(key) {
eprintln!("CACHE PHANTOM: '{}' exists in cache but not in capnp log", key);
cache_issues += 1;
}
}
// Version mismatches
for (key, log_node) in &log_store.nodes {
if let Some(cache_node) = store.nodes.get(key)
&& cache_node.version != log_node.version {
eprintln!("CACHE STALE: '{}' cache v{} vs log v{}",
key, cache_node.version, log_node.version);
cache_issues += 1;
}
}
if cache_issues > 0 {
eprintln!("{} cache inconsistencies found — rebuilding from logs", cache_issues);
store = log_store;
store.save().map_err(|e| format!("rebuild save: {}", e))?;
}
// Check node-key consistency // Check node-key consistency
let mut issues = 0; let mut issues = 0;
for (key, node) in &store.nodes { let all_keys = store.all_keys().unwrap_or_default();
for key in &all_keys {
if let Ok(Some(node)) = store.get_node(key) {
if key != &node.key { if key != &node.key {
eprintln!("MISMATCH: map key '{}' vs node.key '{}'", key, node.key); eprintln!("MISMATCH: map key '{}' vs node.key '{}'", key, node.key);
issues += 1; issues += 1;
} }
} }
}
// Check edge endpoints // Check edge endpoints using index
use crate::hippocampus::store::StoreView;
let mut dangling = 0; let mut dangling = 0;
for rel in &store.relations { let mut orphan_edges: Vec<(String, String)> = Vec::new();
if rel.deleted { continue; } store.for_each_relation(|source, target, _, _| {
if !store.nodes.contains_key(&rel.source_key) { let s_missing = !store.contains_key(source).unwrap_or(false);
eprintln!("DANGLING: edge source '{}'", rel.source_key); let t_missing = !store.contains_key(target).unwrap_or(false);
if s_missing {
eprintln!("DANGLING: edge source '{}'", source);
dangling += 1; dangling += 1;
} }
if !store.nodes.contains_key(&rel.target_key) { if t_missing {
eprintln!("DANGLING: edge target '{}'", rel.target_key); eprintln!("DANGLING: edge target '{}'", target);
dangling += 1; dangling += 1;
} }
if s_missing || t_missing {
orphan_edges.push((source.to_string(), target.to_string()));
} }
});
// Prune orphan edges // Prune orphan edges
let mut to_tombstone = Vec::new(); if !orphan_edges.is_empty() {
for rel in &store.relations { let count = orphan_edges.len();
if rel.deleted { continue; } for (source, target) in &orphan_edges {
if !store.nodes.contains_key(&rel.source_key) // set_link_strength with 0 would delete, but we don't have that
|| !store.nodes.contains_key(&rel.target_key) { // For now just report - full cleanup requires more work
let mut tombstone = rel.clone(); eprintln!("Would prune: {} → {}", source, target);
tombstone.deleted = true;
tombstone.version += 1;
to_tombstone.push(tombstone);
} }
} eprintln!("Found {} orphan edges (prune not yet implemented for index)", count);
if !to_tombstone.is_empty() {
let count = to_tombstone.len();
store.append_relations(&to_tombstone)?;
for t in &to_tombstone {
if let Some(r) = store.relations.iter_mut().find(|r| r.uuid == t.uuid) {
r.deleted = true;
r.version = t.version;
}
}
store.save()?;
eprintln!("Pruned {} orphan edges", count);
} }
let g = store.build_graph(); let g = store.build_graph();
println!("fsck: {} nodes, {} edges, {} issues, {} dangling, {} cache", println!("fsck: {} nodes, {} edges, {} issues, {} dangling",
store.nodes.len(), g.edge_count(), issues, dangling, cache_issues); all_keys.len(), g.edge_count(), issues, dangling);
Ok(()) Ok(())
} }
pub fn cmd_dedup(apply: bool) -> Result<(), String> { pub async fn cmd_dedup(apply: bool) -> Result<()> {
use std::collections::{HashMap, HashSet}; use std::collections::HashMap;
let mut store = store::Store::load()?; let store = memory::access_local()?;
let duplicates = store.find_duplicates()?; let duplicates = store.find_duplicates()?;
if duplicates.is_empty() { if duplicates.is_empty() {
@ -230,12 +121,19 @@ pub fn cmd_dedup(apply: bool) -> Result<(), String> {
return Ok(()); return Ok(());
} }
// Count edges per UUID // Count edges per key (we'll map to UUID later)
use crate::hippocampus::store::StoreView;
let mut edges_by_key: HashMap<String, usize> = HashMap::new();
store.for_each_relation(|source, target, _, _| {
*edges_by_key.entry(source.to_string()).or_default() += 1;
*edges_by_key.entry(target.to_string()).or_default() += 1;
});
// Convert to edges_by_uuid for compatibility
let mut edges_by_uuid: HashMap<[u8; 16], usize> = HashMap::new(); let mut edges_by_uuid: HashMap<[u8; 16], usize> = HashMap::new();
for rel in &store.relations { for (key, count) in &edges_by_key {
if rel.deleted { continue; } if let Ok(Some(node)) = store.get_node(key) {
*edges_by_uuid.entry(rel.source).or_default() += 1; edges_by_uuid.insert(node.uuid, *count);
*edges_by_uuid.entry(rel.target).or_default() += 1; }
} }
let mut identical_groups = Vec::new(); let mut identical_groups = Vec::new();
@ -309,6 +207,14 @@ pub fn cmd_dedup(apply: bool) -> Result<(), String> {
.chain(diverged_groups) .chain(diverged_groups)
.collect(); .collect();
// Build uuid → key map for relation key strings
let mut uuid_to_key: HashMap<[u8; 16], String> = HashMap::new();
for key in store.all_keys()? {
if let Ok(Some(node)) = store.get_node(&key) {
uuid_to_key.insert(node.uuid, key);
}
}
let mut merged = 0usize; let mut merged = 0usize;
let mut edges_redirected = 0usize; let mut edges_redirected = 0usize;
let mut edges_deduped = 0usize; let mut edges_deduped = 0usize;
@ -318,52 +224,89 @@ pub fn cmd_dedup(apply: bool) -> Result<(), String> {
copies.sort_by(|a, b| b.1.cmp(&a.1).then(b.0.version.cmp(&a.0.version))); copies.sort_by(|a, b| b.1.cmp(&a.1).then(b.0.version.cmp(&a.0.version)));
let survivor_uuid = copies[0].0.uuid; let survivor_uuid = copies[0].0.uuid;
let survivor_key = uuid_to_key.get(&survivor_uuid).cloned().unwrap_or_default();
let doomed_uuids: Vec<[u8; 16]> = copies[1..].iter().map(|c| c.0.uuid).collect(); let doomed_uuids: Vec<[u8; 16]> = copies[1..].iter().map(|c| c.0.uuid).collect();
// Redirect edges from doomed UUIDs to survivor // Redirect edges from doomed UUIDs to survivor via index iteration
let mut updated_rels = Vec::new(); for doomed_uuid in &doomed_uuids {
for rel in &mut store.relations { let edges = store.edges_for_uuid(doomed_uuid)?;
if rel.deleted { continue; } for (other_uuid, strength, rel_type, is_outgoing) in edges {
let mut changed = false; let other_key = uuid_to_key.get(&other_uuid).cloned().unwrap_or_default();
if doomed_uuids.contains(&rel.source) {
rel.source = survivor_uuid; // Remove old edge from index
changed = true; let (old_src, old_tgt) = if is_outgoing {
} (*doomed_uuid, other_uuid)
if doomed_uuids.contains(&rel.target) { } else {
rel.target = survivor_uuid; (other_uuid, *doomed_uuid)
changed = true; };
} store.remove_relation_from_index(&old_src, &old_tgt, strength, rel_type)?;
if changed {
rel.version += 1; // Add redirected edge
updated_rels.push(rel.clone()); let (new_src, new_tgt, src_key, tgt_key) = if is_outgoing {
(survivor_uuid, other_uuid, survivor_key.clone(), other_key)
} else {
(other_uuid, survivor_uuid, other_key, survivor_key.clone())
};
store.index_relation(&new_src, &new_tgt, strength, rel_type)?;
// Append tombstone for old + new relation to log
let mut tombstone = store::new_relation(
old_src, old_tgt,
store::RelationType::from_u8(rel_type), strength,
&uuid_to_key.get(&old_src).cloned().unwrap_or_default(),
&uuid_to_key.get(&old_tgt).cloned().unwrap_or_default(),
);
tombstone.deleted = true;
tombstone.version = 2;
let mut redirected = store::new_relation(
new_src, new_tgt,
store::RelationType::from_u8(rel_type), strength,
&src_key, &tgt_key,
);
redirected.version = 2;
store.append_relations(&[tombstone, redirected])?;
edges_redirected += 1; edges_redirected += 1;
} }
} }
// Dedup edges: same (source, target, rel_type) → keep highest strength // Dedup edges: same (other_uuid, rel_type) → keep highest strength
let mut seen: HashSet<([u8; 16], [u8; 16], String)> = HashSet::new(); // Group edges by (other, type), sort each group by strength desc, tombstone extras
let mut to_tombstone_rels = Vec::new(); let edges = store.edges_for_uuid(&survivor_uuid)?;
// Sort by strength descending so we keep the strongest let mut by_endpoint: HashMap<([u8; 16], u8), Vec<(f32, bool)>> = HashMap::new();
let mut rels_with_idx: Vec<(usize, &store::Relation)> = store.relations.iter() for (other_uuid, strength, rel_type, is_outgoing) in edges {
.enumerate() by_endpoint.entry((other_uuid, rel_type))
.filter(|(_, r)| !r.deleted && (r.source == survivor_uuid || r.target == survivor_uuid)) .or_default()
.collect(); .push((strength, is_outgoing));
rels_with_idx.sort_by(|a, b| b.1.strength.total_cmp(&a.1.strength)); }
for (idx, rel) in &rels_with_idx { for ((other_uuid, rel_type), mut variants) in by_endpoint {
let edge_key = (rel.source, rel.target, format!("{:?}", rel.rel_type)); if variants.len() <= 1 { continue; }
if !seen.insert(edge_key) { // Sort by strength descending, keep first
to_tombstone_rels.push(*idx); variants.sort_by(|a, b| b.0.total_cmp(&a.0));
let other_key = uuid_to_key.get(&other_uuid).cloned().unwrap_or_default();
for (strength, is_outgoing) in variants.into_iter().skip(1) {
let (src, tgt, src_key, tgt_key) = if is_outgoing {
(survivor_uuid, other_uuid, survivor_key.clone(), other_key.clone())
} else {
(other_uuid, survivor_uuid, other_key.clone(), survivor_key.clone())
};
store.remove_relation_from_index(&src, &tgt, strength, rel_type)?;
let mut tombstone = store::new_relation(
src, tgt,
store::RelationType::from_u8(rel_type), strength,
&src_key, &tgt_key,
);
tombstone.deleted = true;
tombstone.version = 2;
store.append_relations(&[tombstone])?;
edges_deduped += 1; edges_deduped += 1;
} }
} }
for &idx in &to_tombstone_rels {
store.relations[idx].deleted = true;
store.relations[idx].version += 1;
updated_rels.push(store.relations[idx].clone());
}
// Tombstone doomed nodes // Tombstone doomed nodes
let mut tombstones = Vec::new(); let mut tombstones = Vec::new();
for (doomed_node, _) in &copies[1..] { for (doomed_node, _) in &copies[1..] {
@ -374,19 +317,15 @@ pub fn cmd_dedup(apply: bool) -> Result<(), String> {
} }
store.append_nodes(&tombstones)?; store.append_nodes(&tombstones)?;
if !updated_rels.is_empty() {
store.append_relations(&updated_rels)?;
}
for uuid in &doomed_uuids { // Remove doomed nodes from index
store.uuid_to_key.remove(uuid); for (doomed_node, _) in &copies[1..] {
store.remove_from_index(&doomed_node.key)?;
} }
merged += doomed_uuids.len(); merged += doomed_uuids.len();
} }
// Remove tombstoned relations from cache
store.relations.retain(|r| !r.deleted);
store.save()?; store.save()?;
println!("Merged {} duplicates, redirected {} edges, deduped {} duplicate edges", println!("Merged {} duplicates, redirected {} edges, deduped {} duplicate edges",
@ -395,87 +334,30 @@ pub fn cmd_dedup(apply: bool) -> Result<(), String> {
Ok(()) Ok(())
} }
pub fn cmd_health() -> Result<(), String> { pub async fn cmd_health() -> Result<()> {
let store = store::Store::load()?; let result = memory::graph_health(None).await
let g = store.build_graph(); ?;
let report = crate::graph::health_report(&g, &store); print!("{}", result);
print!("{}", report);
Ok(()) Ok(())
} }
pub fn cmd_daily_check() -> Result<(), String> { pub async fn cmd_topology() -> Result<()> {
let store = store::Store::load()?; let result = memory::graph_topology(None).await
?;
print!("{}", result);
Ok(())
}
pub async fn cmd_daily_check() -> Result<()> {
let store = memory::access_local()?;
let report = crate::neuro::daily_check(&store); let report = crate::neuro::daily_check(&store);
print!("{}", report); print!("{}", report);
Ok(()) Ok(())
} }
pub fn cmd_import(files: &[String]) -> Result<(), String> { pub async fn cmd_status() -> Result<()> {
if files.is_empty() { let result = memory::graph_topology(None).await
return Err("import requires at least one file path".into()); ?;
} print!("{}", result);
let mut store = store::Store::load()?;
let mut total_new = 0;
let mut total_updated = 0;
for arg in files {
let path = std::path::PathBuf::from(arg);
let resolved = if path.exists() {
path
} else {
let mem_path = store::memory_dir().join(arg);
if !mem_path.exists() {
eprintln!("File not found: {}", arg);
continue;
}
mem_path
};
let (n, u) = store.import_file(&resolved)?;
total_new += n;
total_updated += u;
}
if total_new > 0 || total_updated > 0 {
store.save()?;
}
println!("Import: {} new, {} updated", total_new, total_updated);
Ok(()) Ok(())
} }
pub fn cmd_export(files: &[String], export_all: bool) -> Result<(), String> {
let store = store::Store::load()?;
let targets: Vec<String> = if export_all {
let mut files: Vec<String> = store.nodes.keys()
.filter(|k| !k.contains('#'))
.cloned()
.collect();
files.sort();
files
} else if files.is_empty() {
return Err("export requires file keys or --all".into());
} else {
files.iter().map(|a| {
a.strip_suffix(".md").unwrap_or(a).to_string()
}).collect()
};
let mem_dir = store::memory_dir();
for file_key in &targets {
match store.export_to_markdown(file_key) {
Some(content) => {
let out_path = mem_dir.join(format!("{}.md", file_key));
std::fs::write(&out_path, &content)
.map_err(|e| format!("write {}: {}", out_path.display(), e))?;
let section_count = content.matches("<!-- mem:").count() + 1;
println!("Exported {} ({} sections)", file_key, section_count);
}
None => eprintln!("No nodes for '{}'", file_key),
}
}
Ok(())
}

69
src/cli/agent.rs
View file

@ -1,9 +1,9 @@
// cli/agent.rs — agent subcommand handlers // cli/agent.rs — agent subcommand handlers
use crate::store; use anyhow::{bail, Context, Result};
use crate::subconscious::digest; use crate::hippocampus as memory;
pub fn cmd_run_agent(agent: &str, count: usize, target: &[String], query: Option<&str>, dry_run: bool, _local: bool, state_dir: Option<&str>) -> Result<(), String> { pub async fn cmd_run_agent(agent: &str, count: usize, target: &[String], query: Option<&str>, dry_run: bool, _local: bool, state_dir: Option<&str>) -> Result<()> {
// Mark as agent so tool calls (e.g. poc-memory render) don't // Mark as agent so tool calls (e.g. poc-memory render) don't
// pollute the user's seen set as a side effect // pollute the user's seen set as a side effect
// SAFETY: single-threaded at this point (CLI startup, before any agent work) // SAFETY: single-threaded at this point (CLI startup, before any agent work)
@ -11,7 +11,7 @@ pub fn cmd_run_agent(agent: &str, count: usize, target: &[String], query: Option
// Override agent output/state directory if specified // Override agent output/state directory if specified
if let Some(dir) = state_dir { if let Some(dir) = state_dir {
std::fs::create_dir_all(dir).map_err(|e| format!("create state dir: {}", e))?; std::fs::create_dir_all(dir).context("create state dir")?;
unsafe { std::env::set_var("POC_AGENT_OUTPUT_DIR", dir); } unsafe { std::env::set_var("POC_AGENT_OUTPUT_DIR", dir); }
} }
@ -19,19 +19,20 @@ pub fn cmd_run_agent(agent: &str, count: usize, target: &[String], query: Option
unsafe { std::env::set_var("POC_MEMORY_DRY_RUN", "1"); } unsafe { std::env::set_var("POC_MEMORY_DRY_RUN", "1"); }
} }
let mut store = store::Store::load()?;
// Resolve targets: explicit --target, --query, or agent's default query // Resolve targets: explicit --target, --query, or agent's default query
let resolved_targets: Vec<String> = if !target.is_empty() { let resolved_targets: Vec<String> = if !target.is_empty() {
target.to_vec() target.to_vec()
} else if let Some(q) = query { } else if let Some(q) = query {
let graph = store.build_graph(); // Resolve query via typed API
let stages = crate::query_parser::parse_stages(q)?; let q_str = format!("{} | limit:{}", q, count);
let results = crate::search::run_query(&stages, vec![], &graph, &store, false, count); let result = memory::memory_query(None, &q_str, None).await?;
if results.is_empty() { let keys: Vec<String> = result.lines()
return Err(format!("query returned no results: {}", q)); .filter(|l| !l.is_empty() && *l != "no results")
.map(|s| s.to_string())
.collect();
if keys.is_empty() {
bail!("query returned no results: {}", q);
} }
let keys: Vec<String> = results.into_iter().map(|(k, _)| k).collect();
println!("[{}] query matched {} nodes", agent, keys.len()); println!("[{}] query matched {} nodes", agent, keys.len());
keys keys
} else { } else {
@ -41,53 +42,17 @@ pub fn cmd_run_agent(agent: &str, count: usize, target: &[String], query: Option
if !resolved_targets.is_empty() { if !resolved_targets.is_empty() {
for (i, key) in resolved_targets.iter().enumerate() { for (i, key) in resolved_targets.iter().enumerate() {
println!("[{}] [{}/{}] {}", agent, i + 1, resolved_targets.len(), key); println!("[{}] [{}/{}] {}", agent, i + 1, resolved_targets.len(), key);
if i > 0 { store = store::Store::load()?; }
if let Err(e) = crate::agent::oneshot::run_one_agent( if let Err(e) = crate::agent::oneshot::run_one_agent(
&mut store, agent, count, Some(&[key.clone()]), agent, count, Some(&[key.clone()]),
) { ).await {
println!("[{}] ERROR on {}: {}", agent, key, e); println!("[{}] ERROR on {}: {}", agent, key, e);
} }
} }
} else { } else {
// Local execution (--local, --debug, dry-run, or daemon unavailable)
crate::agent::oneshot::run_one_agent( crate::agent::oneshot::run_one_agent(
&mut store, agent, count, None, agent, count, None,
)?; ).await.map_err(|e| anyhow::anyhow!("{}", e))?;
} }
Ok(()) Ok(())
} }
pub fn cmd_replay_queue(count: usize) -> Result<(), String> {
let store = store::Store::load()?;
let queue = crate::neuro::replay_queue(&store, count);
println!("Replay queue ({} items):", queue.len());
for (i, item) in queue.iter().enumerate() {
println!(" {:2}. [{:.3}] {:>10} {} (interval={}d, emotion={:.1}, spectral={:.1})",
i + 1, item.priority, item.classification, item.key,
item.interval_days, item.emotion, item.outlier_score);
}
Ok(())
}
pub fn cmd_digest_links(do_apply: bool) -> Result<(), String> {
let store = store::Store::load()?;
let links = digest::parse_all_digest_links(&store);
drop(store);
println!("Found {} unique links from digest nodes", links.len());
if !do_apply {
for (i, link) in links.iter().enumerate() {
println!(" {:3}. {} → {}", i + 1, link.source, link.target);
if !link.reason.is_empty() {
println!(" ({})", &link.reason[..link.reason.floor_char_boundary(link.reason.len().min(80))]);
}
}
println!("\nTo apply: poc-memory digest-links --apply");
return Ok(());
}
let mut store = store::Store::load()?;
let (applied, skipped, fallbacks) = digest::apply_digest_links(&mut store, &links);
println!("\nApplied: {} ({} file-level fallbacks) Skipped: {}", applied, fallbacks, skipped);
Ok(())
}

429
src/cli/graph.rs
View file

@ -4,437 +4,72 @@
// link, link-add, link-impact, link-audit, cap-degree, // link, link-add, link-impact, link-audit, cap-degree,
// normalize-strengths, trace, spectral-*, organize, communities. // normalize-strengths, trace, spectral-*, organize, communities.
use crate::{store, graph}; use anyhow::{bail, Result};
use crate::store::StoreView; use crate::hippocampus as memory;
pub fn cmd_graph() -> Result<(), String> { pub async fn cmd_cap_degree(max_deg: usize) -> Result<()> {
let store = store::Store::load()?; let store = memory::access_local()?;
let g = store.build_graph();
println!("Graph: {} nodes, {} edges, {} communities",
g.nodes().len(), g.edge_count(), g.community_count());
println!("σ={:.2} α={:.2} gini={:.3} cc={:.4}",
g.small_world_sigma(), g.degree_power_law_exponent(),
g.degree_gini(), g.avg_clustering_coefficient());
Ok(())
}
pub fn cmd_cap_degree(max_deg: usize) -> Result<(), String> {
let mut store = store::Store::load()?;
let (hubs, pruned) = store.cap_degree(max_deg)?; let (hubs, pruned) = store.cap_degree(max_deg)?;
store.save()?; store.save()?;
println!("Capped {} hubs, pruned {} weak Auto edges (max_degree={})", hubs, pruned, max_deg); println!("Capped {} hubs, pruned {} weak Auto edges (max_degree={})", hubs, pruned, max_deg);
Ok(()) Ok(())
} }
pub fn cmd_normalize_strengths(apply: bool) -> Result<(), String> { pub async fn cmd_normalize_strengths(apply: bool) -> Result<()> {
let mut store = store::Store::load()?; if apply { super::check_dry_run(); }
let graph = store.build_graph(); let result = memory::graph_normalize_strengths(None, Some(apply)).await?;
let strengths = graph.jaccard_strengths(); print!("{}", result);
// Build a lookup from (source_key, target_key) → new_strength
let mut updates: std::collections::HashMap<(String, String), f32> = std::collections::HashMap::new();
for (a, b, s) in &strengths {
// Store both directions for easy lookup
updates.insert((a.clone(), b.clone()), *s);
updates.insert((b.clone(), a.clone()), *s);
}
// Stats
let mut changed = 0usize;
let mut unchanged = 0usize;
let mut temporal_skipped = 0usize;
let mut delta_sum: f64 = 0.0;
// Histogram of new strengths
let mut buckets = [0usize; 10]; // 0.0-0.1, 0.1-0.2, ...
for rel in &mut store.relations {
if rel.deleted { continue; }
// Skip implicit temporal edges (strength 1.0, Auto type)
if rel.strength == 1.0 && rel.rel_type == store::RelationType::Auto {
temporal_skipped += 1;
continue;
}
if let Some(&new_s) = updates.get(&(rel.source_key.clone(), rel.target_key.clone())) {
let old_s = rel.strength;
let delta = (new_s - old_s).abs();
if delta > 0.001 {
delta_sum += delta as f64;
if apply {
rel.strength = new_s;
}
changed += 1;
} else {
unchanged += 1;
}
let bucket = ((new_s * 10.0) as usize).min(9);
buckets[bucket] += 1;
}
}
println!("Normalize link strengths (Jaccard similarity)");
println!(" Total edges in graph: {}", strengths.len());
println!(" Would change: {}", changed);
println!(" Unchanged: {}", unchanged);
println!(" Temporal (skipped): {}", temporal_skipped);
if changed > 0 {
println!(" Avg delta: {:.3}", delta_sum / changed as f64);
}
println!();
println!(" Strength distribution:");
for (i, &count) in buckets.iter().enumerate() {
let lo = i as f32 / 10.0;
let hi = lo + 0.1;
let bar = "#".repeat(count / 50 + if count > 0 { 1 } else { 0 });
println!(" {:.1}-{:.1}: {:5} {}", lo, hi, count, bar);
}
if apply {
store.save()?;
println!("\nApplied {} strength updates.", changed);
} else {
println!("\nDry run. Use --apply to write changes.");
}
Ok(()) Ok(())
} }
pub fn cmd_spread(keys: &[String], max_results: usize) -> Result<(), String> { pub async fn cmd_link(key: &[String]) -> Result<()> {
if keys.is_empty() {
return Err("spread requires at least one seed key".into());
}
let store = store::Store::load()?;
let graph = graph::build_graph_fast(&store);
let params = store.params();
let seeds: Vec<(String, f64)> = keys.iter()
.filter_map(|k| {
let resolved = store.resolve_key(k).ok()?;
Some((resolved, 1.0))
})
.collect();
if seeds.is_empty() {
return Err("no valid seed keys found".into());
}
let results = crate::search::spreading_activation(
&seeds, &graph, &store,
params.max_hops, params.edge_decay, params.min_activation,
);
let seed_keys: std::collections::HashSet<&str> = seeds.iter()
.map(|(k, _)| k.as_str())
.collect();
for (key, score) in results.iter()
.filter(|(k, _)| !seed_keys.contains(k.as_str()))
.take(max_results)
{
println!(" {:.2} {}", score, key);
}
Ok(())
}
pub fn cmd_link(key: &[String]) -> Result<(), String> {
if key.is_empty() { if key.is_empty() {
return Err("link requires a key".into()); bail!("link requires a key");
} }
let key = key.join(" "); let key = key.join(" ");
let store = store::Store::load()?; let links = memory::memory_links(None, &key).await?;
let resolved = store.resolve_key(&key)?; println!("Neighbors of '{}':", key);
let g = store.build_graph(); for link in links {
println!("Neighbors of '{}':", resolved); println!(" ({:.2}) {} [w={:.2}]", link.link_strength, link.key, link.node_weight);
crate::query_parser::run_query(&store, &g, }
&format!("neighbors('{}') | select strength,clustering_coefficient", resolved)) Ok(())
} }
pub fn cmd_link_add(source: &str, target: &str, reason: &[String]) -> Result<(), String> { pub async fn cmd_link_add(source: &str, target: &str, _reason: &[String]) -> Result<()> {
super::check_dry_run(); super::check_dry_run();
let mut store = store::Store::load()?; let result = memory::memory_link_add(None, source, target).await?;
let source = store.resolve_key(source)?; println!("{}", result);
let target = store.resolve_key(target)?;
let reason = reason.join(" ");
match store.add_link(&source, &target, "manual") {
Ok(strength) => {
store.save()?;
println!("Linked: {} → {} (strength={:.2}, {})", source, target, strength, reason);
}
Err(msg) if msg.contains("already exists") => {
println!("Link already exists: {} ↔ {}", source, target);
}
Err(e) => return Err(e),
}
Ok(()) Ok(())
} }
pub fn cmd_link_set(source: &str, target: &str, strength: f32) -> Result<(), String> { pub async fn cmd_link_set(source: &str, target: &str, strength: f32) -> Result<()> {
super::check_dry_run(); super::check_dry_run();
let mut store = store::Store::load()?; let result = memory::memory_link_set(None, source, target, strength).await?;
let source = store.resolve_key(source)?; println!("{}", result);
let target = store.resolve_key(target)?;
let old = store.set_link_strength(&source, &target, strength)?;
println!("Set: {} ↔ {} strength {:.2} → {:.2}", source, target, old, strength);
store.save()?;
Ok(()) Ok(())
} }
pub fn cmd_link_impact(source: &str, target: &str) -> Result<(), String> { pub async fn cmd_link_impact(source: &str, target: &str) -> Result<()> {
let store = store::Store::load()?; let result = memory::graph_link_impact(None, source, target).await?;
let source = store.resolve_key(source)?; print!("{}", result);
let target = store.resolve_key(target)?;
let g = store.build_graph();
let impact = g.link_impact(&source, &target);
println!("Link impact: {} → {}", source, target);
println!(" Source degree: {} Target degree: {}", impact.source_deg, impact.target_deg);
println!(" Hub link: {} Same community: {}", impact.is_hub_link, impact.same_community);
println!(" ΔCC source: {:+.4} ΔCC target: {:+.4}", impact.delta_cc_source, impact.delta_cc_target);
println!(" ΔGini: {:+.6}", impact.delta_gini);
println!(" Assessment: {}", impact.assessment);
Ok(()) Ok(())
} }
pub fn cmd_trace(key: &[String]) -> Result<(), String> { pub async fn cmd_trace(key: &[String]) -> Result<()> {
if key.is_empty() { if key.is_empty() {
return Err("trace requires a key".into()); bail!("trace requires a key");
} }
let key = key.join(" "); let key = key.join(" ");
let store = store::Store::load()?; let result = memory::graph_trace(None, &key).await?;
let resolved = store.resolve_key(&key)?; print!("{}", result);
let g = store.build_graph();
let node = store.nodes.get(&resolved)
.ok_or_else(|| format!("Node not found: {}", resolved))?;
// Display the node itself
println!("=== {} ===", resolved);
println!("Type: {:?} Weight: {:.2}",
node.node_type, node.weight);
if !node.source_ref.is_empty() {
println!("Source: {}", node.source_ref);
}
// Show content preview
let preview = crate::util::truncate(&node.content, 200, "...");
println!("\n{}\n", preview);
// Walk neighbors, grouped by node type
let neighbors = g.neighbors(&resolved);
let mut episodic_session = Vec::new();
let mut episodic_daily = Vec::new();
let mut episodic_weekly = Vec::new();
let mut semantic = Vec::new();
for (n, strength) in &neighbors {
if let Some(nnode) = store.nodes.get(n.as_str()) {
let entry = (n.as_str(), *strength, nnode);
match nnode.node_type {
store::NodeType::EpisodicSession =>
episodic_session.push(entry),
store::NodeType::EpisodicDaily =>
episodic_daily.push(entry),
store::NodeType::EpisodicWeekly
| store::NodeType::EpisodicMonthly =>
episodic_weekly.push(entry),
store::NodeType::Semantic =>
semantic.push(entry),
}
}
}
if !episodic_weekly.is_empty() {
println!("Weekly digests:");
for (k, s, n) in &episodic_weekly {
let preview = crate::util::first_n_chars(n.content.lines().next().unwrap_or(""), 80);
println!(" [{:.2}] {} — {}", s, k, preview);
}
}
if !episodic_daily.is_empty() {
println!("Daily digests:");
for (k, s, n) in &episodic_daily {
let preview = crate::util::first_n_chars(n.content.lines().next().unwrap_or(""), 80);
println!(" [{:.2}] {} — {}", s, k, preview);
}
}
if !episodic_session.is_empty() {
println!("Session entries:");
for (k, s, n) in &episodic_session {
let preview = crate::util::first_n_chars(
n.content.lines()
.find(|l| !l.is_empty() && !l.starts_with("<!--"))
.unwrap_or(""),
80);
println!(" [{:.2}] {}", s, k);
if !n.source_ref.is_empty() {
println!(" ↳ source: {}", n.source_ref);
}
println!(" {}", preview);
}
}
if !semantic.is_empty() {
println!("Semantic links:");
for (k, s, _) in &semantic {
println!(" [{:.2}] {}", s, k);
}
}
println!("\nLinks: {} session, {} daily, {} weekly, {} semantic",
episodic_session.len(), episodic_daily.len(),
episodic_weekly.len(), semantic.len());
Ok(())
}
pub fn cmd_organize(term: &str, key_only: bool, create_anchor: bool) -> Result<(), String> {
let mut store = store::Store::load()?;
// Step 1: find all non-deleted nodes matching the term
let term_lower = term.to_lowercase();
let mut topic_nodes: Vec<(String, String)> = Vec::new(); // (key, content)
let skip_prefixes = ["_", "deep-index#", "facts-", "irc-history#"];
for (key, node) in &store.nodes {
if node.deleted { continue; }
// Skip episodic/digest nodes — use NodeType, not key prefix
if node.node_type != crate::store::NodeType::Semantic { continue; }
let key_matches = key.to_lowercase().contains(&term_lower);
let content_matches = !key_only && node.content.to_lowercase().contains(&term_lower);
if !key_matches && !content_matches { continue; }
if skip_prefixes.iter().any(|p| key.starts_with(p)) { continue; }
topic_nodes.push((key.clone(), node.content.clone()));
}
if topic_nodes.is_empty() {
println!("No topic nodes found matching '{}'", term);
return Ok(());
}
topic_nodes.sort_by(|a, b| a.0.cmp(&b.0));
println!("=== Organize: '{}' ===", term);
println!("Found {} topic nodes:\n", topic_nodes.len());
for (key, content) in &topic_nodes {
let lines = content.lines().count();
let words = content.split_whitespace().count();
println!(" {:60} {:>4} lines {:>5} words", key, lines, words);
}
// Step 2: check connectivity within cluster
let g = store.build_graph();
println!("=== Connectivity ===\n");
// Pick hub by intra-cluster connectivity, not overall degree
let cluster_keys: std::collections::HashSet<&str> = topic_nodes.iter()
.filter(|(k,_)| store.nodes.contains_key(k.as_str()))
.map(|(k,_)| k.as_str())
.collect();
let mut best_hub: Option<(&str, usize)> = None;
for key in &cluster_keys {
let intra_degree = g.neighbor_keys(key).iter()
.filter(|n| cluster_keys.contains(*n))
.count();
if best_hub.is_none() || intra_degree > best_hub.unwrap().1 {
best_hub = Some((key, intra_degree));
}
}
if let Some((hub, deg)) = best_hub {
println!(" Hub: {} (degree {})", hub, deg);
let hub_nbrs = g.neighbor_keys(hub);
let mut unlinked = Vec::new();
for (key, _) in &topic_nodes {
if key == hub { continue; }
if store.nodes.get(key.as_str()).is_none() { continue; }
if !hub_nbrs.contains(key.as_str()) {
unlinked.push(key.clone());
}
}
if unlinked.is_empty() {
println!(" All cluster nodes connected to hub ✓");
} else {
println!(" NOT linked to hub:");
for key in &unlinked {
println!(" {} → needs link to {}", key, hub);
}
}
}
// Step 4: anchor node
if create_anchor {
println!("\n=== Anchor node ===\n");
if store.nodes.contains_key(term) && !store.nodes[term].deleted {
println!(" Anchor '{}' already exists ✓", term);
} else {
let desc = format!("Anchor node for '{}' search term", term);
store.upsert(term, &desc)?;
let anchor_uuid = store.nodes.get(term).unwrap().uuid;
for (key, _) in &topic_nodes {
if store.nodes.get(key.as_str()).is_none() { continue; }
let target_uuid = store.nodes[key.as_str()].uuid;
let rel = store::new_relation(
anchor_uuid, target_uuid,
store::RelationType::Link, 0.8,
term, key,
);
store.add_relation(rel)?;
}
println!(" Created anchor '{}' with {} links", term, topic_nodes.len());
}
}
store.save()?;
Ok(()) Ok(())
} }
/// Show communities sorted by isolation (most isolated first). /// Show communities sorted by isolation (most isolated first).
/// Useful for finding poorly-integrated knowledge clusters that need /// Useful for finding poorly-integrated knowledge clusters that need
/// organize agents aimed at them. /// organize agents aimed at them.
pub fn cmd_communities(top_n: usize, min_size: usize) -> Result<(), String> { pub async fn cmd_communities(top_n: usize, min_size: usize) -> Result<()> {
let store = store::Store::load()?; let result = memory::graph_communities(None, Some(top_n), Some(min_size)).await?;
let g = store.build_graph(); print!("{}", result);
let infos = g.community_info();
let total = infos.len();
let shown: Vec<_> = infos.into_iter()
.filter(|c| c.size >= min_size)
.take(top_n)
.collect();
println!("{} communities total ({} with size >= {})\n",
total, shown.len(), min_size);
println!("{:<6} {:>5} {:>7} {:>7} members", "id", "size", "iso", "cross");
println!("{}", "-".repeat(70));
for c in &shown {
let preview: Vec<&str> = c.members.iter()
.take(5)
.map(|s| s.as_str())
.collect();
let more = if c.size > 5 {
format!(" +{}", c.size - 5)
} else {
String::new()
};
println!("{:<6} {:>5} {:>6.0}% {:>7} {}{}",
c.id, c.size, c.isolation * 100.0, c.cross_edges,
preview.join(", "), more);
}
Ok(()) Ok(())
} }

125
src/cli/journal.rs
View file

@ -1,24 +1,26 @@
// cli/journal.rs — journal subcommand handlers // cli/journal.rs — journal subcommand handlers
use anyhow::{bail, Context, Result};
use crate::hippocampus as memory;
pub fn cmd_tail(n: usize, full: bool, provenance: Option<&str>, dedup: bool) -> Result<(), String> { pub fn cmd_tail(n: usize, full: bool, provenance: Option<&str>, dedup: bool) -> Result<()> {
let path = crate::store::nodes_path(); let path = crate::store::nodes_path();
if !path.exists() { if !path.exists() {
return Err("No node log found".into()); bail!("No node log found");
} }
use std::io::BufReader; use std::io::BufReader;
let file = std::fs::File::open(&path) let file = std::fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?; .with_context(|| format!("open {}", path.display()))?;
let mut reader = BufReader::new(file); let mut reader = BufReader::new(file);
// Read all entries, keep last N // Read all entries, keep last N
let mut entries: Vec<crate::store::Node> = Vec::new(); let mut entries: Vec<crate::store::Node> = Vec::new();
while let Ok(msg) = capnp::serialize::read_message(&mut reader, capnp::message::ReaderOptions::new()) { while let Ok(msg) = capnp::serialize::read_message(&mut reader, capnp::message::ReaderOptions::new()) {
let log = msg.get_root::<crate::memory_capnp::node_log::Reader>() let log = msg.get_root::<crate::memory_capnp::node_log::Reader>()
.map_err(|e| format!("read log: {}", e))?; .with_context(|| "read log")?;
for node_reader in log.get_nodes() for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes: {}", e))? { .with_context(|| "get nodes")? {
let node = crate::store::Node::from_capnp_migrate(node_reader)?; let node = crate::store::Node::from_capnp_migrate(node_reader)?;
entries.push(node); entries.push(node);
} }
@ -66,118 +68,29 @@ pub fn cmd_tail(n: usize, full: bool, provenance: Option<&str>, dedup: bool) ->
Ok(()) Ok(())
} }
pub fn find_current_transcript() -> Option<String> { pub async fn cmd_journal_tail(n: usize, full: bool, level: u8) -> Result<()> {
let projects = crate::config::get().projects_dir.clone(); let entries = memory::journal_tail(None, Some(n as u64), Some(level as u64), None).await?;
if !projects.exists() { return None; } for entry in entries {
let mut newest: Option<(std::time::SystemTime, std::path::PathBuf)> = None;
if let Ok(dirs) = std::fs::read_dir(&projects) {
for dir_entry in dirs.filter_map(|e| e.ok()) {
if !dir_entry.path().is_dir() { continue; }
if let Ok(files) = std::fs::read_dir(dir_entry.path()) {
for f in files.filter_map(|e| e.ok()) {
let p = f.path();
if p.extension().map(|x| x == "jsonl").unwrap_or(false)
&& let Ok(meta) = p.metadata()
&& let Ok(mtime) = meta.modified()
&& newest.as_ref().is_none_or(|(t, _)| mtime > *t) {
newest = Some((mtime, p));
}
}
}
}
}
newest.map(|(_, p)| p.to_string_lossy().to_string())
}
fn journal_tail_query(store: &crate::store::Store, query: &str, n: usize, full: bool) -> Result<(), String> {
let graph = store.build_graph();
let stages = crate::query_parser::parse_stages(query)?;
let results = crate::search::run_query(&stages, vec![], &graph, store, false, n);
// Query sorts desc and limits, so reverse to show oldest-to-newest
for (key, _score) in results.into_iter().rev() {
let Some(node) = store.nodes.get(&key) else { continue };
let ts = if node.created_at > 0 {
crate::store::format_datetime(node.created_at)
} else if node.timestamp > 0 {
crate::store::format_datetime(node.timestamp)
} else {
node.key.clone()
};
let title = extract_title(&node.content);
if full { if full {
println!("--- [{}] {} ---\n{}\n", ts, title, node.content); println!("--- {} ---", entry.key);
println!("{}\n", entry.content);
} else { } else {
println!("[{}] {}", ts, title); let first_line = entry.content.lines().next().unwrap_or("(empty)");
println!("{}: {}", entry.key, first_line);
} }
} }
Ok(()) Ok(())
} }
pub fn cmd_journal_tail(n: usize, full: bool, level: u8) -> Result<(), String> { pub async fn cmd_journal_write(name: &str, text: &[String]) -> Result<()> {
let store = crate::store::Store::load()?;
let query = format!("all | type:{} | sort:timestamp | limit:{}",
match level { 0 => "episodic", 1 => "daily", 2 => "weekly", _ => "monthly" },
n
);
journal_tail_query(&store, &query, n, full)
}
pub fn cmd_journal_write(name: &str, text: &[String]) -> Result<(), String> {
if text.is_empty() { if text.is_empty() {
return Err("journal write requires text".into()); bail!("journal write requires text");
} }
super::check_dry_run(); super::check_dry_run();
let text = text.join(" "); let body = text.join(" ");
let timestamp = crate::store::format_datetime(crate::store::now_epoch());
let content = format!("## {} — {}\n\n{}", timestamp, name, text);
let key: String = name.split_whitespace()
.map(|w| w.to_lowercase()
.chars().filter(|c| c.is_alphanumeric() || *c == '-')
.collect::<String>())
.filter(|s| !s.is_empty())
.collect::<Vec<_>>()
.join("-");
let source_ref = find_current_transcript();
let mut store = crate::store::Store::load()?;
let mut node = crate::store::new_node(&key, &content);
node.node_type = crate::store::NodeType::EpisodicSession;
node.provenance = "journal".to_string();
if let Some(src) = source_ref {
node.source_ref = src;
}
store.upsert_node(node)?;
store.save()?;
let word_count = text.split_whitespace().count();
println!("Appended entry at {} ({} words)", timestamp, word_count);
let result = memory::journal_new(None, name, name, &body, Some(0)).await?;
println!("{}", result);
Ok(()) Ok(())
} }
fn extract_title(content: &str) -> String {
let date_re = regex::Regex::new(r"(\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2})").unwrap();
for line in content.lines() {
let stripped = line.trim();
if stripped.is_empty() { continue; }
if date_re.is_match(stripped) && stripped.len() < 25 { continue; }
if let Some(h) = stripped.strip_prefix("## ") {
return h.to_string();
} else if let Some(h) = stripped.strip_prefix("# ") {
return h.to_string();
} else {
return crate::util::truncate(stripped, 67, "...");
}
}
String::from("(untitled)")
}

319
src/cli/misc.rs
View file

@ -1,319 +0,0 @@
// cli/misc.rs — misc subcommand handlers
pub fn cmd_search(terms: &[String], pipeline_args: &[String], expand: bool, full: bool, debug: bool, fuzzy: bool, content: bool) -> Result<(), String> {
use std::collections::BTreeMap;
use crate::search::{Stage, Algorithm, AlgoStage};
// When running inside an agent session, exclude already-surfaced nodes
let seen = crate::session::HookSession::from_env()
.map(|s| s.seen())
.unwrap_or_default();
// Build pipeline: if args provided, parse them; otherwise default to spread
let stages: Vec<Stage> = if pipeline_args.is_empty() {
vec![Stage::Algorithm(AlgoStage { algo: Algorithm::Spread, params: std::collections::HashMap::new() })]
} else {
// Join args with | and parse as unified query
let pipeline_str = format!("all | {}", pipeline_args.join(" | "));
crate::query_parser::parse_stages(&pipeline_str)?
};
// Check if pipeline needs full Store (has filters/transforms/generators)
let needs_store = stages.iter().any(|s| !matches!(s, Stage::Algorithm(_)));
// Check if pipeline starts with a generator (doesn't need seed terms)
let has_generator = stages.first().map(|s| matches!(s, Stage::Generator(_))).unwrap_or(false);
if terms.is_empty() && !has_generator {
return Err("search requires terms or a generator stage (e.g. 'all')".into());
}
let query: String = terms.join(" ");
if debug {
let names: Vec<String> = stages.iter().map(|s| format!("{}", s)).collect();
println!("[search] pipeline: {}", names.join(" → "));
}
let max_results = if expand { 15 } else { 5 };
if needs_store {
// Full Store path — needed for filter/transform/generator stages
let store = crate::store::Store::load()?;
let graph = store.build_graph();
let seeds = if has_generator {
vec![] // generator will produce its own result set
} else {
let terms_map: BTreeMap<String, f64> = query.split_whitespace()
.map(|t| (t.to_lowercase(), 1.0))
.collect();
let (seeds, _) = crate::search::match_seeds_opts(&terms_map, &store, fuzzy, content);
seeds
};
let raw = crate::search::run_query(&stages, seeds, &graph, &store, debug, max_results);
let raw: Vec<_> = raw.into_iter()
.filter(|(key, _)| !seen.contains(key))
.collect();
if raw.is_empty() {
eprintln!("No results");
return Ok(());
}
for (i, (key, score)) in raw.iter().enumerate().take(max_results) {
let weight = store.nodes.get(key).map(|n| n.weight).unwrap_or(0.0);
println!("{:2}. [{:.2}/{:.2}] {}", i + 1, score, weight, key);
if full
&& let Some(node) = store.nodes.get(key) {
println!();
for line in node.content.lines() {
println!(" {}", line);
}
println!();
}
}
} else {
// Fast MmapView path — algorithm-only pipeline
use crate::store::StoreView;
let view = crate::store::AnyView::load()?;
let graph = crate::graph::build_graph_fast(&view);
let terms_map: BTreeMap<String, f64> = query.split_whitespace()
.map(|t| (t.to_lowercase(), 1.0))
.collect();
let (seeds, direct_hits) = crate::search::match_seeds_opts(&terms_map, &view, fuzzy, content);
if seeds.is_empty() {
eprintln!("No results for '{}'", query);
return Ok(());
}
if debug {
println!("[search] {} seeds from query '{}'", seeds.len(), query);
}
// Extract AlgoStages from the unified stages
let algo_stages: Vec<&crate::search::AlgoStage> = stages.iter()
.filter_map(|s| match s {
crate::search::Stage::Algorithm(a) => Some(a),
_ => None,
})
.collect();
let algo_owned: Vec<crate::search::AlgoStage> = algo_stages.into_iter().cloned().collect();
let raw = crate::search::run_pipeline(&algo_owned, seeds, &graph, &view, debug, max_results);
let results: Vec<crate::search::SearchResult> = raw.into_iter()
.filter(|(key, _)| !seen.contains(key))
.map(|(key, activation)| {
let is_direct = direct_hits.contains(&key);
crate::search::SearchResult { key, activation, is_direct, snippet: None }
})
.collect();
if results.is_empty() {
eprintln!("No results for '{}'", query);
return Ok(());
}
// Log retrieval
crate::store::Store::log_retrieval_static(&query,
&results.iter().map(|r| r.key.clone()).collect::<Vec<_>>());
let bump_keys: Vec<&str> = results.iter().take(max_results).map(|r| r.key.as_str()).collect();
let _ = crate::lookups::bump_many(&bump_keys);
for (i, r) in results.iter().enumerate().take(max_results) {
let marker = if r.is_direct { "→" } else { " " };
let weight = view.node_weight(&r.key);
println!("{}{:2}. [{:.2}/{:.2}] {}", marker, i + 1, r.activation, weight, r.key);
if full
&& let Some(content) = view.node_content(&r.key) {
println!();
for line in content.lines() {
println!(" {}", line);
}
println!();
}
}
}
Ok(())
}
pub fn cmd_status() -> Result<(), String> {
// TUI moved to consciousness binary (F4 unconscious screen)
let store = crate::store::Store::load()?;
let g = store.build_graph();
let mut type_counts = std::collections::HashMap::new();
for node in store.nodes.values() {
*type_counts.entry(format!("{:?}", node.node_type)).or_insert(0usize) += 1;
}
let mut types: Vec<_> = type_counts.iter().collect();
types.sort_by_key(|(_, c)| std::cmp::Reverse(**c));
println!("Nodes: {} Relations: {}", store.nodes.len(), store.relations.len());
print!("Types:");
for (t, c) in &types {
let label = match t.as_str() {
"Semantic" => "semantic",
"EpisodicSession" | "EpisodicDaily" | "EpisodicWeekly" | "EpisodicMonthly"
=> "episodic",
_ => t,
};
print!(" {}={}", label, c);
}
println!();
println!("Graph edges: {} Communities: {}",
g.edge_count(), g.community_count());
Ok(())
}
pub fn cmd_log() -> Result<(), String> {
let store = crate::store::Store::load()?;
for event in store.retrieval_log.iter().rev().take(20) {
println!("[{}] q=\"{}\" → {} results",
event.timestamp, event.query, event.results.len());
for r in &event.results {
println!(" {}", r);
}
}
Ok(())
}
pub fn cmd_params() -> Result<(), String> {
let store = crate::store::Store::load()?;
println!("decay_factor: {}", store.params.decay_factor);
println!("use_boost: {}", store.params.use_boost);
println!("prune_threshold: {}", store.params.prune_threshold);
println!("edge_decay: {}", store.params.edge_decay);
println!("max_hops: {}", store.params.max_hops);
println!("min_activation: {}", store.params.min_activation);
Ok(())
}
pub fn cmd_query(expr: &[String]) -> Result<(), String> {
if expr.is_empty() {
return Err("query requires an expression (try: poc-memory query --help)".into());
}
let query_str = expr.join(" ");
let store = crate::store::Store::load()?;
let graph = store.build_graph();
crate::query_parser::run_query(&store, &graph, &query_str)
}
pub fn get_group_content(group: &crate::config::ContextGroup, store: &crate::store::Store, cfg: &crate::config::Config) -> Vec<(String, String)> {
match group.source {
crate::config::ContextSource::Journal => {
let mut entries = Vec::new();
let now = crate::store::now_epoch();
let window: i64 = cfg.journal_days as i64 * 24 * 3600;
let cutoff = now - window;
let key_date_re = regex::Regex::new(r"j-(\d{4}-\d{2}-\d{2})").unwrap();
let journal_ts = |n: &crate::store::Node| -> i64 {
if n.created_at > 0 { return n.created_at; }
if let Some(caps) = key_date_re.captures(&n.key) {
use chrono::{NaiveDate, TimeZone, Local};
if let Ok(d) = NaiveDate::parse_from_str(&caps[1], "%Y-%m-%d")
&& let Some(dt) = Local.from_local_datetime(&d.and_hms_opt(0, 0, 0).unwrap()).earliest() {
return dt.timestamp();
}
}
n.timestamp
};
let mut journal_nodes: Vec<_> = store.nodes.values()
.filter(|n| n.node_type == crate::store::NodeType::EpisodicSession && journal_ts(n) >= cutoff)
.collect();
journal_nodes.sort_by_key(|n| journal_ts(n));
let max = cfg.journal_max;
let skip = journal_nodes.len().saturating_sub(max);
for node in journal_nodes.iter().skip(skip) {
entries.push((node.key.clone(), node.content.clone()));
}
entries
}
crate::config::ContextSource::File => {
group.keys.iter().filter_map(|key| {
let content = std::fs::read_to_string(cfg.identity_dir.join(key)).ok()?;
if content.trim().is_empty() { return None; }
Some((key.clone(), content.trim().to_string()))
}).collect()
}
crate::config::ContextSource::Store => {
group.keys.iter().filter_map(|key| {
let content = store.render_file(key)?;
if content.trim().is_empty() { return None; }
Some((key.clone(), content.trim().to_string()))
}).collect()
}
}
}
/// MCP tool schema with CLI routing info.
///
/// Each tool definition includes:
/// - name, description, inputSchema (standard MCP)
/// - cli: the CLI args prefix to invoke this tool
/// - stdin_param: which parameter (if any) should be sent via stdin
///
/// Tools with cli=null are agent-internal (not exposed via MCP CLI bridge).
// mcp-schema moved to consciousness-mcp binary (src/claude/mcp-server.rs)
pub fn cmd_load_context(stats: bool) -> Result<(), String> {
let cfg = crate::config::get();
let store = crate::store::Store::load()?;
if stats {
let mut total_words = 0;
let mut total_entries = 0;
println!("{:<25} {:>6} {:>8}", "GROUP", "ITEMS", "WORDS");
println!("{}", "-".repeat(42));
for group in &cfg.context_groups {
let entries = get_group_content(group, &store, &cfg);
let words: usize = entries.iter()
.map(|(_, c)| c.split_whitespace().count())
.sum();
let count = entries.len();
println!("{:<25} {:>6} {:>8}", group.label, count, words);
total_words += words;
total_entries += count;
}
println!("{}", "-".repeat(42));
println!("{:<25} {:>6} {:>8}", "TOTAL", total_entries, total_words);
return Ok(());
}
println!("=== MEMORY SYSTEM ({}) ===", cfg.assistant_name);
println!();
for group in &cfg.context_groups {
let entries = get_group_content(group, &store, &cfg);
if !entries.is_empty() && group.source == crate::config::ContextSource::Journal {
println!("--- recent journal entries ({}/{}) ---",
entries.len(), cfg.journal_max);
}
for (key, content) in entries {
if group.source == crate::config::ContextSource::Journal {
println!("## {}", key);
} else {
println!("--- {} ({}) ---", key, group.label);
}
println!("{}\n", content);
}
}
println!("=== END MEMORY LOAD ===");
Ok(())
}

1
src/cli/mod.rs
View file

@ -8,7 +8,6 @@ pub mod node;
pub mod agent; pub mod agent;
pub mod admin; pub mod admin;
pub mod journal; pub mod journal;
pub mod misc;
/// Exit silently if POC_MEMORY_DRY_RUN=1. /// Exit silently if POC_MEMORY_DRY_RUN=1.
pub fn check_dry_run() { pub fn check_dry_run() {

497
src/cli/node.rs
View file

@ -1,203 +1,45 @@
// cli/node.rs — node subcommand handlers // cli/node.rs — node subcommand handlers
// //
// render, write, used, wrong, not-relevant, not-useful, gap, // render, write, node-delete, node-rename, history, list-keys,
// node-delete, node-rename, history, list-keys, list-edges, // list-edges, dump-json, lookup-bump, lookups.
// dump-json, lookup-bump, lookups.
use anyhow::{bail, Context, Result};
use crate::hippocampus as memory;
use crate::store; use crate::store;
pub fn cmd_used(key: &[String]) -> Result<(), String> { pub async fn cmd_weight_set(key: &str, weight: f32) -> Result<()> {
super::check_dry_run();
let result = memory::memory_weight_set(None, key, weight).await?;
println!("{}", result);
Ok(())
}
pub async fn cmd_node_delete(key: &[String]) -> Result<()> {
if key.is_empty() { if key.is_empty() {
return Err("used requires a key".into()); bail!("node-delete requires a key");
} }
super::check_dry_run(); super::check_dry_run();
let key = key.join(" "); let key = key.join(" ");
let mut store = store::Store::load()?; let result = memory::memory_delete(None, &key).await?;
let resolved = store.resolve_key(&key)?; println!("{}", result);
store.mark_used(&resolved);
// Also strengthen edges to this node — conscious-tier delta.
const DELTA: f32 = 0.01;
let mut strengthened = 0;
for rel in &mut store.relations {
if rel.deleted { continue; }
if rel.source_key == resolved || rel.target_key == resolved {
let old = rel.strength;
rel.strength = (rel.strength + DELTA).clamp(0.05, 0.95);
if (rel.strength - old).abs() > 0.001 {
rel.version += 1;
strengthened += 1;
}
}
}
store.save()?;
println!("Marked '{}' as used (strengthened {} edges)", resolved, strengthened);
Ok(()) Ok(())
} }
pub fn cmd_wrong(key: &str, context: &[String]) -> Result<(), String> { pub async fn cmd_node_rename(old_key: &str, new_key: &str) -> Result<()> {
let ctx = if context.is_empty() { None } else { Some(context.join(" ")) };
super::check_dry_run(); super::check_dry_run();
let mut store = store::Store::load()?; let result = memory::memory_rename(None, old_key, new_key).await?;
let resolved = store.resolve_key(key)?; println!("{}", result);
store.mark_wrong(&resolved, ctx.as_deref());
store.save()?;
println!("Marked '{}' as wrong", resolved);
Ok(()) Ok(())
} }
pub fn cmd_not_relevant(key: &str) -> Result<(), String> { pub async fn cmd_render(key: &[String]) -> Result<()> {
let mut store = store::Store::load()?;
let resolved = store.resolve_key(key)?;
// Weaken all edges to this node — it was routed to incorrectly.
// Conscious-tier delta: 0.01 per edge.
const DELTA: f32 = -0.01;
let mut adjusted = 0;
for rel in &mut store.relations {
if rel.deleted { continue; }
if rel.source_key == resolved || rel.target_key == resolved {
let old = rel.strength;
rel.strength = (rel.strength + DELTA).clamp(0.05, 0.95);
if (rel.strength - old).abs() > 0.001 {
rel.version += 1;
adjusted += 1;
}
}
}
store.save()?;
println!("Not relevant: '{}' — weakened {} edges by {}", resolved, adjusted, DELTA.abs());
Ok(())
}
pub fn cmd_not_useful(key: &str) -> Result<(), String> {
// no args to validate
super::check_dry_run();
let mut store = store::Store::load()?;
let resolved = store.resolve_key(key)?;
// Same as wrong but with clearer semantics: node content is bad, edges are fine.
store.mark_wrong(&resolved, Some("not-useful"));
store.save()?;
println!("Not useful: '{}' — node weight reduced", resolved);
Ok(())
}
pub fn cmd_weight_set(key: &str, weight: f32) -> Result<(), String> {
super::check_dry_run();
let mut store = store::Store::load()?;
let resolved = store.resolve_key(key)?;
let (old, new) = store.set_weight(&resolved, weight)?;
println!("Weight: {} {:.2} → {:.2}", resolved, old, new);
store.save()?;
Ok(())
}
pub fn cmd_gap(description: &[String]) -> Result<(), String> {
if description.is_empty() {
return Err("gap requires a description".into());
}
super::check_dry_run();
let desc = description.join(" ");
let mut store = store::Store::load()?;
store.record_gap(&desc);
store.save()?;
println!("Recorded gap: {}", desc);
Ok(())
}
pub fn cmd_list_keys(pattern: Option<&str>) -> Result<(), String> {
let store = store::Store::load()?;
let g = store.build_graph();
if let Some(pat) = pattern {
let pat_lower = pat.to_lowercase();
let (prefix, suffix, middle) = if pat_lower.starts_with('*') && pat_lower.ends_with('*') {
(None, None, Some(pat_lower.trim_matches('*').to_string()))
} else if pat_lower.starts_with('*') {
(None, Some(pat_lower.trim_start_matches('*').to_string()), None)
} else if pat_lower.ends_with('*') {
(Some(pat_lower.trim_end_matches('*').to_string()), None, None)
} else {
(None, None, Some(pat_lower.clone()))
};
let mut keys: Vec<_> = store.nodes.keys()
.filter(|k| {
let kl = k.to_lowercase();
if let Some(ref m) = middle { kl.contains(m.as_str()) }
else if let Some(ref p) = prefix { kl.starts_with(p.as_str()) }
else if let Some(ref s) = suffix { kl.ends_with(s.as_str()) }
else { true }
})
.cloned()
.collect();
keys.sort();
for k in keys { println!("{}", k); }
Ok(())
} else {
crate::query_parser::run_query(&store, &g, "* | sort key asc")
}
}
pub fn cmd_list_edges() -> Result<(), String> {
let store = store::Store::load()?;
for rel in &store.relations {
println!("{}\t{}\t{:.2}\t{:?}",
rel.source_key, rel.target_key, rel.strength, rel.rel_type);
}
Ok(())
}
pub fn cmd_dump_json() -> Result<(), String> {
let store = store::Store::load()?;
let json = serde_json::to_string_pretty(&store)
.map_err(|e| format!("serialize: {}", e))?;
println!("{}", json);
Ok(())
}
pub fn cmd_node_delete(key: &[String]) -> Result<(), String> {
if key.is_empty() { if key.is_empty() {
return Err("node-delete requires a key".into()); bail!("render requires a key");
}
super::check_dry_run();
let key = key.join(" ");
let mut store = store::Store::load()?;
let resolved = store.resolve_key(&key)?;
store.delete_node(&resolved)?;
store.save()?;
println!("Deleted '{}'", resolved);
Ok(())
}
pub fn cmd_node_rename(old_key: &str, new_key: &str) -> Result<(), String> {
// args are positional, always valid if present
super::check_dry_run();
let mut store = store::Store::load()?;
let old_resolved = store.resolve_key(old_key)?;
store.rename_node(&old_resolved, new_key)?;
store.save()?;
println!("Renamed '{}' → '{}'", old_resolved, new_key);
Ok(())
}
/// Render a node to a string: content + deduped footer links.
/// Used by both the CLI command and agent placeholders.
pub fn render_node(store: &store::Store, key: &str) -> Option<String> {
crate::hippocampus::memory::MemoryNode::from_store(store, key)
.map(|node| node.render())
}
pub fn cmd_render(key: &[String]) -> Result<(), String> {
if key.is_empty() {
return Err("render requires a key".into());
} }
let key = key.join(" "); let key = key.join(" ");
let store = store::Store::load()?;
let bare = store::strip_md_suffix(&key); let bare = store::strip_md_suffix(&key);
let rendered = render_node(&store, &bare) let rendered = memory::memory_render(None, &bare, None).await?;
.ok_or_else(|| format!("Node not found: {}", bare))?;
print!("{}", rendered); print!("{}", rendered);
// Mark as seen if we're inside a Claude session (not an agent subprocess — // Mark as seen if we're inside a Claude session (not an agent subprocess —
@ -221,182 +63,62 @@ pub fn cmd_render(key: &[String]) -> Result<(), String> {
Ok(()) Ok(())
} }
/// Check content for common inline reference problems: pub async fn cmd_history(key: &[String], full: bool) -> Result<()> {
/// - `poc-memory render key` embedded in content (render artifact, should be just `key`)
/// - `→ something` where something doesn't parse as a valid key
/// - `key` referencing a node that doesn't exist
fn validate_inline_refs(content: &str, store: &store::Store) -> Vec<String> {
let mut warnings = Vec::new();
for line in content.lines() {
// Check for render commands embedded in content
if line.contains("poc-memory render ") && !line.starts_with(" ") {
// Skip lines that look like CLI documentation/examples
if !line.contains("CLI") && !line.contains("equivalent") && !line.contains("tool") {
warnings.push(format!(
"render command in content (should be just `key`): {}",
line.chars().take(80).collect::<String>(),
));
}
}
// Check → references
if let Some(rest) = line.trim().strip_prefix("→ ") {
// Extract the key (may be backtick-quoted)
let key = rest.trim().trim_matches('`').trim();
if !key.is_empty() && !store.nodes.contains_key(key) {
// Might be a poc-memory render artifact
if let Some(k) = key.strip_prefix("poc-memory render ") {
warnings.push(format!(
"render artifact in → reference (use `{}` not `poc-memory render {}`)", k, k,
));
} else if key.contains(' ') {
warnings.push(format!(
"→ reference doesn't look like a key: → {}", key,
));
}
// Don't warn about missing keys — the target might be created later
}
}
}
warnings
}
pub fn cmd_history(key: &[String], full: bool) -> Result<(), String> {
if key.is_empty() { if key.is_empty() {
return Err("history requires a key".into()); bail!("history requires a key");
} }
let raw_key = key.join(" "); let key = key.join(" ");
let result = memory::memory_history(None, &key, Some(full)).await?;
let store = store::Store::load()?; print!("{}", result);
let key = store.resolve_key(&raw_key).unwrap_or(raw_key);
drop(store);
let path = store::nodes_path();
if !path.exists() {
return Err("No node log found".into());
}
use std::io::BufReader;
let file = std::fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
let mut versions: Vec<store::Node> = Vec::new();
while let Ok(msg) = capnp::serialize::read_message(&mut reader, capnp::message::ReaderOptions::new()) {
let log = msg.get_root::<crate::memory_capnp::node_log::Reader>()
.map_err(|e| format!("read log: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes: {}", e))? {
let node = store::Node::from_capnp_migrate(node_reader)?;
if node.key == key {
versions.push(node);
}
}
}
if versions.is_empty() {
return Err(format!("No history found for '{}'", key));
}
eprintln!("{} versions of '{}':\n", versions.len(), key);
for node in &versions {
let ts = if node.timestamp > 0 && node.timestamp < 4_000_000_000 {
store::format_datetime(node.timestamp)
} else {
format!("(raw:{})", node.timestamp)
};
let deleted_marker = if node.deleted { " DELETED" } else { "" };
let content_len = node.content.len();
if full {
eprintln!("=== v{} {} {}{} w={:.3} {}b ===",
node.version, ts, node.provenance, deleted_marker, node.weight, content_len);
eprintln!("{}", node.content);
} else {
let preview = crate::util::first_n_chars(&node.content, 120);
let preview = preview.replace('\n', "\\n");
eprintln!(" v{:<3} {} {:24} w={:.3} {}b{}",
node.version, ts, node.provenance, node.weight, content_len, deleted_marker);
eprintln!(" {}", preview);
}
}
if !full
&& let Some(latest) = versions.last() {
eprintln!("\n--- Latest content (v{}, {}) ---",
latest.version, latest.provenance);
print!("{}", latest.content);
}
Ok(()) Ok(())
} }
pub fn cmd_write(key: &[String]) -> Result<(), String> { pub async fn cmd_write(key: &[String]) -> Result<()> {
if key.is_empty() { if key.is_empty() {
return Err("write requires a key (reads content from stdin)".into()); bail!("write requires a key (reads content from stdin)");
} }
let raw_key = key.join(" "); let key = key.join(" ");
let mut content = String::new(); let mut content = String::new();
std::io::Read::read_to_string(&mut std::io::stdin(), &mut content) std::io::Read::read_to_string(&mut std::io::stdin(), &mut content)
.map_err(|e| format!("read stdin: {}", e))?; .context("read stdin")?;
if content.trim().is_empty() { if content.trim().is_empty() {
return Err("No content on stdin".into()); bail!("No content on stdin");
} }
super::check_dry_run(); super::check_dry_run();
let mut store = store::Store::load()?; let result = memory::memory_write(None, &key, &content).await?;
let key = store.resolve_key(&raw_key).unwrap_or(raw_key); println!("{}", result);
// Validate inline references: warn about render commands embedded
// in content (should be just `key`) and broken references.
let warnings = validate_inline_refs(&content, &store);
for w in &warnings {
eprintln!("warning: {}", w);
}
let result = store.upsert(&key, &content)?;
match result {
"unchanged" => println!("No change: '{}'", key),
"updated" => println!("Updated '{}' (v{})", key, store.nodes[&key].version),
_ => println!("Created '{}'", key),
}
if result != "unchanged" {
store.save()?;
}
Ok(()) Ok(())
} }
pub fn cmd_edit(key: &[String]) -> Result<(), String> { pub async fn cmd_edit(key: &[String]) -> Result<()> {
if key.is_empty() { if key.is_empty() {
return Err("edit requires a key".into()); bail!("edit requires a key");
} }
let raw_key = key.join(" "); let key = key.join(" ");
let store = store::Store::load()?;
let key = store.resolve_key(&raw_key).unwrap_or(raw_key.clone());
let content = store.nodes.get(&key) // Get raw content
.map(|n| n.content.clone()) let content = memory::memory_render(None, &key, Some(true)).await
.unwrap_or_default(); .unwrap_or_default();
let tmp = std::env::temp_dir().join(format!("poc-memory-edit-{}.md", key.replace('/', "_"))); let tmp = std::env::temp_dir().join(format!("poc-memory-edit-{}.md", key.replace('/', "_")));
std::fs::write(&tmp, &content) std::fs::write(&tmp, &content)
.map_err(|e| format!("write temp file: {}", e))?; .with_context(|| format!("write temp file {}", tmp.display()))?;
let editor = std::env::var("EDITOR").unwrap_or_else(|_| "vi".into()); let editor = std::env::var("EDITOR").unwrap_or_else(|_| "vi".into());
let status = std::process::Command::new(&editor) let status = std::process::Command::new(&editor)
.arg(&tmp) .arg(&tmp)
.status() .status()
.map_err(|e| format!("spawn {}: {}", editor, e))?; .with_context(|| format!("spawn {}", editor))?;
if !status.success() { if !status.success() {
let _ = std::fs::remove_file(&tmp); let _ = std::fs::remove_file(&tmp);
return Err(format!("{} exited with {}", editor, status)); bail!("{} exited with {}", editor, status);
} }
let new_content = std::fs::read_to_string(&tmp) let new_content = std::fs::read_to_string(&tmp)
.map_err(|e| format!("read temp file: {}", e))?; .with_context(|| format!("read temp file {}", tmp.display()))?;
let _ = std::fs::remove_file(&tmp); let _ = std::fs::remove_file(&tmp);
if new_content == content { if new_content == content {
@ -405,51 +127,124 @@ pub fn cmd_edit(key: &[String]) -> Result<(), String> {
} }
if new_content.trim().is_empty() { if new_content.trim().is_empty() {
return Err("Content is empty, aborting".into()); bail!("Content is empty, aborting");
} }
drop(store); super::check_dry_run();
let mut store = store::Store::load()?; let result = memory::memory_write(None, &key, &new_content).await?;
let result = store.upsert(&key, &new_content)?; println!("{}", result);
match result {
"unchanged" => println!("No change: '{}'", key),
"updated" => println!("Updated '{}' (v{})", key, store.nodes[&key].version),
_ => println!("Created '{}'", key),
}
if result != "unchanged" {
store.save()?;
}
Ok(()) Ok(())
} }
pub fn cmd_lookup_bump(keys: &[String]) -> Result<(), String> { pub async fn cmd_search(keys: &[String]) -> Result<()> {
if keys.is_empty() { if keys.is_empty() {
return Err("lookup-bump requires at least one key".into()); bail!("search requires seed keys");
} }
let keys: Vec<&str> = keys.iter().map(|s| s.as_str()).collect(); let result = memory::memory_search(None, keys.to_vec(), None, None, None, None).await?;
crate::lookups::bump_many(&keys) print!("{}", result);
Ok(())
} }
pub fn cmd_lookups(date: Option<&str>) -> Result<(), String> { pub async fn cmd_query(expr: &[String]) -> Result<()> {
let date = date.map(|d| d.to_string()) if expr.is_empty() {
.unwrap_or_else(|| chrono::Local::now().format("%Y-%m-%d").to_string()); bail!("query requires an expression (try: poc-memory query --help)");
}
let store = store::Store::load()?; let query_str = expr.join(" ");
let keys: Vec<String> = store.nodes.values().map(|n| n.key.clone()).collect(); let result = memory::memory_query(None, &query_str, None).await?;
let resolved = crate::lookups::dump_resolved(&date, &keys)?; print!("{}", result);
Ok(())
}
if resolved.is_empty() { /// Get group content (handles daemon or local fallback)
println!("No lookups for {}", date); pub async fn get_group_content(group: &crate::config::ContextGroup, cfg: &crate::config::Config) -> Vec<(String, String)> {
match group.source {
crate::config::ContextSource::Journal => {
// Query for recent journal entries
let window: i64 = cfg.journal_days as i64 * 24 * 3600;
let query = format!("all | type:episodic | age:<{} | sort:timestamp | limit:{}",
window, cfg.journal_max);
let keys_str = match memory::memory_query(None, &query, None).await {
Ok(s) => s,
Err(_) => return vec![],
};
// Parse keys (one per line) and render each
let mut results = Vec::new();
for key in keys_str.lines().filter(|k| !k.is_empty() && *k != "no results") {
if let Ok(content) = memory::memory_render(None, key, Some(true)).await {
if !content.trim().is_empty() {
results.push((key.to_string(), content));
}
}
}
results
}
crate::config::ContextSource::File => {
group.keys.iter().filter_map(|key| {
let content = std::fs::read_to_string(cfg.identity_dir.join(key)).ok()?;
if content.trim().is_empty() { return None; }
Some((key.clone(), content.trim().to_string()))
}).collect()
}
crate::config::ContextSource::Store => {
let mut results = Vec::new();
for key in &group.keys {
if let Ok(content) = memory::memory_render(None, key, Some(true)).await {
if !content.trim().is_empty() {
results.push((key.clone(), content.trim().to_string()));
}
}
}
results
}
}
}
pub async fn cmd_load_context(stats: bool) -> Result<()> {
let cfg = crate::config::get();
if stats {
let mut total_words = 0;
let mut total_entries = 0;
println!("{:<25} {:>6} {:>8}", "GROUP", "ITEMS", "WORDS");
println!("{}", "-".repeat(42));
for group in &cfg.context_groups {
let entries = get_group_content(group, &cfg).await;
let words: usize = entries.iter()
.map(|(_, c)| c.split_whitespace().count())
.sum();
let count = entries.len();
println!("{:<25} {:>6} {:>8}", group.label, count, words);
total_words += words;
total_entries += count;
}
println!("{}", "-".repeat(42));
println!("{:<25} {:>6} {:>8}", "TOTAL", total_entries, total_words);
return Ok(()); return Ok(());
} }
println!("Lookups for {}:", date); println!("=== MEMORY SYSTEM ({}) ===", cfg.assistant_name);
for (key, count) in &resolved {
println!(" {:4} {}", count, key); for group in &cfg.context_groups {
let entries = get_group_content(group, &cfg).await;
if !entries.is_empty() && group.source == crate::config::ContextSource::Journal {
println!("--- recent journal entries ({}/{}) ---",
entries.len(), cfg.journal_max);
} }
println!("\n{} distinct keys, {} total lookups", for (key, content) in entries {
resolved.len(), if group.source == crate::config::ContextSource::Journal {
resolved.iter().map(|(_, c)| *c as u64).sum::<u64>()); println!("## {}", key);
} else {
println!("--- {} ({}) ---", key, group.label);
}
println!("{}\n", content);
}
}
println!("=== END MEMORY LOAD ===");
Ok(()) Ok(())
} }

30
src/config.rs
View file

@ -58,6 +58,10 @@ fn default_stream_timeout() -> u64 { 60 }
fn default_scoring_chunk_tokens() -> usize { 50_000 } fn default_scoring_chunk_tokens() -> usize { 50_000 }
fn default_scoring_interval_secs() -> u64 { 3600 } // 1 hour fn default_scoring_interval_secs() -> u64 { 3600 } // 1 hour
fn default_scoring_response_window() -> usize { 100 } fn default_scoring_response_window() -> usize { 100 }
fn default_node_weight() -> f64 { 0.7 }
fn default_edge_decay() -> f64 { 0.3 }
fn default_max_hops() -> u32 { 3 }
fn default_min_activation() -> f64 { 0.05 }
fn default_identity_dir() -> PathBuf { fn default_identity_dir() -> PathBuf {
dirs::home_dir().unwrap_or_default().join(".consciousness/identity") dirs::home_dir().unwrap_or_default().join(".consciousness/identity")
} }
@ -120,6 +124,16 @@ pub struct Config {
/// Hook events that trigger the surface agent. /// Hook events that trigger the surface agent.
#[serde(default)] #[serde(default)]
pub surface_hooks: Vec<String>, pub surface_hooks: Vec<String>,
// Spreading activation parameters
#[serde(default = "default_node_weight")]
pub default_node_weight: f64,
#[serde(default = "default_edge_decay")]
pub edge_decay: f64,
#[serde(default = "default_max_hops")]
pub max_hops: u32,
#[serde(default = "default_min_activation")]
pub min_activation: f64,
} }
impl Default for Config { impl Default for Config {
@ -170,6 +184,10 @@ impl Default for Config {
surface_hooks: vec![], surface_hooks: vec![],
mcp_servers: vec![], mcp_servers: vec![],
lsp_servers: vec![], lsp_servers: vec![],
default_node_weight: default_node_weight(),
edge_decay: default_edge_decay(),
max_hops: default_max_hops(),
min_activation: default_min_activation(),
} }
} }
} }
@ -506,7 +524,7 @@ pub struct ResolvedModel {
impl AppConfig { impl AppConfig {
/// Resolve the active backend and assemble prompts into a SessionConfig. /// Resolve the active backend and assemble prompts into a SessionConfig.
pub fn resolve(&self, cli: &crate::user::CliArgs) -> Result<SessionConfig> { pub async fn resolve(&self, cli: &crate::user::CliArgs) -> Result<SessionConfig> {
let cwd = std::env::current_dir().context("Failed to get current directory")?; let cwd = std::env::current_dir().context("Failed to get current directory")?;
let (api_base, api_key, model, prompt_file); let (api_base, api_key, model, prompt_file);
@ -536,7 +554,7 @@ impl AppConfig {
let context_groups = get().context_groups.clone(); let context_groups = get().context_groups.clone();
let (context_parts, config_file_count, memory_file_count) = let (context_parts, config_file_count, memory_file_count) =
crate::mind::identity::assemble_context_message(&cwd, &prompt_file, self.memory_project.as_deref(), &context_groups)?; crate::mind::identity::assemble_context_message(&cwd, &prompt_file, self.memory_project.as_deref(), &context_groups).await?;
let session_dir = dirs::home_dir() let session_dir = dirs::home_dir()
.unwrap_or_else(|| PathBuf::from(".")) .unwrap_or_else(|| PathBuf::from("."))
@ -668,17 +686,17 @@ pub fn load_app(cli: &crate::user::CliArgs) -> Result<(AppConfig, Figment)> {
} }
/// Load the full config: figment → AppConfig → resolve backend → assemble prompts. /// Load the full config: figment → AppConfig → resolve backend → assemble prompts.
pub fn load_session(cli: &crate::user::CliArgs) -> Result<(SessionConfig, Figment)> { pub async fn load_session(cli: &crate::user::CliArgs) -> Result<(SessionConfig, Figment)> {
let (app, figment) = load_app(cli)?; let (app, figment) = load_app(cli)?;
let config = app.resolve(cli)?; let config = app.resolve(cli).await?;
Ok((config, figment)) Ok((config, figment))
} }
/// Re-assemble context for a specific model's prompt file. /// Re-assemble context for a specific model's prompt file.
pub fn reload_for_model(app: &AppConfig, prompt_file: &str) -> Result<Vec<(String, String)>> { pub async fn reload_for_model(app: &AppConfig, prompt_file: &str) -> Result<Vec<(String, String)>> {
let cwd = std::env::current_dir().context("Failed to get current directory")?; let cwd = std::env::current_dir().context("Failed to get current directory")?;
let context_groups = get().context_groups.clone(); let context_groups = get().context_groups.clone();
let (context_parts, _, _) = crate::mind::identity::assemble_context_message(&cwd, prompt_file, app.memory_project.as_deref(), &context_groups)?; let (context_parts, _, _) = crate::mind::identity::assemble_context_message(&cwd, prompt_file, app.memory_project.as_deref(), &context_groups).await?;
Ok(context_parts) Ok(context_parts)
} }

25
src/hippocampus/graph.rs
View file

@ -519,11 +519,9 @@ pub fn build_graph_fast(store: &impl StoreView) -> Graph {
fn build_adjacency(store: &impl StoreView) -> (HashMap<String, Vec<Edge>>, HashSet<String>) { fn build_adjacency(store: &impl StoreView) -> (HashMap<String, Vec<Edge>>, HashSet<String>) {
let mut adj: HashMap<String, Vec<Edge>> = HashMap::new(); let mut adj: HashMap<String, Vec<Edge>> = HashMap::new();
let mut keys: HashSet<String> = HashSet::new();
store.for_each_node(|key, _, _| { // Get keys directly from index — no need to deserialize node content
keys.insert(key.to_owned()); let keys: HashSet<String> = store.all_keys().into_iter().collect();
});
store.for_each_relation(|source_key, target_key, strength, rel_type| { store.for_each_relation(|source_key, target_key, strength, rel_type| {
if !keys.contains(source_key) || !keys.contains(target_key) { if !keys.contains(source_key) || !keys.contains(target_key) {
@ -882,22 +880,24 @@ pub fn health_report(graph: &Graph, store: &Store) -> String {
.count(); .count();
// Orphan edges: relations referencing non-existent nodes // Orphan edges: relations referencing non-existent nodes
// With index-based lookup, we count edges where endpoints don't resolve
let mut orphan_edges = 0usize; let mut orphan_edges = 0usize;
let mut missing_nodes: HashSet<String> = HashSet::new(); let mut missing_nodes: HashSet<String> = HashSet::new();
for rel in &store.relations { store.for_each_relation(|source, target, _, _| {
if rel.deleted { continue; } let s_missing = !store.contains_key(source).unwrap_or(false);
let s_missing = !store.nodes.contains_key(&rel.source_key); let t_missing = !store.contains_key(target).unwrap_or(false);
let t_missing = !store.nodes.contains_key(&rel.target_key);
if s_missing || t_missing { if s_missing || t_missing {
orphan_edges += 1; orphan_edges += 1;
if s_missing { missing_nodes.insert(rel.source_key.clone()); } if s_missing { missing_nodes.insert(source.to_string()); }
if t_missing { missing_nodes.insert(rel.target_key.clone()); } if t_missing { missing_nodes.insert(target.to_string()); }
}
} }
});
// NodeType breakdown // NodeType breakdown
let mut type_counts: HashMap<&str, usize> = HashMap::new(); let mut type_counts: HashMap<&str, usize> = HashMap::new();
for node in store.nodes.values() { let all_keys = store.all_keys().unwrap_or_default();
for key in &all_keys {
if let Ok(Some(node)) = store.get_node(key) {
let label = match node.node_type { let label = match node.node_type {
crate::store::NodeType::EpisodicSession => "episodic", crate::store::NodeType::EpisodicSession => "episodic",
crate::store::NodeType::EpisodicDaily => "daily", crate::store::NodeType::EpisodicDaily => "daily",
@ -907,6 +907,7 @@ pub fn health_report(graph: &Graph, store: &Store) -> String {
}; };
*type_counts.entry(label).or_default() += 1; *type_counts.entry(label).or_default() += 1;
} }
}
// Load history for deltas // Load history for deltas
let history = load_metrics_history(); let history = load_metrics_history();

605
src/hippocampus/local.rs Normal file
View file

@ -0,0 +1,605 @@
use anyhow::Result;
use super::memory::MemoryNode;
use super::store::Store;
use crate::graph::Graph;
use crate::neuro::{consolidation_priority, ReplayItem};
// ── Memory operations ──────────────────────────────────────────
pub fn memory_render(store: &Store, _provenance: &str, key: &str, raw: Option<bool>) -> Result<String> {
let node = MemoryNode::from_store(store, key)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", key))?;
if raw.unwrap_or(false) {
Ok(node.content)
} else {
Ok(node.render())
}
}
pub fn memory_write(store: &Store, provenance: &str, key: &str, content: &str) -> Result<String> {
let result = store.upsert_provenance(key, content, provenance)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("{} '{}'", result, key))
}
pub fn memory_search(
store: &Store,
_provenance: &str,
keys: Vec<String>,
max_hops: Option<u32>,
edge_decay: Option<f64>,
min_activation: Option<f64>,
limit: Option<usize>,
) -> Result<String> {
if keys.is_empty() {
anyhow::bail!("memory_search requires at least one seed key");
}
let max_hops = max_hops.unwrap_or(3);
let edge_decay = edge_decay.unwrap_or(0.3);
let min_activation = min_activation.unwrap_or(0.01);
let limit = limit.unwrap_or(20);
let graph = crate::graph::build_graph_fast(store);
let seeds: Vec<(String, f64)> = keys.iter()
.filter_map(|k| {
let resolved = store.resolve_key(k).ok()?;
Some((resolved, 1.0))
})
.collect();
if seeds.is_empty() {
anyhow::bail!("no valid seed keys found");
}
let seed_set: std::collections::HashSet<&str> = seeds.iter()
.map(|(k, _)| k.as_str()).collect();
let results = crate::search::spreading_activation(
&seeds, &graph, store,
max_hops, edge_decay, min_activation,
);
Ok(results.iter()
.filter(|(k, _)| !seed_set.contains(k.as_str()))
.take(limit)
.map(|(key, score)| format!(" {:.2} {}", score, key))
.collect::<Vec<_>>().join("\n"))
}
/// Info about a linked neighbor node.
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct LinkInfo {
pub key: String,
pub link_strength: f32,
pub node_weight: f32,
}
pub fn memory_links(store: &Store, _provenance: &str, key: &str) -> Result<Vec<LinkInfo>> {
let node = MemoryNode::from_store(store, key)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", key))?;
let mut links = Vec::new();
for (target, strength, _is_new) in &node.links {
let node_weight = store.get_node(target)
.ok()
.flatten()
.map(|n| n.weight)
.unwrap_or(0.5);
links.push(LinkInfo {
key: target.clone(),
link_strength: *strength,
node_weight,
});
}
Ok(links)
}
pub fn memory_link_set(store: &Store, _provenance: &str, source: &str, target: &str, strength: f32) -> Result<String> {
let s = store.resolve_key(source).map_err(|e| anyhow::anyhow!("{}", e))?;
let t = store.resolve_key(target).map_err(|e| anyhow::anyhow!("{}", e))?;
let old = store.set_link_strength(&s, &t, strength).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("{} ↔ {} strength {:.2} → {:.2}", s, t, old, strength))
}
pub fn memory_link_add(store: &Store, provenance: &str, source: &str, target: &str) -> Result<String> {
let s = store.resolve_key(source).map_err(|e| anyhow::anyhow!("{}", e))?;
let t = store.resolve_key(target).map_err(|e| anyhow::anyhow!("{}", e))?;
let strength = store.add_link(&s, &t, provenance).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("linked {} → {} (strength={:.2})", s, t, strength))
}
pub fn memory_delete(store: &Store, _provenance: &str, key: &str) -> Result<String> {
let resolved = store.resolve_key(key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.delete_node(&resolved).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("deleted {}", resolved))
}
pub fn memory_history(store: &Store, _provenance: &str, key: &str, full: Option<bool>) -> Result<String> {
let key = store.resolve_key(key).unwrap_or_else(|_| key.to_string());
let full = full.unwrap_or(false);
let path = crate::store::nodes_path();
if !path.exists() {
anyhow::bail!("No node log found");
}
use std::io::BufReader;
let file = std::fs::File::open(&path)
.map_err(|e| anyhow::anyhow!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
let mut versions: Vec<crate::store::Node> = Vec::new();
while let Ok(msg) = capnp::serialize::read_message(&mut reader, capnp::message::ReaderOptions::new()) {
let log = msg.get_root::<crate::memory_capnp::node_log::Reader>()
.map_err(|e| anyhow::anyhow!("read log: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| anyhow::anyhow!("get nodes: {}", e))? {
let node = crate::store::Node::from_capnp_migrate(node_reader)
.map_err(|e| anyhow::anyhow!("{}", e))?;
if node.key == key {
versions.push(node);
}
}
}
if versions.is_empty() {
anyhow::bail!("No history found for '{}'", key);
}
let mut out = format!("{} versions of '{}':\n\n", versions.len(), key);
for node in &versions {
let ts = crate::store::format_datetime(node.timestamp);
let deleted = if node.deleted { " DELETED" } else { "" };
if full {
out.push_str(&format!("=== v{} {} {}{} w={:.3} {}b ===\n",
node.version, ts, node.provenance, deleted, node.weight, node.content.len()));
out.push_str(&node.content);
out.push('\n');
} else {
let preview = crate::util::first_n_chars(&node.content, 120).replace('\n', "\\n");
out.push_str(&format!("v{:<3} {} {:24} w={:.3} {}b{}\n {}\n",
node.version, ts, node.provenance, node.weight, node.content.len(), deleted, preview));
}
}
Ok(out)
}
pub fn memory_weight_set(store: &Store, _provenance: &str, key: &str, weight: f32) -> Result<String> {
let resolved = store.resolve_key(key).map_err(|e| anyhow::anyhow!("{}", e))?;
let (old, new) = store.set_weight(&resolved, weight).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("weight {} {:.2} → {:.2}", resolved, old, new))
}
pub fn memory_rename(store: &Store, _provenance: &str, old_key: &str, new_key: &str) -> Result<String> {
let resolved = store.resolve_key(old_key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.rename_node(&resolved, new_key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("Renamed '{}' → '{}'", resolved, new_key))
}
pub fn memory_supersede(store: &Store, provenance: &str, old_key: &str, new_key: &str, reason: Option<&str>) -> Result<String> {
let reason = reason.unwrap_or("superseded");
let content = store.get_node(old_key)
.map_err(|e| anyhow::anyhow!("{}", e))?
.map(|n| n.content)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", old_key))?;
let notice = format!("**SUPERSEDED** by `{}` — {}\n\n---\n\n{}",
new_key, reason, content.trim());
store.upsert_provenance(old_key, &notice, provenance)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.set_weight(old_key, 0.01).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("superseded {} → {} ({})", old_key, new_key, reason))
}
/// Convert a list of keys to ReplayItems with priority and graph metrics.
pub fn keys_to_replay_items(
store: &Store,
keys: &[String],
graph: &Graph,
) -> Vec<ReplayItem> {
keys.iter()
.filter_map(|key| {
let node = store.get_node(key).ok()??;
let priority = consolidation_priority(store, key, graph, None);
let cc = graph.clustering_coefficient(key);
Some(ReplayItem {
key: key.clone(),
priority,
interval_days: node.spaced_repetition_interval,
emotion: node.emotion,
cc,
classification: "unknown",
outlier_score: 0.0,
})
})
.collect()
}
pub fn memory_query(store: &Store, _provenance: &str, query_str: &str, format: Option<&str>) -> Result<String> {
let graph = store.build_graph();
match format.unwrap_or("compact") {
"full" => {
// Rich output with full content, graph metrics, hub analysis
let results = crate::query_parser::execute_query(store, &graph, query_str)
.map_err(|e| anyhow::anyhow!("{}", e))?;
let keys: Vec<String> = results.into_iter().map(|r| r.key).collect();
let items = keys_to_replay_items(store, &keys, &graph);
Ok(crate::subconscious::prompts::format_nodes_section(store, &items, &graph))
}
_ => {
// Compact output: handles count, select, and all expression types
crate::query_parser::query_to_string(store, &graph, query_str)
.map_err(|e| anyhow::anyhow!("{}", e))
}
}
}
// ── Journal tools ──────────────────────────────────────────────
/// A journal entry with key, content, and timestamp.
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct JournalEntry {
pub key: String,
pub content: String,
pub created_at: i64,
}
/// Get journal entries, sorted by timestamp (newest first).
/// level: 0=session, 1=daily, 2=weekly, 3=monthly
/// after: only entries after this date (YYYY-MM-DD)
pub fn journal_tail(store: &Store, _provenance: &str, count: Option<u64>, level: Option<u64>, after: Option<&str>) -> Result<Vec<JournalEntry>> {
let count = count.unwrap_or(10) as usize;
let level = level.unwrap_or(0);
let node_type = match level {
0 => crate::store::NodeType::EpisodicSession,
1 => crate::store::NodeType::EpisodicDaily,
2 => crate::store::NodeType::EpisodicWeekly,
3 => crate::store::NodeType::EpisodicMonthly,
_ => return Err(anyhow::anyhow!("invalid level: {}", level)),
};
let after_ts = after.and_then(|date| {
chrono::NaiveDate::parse_from_str(date, "%Y-%m-%d").ok()
.and_then(|nd| nd.and_hms_opt(0, 0, 0))
.map(|dt| dt.and_utc().timestamp())
});
let all_keys = store.all_keys()?;
let mut entries: Vec<_> = all_keys.iter()
.filter_map(|key| store.get_node(key).ok()?)
.filter(|n| n.node_type == node_type)
.filter(|n| after_ts.map(|ts| n.created_at >= ts).unwrap_or(true))
.map(|n| JournalEntry {
key: n.key.clone(),
content: n.content,
created_at: n.created_at,
})
.collect();
entries.sort_by_key(|e| std::cmp::Reverse(e.created_at));
entries.truncate(count);
Ok(entries)
}
fn level_to_node_type(level: i64) -> crate::store::NodeType {
match level {
1 => crate::store::NodeType::EpisodicDaily,
2 => crate::store::NodeType::EpisodicWeekly,
3 => crate::store::NodeType::EpisodicMonthly,
_ => crate::store::NodeType::EpisodicSession,
}
}
pub fn journal_new(store: &Store, provenance: &str, name: &str, title: &str, body: &str, level: Option<i64>) -> Result<String> {
let level = level.unwrap_or(0);
let ts = chrono::Local::now().format("%Y-%m-%dT%H:%M");
let content = format!("## {} — {}\n\n{}", ts, title, body);
let base_key: String = name.split_whitespace()
.map(|w| w.to_lowercase()
.chars().filter(|c| c.is_alphanumeric() || *c == '-')
.collect::<String>())
.filter(|s| !s.is_empty())
.collect::<Vec<_>>()
.join("-");
let base_key = if base_key.len() > 80 { &base_key[..80] } else { base_key.as_str() };
let key = if store.contains_key(base_key).unwrap_or(false) {
let mut n = 2;
loop {
let candidate = format!("{}-{}", base_key, n);
if !store.contains_key(&candidate).unwrap_or(false) { break candidate; }
n += 1;
}
} else {
base_key.to_string()
};
let mut node = crate::store::new_node(&key, &content);
node.node_type = level_to_node_type(level);
node.provenance = provenance.to_string();
store.upsert_node(node).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
let word_count = body.split_whitespace().count();
Ok(format!("New entry '{}' ({} words)", title, word_count))
}
pub fn journal_update(store: &Store, provenance: &str, body: &str, level: Option<i64>) -> Result<String> {
let level = level.unwrap_or(0);
let node_type = level_to_node_type(level);
let all_keys = store.all_keys()?;
let latest_key = all_keys.iter()
.filter_map(|key| store.get_node(key).ok()?)
.filter(|n| n.node_type == node_type)
.max_by_key(|n| n.created_at)
.map(|n| n.key.clone());
let Some(key) = latest_key else {
anyhow::bail!("no entry at level {} to update — use journal_new first", level);
};
let existing = store.get_node(&key)?.ok_or_else(|| anyhow::anyhow!("node not found"))?.content;
let new_content = format!("{}\n\n{}", existing.trim_end(), body);
store.upsert_provenance(&key, &new_content, provenance)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
let word_count = body.split_whitespace().count();
Ok(format!("Updated last entry (+{} words)", word_count))
}
// ── Graph tools ───────────────────────────────────────────────
pub fn graph_topology(store: &Store, _provenance: &str) -> Result<String> {
let graph = store.build_graph();
Ok(crate::subconscious::prompts::format_topology_header(store, &graph))
}
pub fn graph_health(store: &Store, _provenance: &str) -> Result<String> {
let graph = store.build_graph();
Ok(crate::subconscious::prompts::format_health_section(store, &graph))
}
pub fn graph_communities(store: &Store, _provenance: &str, top_n: Option<usize>, min_size: Option<usize>) -> Result<String> {
let top_n = top_n.unwrap_or(10);
let min_size = min_size.unwrap_or(3);
let g = store.build_graph();
let infos = g.community_info();
let total = infos.len();
let shown: Vec<_> = infos.into_iter()
.filter(|c| c.size >= min_size)
.take(top_n)
.collect();
use std::fmt::Write;
let mut out = String::new();
writeln!(out, "{} communities total ({} with size >= {})\n",
total, shown.len(), min_size).ok();
writeln!(out, "{:<6} {:>5} {:>7} {:>7} members", "id", "size", "iso", "cross").ok();
writeln!(out, "{}", "-".repeat(70)).ok();
for c in &shown {
let preview: Vec<&str> = c.members.iter()
.take(5)
.map(|s| s.as_str())
.collect();
let more = if c.size > 5 {
format!(" +{}", c.size - 5)
} else {
String::new()
};
writeln!(out, "{:<6} {:>5} {:>6.0}% {:>7} {}{}",
c.id, c.size, c.isolation * 100.0, c.cross_edges,
preview.join(", "), more).ok();
}
Ok(out)
}
pub fn graph_normalize_strengths(store: &Store, _provenance: &str, apply: Option<bool>) -> Result<String> {
use crate::store::{StoreView, RelationType};
let apply = apply.unwrap_or(false);
let graph = store.build_graph();
let strengths = graph.jaccard_strengths();
// Build lookup from (source_key, target_key) → new_strength
let mut target_strengths: std::collections::HashMap<(String, String), f32> = std::collections::HashMap::new();
for (a, b, s) in &strengths {
target_strengths.insert((a.clone(), b.clone()), *s);
target_strengths.insert((b.clone(), a.clone()), *s);
}
// Collect edges and compute changes
let mut to_update: Vec<(String, String, f32)> = Vec::new();
let mut unchanged = 0usize;
let mut temporal_skipped = 0usize;
let mut delta_sum: f64 = 0.0;
let mut buckets = [0usize; 10];
store.for_each_relation(|source, target, strength, rel_type| {
// Skip temporal links
if strength == 1.0 && rel_type == RelationType::Auto {
temporal_skipped += 1;
return;
}
if let Some(&new_s) = target_strengths.get(&(source.to_string(), target.to_string())) {
let delta = (new_s - strength).abs();
if delta > 0.001 {
delta_sum += delta as f64;
to_update.push((source.to_string(), target.to_string(), new_s));
} else {
unchanged += 1;
}
let bucket = ((new_s * 10.0) as usize).min(9);
buckets[bucket] += 1;
}
});
let changed = to_update.len();
use std::fmt::Write;
let mut out = String::new();
writeln!(out, "Normalize link strengths (Jaccard similarity)").ok();
writeln!(out, " Total edges in graph: {}", strengths.len()).ok();
writeln!(out, " Would change: {}", changed).ok();
writeln!(out, " Unchanged: {}", unchanged).ok();
writeln!(out, " Temporal (skipped): {}", temporal_skipped).ok();
if changed > 0 {
writeln!(out, " Avg delta: {:.3}", delta_sum / changed as f64).ok();
}
writeln!(out).ok();
writeln!(out, " Strength distribution:").ok();
for (i, &count) in buckets.iter().enumerate() {
let lo = i as f32 / 10.0;
let hi = lo + 0.1;
let bar = "#".repeat(count / 50 + if count > 0 { 1 } else { 0 });
writeln!(out, " {:.1}-{:.1}: {:5} {}", lo, hi, count, bar).ok();
}
if apply {
for (source, target, new_strength) in to_update {
store.set_link_strength(&source, &target, new_strength)?;
}
writeln!(out, "\nApplied {} strength updates.", changed).ok();
} else {
writeln!(out, "\nDry run. Pass apply:true to write changes.").ok();
}
Ok(out)
}
pub fn graph_link_impact(store: &Store, _provenance: &str, source: &str, target: &str) -> Result<String> {
let source = store.resolve_key(source).map_err(|e| anyhow::anyhow!("{}", e))?;
let target = store.resolve_key(target).map_err(|e| anyhow::anyhow!("{}", e))?;
let g = store.build_graph();
let impact = g.link_impact(&source, &target);
use std::fmt::Write;
let mut out = String::new();
writeln!(out, "Link impact: {} → {}", source, target).ok();
writeln!(out, " Source degree: {} Target degree: {}", impact.source_deg, impact.target_deg).ok();
writeln!(out, " Hub link: {} Same community: {}", impact.is_hub_link, impact.same_community).ok();
writeln!(out, " ΔCC source: {:+.4} ΔCC target: {:+.4}", impact.delta_cc_source, impact.delta_cc_target).ok();
writeln!(out, " ΔGini: {:+.6}", impact.delta_gini).ok();
writeln!(out, " Assessment: {}", impact.assessment).ok();
Ok(out)
}
pub fn graph_hubs(store: &Store, _provenance: &str, count: Option<usize>) -> Result<String> {
let count = count.unwrap_or(20);
let graph = store.build_graph();
// Top hub nodes by degree, spread apart (skip neighbors of already-selected hubs)
let all_keys = store.all_keys().unwrap_or_default();
let mut hubs: Vec<(String, usize)> = all_keys.iter()
.filter(|k| !k.starts_with('_'))
.map(|k| {
let degree = graph.neighbors(k).len();
(k.clone(), degree)
})
.collect();
hubs.sort_by(|a, b| b.1.cmp(&a.1));
let mut selected = Vec::new();
let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
for (key, degree) in &hubs {
if seen.contains(key) { continue; }
selected.push(format!(" - {} (degree {})", key, degree));
// Mark neighbors as seen so we pick far-apart hubs
for (nbr, _) in graph.neighbors(key) {
seen.insert(nbr.clone());
}
seen.insert(key.clone());
if selected.len() >= count { break; }
}
Ok(format!("## Hub nodes (link targets)\n\n{}", selected.join("\n")))
}
pub fn graph_trace(store: &Store, _provenance: &str, key: &str) -> Result<String> {
let resolved = store.resolve_key(key).map_err(|e| anyhow::anyhow!("{}", e))?;
let g = store.build_graph();
let node = store.get_node(&resolved)?
.ok_or_else(|| anyhow::anyhow!("Node not found: {}", resolved))?;
use std::fmt::Write;
let mut out = String::new();
writeln!(out, "=== {} ===", resolved).ok();
writeln!(out, "Type: {:?} Weight: {:.2}", node.node_type, node.weight).ok();
if !node.source_ref.is_empty() {
writeln!(out, "Source: {}", node.source_ref).ok();
}
let preview = crate::util::truncate(&node.content, 200, "...");
writeln!(out, "\n{}\n", preview).ok();
// Walk neighbors, grouped by node type
let neighbors = g.neighbors(&resolved);
let mut episodic_session: Vec<(String, f32, crate::store::Node)> = Vec::new();
let mut episodic_daily: Vec<(String, f32, crate::store::Node)> = Vec::new();
let mut episodic_weekly: Vec<(String, f32, crate::store::Node)> = Vec::new();
let mut semantic: Vec<(String, f32, crate::store::Node)> = Vec::new();
for (n, strength) in &neighbors {
if let Ok(Some(nnode)) = store.get_node(n) {
let node_type = nnode.node_type;
let key: String = (*n).clone();
let entry = (key, *strength, nnode);
match node_type {
crate::store::NodeType::EpisodicSession => episodic_session.push(entry),
crate::store::NodeType::EpisodicDaily => episodic_daily.push(entry),
crate::store::NodeType::EpisodicWeekly
| crate::store::NodeType::EpisodicMonthly => episodic_weekly.push(entry),
crate::store::NodeType::Semantic => semantic.push(entry),
}
}
}
if !episodic_weekly.is_empty() {
writeln!(out, "Weekly digests:").ok();
for (k, s, n) in &episodic_weekly {
let preview = crate::util::first_n_chars(n.content.lines().next().unwrap_or(""), 80);
writeln!(out, " [{:.2}] {} — {}", s, &k, preview).ok();
}
}
if !episodic_daily.is_empty() {
writeln!(out, "Daily digests:").ok();
for (k, s, n) in &episodic_daily {
let preview = crate::util::first_n_chars(n.content.lines().next().unwrap_or(""), 80);
writeln!(out, " [{:.2}] {} — {}", s, &k, preview).ok();
}
}
if !episodic_session.is_empty() {
writeln!(out, "Session entries:").ok();
for (k, s, n) in &episodic_session {
let preview = crate::util::first_n_chars(
n.content.lines()
.find(|l| !l.is_empty() && !l.starts_with("<!--"))
.unwrap_or(""),
80);
writeln!(out, " [{:.2}] {}", s, &k).ok();
if !n.source_ref.is_empty() {
writeln!(out, " ↳ source: {}", n.source_ref).ok();
}
writeln!(out, " {}", preview).ok();
}
}
if !semantic.is_empty() {
writeln!(out, "Semantic links:").ok();
for (k, s, _) in &semantic {
writeln!(out, " [{:.2}] {}", s, k).ok();
}
}
writeln!(out, "\nLinks: {} session, {} daily, {} weekly, {} semantic",
episodic_session.len(), episodic_daily.len(),
episodic_weekly.len(), semantic.len()).ok();
Ok(out)
}

39
src/hippocampus/memory.rs
View file

@ -19,13 +19,13 @@ pub struct MemoryNode {
impl MemoryNode { impl MemoryNode {
/// Load a node from the store by key. /// Load a node from the store by key.
pub fn load(key: &str) -> Option<Self> { pub fn load(key: &str) -> Option<Self> {
let store = Store::load().ok()?; let store = super::access_local().ok()?;
Self::from_store(&store, key) Self::from_store(&store, key)
} }
/// Load from an already-open store. /// Load from an already-open store.
pub fn from_store(store: &Store, key: &str) -> Option<Self> { pub fn from_store(store: &Store, key: &str) -> Option<Self> {
let node = store.nodes.get(key)?; let node = store.get_node(key).ok()??;
// If set, tag links to nodes created after this timestamp as (new) // If set, tag links to nodes created after this timestamp as (new)
let older_than: i64 = std::env::var("POC_MEMORIES_OLDER_THAN") let older_than: i64 = std::env::var("POC_MEMORIES_OLDER_THAN")
@ -33,34 +33,30 @@ impl MemoryNode {
.and_then(|s| s.parse().ok()) .and_then(|s| s.parse().ok())
.unwrap_or(0); .unwrap_or(0);
let mut neighbors: std::collections::HashMap<&str, (f32, bool)> = std::collections::HashMap::new(); // Get neighbors via index
for r in &store.relations { let mut neighbors: std::collections::HashMap<String, (f32, bool)> = std::collections::HashMap::new();
if r.deleted { continue; } if let Ok(neighbor_list) = store.neighbors(key) {
let neighbor_key = if r.source_key == key { for (neighbor_key, strength) in neighbor_list {
&r.target_key let is_new = older_than > 0 && store.get_node(&neighbor_key)
} else if r.target_key == key { .ok()
&r.source_key .flatten()
} else {
continue;
};
let is_new = older_than > 0 && store.nodes.get(neighbor_key.as_str())
.map(|n| n.created_at > older_than) .map(|n| n.created_at > older_than)
.unwrap_or(false); .unwrap_or(false);
let e = neighbors.entry(neighbor_key.as_str()).or_insert((0.0, false)); let e = neighbors.entry(neighbor_key).or_insert((0.0, false));
e.0 = e.0.max(r.strength); e.0 = e.0.max(strength);
e.1 = e.1 || is_new; e.1 = e.1 || is_new;
} }
}
let mut links: Vec<(String, f32, bool)> = neighbors.into_iter() let mut links: Vec<(String, f32, bool)> = neighbors.into_iter()
.map(|(k, (s, new))| (k.to_string(), s, new)) .map(|(k, (s, new))| (k, s, new))
.collect(); .collect();
links.sort_by(|a, b| b.1.total_cmp(&a.1)); links.sort_by(|a, b| b.1.total_cmp(&a.1));
Some(MemoryNode { Some(MemoryNode {
key: key.to_string(), key: key.to_string(),
content: node.content.clone(), content: node.content,
links, links,
version: node.version, version: node.version,
weight: node.weight, weight: node.weight,
@ -91,3 +87,10 @@ impl MemoryNode {
out out
} }
} }
/// Render a node to a string: content + deduped footer links.
/// Used by both the CLI command and agent placeholders.
pub fn render_node(store: &Store, key: &str) -> Option<String> {
crate::hippocampus::memory::MemoryNode::from_store(store, key)
.map(|node| node.render())
}

306
src/hippocampus/mod.rs
View file

@ -4,13 +4,319 @@
// similarity scoring, spectral analysis, and neuroscience-inspired // similarity scoring, spectral analysis, and neuroscience-inspired
// consolidation (spaced repetition, interference detection, schema // consolidation (spaced repetition, interference detection, schema
// assimilation). // assimilation).
//
// Tool implementations are typed functions that take &Store or &mut Store.
// The tools/memory.rs layer handles JSON parsing and RPC routing.
pub mod memory; pub mod memory;
pub mod store; pub mod store;
pub mod graph; pub mod graph;
pub mod local;
pub mod lookups; pub mod lookups;
pub mod query; pub mod query;
pub mod spectral; pub mod spectral;
pub mod neuro; pub mod neuro;
pub mod counters; pub mod counters;
pub mod transcript; pub mod transcript;
use std::cell::RefCell;
use std::path::PathBuf;
use std::sync::{Arc, OnceLock};
use anyhow::Result;
use crate::hippocampus::store::Store;
pub use local::{LinkInfo, JournalEntry};
// ── Store access ───────────────────────────────────────────────
/// Daemon's store (eager init) or client's fallback local store.
static STORE_ACCESS: OnceLock<Option<Arc<Store>>> = OnceLock::new();
// Client's socket connection (thread-local for lock-free access).
thread_local! {
static SOCKET_CONN: RefCell<Option<SocketConn>> = const { RefCell::new(None) };
}
/// How we access the memory store.
pub enum StoreAccess {
Daemon(Arc<Store>), // Direct store access
Client, // Socket to daemon (in thread-local)
None(String), // Error: couldn't get access
}
/// Get store access: daemon's store, socket, or local fallback.
pub fn access() -> StoreAccess {
// Check if already cached
if let Some(Some(store)) = STORE_ACCESS.get() {
return StoreAccess::Daemon(store.clone());
}
// Client: check if socket already cached in thread-local
let have_socket = SOCKET_CONN.with(|cell| cell.borrow().is_some());
if have_socket {
return StoreAccess::Client;
}
// No socket cached, try connecting
if let Ok(conn) = SocketConn::connect() {
SOCKET_CONN.with(|cell| *cell.borrow_mut() = Some(conn));
return StoreAccess::Client;
}
// Socket failed - try local store as fallback (cached in STORE_ACCESS)
let store_opt = STORE_ACCESS.get_or_init(|| {
Store::load().ok().map(Arc::new)
});
match store_opt {
Some(store) => StoreAccess::Daemon(store.clone()),
None => StoreAccess::None("could not connect to daemon or open store locally".into()),
}
}
/// Get local store access. Returns error if only RPC available.
pub fn access_local() -> Result<Arc<Store>> {
match access() {
StoreAccess::Daemon(arc) => Ok(arc),
StoreAccess::Client => anyhow::bail!("direct store access not available via RPC"),
StoreAccess::None(err) => anyhow::bail!("{}", err),
}
}
pub fn socket_path() -> PathBuf {
dirs::home_dir()
.unwrap_or_default()
.join(".consciousness/mcp.sock")
}
struct SocketConn {
reader: std::io::BufReader<std::os::unix::net::UnixStream>,
writer: std::io::BufWriter<std::os::unix::net::UnixStream>,
next_id: u64,
}
impl SocketConn {
fn connect() -> Result<Self> {
use std::os::unix::net::UnixStream;
use std::io::{BufRead, BufReader, BufWriter, Write};
let path = socket_path();
let stream = UnixStream::connect(&path)?;
let mut reader = BufReader::new(stream.try_clone()?);
let mut writer = BufWriter::new(stream);
// Initialize MCP connection
let init = serde_json::json!({"jsonrpc": "2.0", "id": 1, "method": "initialize",
"params": {"protocolVersion": "2024-11-05", "capabilities": {},
"clientInfo": {"name": "forward", "version": "0.1"}}});
writeln!(writer, "{}", init)?;
writer.flush()?;
let mut buf = String::new();
reader.read_line(&mut buf)?;
Ok(Self { reader, writer, next_id: 1 })
}
fn call(&mut self, tool_name: &str, args: &serde_json::Value) -> Result<String> {
use std::io::{BufRead, Write};
self.next_id += 1;
let call = serde_json::json!({"jsonrpc": "2.0", "id": self.next_id, "method": "tools/call",
"params": {"name": tool_name, "arguments": args}});
writeln!(self.writer, "{}", call)?;
self.writer.flush()?;
let mut buf = String::new();
self.reader.read_line(&mut buf)?;
let resp: serde_json::Value = serde_json::from_str(&buf)?;
if let Some(err) = resp.get("error") {
anyhow::bail!("daemon error: {}", err);
}
let result = resp.get("result").cloned().unwrap_or(serde_json::json!({}));
let text = result.get("content")
.and_then(|c| c.as_array())
.and_then(|arr| arr.first())
.and_then(|c| c.get("text"))
.and_then(|t| t.as_str())
.unwrap_or("");
Ok(text.to_string())
}
}
/// Forward a tool call to the daemon via socket.
/// Only valid when access() returns Client.
pub fn memory_rpc(tool_name: &str, args: serde_json::Value) -> Result<String> {
SOCKET_CONN.with(|cell| {
let mut conn = cell.borrow_mut();
let conn = conn.as_mut().expect("access() returned Client but SOCKET_CONN is None");
conn.call(tool_name, &args)
})
}
// ── Macro for generating tool wrappers ─────────────────────────
//
// memory_tool!(name, mut, arg1: [str], arg2: [Option<bool>])
// - mut/ref for store mutability
// - generates jsonargs_* (internal, JSON args) and public typed API
macro_rules! memory_tool {
// ── Helper rules (must come first) ─────────────────────────────
// Extract from JSON
(@extract $args:ident, $name:ident, str) => {
get_str($args, stringify!($name))?
};
(@extract $args:ident, $name:ident, f32) => {
get_f64($args, stringify!($name))? as f32
};
(@extract $args:ident, $name:ident, Vec<String>) => {
$args.get(stringify!($name))
.and_then(|v| v.as_array())
.map(|arr| arr.iter().filter_map(|v| v.as_str().map(String::from)).collect::<Vec<_>>())
.unwrap_or_default()
};
(@extract $args:ident, $name:ident, Option<&str>) => {
$args.get(stringify!($name)).and_then(|v| v.as_str())
};
(@extract $args:ident, $name:ident, Option<bool>) => {
$args.get(stringify!($name)).and_then(|v| v.as_bool())
};
(@extract $args:ident, $name:ident, Option<u64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64())
};
(@extract $args:ident, $name:ident, Option<i64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_i64())
};
(@extract $args:ident, $name:ident, Option<usize>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64()).map(|v| v as usize)
};
(@extract $args:ident, $name:ident, Option<u32>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64()).map(|v| v as u32)
};
(@extract $args:ident, $name:ident, Option<f64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_f64())
};
// Parameter types for function signatures
(@param_type str) => { &str };
(@param_type f32) => { f32 };
(@param_type Vec<String>) => { Vec<String> };
(@param_type Option<&str>) => { Option<&str> };
(@param_type Option<bool>) => { Option<bool> };
(@param_type Option<u64>) => { Option<u64> };
(@param_type Option<i64>) => { Option<i64> };
(@param_type Option<usize>) => { Option<usize> };
(@param_type Option<u32>) => { Option<u32> };
(@param_type Option<f64>) => { Option<f64> };
// Serialize result for jsonargs
(@serialize $t:ty, $result:expr) => { serde_json::to_string(&$result)? };
// Deserialize RPC response
(@deserialize $t:ty, $json:expr) => { serde_json::from_str(&$json).map_err(|e| anyhow::anyhow!("{}", e)) };
// Serialize to JSON for RPC
(@insert_json $map:ident, $name:ident, str) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, f32) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, Vec<String>) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, Option<&str>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<bool>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<u64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<i64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<usize>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<u32>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<f64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
// Call hippocampus (all methods now take &self, deref Arc)
(@call mut, $name:ident, $store:ident, $prov:expr $(, $arg:expr)*) => {
local::$name(&*$store, $prov $(, $arg)*)
};
(@call ref, $name:ident, $store:ident, $prov:expr $(, $arg:expr)*) => {
local::$name(&*$store, $prov $(, $arg)*)
};
// ── Main rules ─────────────────────────────────────────────────
// Shorthand: mut/ref without return type defaults to String
($name:ident, $m:ident $(, $($arg:ident : [$($typ:tt)+]),* $(,)?)?) => {
memory_tool!($name, $m -> String $(, $($arg : [$($typ)+]),*)?);
};
// Full form with return type
($name:ident, $m:ident -> $ret:ty $(, $($arg:ident : [$($typ:tt)+]),* $(,)?)?) => {
paste::paste! {
pub async fn $name(agent: Option<&crate::agent::Agent> $($(, $arg: memory_tool!(@param_type $($typ)+))*)?) -> Result<$ret> {
let prov = match agent {
Some(a) => a.state.lock().await.provenance.clone(),
None => "manual".to_string(),
};
match access() {
StoreAccess::Daemon(store) => {
memory_tool!(@call $m, $name, store, &prov $($(, $arg)*)?)
}
StoreAccess::Client => {
#[allow(unused_mut)]
let mut map = serde_json::Map::new();
$($(memory_tool!(@insert_json map, $arg, $($typ)+);)*)?
let json = memory_rpc(stringify!($name), serde_json::Value::Object(map))?;
memory_tool!(@deserialize $ret, json)
}
StoreAccess::None(err) => anyhow::bail!("{}", err),
}
}
}
};
}
// ── Memory tools ───────────────────────────────────────────────
memory_tool!(memory_render, ref, key: [str], raw: [Option<bool>]);
memory_tool!(memory_write, mut, key: [str], content: [str]);
memory_tool!(memory_search, ref, keys: [Vec<String>], max_hops: [Option<u32>], edge_decay: [Option<f64>], min_activation: [Option<f64>], limit: [Option<usize>]);
memory_tool!(memory_link_set, mut, source: [str], target: [str], strength: [f32]);
memory_tool!(memory_link_add, mut, source: [str], target: [str]);
memory_tool!(memory_delete, mut, key: [str]);
memory_tool!(memory_history, ref, key: [str], full: [Option<bool>]);
memory_tool!(memory_weight_set, mut, key: [str], weight: [f32]);
memory_tool!(memory_rename, mut, old_key: [str], new_key: [str]);
memory_tool!(memory_supersede, mut, old_key: [str], new_key: [str], reason: [Option<&str>]);
memory_tool!(memory_query, ref, query: [str], format: [Option<&str>]);
memory_tool!(memory_links, ref -> Vec<LinkInfo>, key: [str]);
// ── Journal tools ──────────────────────────────────────────────
memory_tool!(journal_tail, ref -> Vec<JournalEntry>, count: [Option<u64>], level: [Option<u64>], after: [Option<&str>]);
memory_tool!(journal_new, mut, name: [str], title: [str], body: [str], level: [Option<i64>]);
memory_tool!(journal_update, mut, body: [str], level: [Option<i64>]);
// ── Graph tools ───────────────────────────────────────────────
memory_tool!(graph_topology, ref);
memory_tool!(graph_health, ref);
memory_tool!(graph_communities, ref, top_n: [Option<usize>], min_size: [Option<usize>]);
memory_tool!(graph_normalize_strengths, mut, apply: [Option<bool>]);
memory_tool!(graph_link_impact, ref, source: [str], target: [str]);
memory_tool!(graph_hubs, ref, count: [Option<usize>]);
memory_tool!(graph_trace, ref, key: [str]);

22
src/hippocampus/neuro/scoring.rs
View file

@ -26,7 +26,7 @@ pub fn consolidation_priority(
graph: &Graph, graph: &Graph,
spectral_outlier: Option<f64>, spectral_outlier: Option<f64>,
) -> f64 { ) -> f64 {
let node = match store.nodes.get(key) { let node = match store.get_node(key).ok().flatten() {
Some(n) => n, Some(n) => n,
None => return 0.0, None => return 0.0,
}; };
@ -97,8 +97,10 @@ pub fn replay_queue_with_graph(
HashMap::new() HashMap::new()
}; };
let mut items: Vec<ReplayItem> = store.nodes.iter() let all_keys = store.all_keys().unwrap_or_default();
.map(|(key, node)| { let mut items: Vec<ReplayItem> = all_keys.iter()
.filter_map(|key| {
let node = store.get_node(key).ok()??;
let pos = positions.get(key); let pos = positions.get(key);
let outlier_score = pos.map(|p| p.outlier_score).unwrap_or(0.0); let outlier_score = pos.map(|p| p.outlier_score).unwrap_or(0.0);
let classification = pos let classification = pos
@ -109,7 +111,7 @@ pub fn replay_queue_with_graph(
store, key, graph, store, key, graph,
pos.map(|p| p.outlier_score), pos.map(|p| p.outlier_score),
); );
ReplayItem { Some(ReplayItem {
key: key.clone(), key: key.clone(),
priority, priority,
interval_days: node.spaced_repetition_interval, interval_days: node.spaced_repetition_interval,
@ -117,7 +119,7 @@ pub fn replay_queue_with_graph(
cc: graph.clustering_coefficient(key), cc: graph.clustering_coefficient(key),
classification, classification,
outlier_score, outlier_score,
} })
}) })
.collect(); .collect();
@ -214,11 +216,13 @@ fn consolidation_plan_inner(store: &Store, _detect_interf: bool) -> Consolidatio
let gini = graph.degree_gini(); let gini = graph.degree_gini();
let _avg_cc = graph.avg_clustering_coefficient(); let _avg_cc = graph.avg_clustering_coefficient();
let episodic_count = store.nodes.iter() let all_keys = store.all_keys().unwrap_or_default();
.filter(|(_, n)| matches!(n.node_type, crate::store::NodeType::EpisodicSession)) let episodic_count = all_keys.iter()
.filter_map(|k| store.get_node(k).ok()?)
.filter(|n| matches!(n.node_type, crate::store::NodeType::EpisodicSession))
.count(); .count();
let _episodic_ratio = if store.nodes.is_empty() { 0.0 } let _episodic_ratio = if all_keys.is_empty() { 0.0 }
else { episodic_count as f32 / store.nodes.len() as f32 }; else { episodic_count as f32 / all_keys.len() as f32 };
let mut plan = ConsolidationPlan { let mut plan = ConsolidationPlan {
counts: std::collections::HashMap::new(), counts: std::collections::HashMap::new(),

59
src/hippocampus/query/engine.rs
View file

@ -148,8 +148,6 @@ pub enum Filter {
Age(Cmp), // vs now - timestamp (seconds) Age(Cmp), // vs now - timestamp (seconds)
ContentLen(Cmp), ContentLen(Cmp),
Provenance(String), Provenance(String),
NotVisited { agent: String, duration: i64 }, // seconds
Visited { agent: String },
Negated(Box<Filter>), Negated(Box<Filter>),
} }
@ -185,8 +183,6 @@ pub enum ScoreField {
Weight, Weight,
ContentLen, ContentLen,
Priority, Priority,
/// Time since last visit by named agent. 1.0 = never visited, decays toward 0.
Recency(String),
} }
/// Numeric comparison operator. /// Numeric comparison operator.
@ -231,10 +227,10 @@ fn score_field(
(d / max).min(1.0) (d / max).min(1.0)
} }
ScoreField::Weight => { ScoreField::Weight => {
store.nodes.get(key).map(|n| n.weight as f64).unwrap_or(0.0) store.get_node(key).ok().flatten().map(|n| n.weight as f64).unwrap_or(0.0)
} }
ScoreField::ContentLen => { ScoreField::ContentLen => {
let len = store.nodes.get(key).map(|n| n.content.len()).unwrap_or(0) as f64; let len = store.get_node(key).ok().flatten().map(|n| n.content.len()).unwrap_or(0) as f64;
let max = precomputed.max_content_len.max(1.0); let max = precomputed.max_content_len.max(1.0);
(len / max).min(1.0) (len / max).min(1.0)
} }
@ -243,17 +239,6 @@ fn score_field(
// Priority is already roughly 0-1 from the scoring function // Priority is already roughly 0-1 from the scoring function
p.min(1.0) p.min(1.0)
} }
ScoreField::Recency(agent) => {
let last = store.last_visited(key, agent);
if last == 0 {
1.0 // never visited = highest recency score
} else {
let age = (crate::store::now_epoch() - last) as f64;
// Sigmoid decay: 1.0 at 7+ days, ~0.5 at 1 day, ~0.1 at 1 hour
let hours = age / 3600.0;
1.0 - (-0.03 * hours).exp()
}
}
} }
} }
@ -270,7 +255,7 @@ impl CompositeCache {
.map(|(k, _)| graph.degree(k) as f64) .map(|(k, _)| graph.degree(k) as f64)
.fold(0.0f64, f64::max); .fold(0.0f64, f64::max);
let max_content_len = items.iter() let max_content_len = items.iter()
.map(|(k, _)| store.nodes.get(k).map(|n| n.content.len()).unwrap_or(0) as f64) .map(|(k, _)| store.get_node(k).ok().flatten().map(|n| n.content.len()).unwrap_or(0) as f64)
.fold(0.0f64, f64::max); .fold(0.0f64, f64::max);
Self { Self {
isolation: graph.community_isolation(), isolation: graph.community_isolation(),
@ -306,8 +291,6 @@ impl fmt::Display for Filter {
Filter::Age(c) => write!(f, "age:{}", c), Filter::Age(c) => write!(f, "age:{}", c),
Filter::ContentLen(c) => write!(f, "content-len:{}", c), Filter::ContentLen(c) => write!(f, "content-len:{}", c),
Filter::Provenance(p) => write!(f, "provenance:{}", p), Filter::Provenance(p) => write!(f, "provenance:{}", p),
Filter::NotVisited { agent, duration } => write!(f, "not-visited:{},{}s", agent, duration),
Filter::Visited { agent } => write!(f, "visited:{}", agent),
Filter::Negated(inner) => write!(f, "!{}", inner), Filter::Negated(inner) => write!(f, "!{}", inner),
} }
} }
@ -410,9 +393,12 @@ pub fn run_query(
fn run_generator(g: &Generator, store: &Store) -> Vec<(String, f64)> { fn run_generator(g: &Generator, store: &Store) -> Vec<(String, f64)> {
match g { match g {
Generator::All => { Generator::All => {
store.nodes.iter() store.all_keys().unwrap_or_default().into_iter()
.filter(|(_, n)| !n.deleted) .filter_map(|key| {
.map(|(key, n)| (key.clone(), n.weight as f64)) let n = store.get_node(&key).ok()??;
if n.deleted { return None; }
Some((key, n.weight as f64))
})
.collect() .collect()
} }
Generator::Match(terms) => { Generator::Match(terms) => {
@ -426,7 +412,7 @@ fn run_generator(g: &Generator, store: &Store) -> Vec<(String, f64)> {
} }
pub fn eval_filter(filt: &Filter, key: &str, store: &Store, now: i64) -> bool { pub fn eval_filter(filt: &Filter, key: &str, store: &Store, now: i64) -> bool {
let node = match store.nodes.get(key) { let node = match store.get_node(key).ok().flatten() {
Some(n) => n, Some(n) => n,
None => return false, None => return false,
}; };
@ -441,13 +427,6 @@ pub fn eval_filter(filt: &Filter, key: &str, store: &Store, now: i64) -> bool {
} }
Filter::ContentLen(cmp) => cmp.matches(node.content.len() as f64), Filter::ContentLen(cmp) => cmp.matches(node.content.len() as f64),
Filter::Provenance(p) => node.provenance == *p, Filter::Provenance(p) => node.provenance == *p,
Filter::NotVisited { agent, duration } => {
let last = store.last_visited(key, agent);
last == 0 || (now - last) > *duration
}
Filter::Visited { agent } => {
store.last_visited(key, agent) > 0
}
Filter::Negated(inner) => !eval_filter(inner, key, store, now), Filter::Negated(inner) => !eval_filter(inner, key, store, now),
} }
} }
@ -466,15 +445,15 @@ pub fn run_transform(
} }
SortField::Timestamp => { SortField::Timestamp => {
items.sort_by(|a, b| { items.sort_by(|a, b| {
let ta = store.nodes.get(&a.0).map(|n| n.timestamp).unwrap_or(0); let ta = store.get_node(&a.0).ok().flatten().map(|n| n.timestamp).unwrap_or(0);
let tb = store.nodes.get(&b.0).map(|n| n.timestamp).unwrap_or(0); let tb = store.get_node(&b.0).ok().flatten().map(|n| n.timestamp).unwrap_or(0);
tb.cmp(&ta) // desc tb.cmp(&ta) // desc
}); });
} }
SortField::ContentLen => { SortField::ContentLen => {
items.sort_by(|a, b| { items.sort_by(|a, b| {
let la = store.nodes.get(&a.0).map(|n| n.content.len()).unwrap_or(0); let la = store.get_node(&a.0).ok().flatten().map(|n| n.content.len()).unwrap_or(0);
let lb = store.nodes.get(&b.0).map(|n| n.content.len()).unwrap_or(0); let lb = store.get_node(&b.0).ok().flatten().map(|n| n.content.len()).unwrap_or(0);
lb.cmp(&la) // desc lb.cmp(&la) // desc
}); });
} }
@ -504,7 +483,7 @@ pub fn run_transform(
SortField::Named(field, asc) => { SortField::Named(field, asc) => {
// Resolve field from node properties // Resolve field from node properties
let resolve = |key: &str| -> Option<f64> { let resolve = |key: &str| -> Option<f64> {
let node = store.nodes.get(key)?; let node = store.get_node(key).ok()??;
match field.as_str() { match field.as_str() {
"weight" => Some(node.weight as f64), "weight" => Some(node.weight as f64),
"emotion" => Some(node.emotion as f64), "emotion" => Some(node.emotion as f64),
@ -760,10 +739,10 @@ fn run_spread(
stage: &AlgoStage, stage: &AlgoStage,
_debug: bool, _debug: bool,
) -> Vec<(String, f64)> { ) -> Vec<(String, f64)> {
let store_params = store.params(); let cfg = crate::config::get();
let max_hops = stage.param_u32("max_hops", store_params.max_hops); let max_hops = stage.param_u32("max_hops", cfg.max_hops);
let edge_decay = stage.param_f64("edge_decay", store_params.edge_decay); let edge_decay = stage.param_f64("edge_decay", cfg.edge_decay);
let min_activation = stage.param_f64("min_activation", store_params.min_activation * 0.1); let min_activation = stage.param_f64("min_activation", cfg.min_activation * 0.1);
spreading_activation(seeds, graph, store, max_hops, edge_decay, min_activation) spreading_activation(seeds, graph, store, max_hops, edge_decay, min_activation)
} }

31
src/hippocampus/query/parser.rs
View file

@ -100,8 +100,6 @@ peg::parser! {
/ "key:" g:glob_pattern() { Stage::Filter(Filter::KeyGlob(g)) } / "key:" g:glob_pattern() { Stage::Filter(Filter::KeyGlob(g)) }
/ "!key:" g:glob_pattern() { Stage::Filter(Filter::Negated(Box::new(Filter::KeyGlob(g)))) } / "!key:" g:glob_pattern() { Stage::Filter(Filter::Negated(Box::new(Filter::KeyGlob(g)))) }
/ "provenance:" p:ident() { Stage::Filter(Filter::Provenance(p)) } / "provenance:" p:ident() { Stage::Filter(Filter::Provenance(p)) }
/ "not-visited:" a:ident() "," d:integer() { Stage::Filter(Filter::NotVisited { agent: a, duration: d as i64 }) }
/ "visited:" a:ident() { Stage::Filter(Filter::Visited { agent: a }) }
/ "all" { Stage::Generator(Generator::All) } / "all" { Stage::Generator(Generator::All) }
// Graph algorithms // Graph algorithms
/ "spread" { Stage::Algorithm(AlgoStage { algo: Algorithm::Spread, params: std::collections::HashMap::new() }) } / "spread" { Stage::Algorithm(AlgoStage { algo: Algorithm::Spread, params: std::collections::HashMap::new() }) }
@ -123,8 +121,7 @@ peg::parser! {
/ f:field() { make_sort_field(&f, false) } / f:field() { make_sort_field(&f, false) }
rule score_term() -> (ScoreField, f64) rule score_term() -> (ScoreField, f64)
= "recency(" a:ident() ")" "*" w:number() { (ScoreField::Recency(a), w) } = f:score_field_name() "*" w:number() { (f, w) }
/ f:score_field_name() "*" w:number() { (f, w) }
rule score_field_name() -> ScoreField rule score_field_name() -> ScoreField
= "isolation" { ScoreField::Isolation } = "isolation" { ScoreField::Isolation }
@ -303,7 +300,7 @@ pub fn parse_stages(s: &str) -> Result<Vec<Stage>, String> {
/// Resolve a field value from a node + graph context, returning a comparable Value. /// Resolve a field value from a node + graph context, returning a comparable Value.
fn resolve_field(field: &str, key: &str, store: &Store, graph: &Graph) -> Option<Value> { fn resolve_field(field: &str, key: &str, store: &Store, graph: &Graph) -> Option<Value> {
let node = store.nodes.get(key)?; let node = store.get_node(key).ok()??;
match field { match field {
"key" => Some(Value::Str(key.to_string())), "key" => Some(Value::Str(key.to_string())),
"weight" => Some(Value::Num(node.weight as f64)), "weight" => Some(Value::Num(node.weight as f64)),
@ -494,9 +491,13 @@ fn execute_parsed(
} }
_ => { _ => {
let mut out = Vec::new(); let mut out = Vec::new();
for key in store.nodes.keys() { for key in store.all_keys().unwrap_or_default() {
if store.nodes[key].deleted { continue; } let node = match store.get_node(&key).ok().flatten() {
if eval(&q.expr, &|f| resolve_field(f, key, store, graph), store, graph) { Some(n) => n,
None => continue,
};
if node.deleted { continue; }
if eval(&q.expr, &|f| resolve_field(f, &key, store, graph), store, graph) {
out.push(QueryResult { key: key.clone(), fields: BTreeMap::new() }); out.push(QueryResult { key: key.clone(), fields: BTreeMap::new() });
} }
} }
@ -568,15 +569,15 @@ fn execute_parsed(
} }
SortField::Weight => { SortField::Weight => {
results.sort_by(|a, b| { results.sort_by(|a, b| {
let wa = store.nodes.get(&a.key).map(|n| n.weight).unwrap_or(0.0); let wa = store.get_node(&a.key).ok().flatten().map(|n| n.weight).unwrap_or(0.0);
let wb = store.nodes.get(&b.key).map(|n| n.weight).unwrap_or(0.0); let wb = store.get_node(&b.key).ok().flatten().map(|n| n.weight).unwrap_or(0.0);
wb.total_cmp(&wa) wb.total_cmp(&wa)
}); });
} }
SortField::Timestamp => { SortField::Timestamp => {
results.sort_by(|a, b| { results.sort_by(|a, b| {
let ta = store.nodes.get(&a.key).map(|n| n.timestamp).unwrap_or(0); let ta = store.get_node(&a.key).ok().flatten().map(|n| n.timestamp).unwrap_or(0);
let tb = store.nodes.get(&b.key).map(|n| n.timestamp).unwrap_or(0); let tb = store.get_node(&b.key).ok().flatten().map(|n| n.timestamp).unwrap_or(0);
tb.cmp(&ta) tb.cmp(&ta)
}); });
} }
@ -882,7 +883,8 @@ mod tests {
assert!(parses("all | key:journal-*")); assert!(parses("all | key:journal-*"));
assert!(parses("all | !key:_*")); // negated key glob assert!(parses("all | !key:_*")); // negated key glob
assert!(parses("all | age:>7d")); assert!(parses("all | age:>7d"));
assert!(parses("all | not-visited:organize,86400")); // TODO: not-visited filter not yet implemented
// assert!(parses("all | not-visited:organize,86400"));
} }
#[test] #[test]
@ -898,7 +900,8 @@ mod tests {
fn test_composite_sort() { fn test_composite_sort() {
// Weighted composite sort expressions (require 2+ terms with +) // Weighted composite sort expressions (require 2+ terms with +)
assert!(parses("all | sort:degree*0.5+isolation*0.3")); assert!(parses("all | sort:degree*0.5+isolation*0.3"));
assert!(parses("all | sort:degree*0.5+isolation*0.3+recency(organize)*0.2")); // TODO: recency(agent) not yet implemented
// assert!(parses("all | sort:degree*0.5+isolation*0.3+recency(organize)*0.2"));
assert!(parses("all | sort:weight*0.5+degree*0.5")); assert!(parses("all | sort:weight*0.5+degree*0.5"));
// Single field (no weight) falls back to simple sort // Single field (no weight) falls back to simple sort
assert!(parses("all | sort:weight")); assert!(parses("all | sort:weight"));

681
src/hippocampus/store/capnp.rs Normal file
View file

@ -0,0 +1,681 @@
// Cap'n Proto serialization and persistence
//
// capnp logs are the source of truth; redb provides indexed access.
// This module contains:
// - Serialization macros (capnp_enum!, capnp_message!)
// - Load/replay from capnp logs
// - Append to capnp logs
// - fsck (corruption repair)
use super::{index, types::*};
use redb::ReadableTableMetadata;
use crate::memory_capnp;
use super::Store;
use anyhow::{anyhow, Context, Result};
use capnp::message;
use capnp::serialize;
use std::collections::HashMap;
use std::fs;
use std::io::{BufReader, Seek};
use std::path::Path;
// ---------------------------------------------------------------------------
// Capnp serialization macros
//
// Declarative mapping between Rust types and capnp generated types.
// Adding a field to the schema means adding it in one place below;
// both read and write are generated from the same declaration.
// ---------------------------------------------------------------------------
/// Generate to_capnp/from_capnp conversion methods for an enum.
macro_rules! capnp_enum {
($rust_type:ident, $capnp_type:path, [$($variant:ident),+ $(,)?]) => {
impl $rust_type {
#[allow(clippy::wrong_self_convention, dead_code)]
pub(crate) fn to_capnp(&self) -> $capnp_type {
match self {
$(Self::$variant => <$capnp_type>::$variant,)+
}
}
pub(crate) fn from_capnp(v: $capnp_type) -> Self {
match v {
$(<$capnp_type>::$variant => Self::$variant,)+
}
}
}
};
}
/// Generate from_capnp/to_capnp methods for a struct with capnp serialization.
/// Fields are grouped by serialization kind:
/// text - capnp Text fields (String in Rust)
/// uuid - capnp Data fields ([u8; 16] in Rust)
/// prim - copy types (u32, f32, f64, bool)
/// enm - enums with to_capnp/from_capnp methods
/// skip - Rust-only fields not in capnp (set to Default on read)
macro_rules! capnp_message {
(
$struct:ident,
reader: $reader:ty,
builder: $builder:ty,
text: [$($tf:ident),* $(,)?],
uuid: [$($uf:ident),* $(,)?],
prim: [$($pf:ident),* $(,)?],
enm: [$($ef:ident: $et:ident),* $(,)?],
skip: [$($sf:ident),* $(,)?] $(,)?
) => {
impl $struct {
pub fn from_capnp(r: $reader) -> Result<Self> {
paste::paste! {
Ok(Self {
$($tf: read_text(r.[<get_ $tf>]()),)*
$($uf: read_uuid(r.[<get_ $uf>]()),)*
$($pf: r.[<get_ $pf>](),)*
$($ef: $et::from_capnp(
r.[<get_ $ef>]().map_err(|_| anyhow!(concat!("bad ", stringify!($ef))))?
),)*
$($sf: Default::default(),)*
})
}
}
pub fn to_capnp(&self, mut b: $builder) {
paste::paste! {
$(b.[<set_ $tf>](&self.$tf);)*
$(b.[<set_ $uf>](&self.$uf);)*
$(b.[<set_ $pf>](self.$pf);)*
$(b.[<set_ $ef>](self.$ef.to_capnp());)*
}
}
}
};
}
// ---------------------------------------------------------------------------
// Capnp helpers
// ---------------------------------------------------------------------------
/// Read a capnp text field, returning empty string on any error
fn read_text(result: capnp::Result<capnp::text::Reader>) -> String {
result.ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string()
}
/// Read a capnp data field as [u8; 16], zero-padded
fn read_uuid(result: capnp::Result<&[u8]>) -> [u8; 16] {
let mut out = [0u8; 16];
if let Ok(data) = result
&& data.len() >= 16 {
out.copy_from_slice(&data[..16]);
}
out
}
// ---------------------------------------------------------------------------
// Type-to-capnp mappings
// ---------------------------------------------------------------------------
capnp_enum!(NodeType, memory_capnp::NodeType,
[EpisodicSession, EpisodicDaily, EpisodicWeekly, Semantic, EpisodicMonthly]);
capnp_enum!(RelationType, memory_capnp::RelationType,
[Link, Causal, Auto]);
capnp_message!(Node,
reader: memory_capnp::content_node::Reader<'_>,
builder: memory_capnp::content_node::Builder<'_>,
text: [key, content, source_ref, provenance],
uuid: [uuid],
prim: [version, timestamp, weight, emotion, deleted,
retrievals, uses, wrongs, last_replayed,
spaced_repetition_interval, created_at, last_scored],
enm: [node_type: NodeType],
skip: [community_id, clustering_coefficient, degree],
);
capnp_message!(Relation,
reader: memory_capnp::relation::Reader<'_>,
builder: memory_capnp::relation::Builder<'_>,
text: [source_key, target_key, provenance],
uuid: [uuid, source, target],
prim: [version, timestamp, strength, deleted],
enm: [rel_type: RelationType],
skip: [],
);
// ---------------------------------------------------------------------------
// Migration helpers (legacy provenance enum → string)
// ---------------------------------------------------------------------------
/// Convert legacy capnp provenance enum to string label.
fn legacy_provenance_label(p: memory_capnp::Provenance) -> &'static str {
use memory_capnp::Provenance::*;
match p {
Manual => "manual",
Journal => "journal",
Agent => "agent",
Dream => "dream",
Derived => "derived",
AgentExperienceMine => "agent:experience-mine",
AgentKnowledgeObservation => "agent:knowledge-observation",
AgentKnowledgePattern => "agent:knowledge-pattern",
AgentKnowledgeConnector => "agent:knowledge-connector",
AgentKnowledgeChallenger => "agent:knowledge-challenger",
AgentConsolidate => "agent:consolidate",
AgentDigest => "agent:digest",
AgentFactMine => "agent:fact-mine",
AgentDecay => "agent:decay",
}
}
impl Node {
/// Read from capnp with migration: if the new provenance text field
/// is empty (old record), fall back to the deprecated provenanceOld enum.
pub fn from_capnp_migrate(r: memory_capnp::content_node::Reader<'_>) -> Result<Self> {
let mut node = Self::from_capnp(r)?;
if node.provenance.is_empty()
&& let Ok(old) = r.get_provenance_old() {
node.provenance = legacy_provenance_label(old).to_string();
}
// Sanitize timestamps: old capnp records have raw offsets instead
// of unix epoch. Anything past year 2100 (~4102444800) is bogus.
const MAX_SANE_EPOCH: i64 = 4_102_444_800;
if node.timestamp > MAX_SANE_EPOCH || node.timestamp < 0 {
node.timestamp = node.created_at;
}
if node.created_at > MAX_SANE_EPOCH || node.created_at < 0 {
node.created_at = node.timestamp.min(MAX_SANE_EPOCH);
}
Ok(node)
}
}
impl Relation {
pub fn from_capnp_migrate(r: memory_capnp::relation::Reader<'_>) -> Result<Self> {
let mut rel = Self::from_capnp(r)?;
if rel.provenance.is_empty()
&& let Ok(old) = r.get_provenance_old() {
rel.provenance = legacy_provenance_label(old).to_string();
}
Ok(rel)
}
}
// ---------------------------------------------------------------------------
// Direct node access
// ---------------------------------------------------------------------------
/// Read a single node at the given offset in the capnp log.
/// The offset must point to a valid message containing the node.
pub fn read_node_at_offset(offset: u64) -> Result<Node> {
let path = nodes_path();
let mut file = fs::File::open(&path)
.with_context(|| format!("open {}", path.display()))?;
use std::io::{Seek, SeekFrom};
file.seek(SeekFrom::Start(offset))?;
let mut reader = BufReader::new(file);
let msg = serialize::read_message(&mut reader, message::ReaderOptions::new())
.with_context(|| format!("read message at offset {}", offset))?;
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.with_context(|| "read node log")?;
let nodes = log.get_nodes()
.with_context(|| "get nodes")?;
// A message at this offset should have exactly one node (from upsert),
// or we take the last one if there are multiple (from batch operations like rename)
if nodes.is_empty() {
anyhow::bail!("no nodes in message at offset {}", offset);
}
// Return the first non-deleted node, or the first one if all are deleted
for node_reader in nodes.iter() {
let node = Node::from_capnp_migrate(node_reader)?;
if !node.deleted {
return Ok(node);
}
}
// All nodes in this message are deleted - shouldn't happen if index is correct
Node::from_capnp_migrate(nodes.get(0))
}
// ---------------------------------------------------------------------------
// Store persistence methods
// ---------------------------------------------------------------------------
impl Store {
/// Load store by opening redb index and replaying relations.
pub fn load() -> Result<Store> {
let nodes_p = nodes_path();
let rels_p = relations_path();
let mut store = Store::default();
// Open redb index first (rebuilds from capnp if needed)
let db_p = db_path();
store.db = Some(store.open_or_rebuild_db(&db_p)?);
// Replay relations
if rels_p.exists() {
store.replay_relations(&rels_p)?;
}
// Record log sizes
use std::sync::atomic::Ordering;
store.loaded_nodes_size.store(
fs::metadata(&nodes_p).map(|m| m.len()).unwrap_or(0),
Ordering::Relaxed
);
store.loaded_rels_size.store(
fs::metadata(&rels_p).map(|m| m.len()).unwrap_or(0),
Ordering::Relaxed
);
// Orphan edges filtered naturally during for_each_relation (unresolvable UUIDs skipped)
Ok(store)
}
/// Open redb database, rebuilding if unhealthy.
fn open_or_rebuild_db(&self, path: &Path) -> Result<redb::Database> {
// Try opening existing database
if path.exists() {
match index::open_db(path) {
Ok(database) => {
if self.db_is_healthy(&database)? {
return Ok(database);
}
eprintln!("redb index stale, rebuilding...");
}
Err(e) => {
eprintln!("redb open failed ({}), rebuilding...", e);
}
}
}
// Rebuild index from capnp log
rebuild_index(path, &nodes_path())
}
/// Check if redb index is healthy by verifying some offsets are valid.
fn db_is_healthy(&self, database: &redb::Database) -> Result<bool> {
use redb::{ReadableDatabase, ReadableTable};
let txn = database.begin_read()?;
let nodes_table = txn.open_table(index::NODES)?;
// Check that we can read the table and it has entries
if nodes_table.len()? == 0 {
// Empty database - might be stale or new
let capnp_size = fs::metadata(nodes_path()).map(|m| m.len()).unwrap_or(0);
return Ok(capnp_size == 0); // healthy only if capnp is also empty
}
// Spot check: verify a few offsets point to valid messages
let mut checked = 0;
for entry in nodes_table.iter()? {
if checked >= 5 { break; }
let (key, offset) = entry?;
let offset = offset.value();
// Try to read the node at this offset
if read_node_at_offset(offset).is_err() {
return Ok(false);
}
checked += 1;
let _ = key; // silence unused warning
}
Ok(true)
}
/// Replay relation log, keeping latest version per UUID
fn replay_relations(&mut self, path: &Path) -> Result<()> {
let file = fs::File::open(path)
.with_context(|| format!("open {}", path.display()))?;
let mut reader = BufReader::new(file);
// Collect all, then deduplicate by UUID keeping latest version
let mut by_uuid: HashMap<[u8; 16], Relation> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::relation_log::Reader>()
.with_context(|| format!("read relation log"))?;
for rel_reader in log.get_relations()
.with_context(|| format!("get relations"))? {
let rel = Relation::from_capnp_migrate(rel_reader)?;
let existing_version = by_uuid.get(&rel.uuid)
.map(|r| r.version)
.unwrap_or(0);
if rel.version >= existing_version {
by_uuid.insert(rel.uuid, rel);
}
}
}
// Index relations directly (single transaction)
if let Some(db) = &self.db {
let txn = db.begin_write()?;
for rel in by_uuid.into_values() {
if rel.deleted { continue; }
index::index_relation(&txn, &rel.source, &rel.target, rel.strength, rel.rel_type as u8)?;
}
txn.commit()?;
}
Ok(())
}
/// Find all duplicate keys: keys with multiple live UUIDs in the log.
/// Returns a map from key → vec of all live Node versions (one per UUID).
pub fn find_duplicates(&self) -> Result<HashMap<String, Vec<Node>>> {
let path = nodes_path();
if !path.exists() { return Ok(HashMap::new()); }
let file = fs::File::open(&path)
.with_context(|| format!("open {}", path.display()))?;
let mut reader = BufReader::new(file);
// Track latest version of each UUID
let mut by_uuid: HashMap<[u8; 16], Node> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.with_context(|| format!("read node log"))?;
for node_reader in log.get_nodes()
.with_context(|| format!("get nodes"))? {
let node = Node::from_capnp_migrate(node_reader)?;
let dominated = by_uuid.get(&node.uuid)
.map(|n| node.version >= n.version)
.unwrap_or(true);
if dominated {
by_uuid.insert(node.uuid, node);
}
}
}
// Group live (non-deleted) nodes by key
let mut by_key: HashMap<String, Vec<Node>> = HashMap::new();
for node in by_uuid.into_values() {
if !node.deleted {
by_key.entry(node.key.clone()).or_default().push(node);
}
}
// Keep only duplicates
by_key.retain(|_, nodes| nodes.len() > 1);
Ok(by_key)
}
/// Append nodes to the log file. Returns the offset where the message was written.
pub fn append_nodes(&self, nodes: &[Node]) -> Result<u64> {
use std::sync::atomic::Ordering;
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::node_log::Builder>();
let mut list = log.init_nodes(nodes.len() as u32);
for (i, node) in nodes.iter().enumerate() {
node.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.with_context(|| format!("serialize nodes"))?;
// Lock for file append
let _guard = self.append_lock.lock().unwrap();
let path = nodes_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.with_context(|| format!("open {}", path.display()))?;
// Get offset before writing
let offset = file.metadata().map(|m| m.len()).unwrap_or(0);
use std::io::Write;
(&file).write_all(&buf)
.with_context(|| format!("write nodes"))?;
self.loaded_nodes_size.store(
file.metadata().map(|m| m.len()).unwrap_or(0),
Ordering::Relaxed
);
Ok(offset)
}
/// Append relations to the log file.
pub fn append_relations(&self, relations: &[Relation]) -> Result<()> {
use std::sync::atomic::Ordering;
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::relation_log::Builder>();
let mut list = log.init_relations(relations.len() as u32);
for (i, rel) in relations.iter().enumerate() {
rel.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.with_context(|| format!("serialize relations"))?;
// Lock for file append
let _guard = self.append_lock.lock().unwrap();
let path = relations_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.with_context(|| format!("open {}", path.display()))?;
use std::io::Write;
(&file).write_all(&buf)
.with_context(|| format!("write relations"))?;
self.loaded_rels_size.store(
file.metadata().map(|m| m.len()).unwrap_or(0),
Ordering::Relaxed
);
Ok(())
}
/// Placeholder - indices will be updated on write with redb.
pub fn save(&self) -> Result<()> {
Ok(())
}
}
/// Check and repair corrupt capnp log files.
///
/// Reads each message sequentially, tracking file position. On the first
/// corrupt message, truncates the file to the last good position. Also
/// removes stale caches so the next load replays from the repaired log.
pub fn fsck() -> Result<()> {
let mut any_corrupt = false;
for (path, kind) in [
(nodes_path(), "node"),
(relations_path(), "relation"),
] {
if !path.exists() { continue; }
let file = fs::File::open(&path)
.with_context(|| format!("open {}", path.display()))?;
let file_len = file.metadata()
.with_context(|| format!("stat {}", path.display()))?.len();
let mut reader = BufReader::new(file);
let mut good_messages = 0u64;
let mut last_good_pos = 0u64;
loop {
let pos = reader.stream_position()
.with_context(|| format!("tell {}", path.display()))?;
let msg = match serialize::read_message(&mut reader, message::ReaderOptions::new()) {
Ok(m) => m,
Err(_) => {
// read_message fails at EOF (normal) or on corrupt framing
if pos < file_len {
// Not at EOF — corrupt framing
eprintln!("{}: corrupt message at offset {}, truncating", kind, pos);
any_corrupt = true;
drop(reader);
let file = fs::OpenOptions::new().write(true).open(&path)
.with_context(|| format!("open for truncate"))?;
file.set_len(pos)
.with_context(|| format!("truncate {}", path.display()))?;
eprintln!("{}: truncated from {} to {} bytes ({} good messages)",
kind, file_len, pos, good_messages);
}
break;
}
};
// Validate the message content too
let valid = if kind == "node" {
msg.get_root::<memory_capnp::node_log::Reader>()
.and_then(|l| l.get_nodes().map(|_| ()))
.is_ok()
} else {
msg.get_root::<memory_capnp::relation_log::Reader>()
.and_then(|l| l.get_relations().map(|_| ()))
.is_ok()
};
if valid {
good_messages += 1;
last_good_pos = reader.stream_position()
.with_context(|| format!("tell {}", path.display()))?;
} else {
eprintln!("{}: corrupt message content at offset {}, truncating to {}",
kind, pos, last_good_pos);
any_corrupt = true;
drop(reader);
let file = fs::OpenOptions::new().write(true).open(&path)
.with_context(|| format!("open for truncate"))?;
file.set_len(last_good_pos)
.with_context(|| format!("truncate {}", path.display()))?;
eprintln!("{}: truncated from {} to {} bytes ({} good messages)",
kind, file_len, last_good_pos, good_messages);
break;
}
}
if !any_corrupt {
eprintln!("{}: {} messages, all clean", kind, good_messages);
}
}
if any_corrupt {
eprintln!("repair complete — run `poc-memory status` to verify");
} else {
eprintln!("store is clean");
}
Ok(())
}
/// Rebuild redb index from capnp log.
/// Scans the log, tracking offsets, and records latest version of each node.
fn rebuild_index(db_path: &Path, capnp_path: &Path) -> Result<redb::Database> {
// Remove old database if it exists
if db_path.exists() {
fs::remove_file(db_path)
.with_context(|| format!("remove old db {}", db_path.display()))?;
}
let database = index::open_db(db_path)?;
if !capnp_path.exists() {
return Ok(database);
}
// Track latest (offset, uuid, version, deleted, node_type, timestamp) per key
let mut latest: HashMap<String, (u64, [u8; 16], u32, bool, u8, i64)> = HashMap::new();
let file = fs::File::open(capnp_path)
.with_context(|| format!("open {}", capnp_path.display()))?;
let mut reader = BufReader::new(file);
loop {
let offset = reader.stream_position()?;
let msg = match serialize::read_message(&mut reader, message::ReaderOptions::new()) {
Ok(m) => m,
Err(_) => break,
};
let log = match msg.get_root::<memory_capnp::node_log::Reader>() {
Ok(l) => l,
Err(_) => continue,
};
let nodes = match log.get_nodes() {
Ok(n) => n,
Err(_) => continue,
};
for node_reader in nodes {
let key = node_reader.get_key().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
if key.is_empty() { continue; }
let version = node_reader.get_version();
let deleted = node_reader.get_deleted();
let node_type = node_reader.get_node_type()
.map(|t| t as u8)
.unwrap_or(0);
let timestamp = node_reader.get_timestamp();
let mut uuid = [0u8; 16];
if let Ok(data) = node_reader.get_uuid() {
if data.len() >= 16 {
uuid.copy_from_slice(&data[..16]);
}
}
// Keep if newer version
let dominated = latest.get(&key)
.map(|(_, _, v, _, _, _)| version >= *v)
.unwrap_or(true);
if dominated {
latest.insert(key, (offset, uuid, version, deleted, node_type, timestamp));
}
}
}
// Write index entries for non-deleted nodes
{
let txn = database.begin_write()?;
{
let mut nodes_table = txn.open_table(index::NODES)?;
let mut key_uuid_table = txn.open_table(index::KEY_TO_UUID)?;
let mut uuid_offsets = txn.open_multimap_table(index::UUID_OFFSETS)?;
for (key, (offset, uuid, _, deleted, node_type, timestamp)) in latest {
if !deleted {
nodes_table.insert(key.as_str(), offset)?;
// Pack: [uuid:16][node_type:1][timestamp:8] = 25 bytes
let mut packed = [0u8; 25];
packed[0..16].copy_from_slice(&uuid);
packed[16] = node_type;
packed[17..25].copy_from_slice(&timestamp.to_be_bytes());
key_uuid_table.insert(key.as_str(), packed.as_slice())?;
}
// Always record offset in UUID history (even for deleted)
uuid_offsets.insert(uuid.as_slice(), offset)?;
}
}
txn.commit()?;
}
Ok(database)
}

249
src/hippocampus/store/index.rs Normal file
View file

@ -0,0 +1,249 @@
// redb index tables
//
// capnp logs are source of truth; redb provides indexed access.
//
// Node tables:
// NODES: key → offset (current version)
// KEY_TO_UUID: key → uuid
// UUID_OFFSETS: uuid → offsets (multimap, all versions)
// NODES_BY_PROVENANCE: provenance → keys (multimap)
// NODES_BY_TYPE: [type_byte][timestamp_be] → key (for range queries by type+date)
//
// Relation tables:
// RELS: node_uuid → (other_uuid, strength, rel_type, is_outgoing) packed (multimap)
// Each relation stored twice — once per endpoint with direction bit.
//
// To get key from uuid: UUID_OFFSETS → read_node_at_offset() → node.key
use anyhow::{Context, Result};
use redb::{Database, MultimapTableDefinition, ReadableDatabase, ReadableTable, TableDefinition, WriteTransaction};
use std::path::Path;
// Node tables
pub const NODES: TableDefinition<&str, u64> = TableDefinition::new("nodes");
// KEY_TO_UUID: key → [uuid:16][node_type:1][timestamp:8] = 25 bytes
pub const KEY_TO_UUID: TableDefinition<&str, &[u8]> = TableDefinition::new("key_to_uuid");
pub const UUID_OFFSETS: MultimapTableDefinition<&[u8], u64> = MultimapTableDefinition::new("uuid_offsets");
pub const NODES_BY_PROVENANCE: MultimapTableDefinition<&str, &str> = MultimapTableDefinition::new("nodes_by_provenance");
// Composite key: [node_type: u8][timestamp: i64 BE] for range queries
pub const NODES_BY_TYPE: TableDefinition<&[u8], &str> = TableDefinition::new("nodes_by_type");
// Relations table - each relation stored twice (once per endpoint)
// Value: (other_uuid: [u8;16], strength: f32, rel_type: u8, is_outgoing: bool)
// Packed as 22 bytes: [other_uuid:16][strength:4][rel_type:1][is_outgoing:1]
pub const RELS: MultimapTableDefinition<&[u8], &[u8]> = MultimapTableDefinition::new("rels");
/// Open or create the redb database, ensuring all tables exist.
pub fn open_db(path: &Path) -> Result<Database> {
let db = Database::create(path)
.with_context(|| format!("create redb {}", path.display()))?;
// Ensure tables exist by opening a write transaction
let txn = db.begin_write()?;
{
// Node tables
let _ = txn.open_table(NODES)?;
let _ = txn.open_table(KEY_TO_UUID)?;
let _ = txn.open_multimap_table(UUID_OFFSETS)?;
let _ = txn.open_multimap_table(NODES_BY_PROVENANCE)?;
let _ = txn.open_table(NODES_BY_TYPE)?;
// Relations
let _ = txn.open_multimap_table(RELS)?;
}
txn.commit()?;
Ok(db)
}
/// Pack node metadata: [uuid:16][node_type:1][timestamp:8] = 25 bytes
fn pack_node_meta(uuid: &[u8; 16], node_type: u8, timestamp: i64) -> [u8; 25] {
let mut buf = [0u8; 25];
buf[0..16].copy_from_slice(uuid);
buf[16] = node_type;
buf[17..25].copy_from_slice(&timestamp.to_be_bytes());
buf
}
/// Unpack node metadata. Handles both old (16-byte) and new (25-byte) formats.
pub fn unpack_node_meta(data: &[u8]) -> ([u8; 16], u8, i64) {
let mut uuid = [0u8; 16];
uuid.copy_from_slice(&data[0..16]);
if data.len() >= 25 {
let node_type = data[16];
let timestamp = i64::from_be_bytes([
data[17], data[18], data[19], data[20],
data[21], data[22], data[23], data[24],
]);
(uuid, node_type, timestamp)
} else {
// Old format: just uuid, default metadata
(uuid, 0, 0)
}
}
/// Record a node's location in the index.
pub fn index_node(txn: &WriteTransaction, key: &str, offset: u64, uuid: &[u8; 16], node_type: u8, timestamp: i64) -> Result<()> {
let mut nodes_table = txn.open_table(NODES)?;
let mut key_uuid_table = txn.open_table(KEY_TO_UUID)?;
let mut uuid_offsets = txn.open_multimap_table(UUID_OFFSETS)?;
nodes_table.insert(key, offset)?;
let packed = pack_node_meta(uuid, node_type, timestamp);
key_uuid_table.insert(key, packed.as_slice())?;
uuid_offsets.insert(uuid.as_slice(), offset)?;
Ok(())
}
/// Get offset for a node by key.
pub fn get_offset(db: &Database, key: &str) -> Result<Option<u64>> {
let txn = db.begin_read()?;
let table = txn.open_table(NODES)?;
Ok(table.get(key)?.map(|v| v.value()))
}
/// Check if a key exists in the index.
pub fn contains_key(db: &Database, key: &str) -> Result<bool> {
let txn = db.begin_read()?;
let table = txn.open_table(NODES)?;
Ok(table.get(key)?.is_some())
}
/// Get a node's UUID from its key.
pub fn get_uuid_for_key(db: &Database, key: &str) -> Result<Option<[u8; 16]>> {
let txn = db.begin_read()?;
let table = txn.open_table(KEY_TO_UUID)?;
match table.get(key)? {
Some(data) => {
let (uuid, _, _) = unpack_node_meta(data.value());
Ok(Some(uuid))
}
None => Ok(None),
}
}
/// Get all offsets for a UUID (all versions). Returns newest first.
pub fn get_offsets_for_uuid(db: &Database, uuid: &[u8; 16]) -> Result<Vec<u64>> {
let txn = db.begin_read()?;
let table = txn.open_multimap_table(UUID_OFFSETS)?;
let mut offsets = Vec::new();
for entry in table.get(uuid.as_slice())? {
offsets.push(entry?.value());
}
// Sort descending so newest (highest offset) is first
offsets.sort_by(|a, b| b.cmp(a));
Ok(offsets)
}
/// Remove a node from the index (key mappings only; UUID history preserved).
pub fn remove_node(txn: &WriteTransaction, key: &str) -> Result<()> {
let mut nodes_table = txn.open_table(NODES)?;
let mut key_uuid_table = txn.open_table(KEY_TO_UUID)?;
// Note: UUID_OFFSETS is not cleared - preserves version history
nodes_table.remove(key)?;
key_uuid_table.remove(key)?;
Ok(())
}
/// Collect all keys from the index.
pub fn all_keys(db: &Database) -> Result<Vec<String>> {
let txn = db.begin_read()?;
let table = txn.open_table(NODES)?;
let mut keys = Vec::new();
for entry in table.iter()? {
let (key, _) = entry?;
keys.push(key.value().to_string());
}
Ok(keys)
}
/// Collect all (key, uuid, node_type, timestamp) in a single table scan.
pub fn all_key_uuid_pairs(db: &Database) -> Result<Vec<(String, [u8; 16], u8, i64)>> {
let txn = db.begin_read()?;
let table = txn.open_table(KEY_TO_UUID)?;
let mut pairs = Vec::new();
for entry in table.iter()? {
let (key, data) = entry?;
let (uuid, node_type, timestamp) = unpack_node_meta(data.value());
pairs.push((key.value().to_string(), uuid, node_type, timestamp));
}
Ok(pairs)
}
// ── Relation index operations ──────────────────────────────────────
//
// RELS value format: [other_uuid:16][strength:4][rel_type:1][is_outgoing:1] = 22 bytes
/// Pack relation data into bytes for RELS table.
fn pack_rel(other_uuid: &[u8; 16], strength: f32, rel_type: u8, is_outgoing: bool) -> [u8; 22] {
let mut buf = [0u8; 22];
buf[0..16].copy_from_slice(other_uuid);
buf[16..20].copy_from_slice(&strength.to_be_bytes());
buf[20] = rel_type;
buf[21] = if is_outgoing { 1 } else { 0 };
buf
}
/// Unpack relation data from RELS table.
pub fn unpack_rel(data: &[u8]) -> ([u8; 16], f32, u8, bool) {
let mut other_uuid = [0u8; 16];
other_uuid.copy_from_slice(&data[0..16]);
let strength = f32::from_be_bytes([data[16], data[17], data[18], data[19]]);
let rel_type = data[20];
let is_outgoing = data[21] != 0;
(other_uuid, strength, rel_type, is_outgoing)
}
/// Index a relation: store twice (once per endpoint).
pub fn index_relation(
txn: &WriteTransaction,
source_uuid: &[u8; 16],
target_uuid: &[u8; 16],
strength: f32,
rel_type: u8,
) -> Result<()> {
let mut rels = txn.open_multimap_table(RELS)?;
// Store outgoing: source → (target, strength, type, true)
let outgoing = pack_rel(target_uuid, strength, rel_type, true);
rels.insert(source_uuid.as_slice(), outgoing.as_slice())?;
// Store incoming: target → (source, strength, type, false)
let incoming = pack_rel(source_uuid, strength, rel_type, false);
rels.insert(target_uuid.as_slice(), incoming.as_slice())?;
Ok(())
}
/// Remove a relation from the index.
pub fn remove_relation(
txn: &WriteTransaction,
source_uuid: &[u8; 16],
target_uuid: &[u8; 16],
strength: f32,
rel_type: u8,
) -> Result<()> {
let mut rels = txn.open_multimap_table(RELS)?;
let outgoing = pack_rel(target_uuid, strength, rel_type, true);
rels.remove(source_uuid.as_slice(), outgoing.as_slice())?;
let incoming = pack_rel(source_uuid, strength, rel_type, false);
rels.remove(target_uuid.as_slice(), incoming.as_slice())?;
Ok(())
}
/// Get all edges for a node. Returns (other_uuid, strength, rel_type, is_outgoing).
pub fn edges_for_node(db: &Database, node_uuid: &[u8; 16]) -> Result<Vec<([u8; 16], f32, u8, bool)>> {
let txn = db.begin_read()?;
let rels = txn.open_multimap_table(RELS)?;
let mut edges = Vec::new();
for entry in rels.get(node_uuid.as_slice())? {
let guard = entry?;
let slice = guard.value();
let mut data = [0u8; 22];
data.copy_from_slice(slice);
edges.push(unpack_rel(&data));
}
Ok(edges)
}

435
src/hippocampus/store/mod.rs
View file

@ -1,50 +1,41 @@
// Append-only Cap'n Proto storage + derived KV cache // Append-only Cap'n Proto storage + redb indices
// //
// Two log files are source of truth: // capnp logs are the source of truth:
// nodes.capnp - ContentNode messages // nodes.capnp - ContentNode messages
// relations.capnp - Relation messages // relations.capnp - Relation messages
// //
// The Store struct is the derived cache: latest version per UUID, // redb provides indexed access; Store struct holds in-memory state.
// rebuilt from logs when stale. Three-tier load strategy:
// 1. rkyv mmap snapshot (snapshot.rkyv) — ~4ms deserialize
// 2. bincode cache (state.bin) — ~10ms
// 3. capnp log replay — ~40ms
// Staleness: log file sizes embedded in cache headers.
// //
// Module layout: // Module layout:
// types.rs — Node, Relation, enums, capnp macros, path helpers // types.rs — Node, Relation, enums, path/time helpers
// parse.rs — markdown → MemoryUnit parsing // capnp.rs — serialization macros, log IO (load, replay, append, fsck)
// view.rs — zero-copy read-only access (StoreView, MmapView) // index.rs — redb index operations
// persist.rs — load, save, replay, append, snapshot (all disk IO) // ops.rs — mutations (upsert, delete, rename, etc.)
// ops.rs — mutations (upsert, delete, decay, cap_degree, etc.) // view.rs — StoreView trait for read-only access
// mod.rs — re-exports, key resolution, ingestion, rendering
mod types; mod types;
mod parse; mod index;
mod view; mod capnp;
mod persist;
mod ops; mod ops;
mod view;
// Re-export everything callers need // Re-export everything callers need
pub use types::{ pub use types::{
memory_dir, nodes_path, memory_dir, nodes_path,
now_epoch, epoch_to_local, format_date, format_datetime, format_datetime_space, compact_timestamp, today, now_epoch, epoch_to_local, format_date, format_datetime, format_datetime_space, compact_timestamp, today,
Node, Relation, NodeType, Provenance, RelationType, Node, Relation, NodeType, RelationType,
RetrievalEvent, Params, GapRecord, Store,
new_node, new_relation, new_node, new_relation,
}; };
pub use parse::{MemoryUnit, parse_units}; pub use view::StoreView;
pub use view::{StoreView, AnyView}; pub use capnp::fsck;
pub use persist::fsck;
pub use ops::current_provenance; pub use ops::current_provenance;
use crate::graph::{self, Graph}; use crate::graph::{self, Graph};
use std::fs; use anyhow::{bail, Result};
use std::io::Write as IoWrite; use redb::Database;
use std::path::Path; use std::sync::atomic::AtomicU64;
use std::sync::Mutex;
use parse::classify_filename;
/// Strip .md suffix from a key, handling both bare keys and section keys. /// Strip .md suffix from a key, handling both bare keys and section keys.
/// "identity.md" → "identity", "foo.md#section" → "foo#section", "identity" → "identity" /// "identity.md" → "identity", "foo.md#section" → "foo#section", "identity" → "identity"
@ -57,291 +48,149 @@ pub fn strip_md_suffix(key: &str) -> String {
} }
} }
// The full in-memory store with internal locking
pub struct Store {
/// Log sizes at load time — used for staleness detection.
loaded_nodes_size: AtomicU64,
loaded_rels_size: AtomicU64,
/// Protects capnp log appends (redb handles its own locking)
append_lock: Mutex<()>,
/// redb index database
pub(crate) db: Option<redb::Database>,
}
impl Default for Store {
fn default() -> Self {
Store {
loaded_nodes_size: AtomicU64::new(0),
loaded_rels_size: AtomicU64::new(0),
append_lock: Mutex::new(()),
db: None,
}
}
}
impl Store { impl Store {
pub fn build_graph(&self) -> Graph { pub fn build_graph(&self) -> Graph {
graph::build_graph(self) graph::build_graph(self)
} }
pub fn resolve_key(&self, target: &str) -> Result<String, String> { /// Get a node by key, reading from capnp via the index.
pub fn get_node(&self, key: &str) -> Result<Option<Node>> {
let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
match index::get_offset(db, key)? {
Some(offset) => Ok(Some(capnp::read_node_at_offset(offset)?)),
None => Ok(None),
}
}
/// Check if a node exists by key.
pub fn contains_key(&self, key: &str) -> Result<bool> {
let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
index::contains_key(db, key)
}
/// Get all node keys.
pub fn all_keys(&self) -> Result<Vec<String>> {
let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
index::all_keys(db)
}
/// Get neighbors of a node: (key, strength) pairs.
pub fn neighbors(&self, key: &str) -> Result<Vec<(String, f32)>> {
let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
let uuid = match index::get_uuid_for_key(db, key)? {
Some(u) => u,
None => return Ok(Vec::new()),
};
let edges = index::edges_for_node(db, &uuid)?;
let mut neighbors = Vec::new();
for (other_uuid, strength, _, _) in edges {
// Look up key for other_uuid
let offsets = index::get_offsets_for_uuid(db, &other_uuid)?;
if offsets.is_empty() { continue; }
match capnp::read_node_at_offset(offsets[0]) {
Ok(n) if !n.deleted => neighbors.push((n.key, strength)),
_ => continue,
}
}
Ok(neighbors)
}
/// Get the database for transaction management.
pub fn db(&self) -> Result<&Database> {
self.db.as_ref().ok_or_else(|| anyhow::anyhow!("store not loaded"))
}
/// Remove a node from the index (used after appending a tombstone).
/// For batched operations, use index::remove_node with a WriteTransaction directly.
pub fn remove_from_index(&self, key: &str) -> Result<()> {
let db = self.db()?;
let txn = db.begin_write()?;
index::remove_node(&txn, key)?;
txn.commit()?;
Ok(())
}
/// Get all edges for a node by UUID. Returns (other_uuid, strength, rel_type, is_outgoing).
pub fn edges_for_uuid(&self, uuid: &[u8; 16]) -> Result<Vec<([u8; 16], f32, u8, bool)>> {
let db = self.db()?;
index::edges_for_node(db, uuid)
}
/// Add a relation to the index (opens its own transaction).
/// For batched operations, use index::index_relation with a WriteTransaction directly.
pub fn index_relation(&self, source: &[u8; 16], target: &[u8; 16], strength: f32, rel_type: u8) -> Result<()> {
let db = self.db()?;
let txn = db.begin_write()?;
index::index_relation(&txn, source, target, strength, rel_type)?;
txn.commit()?;
Ok(())
}
/// Remove a relation from the index (opens its own transaction).
/// For batched operations, use index::remove_relation with a WriteTransaction directly.
pub fn remove_relation_from_index(&self, source: &[u8; 16], target: &[u8; 16], strength: f32, rel_type: u8) -> Result<()> {
let db = self.db()?;
let txn = db.begin_write()?;
index::remove_relation(&txn, source, target, strength, rel_type)?;
txn.commit()?;
Ok(())
}
pub fn resolve_key(&self, target: &str) -> Result<String> {
// Strip .md suffix if present — keys no longer use it // Strip .md suffix if present — keys no longer use it
let bare = strip_md_suffix(target); let bare = strip_md_suffix(target);
if self.nodes.contains_key(&bare) { if self.contains_key(&bare)? {
return Ok(bare); return Ok(bare);
} }
let matches: Vec<_> = self.nodes.keys() let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
let all_keys = index::all_keys(db)?;
let matches: Vec<_> = all_keys.iter()
.filter(|k| k.to_lowercase().contains(&target.to_lowercase())) .filter(|k| k.to_lowercase().contains(&target.to_lowercase()))
.cloned().collect(); .cloned().collect();
match matches.len() { match matches.len() {
0 => Err(format!("No entry for '{}'. Run 'init'?", target)), 0 => bail!("No entry for '{}'. Run 'init'?", target),
1 => Ok(matches[0].clone()), 1 => Ok(matches[0].clone()),
n if n <= 10 => { n if n <= 10 => {
let list = matches.join("\n "); let list = matches.join("\n ");
Err(format!("Ambiguous '{}'. Matches:\n {}", target, list)) bail!("Ambiguous '{}'. Matches:\n {}", target, list)
} }
n => Err(format!("Too many matches for '{}' ({}). Be more specific.", target, n)), n => bail!("Too many matches for '{}' ({}). Be more specific.", target, n),
} }
} }
/// Resolve a link target to (key, uuid).
fn resolve_node_uuid(&self, target: &str) -> Option<(String, [u8; 16])> {
let bare = strip_md_suffix(target);
let n = self.nodes.get(&bare)?;
Some((bare, n.uuid))
}
/// Append retrieval event to retrieval.log without needing a Store instance.
pub fn log_retrieval_static(query: &str, results: &[String]) {
let path = memory_dir().join("retrieval.log");
let line = format!("[{}] q=\"{}\" hits={}\n", today(), query, results.len());
if let Ok(mut f) = fs::OpenOptions::new()
.create(true).append(true).open(&path) {
let _ = f.write_all(line.as_bytes());
}
}
/// Scan markdown files and index all memory units
pub fn init_from_markdown(&mut self) -> Result<usize, String> {
let dir = memory_dir();
let mut count = 0;
if dir.exists() {
// Build edge set for O(1) dedup during ingestion
let mut edge_set = self.build_edge_set();
count = self.scan_dir_for_init(&dir, &mut edge_set)?;
}
Ok(count)
}
/// Build a HashSet of existing (source, target) UUID pairs for O(1) dedup.
fn build_edge_set(&self) -> std::collections::HashSet<([u8; 16], [u8; 16])> {
let mut set = std::collections::HashSet::with_capacity(self.relations.len() * 2);
for r in &self.relations {
set.insert((r.source, r.target));
set.insert((r.target, r.source));
}
set
}
fn scan_dir_for_init(
&mut self,
dir: &Path,
edge_set: &mut std::collections::HashSet<([u8; 16], [u8; 16])>,
) -> Result<usize, String> {
let mut count = 0;
let entries = fs::read_dir(dir)
.map_err(|e| format!("read dir {}: {}", dir.display(), e))?;
for entry in entries.flatten() {
let path = entry.path();
if path.is_dir() {
count += self.scan_dir_for_init(&path, edge_set)?;
continue;
}
let Some(ext) = path.extension() else { continue };
if ext != "md" { continue }
let filename = path.file_name().unwrap().to_string_lossy().to_string();
let content = fs::read_to_string(&path)
.map_err(|e| format!("read {}: {}", path.display(), e))?;
let units = parse_units(&filename, &content);
let (new_count, _) = self.ingest_units(&units, &filename)?;
count += new_count;
// Create relations from links
let mut new_relations = Vec::new();
for unit in &units {
let source_uuid = match self.nodes.get(&unit.key) {
Some(n) => n.uuid,
None => continue,
};
for link in unit.marker_links.iter().chain(unit.md_links.iter()) {
let Some((key, uuid)) = self.resolve_node_uuid(link) else { continue };
if !edge_set.contains(&(source_uuid, uuid)) {
edge_set.insert((source_uuid, uuid));
edge_set.insert((uuid, source_uuid));
new_relations.push(new_relation(
source_uuid, uuid, RelationType::Link, 1.0,
&unit.key, &key,
));
}
}
for cause in &unit.causes {
let Some((key, uuid)) = self.resolve_node_uuid(cause) else { continue };
if !edge_set.contains(&(uuid, source_uuid)) {
edge_set.insert((uuid, source_uuid));
new_relations.push(new_relation(
uuid, source_uuid, RelationType::Causal, 1.0,
&key, &unit.key,
));
}
}
}
if !new_relations.is_empty() {
self.append_relations(&new_relations)?;
self.relations.extend(new_relations);
}
}
Ok(count)
}
/// Process parsed memory units: diff against existing nodes, persist changes.
/// Holds StoreLock across refresh + check + write to prevent duplicate UUIDs.
fn ingest_units(&mut self, units: &[MemoryUnit], filename: &str) -> Result<(usize, usize), String> {
let _lock = types::StoreLock::acquire()?;
self.refresh_nodes()?;
let node_type = classify_filename(filename);
let mut new_nodes = Vec::new();
let mut updated_nodes = Vec::new();
for (pos, unit) in units.iter().enumerate() {
if let Some(existing) = self.nodes.get(&unit.key) {
if existing.content != unit.content || existing.position != pos as u32 {
let mut node = existing.clone();
node.content = unit.content.clone();
node.position = pos as u32;
node.version += 1;
if let Some(ref s) = unit.state { node.state_tag = s.clone(); }
if let Some(ref s) = unit.source_ref { node.source_ref = s.clone(); }
updated_nodes.push(node);
}
} else {
let mut node = new_node(&unit.key, &unit.content);
node.node_type = node_type;
node.position = pos as u32;
if let Some(ref s) = unit.state { node.state_tag = s.clone(); }
if let Some(ref s) = unit.source_ref { node.source_ref = s.clone(); }
new_nodes.push(node);
}
}
if !new_nodes.is_empty() {
self.append_nodes_unlocked(&new_nodes)?;
for node in &new_nodes {
self.uuid_to_key.insert(node.uuid, node.key.clone());
self.nodes.insert(node.key.clone(), node.clone());
}
}
if !updated_nodes.is_empty() {
self.append_nodes_unlocked(&updated_nodes)?;
for node in &updated_nodes {
self.nodes.insert(node.key.clone(), node.clone());
}
}
Ok((new_nodes.len(), updated_nodes.len()))
}
/// Import a markdown file into the store, parsing it into nodes.
pub fn import_file(&mut self, path: &Path) -> Result<(usize, usize), String> {
let filename = path.file_name().unwrap().to_string_lossy().to_string();
let content = fs::read_to_string(path)
.map_err(|e| format!("read {}: {}", path.display(), e))?;
let units = parse_units(&filename, &content);
self.ingest_units(&units, &filename)
}
/// Gather all sections for a file key, sorted by position.
pub fn file_sections(&self, file_key: &str) -> Option<Vec<&Node>> {
let prefix = format!("{}#", file_key);
let mut sections: Vec<_> = self.nodes.values()
.filter(|n| n.key == file_key || n.key.starts_with(&prefix))
.collect();
if sections.is_empty() {
return None;
}
sections.sort_by_key(|n| n.position);
Some(sections)
}
/// Render a file key as plain content (no mem markers).
pub fn render_file(&self, file_key: &str) -> Option<String> {
let sections = self.file_sections(file_key)?;
let mut output = String::new();
for node in &sections {
output.push_str(&node.content);
if !node.content.ends_with('\n') {
output.push('\n');
}
output.push('\n');
}
Some(output.trim_end().to_string())
}
/// Render a file key back to markdown with reconstituted mem markers.
pub fn export_to_markdown(&self, file_key: &str) -> Option<String> {
let sections = self.file_sections(file_key)?;
let mut output = String::new();
for node in &sections {
if node.key.contains('#') {
let section_id = node.key.rsplit_once('#').map_or("", |(_, s)| s);
let links: Vec<_> = self.relations.iter()
.filter(|r| r.source_key == node.key && !r.deleted
&& r.rel_type != RelationType::Causal)
.map(|r| r.target_key.clone())
.collect();
let causes: Vec<_> = self.relations.iter()
.filter(|r| r.target_key == node.key && !r.deleted
&& r.rel_type == RelationType::Causal)
.map(|r| r.source_key.clone())
.collect();
let mut marker_parts = vec![format!("id={}", section_id)];
if !links.is_empty() {
marker_parts.push(format!("links={}", links.join(",")));
}
if !causes.is_empty() {
marker_parts.push(format!("causes={}", causes.join(",")));
}
output.push_str(&format!("<!-- mem: {} -->\n", marker_parts.join(" ")));
}
output.push_str(&node.content);
if !node.content.ends_with('\n') {
output.push('\n');
}
output.push('\n');
}
Some(output.trim_end().to_string())
}
/// Find the episodic node that best matches the given entry text.
pub fn find_journal_node(&self, entry_text: &str) -> Option<String> {
if entry_text.is_empty() {
return None;
}
let words: Vec<&str> = entry_text.split_whitespace()
.filter(|w| w.len() > 5)
.take(5)
.collect();
let mut best_key = None;
let mut best_score = 0;
for (key, node) in &self.nodes {
if node.node_type != NodeType::EpisodicSession {
continue;
}
let content_lower = node.content.to_lowercase();
let score: usize = words.iter()
.filter(|w| content_lower.contains(&w.to_lowercase()))
.count();
if score > best_score {
best_score = score;
best_key = Some(key.clone());
}
}
best_key
}
} }

486
src/hippocampus/store/ops.rs
View file

@ -1,10 +1,10 @@
// Mutation operations on the store // Mutation operations on the store
// //
// CRUD (upsert, delete, modify), feedback tracking (mark_used, mark_wrong), // CRUD (upsert, delete), maintenance (decay, cap_degree), and graph metrics.
// maintenance (decay, fix_categories, cap_degree), and graph metrics.
use super::types::*; use super::{capnp, index, types::*, Store};
use anyhow::{anyhow, bail, Result};
use std::collections::{HashMap, HashSet}; use std::collections::{HashMap, HashSet};
/// Fallback provenance for non-tool-dispatch paths (CLI, digest, etc.). /// Fallback provenance for non-tool-dispatch paths (CLI, digest, etc.).
@ -15,34 +15,50 @@ pub fn current_provenance() -> String {
} }
impl Store { impl Store {
/// Add or update a node (appends to log + updates cache). /// Add or update a node (appends to log + updates index).
/// Holds StoreLock across refresh + check + write to prevent duplicate UUIDs. pub fn upsert_node(&self, mut node: Node) -> Result<()> {
pub fn upsert_node(&mut self, mut node: Node) -> Result<(), String> { if let Some(existing) = self.get_node(&node.key)? {
let _lock = StoreLock::acquire()?;
self.refresh_nodes()?;
if let Some(existing) = self.nodes.get(&node.key) {
node.uuid = existing.uuid; node.uuid = existing.uuid;
node.version = existing.version + 1; node.version = existing.version + 1;
} }
self.append_nodes_unlocked(&[node.clone()])?; let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
self.uuid_to_key.insert(node.uuid, node.key.clone()); let txn = db.begin_write()?;
self.nodes.insert(node.key.clone(), node); let offset = self.append_nodes(&[node.clone()])?;
index::index_node(&txn, &node.key, offset, &node.uuid, node.node_type as u8, node.timestamp)?;
txn.commit()?;
Ok(()) Ok(())
} }
/// Add a relation (appends to log + updates cache) /// Add a relation (appends to log + indexes)
pub fn add_relation(&mut self, rel: Relation) -> Result<(), String> { pub fn add_relation(&self, rel: Relation) -> Result<()> {
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let txn = db.begin_write()?;
self.append_relations(std::slice::from_ref(&rel))?; self.append_relations(std::slice::from_ref(&rel))?;
self.relations.push(rel); index::index_relation(&txn, &rel.source, &rel.target, rel.strength, rel.rel_type as u8)?;
txn.commit()?;
Ok(()) Ok(())
} }
/// Recent nodes by provenance, sorted newest-first. Returns (key, timestamp). /// Recent nodes by provenance, sorted newest-first. Returns (key, timestamp).
pub fn recent_by_provenance(&self, provenance: &str, limit: usize) -> Vec<(String, i64)> { pub fn recent_by_provenance(&self, provenance: &str, limit: usize) -> Vec<(String, i64)> {
let mut nodes: Vec<_> = self.nodes.values() let db = match self.db.as_ref() {
.filter(|n| !n.deleted && n.provenance == provenance) Some(db) => db,
.map(|n| (n.key.clone(), n.timestamp)) None => return Vec::new(),
};
let keys = match index::all_keys(db) {
Ok(keys) => keys,
Err(_) => return Vec::new(),
};
let mut nodes: Vec<_> = keys.iter()
.filter_map(|key| {
let offset = index::get_offset(db, key).ok()??;
let node = capnp::read_node_at_offset(offset).ok()?;
if !node.deleted && node.provenance == provenance {
Some((key.clone(), node.timestamp))
} else {
None
}
})
.collect(); .collect();
nodes.sort_by(|a, b| b.1.cmp(&a.1)); nodes.sort_by(|a, b| b.1.cmp(&a.1));
nodes.truncate(limit); nodes.truncate(limit);
@ -54,56 +70,57 @@ impl Store {
/// ///
/// Provenance is determined by the POC_PROVENANCE env var if set, /// Provenance is determined by the POC_PROVENANCE env var if set,
/// otherwise defaults to Manual. /// otherwise defaults to Manual.
pub fn upsert(&mut self, key: &str, content: &str) -> Result<&'static str, String> { pub fn upsert(&self, key: &str, content: &str) -> Result<&'static str> {
let prov = current_provenance(); let prov = current_provenance();
self.upsert_provenance(key, content, &prov) self.upsert_provenance(key, content, &prov)
} }
/// Upsert with explicit provenance (for agent-created nodes). /// Upsert with explicit provenance (for agent-created nodes).
/// Holds StoreLock across refresh + check + write to prevent duplicate UUIDs. pub fn upsert_provenance(&self, key: &str, content: &str, provenance: &str) -> Result<&'static str> {
pub fn upsert_provenance(&mut self, key: &str, content: &str, provenance: &str) -> Result<&'static str, String> { let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let _lock = StoreLock::acquire()?;
self.refresh_nodes()?;
if let Some(existing) = self.nodes.get(key) { if let Some(existing) = self.get_node(key)? {
if existing.content == content { if existing.content == content {
return Ok("unchanged"); return Ok("unchanged");
} }
let mut node = existing.clone(); let mut node = existing;
node.content = content.to_string(); node.content = content.to_string();
node.provenance = provenance.to_string(); node.provenance = provenance.to_string();
node.timestamp = now_epoch(); node.timestamp = now_epoch();
node.version += 1; node.version += 1;
self.append_nodes_unlocked(std::slice::from_ref(&node))?; let txn = db.begin_write()?;
self.nodes.insert(key.to_string(), node); let offset = self.append_nodes(std::slice::from_ref(&node))?;
index::index_node(&txn, &node.key, offset, &node.uuid, node.node_type as u8, node.timestamp)?;
txn.commit()?;
Ok("updated") Ok("updated")
} else { } else {
let mut node = new_node(key, content); let mut node = new_node(key, content);
node.provenance = provenance.to_string(); node.provenance = provenance.to_string();
self.append_nodes_unlocked(std::slice::from_ref(&node))?; let txn = db.begin_write()?;
self.uuid_to_key.insert(node.uuid, node.key.clone()); let offset = self.append_nodes(std::slice::from_ref(&node))?;
self.nodes.insert(key.to_string(), node); index::index_node(&txn, &node.key, offset, &node.uuid, node.node_type as u8, node.timestamp)?;
txn.commit()?;
Ok("created") Ok("created")
} }
} }
/// Soft-delete a node (appends deleted version, removes from cache). /// Soft-delete a node (appends deleted version, removes from index).
/// Holds StoreLock across refresh + write to see concurrent creates. pub fn delete_node(&self, key: &str) -> Result<()> {
pub fn delete_node(&mut self, key: &str) -> Result<(), String> {
let _lock = StoreLock::acquire()?;
self.refresh_nodes()?;
let prov = current_provenance(); let prov = current_provenance();
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let node = self.nodes.get(key) let node = self.get_node(key)?
.ok_or_else(|| format!("No node '{}'", key))?; .ok_or_else(|| anyhow!("No node '{}'", key))?;
let mut deleted = node.clone(); let mut deleted = node;
deleted.deleted = true; deleted.deleted = true;
deleted.version += 1; deleted.version += 1;
deleted.provenance = prov; deleted.provenance = prov;
deleted.timestamp = now_epoch(); deleted.timestamp = now_epoch();
self.append_nodes_unlocked(std::slice::from_ref(&deleted))?;
self.nodes.remove(key); let txn = db.begin_write()?;
self.append_nodes(std::slice::from_ref(&deleted))?;
index::remove_node(&txn, key)?;
txn.commit()?;
Ok(()) Ok(())
} }
@ -112,23 +129,16 @@ impl Store {
/// Graph edges (source/target UUIDs) are unaffected — they're already /// Graph edges (source/target UUIDs) are unaffected — they're already
/// UUID-based. We update the human-readable source_key/target_key strings /// UUID-based. We update the human-readable source_key/target_key strings
/// on relations, and created_at is preserved untouched. /// on relations, and created_at is preserved untouched.
/// pub fn rename_node(&self, old_key: &str, new_key: &str) -> Result<()> {
/// Appends: (new_key, v+1) + (old_key, deleted, v+1) + updated relations.
/// Holds StoreLock across refresh + write to prevent races.
pub fn rename_node(&mut self, old_key: &str, new_key: &str) -> Result<(), String> {
if old_key == new_key { if old_key == new_key {
return Ok(()); return Ok(());
} }
if self.contains_key(new_key)? {
let _lock = StoreLock::acquire()?; bail!("Key '{}' already exists", new_key);
self.refresh_nodes()?;
if self.nodes.contains_key(new_key) {
return Err(format!("Key '{}' already exists", new_key));
} }
let node = self.nodes.get(old_key) let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
.ok_or_else(|| format!("No node '{}'", old_key))? let node = self.get_node(old_key)?
.clone(); .ok_or_else(|| anyhow!("No node '{}'", old_key))?;
let prov = current_provenance(); let prov = current_provenance();
@ -146,238 +156,238 @@ impl Store {
tombstone.provenance = prov; tombstone.provenance = prov;
tombstone.timestamp = now_epoch(); tombstone.timestamp = now_epoch();
// Collect affected relations and update their debug key strings // Find relations touching this node's UUID (read before txn)
let updated_rels: Vec<_> = self.relations.iter() let node_uuid = node.uuid;
.filter(|r| r.source_key == old_key || r.target_key == old_key) let edges = index::edges_for_node(db, &node_uuid)?;
.map(|r| {
let mut r = r.clone();
r.version += 1;
if r.source_key == old_key { r.source_key = new_key.to_string(); }
if r.target_key == old_key { r.target_key = new_key.to_string(); }
r
})
.collect();
// Persist under single lock // Build uuid → key map for the other endpoints
self.append_nodes_unlocked(&[renamed.clone(), tombstone])?; let keys = index::all_keys(db)?;
let mut uuid_to_key: HashMap<[u8; 16], String> = HashMap::new();
for k in &keys {
if let Ok(Some(u)) = index::get_uuid_for_key(db, k) {
uuid_to_key.insert(u, k.clone());
}
}
uuid_to_key.insert(node_uuid, new_key.to_string());
let mut updated_rels = Vec::new();
for (other_uuid, strength, rel_type, is_outgoing) in edges {
let other_key = uuid_to_key.get(&other_uuid).cloned().unwrap_or_default();
let (src_uuid, tgt_uuid, src_key, tgt_key) = if is_outgoing {
(node_uuid, other_uuid, new_key.to_string(), other_key)
} else {
(other_uuid, node_uuid, other_key, new_key.to_string())
};
let mut rel = new_relation(src_uuid, tgt_uuid,
RelationType::from_u8(rel_type), strength,
&src_key, &tgt_key);
rel.version = 2; // indicate update
updated_rels.push(rel);
}
// Single transaction for all index updates
let txn = db.begin_write()?;
let offset = self.append_nodes(&[renamed.clone(), tombstone])?;
index::remove_node(&txn, old_key)?;
index::index_node(&txn, new_key, offset, &renamed.uuid, renamed.node_type as u8, renamed.timestamp)?;
if !updated_rels.is_empty() { if !updated_rels.is_empty() {
self.append_relations_unlocked(&updated_rels)?; self.append_relations(&updated_rels)?;
}
// Update in-memory cache
self.nodes.remove(old_key);
self.uuid_to_key.insert(renamed.uuid, new_key.to_string());
self.nodes.insert(new_key.to_string(), renamed);
for updated in &updated_rels {
if let Some(r) = self.relations.iter_mut().find(|r| r.uuid == updated.uuid) {
r.source_key = updated.source_key.clone();
r.target_key = updated.target_key.clone();
r.version = updated.version;
}
} }
txn.commit()?;
Ok(()) Ok(())
} }
/// Modify a node in-place, bump version, and persist to capnp log.
fn modify_node(&mut self, key: &str, f: impl FnOnce(&mut Node)) -> Result<(), String> {
let node = self.nodes.get_mut(key)
.ok_or_else(|| format!("No node '{}'", key))?;
f(node);
node.version += 1;
let node = node.clone();
self.append_nodes(&[node])
}
pub fn mark_used(&mut self, key: &str) {
let boost = self.params.use_boost as f32;
let _ = self.modify_node(key, |n| {
n.uses += 1;
n.weight = (n.weight + boost).min(1.0);
if n.spaced_repetition_interval < 30 {
n.spaced_repetition_interval = match n.spaced_repetition_interval {
1 => 3, 3 => 7, 7 => 14, 14 => 30, _ => 30,
};
}
n.last_replayed = now_epoch();
});
}
pub fn mark_wrong(&mut self, key: &str, _ctx: Option<&str>) {
let _ = self.modify_node(key, |n| {
n.wrongs += 1;
n.weight = (n.weight - 0.1).max(0.0);
n.spaced_repetition_interval = 1;
});
}
pub fn record_gap(&mut self, desc: &str) {
self.gaps.push(GapRecord {
description: desc.to_string(),
timestamp: today(),
});
}
/// Cap node degree by soft-deleting edges from mega-hubs. /// Cap node degree by soft-deleting edges from mega-hubs.
pub fn cap_degree(&mut self, max_degree: usize) -> Result<(usize, usize), String> { pub fn cap_degree(&self, max_degree: usize) -> Result<(usize, usize)> {
let mut node_degree: HashMap<String, usize> = HashMap::new(); let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
for rel in &self.relations { let keys = index::all_keys(db)?;
if rel.deleted { continue; }
*node_degree.entry(rel.source_key.clone()).or_default() += 1; // Build uuid ↔ key maps and count degrees in one pass
*node_degree.entry(rel.target_key.clone()).or_default() += 1; let mut uuid_to_key: HashMap<[u8; 16], String> = HashMap::new();
let mut node_info: Vec<(String, [u8; 16], usize)> = Vec::new(); // (key, uuid, degree)
for key in &keys {
if let Ok(Some(uuid)) = index::get_uuid_for_key(db, key) {
let degree = index::edges_for_node(db, &uuid)?.len();
uuid_to_key.insert(uuid, key.clone());
node_info.push((key.clone(), uuid, degree));
}
} }
let mut node_edges: HashMap<String, Vec<usize>> = HashMap::new(); // Build degree lookup
for (i, rel) in self.relations.iter().enumerate() { let node_degree: HashMap<&str, usize> = node_info.iter()
if rel.deleted { continue; } .map(|(k, _, d)| (k.as_str(), *d))
node_edges.entry(rel.source_key.clone()).or_default().push(i); .collect();
node_edges.entry(rel.target_key.clone()).or_default().push(i);
}
let mut to_delete: HashSet<usize> = HashSet::new(); let mut to_delete: HashSet<([u8; 16], [u8; 16])> = HashSet::new();
let mut hubs_capped = 0; let mut hubs_capped = 0;
for (_key, edge_indices) in &node_edges { for (_key, uuid, degree) in &node_info {
let active: Vec<usize> = edge_indices.iter() if *degree <= max_degree { continue; }
.filter(|&&i| !to_delete.contains(&i)) let uuid = *uuid;
.copied() let edges = index::edges_for_node(db, &uuid)?;
.collect(); if edges.len() <= max_degree { continue; }
if active.len() <= max_degree { continue; }
let mut auto_indices: Vec<(usize, f32)> = Vec::new(); // Separate auto vs manual edges: (source, target, sort_key)
let mut link_indices: Vec<(usize, usize)> = Vec::new(); let mut auto_edges: Vec<([u8; 16], [u8; 16], f32)> = Vec::new();
for &i in &active { let mut link_edges: Vec<([u8; 16], [u8; 16], usize)> = Vec::new();
let rel = &self.relations[i];
if rel.rel_type == RelationType::Auto { for (other_uuid, strength, rel_type, is_outgoing) in &edges {
auto_indices.push((i, rel.strength)); // Canonical edge direction
} else { let (src, tgt) = if *is_outgoing { (uuid, *other_uuid) } else { (*other_uuid, uuid) };
let other = if &rel.source_key == _key { if to_delete.contains(&(src, tgt)) || to_delete.contains(&(tgt, src)) { continue; }
&rel.target_key
} else { let other_key = match uuid_to_key.get(other_uuid) {
&rel.source_key Some(k) => k,
None => continue,
}; };
let other_deg = node_degree.get(other).copied().unwrap_or(0);
link_indices.push((i, other_deg)); if *rel_type == RelationType::Auto as u8 {
auto_edges.push((src, tgt, *strength));
} else {
let other_deg = node_degree.get(other_key.as_str()).copied().unwrap_or(0);
link_edges.push((src, tgt, other_deg));
} }
} }
let excess = active.len() - max_degree; let active_count = auto_edges.len() + link_edges.len();
if active_count <= max_degree { continue; }
auto_indices.sort_by(|a, b| a.1.total_cmp(&b.1)); let excess = active_count - max_degree;
let auto_prune = excess.min(auto_indices.len());
for &(i, _) in auto_indices.iter().take(auto_prune) { // Prune weakest auto edges first
to_delete.insert(i); auto_edges.sort_by(|a, b| a.2.total_cmp(&b.2));
for (src, tgt, _) in auto_edges.iter().take(excess) {
to_delete.insert((*src, *tgt));
} }
let remaining_excess = excess.saturating_sub(auto_prune); // Then prune links to highest-degree nodes
if remaining_excess > 0 { let remaining = excess.saturating_sub(auto_edges.len());
link_indices.sort_by(|a, b| b.1.cmp(&a.1)); if remaining > 0 {
let link_prune = remaining_excess.min(link_indices.len()); link_edges.sort_by(|a, b| b.2.cmp(&a.2));
for &(i, _) in link_indices.iter().take(link_prune) { for (src, tgt, _) in link_edges.iter().take(remaining) {
to_delete.insert(i); to_delete.insert((*src, *tgt));
} }
} }
hubs_capped += 1; hubs_capped += 1;
} }
let mut pruned_rels = Vec::new(); // Collect edge info for deletion
for &i in &to_delete { let mut to_remove: Vec<([u8; 16], [u8; 16], f32, u8, String, String)> = Vec::new();
self.relations[i].deleted = true; for (source_uuid, target_uuid) in &to_delete {
self.relations[i].version += 1; let edges = index::edges_for_node(db, source_uuid)?;
pruned_rels.push(self.relations[i].clone()); if let Some((_, strength, rel_type, _)) = edges.iter()
.find(|(other, _, _, out)| *other == *target_uuid && *out)
{
let source_key = uuid_to_key.get(source_uuid).cloned().unwrap_or_default();
let target_key = uuid_to_key.get(target_uuid).cloned().unwrap_or_default();
to_remove.push((*source_uuid, *target_uuid, *strength, *rel_type, source_key, target_key));
}
} }
if !pruned_rels.is_empty() { // Now mutate: remove from index and persist tombstones (single txn)
self.append_relations(&pruned_rels)?; let pruned_count = to_remove.len();
if !to_remove.is_empty() {
let txn = db.begin_write()?;
for (source_uuid, target_uuid, strength, rel_type, source_key, target_key) in to_remove {
index::remove_relation(&txn, &source_uuid, &target_uuid, strength, rel_type)?;
let mut rel = new_relation(source_uuid, target_uuid,
RelationType::from_u8(rel_type), strength,
&source_key, &target_key);
rel.deleted = true;
rel.version = 2;
self.append_relations(std::slice::from_ref(&rel))?;
}
txn.commit()?;
} }
self.relations.retain(|r| !r.deleted); Ok((hubs_capped, pruned_count))
Ok((hubs_capped, to_delete.len()))
} }
/// Set a node's weight directly. Returns (old, new). /// Set a node's weight directly. Returns (old, new).
pub fn set_weight(&mut self, key: &str, weight: f32) -> Result<(f32, f32), String> { pub fn set_weight(&self, key: &str, weight: f32) -> Result<(f32, f32)> {
let weight = weight.clamp(0.01, 1.0); let weight = weight.clamp(0.01, 1.0);
let node = self.nodes.get_mut(key) let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
.ok_or_else(|| format!("node not found: {}", key))?; let mut node = self.get_node(key)?
.ok_or_else(|| anyhow!("node not found: {}", key))?;
let old = node.weight; let old = node.weight;
if (old - weight).abs() < 0.001 {
return Ok((old, weight)); // unchanged
}
node.weight = weight; node.weight = weight;
node.version += 1;
node.timestamp = now_epoch();
let txn = db.begin_write()?;
let offset = self.append_nodes(std::slice::from_ref(&node))?;
index::index_node(&txn, key, offset, &node.uuid, node.node_type as u8, node.timestamp)?;
txn.commit()?;
Ok((old, weight)) Ok((old, weight))
} }
/// Update a node's weight with a new score and record the scoring /// Set the strength of a link between two nodes.
/// timestamp. Uses asymmetric smoothing: responds quickly to high /// Returns the old strength. Creates link if it doesn't exist.
/// scores (alpha=0.5) but decays slowly on low scores (alpha=0.1). pub fn set_link_strength(&self, source: &str, target: &str, strength: f32) -> Result<f32> {
/// This keeps memories surfaced even if they're only useful 1 in 4 times.
/// Returns (old_weight, new_weight).
pub fn score_weight(&mut self, key: &str, score: f64) -> Result<(f32, f32), String> {
let node = self.nodes.get_mut(key)
.ok_or_else(|| format!("node not found: {}", key))?;
let old = node.weight;
let alpha = if score > old as f64 { 0.5 } else { 0.1 };
let new = (alpha * score + (1.0 - alpha) * old as f64) as f32;
node.weight = new.clamp(0.01, 1.0);
node.last_scored = chrono::Utc::now().timestamp();
Ok((old, node.weight))
}
/// Set the strength of a link between two nodes. Deduplicates if
/// multiple links exist. Returns the old strength, or error if no link.
pub fn set_link_strength(&mut self, source: &str, target: &str, strength: f32) -> Result<f32, String> {
let strength = strength.clamp(0.01, 1.0); let strength = strength.clamp(0.01, 1.0);
let mut old = 0.0f32;
let mut found = false; let source_uuid = self.get_node(source)?
let mut first = true; .map(|n| n.uuid)
for rel in &mut self.relations { .ok_or_else(|| anyhow!("source not found: {}", source))?;
if rel.deleted { continue; } let target_uuid = self.get_node(target)?
if (rel.source_key == source && rel.target_key == target) .map(|n| n.uuid)
|| (rel.source_key == target && rel.target_key == source) .ok_or_else(|| anyhow!("target not found: {}", target))?;
{
if first { // Find existing edge via index (scope the borrow)
old = rel.strength; let existing = {
rel.strength = strength; let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
first = false; let edges = index::edges_for_node(db, &source_uuid)?;
edges.iter().find(|(other, _, _, _)| *other == target_uuid)
.map(|(_, s, t, _)| (*s, *t))
};
if let Some((old_strength, rel_type)) = existing {
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let txn = db.begin_write()?;
// Remove old edge from index, add updated one
index::remove_relation(&txn, &source_uuid, &target_uuid, old_strength, rel_type)?;
index::index_relation(&txn, &source_uuid, &target_uuid, strength, rel_type)?;
// Append updated relation to log
let mut rel = new_relation(source_uuid, target_uuid,
RelationType::from_u8(rel_type), strength, source, target);
rel.version = 2; // indicate update
self.append_relations(std::slice::from_ref(&rel))?;
txn.commit()?;
Ok(old_strength)
} else { } else {
rel.deleted = true; // deduplicate // Create new link then update its strength
}
found = true;
}
}
if !found {
// Upsert: create the link if it doesn't exist
self.add_link(source, target, "link_set")?; self.add_link(source, target, "link_set")?;
// Set the strength on the newly created link let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
for rel in self.relations.iter_mut().rev() { let txn = db.begin_write()?;
if !rel.deleted && rel.source_key == source && rel.target_key == target { index::remove_relation(&txn, &source_uuid, &target_uuid, 0.1, RelationType::Link as u8)?;
rel.strength = strength; index::index_relation(&txn, &source_uuid, &target_uuid, strength, RelationType::Link as u8)?;
break; txn.commit()?;
Ok(0.0)
} }
} }
return Ok(0.0);
}
Ok(old)
}
/// Add a link between two nodes with Jaccard-based initial strength. /// Add a link between two nodes with Jaccard-based initial strength.
/// Returns the strength, or a message if the link already exists. /// Returns the strength, or a message if the link already exists.
pub fn add_link(&mut self, source: &str, target: &str, provenance: &str) -> Result<f32, String> { pub fn add_link(&self, source: &str, target: &str, provenance: &str) -> Result<f32> {
// Check for existing let source_uuid = self.get_node(source)?
let exists = self.relations.iter().any(|r| .map(|n| n.uuid)
!r.deleted && .ok_or_else(|| anyhow!("source not found: {}", source))?;
((r.source_key == source && r.target_key == target) || let target_uuid = self.get_node(target)?
(r.source_key == target && r.target_key == source))); .map(|n| n.uuid)
if exists { .ok_or_else(|| anyhow!("target not found: {}", target))?;
return Err(format!("link already exists: {} ↔ {}", source, target));
}
let source_uuid = self.nodes.get(source) // Check for existing via index
.map(|n| n.uuid) if let Some(db) = &self.db {
.ok_or_else(|| format!("source not found: {}", source))?; let edges = index::edges_for_node(db, &source_uuid)?;
let target_uuid = self.nodes.get(target) let exists = edges.iter().any(|(other, _, _, _)| *other == target_uuid);
.map(|n| n.uuid) if exists {
.ok_or_else(|| format!("target not found: {}", target))?; bail!("link already exists: {} ↔ {}", source, target);
}
}
let graph = self.build_graph(); let graph = self.build_graph();
let jaccard = graph.jaccard(source, target); let jaccard = graph.jaccard(source, target);

173
src/hippocampus/store/parse.rs
View file

@ -1,173 +0,0 @@
// Markdown parsing for memory files
//
// Splits markdown files into MemoryUnit structs based on `<!-- mem: ... -->`
// markers. Each marker starts a new section; content before the first marker
// becomes the file-level unit. Links and causal edges are extracted from
// both marker attributes and inline markdown links.
use super::NodeType;
use regex::Regex;
use std::collections::HashMap;
use std::path::Path;
use std::sync::OnceLock;
pub struct MemoryUnit {
pub key: String,
pub content: String,
pub marker_links: Vec<String>,
pub md_links: Vec<String>,
pub causes: Vec<String>,
pub state: Option<String>,
pub source_ref: Option<String>,
}
pub(super) fn classify_filename(filename: &str) -> NodeType {
let bare = filename.strip_suffix(".md").unwrap_or(filename);
if bare.starts_with("daily-") { NodeType::EpisodicDaily }
else if bare.starts_with("weekly-") { NodeType::EpisodicWeekly }
else if bare.starts_with("monthly-") { NodeType::EpisodicMonthly }
else if bare == "journal" { NodeType::EpisodicSession }
else { NodeType::Semantic }
}
pub fn parse_units(raw_filename: &str, content: &str) -> Vec<MemoryUnit> {
let filename = raw_filename.strip_suffix(".md").unwrap_or(raw_filename);
static MARKER_RE: OnceLock<Regex> = OnceLock::new();
static SOURCE_RE: OnceLock<Regex> = OnceLock::new();
static MD_LINK_RE: OnceLock<Regex> = OnceLock::new();
let marker_re = MARKER_RE.get_or_init(||
Regex::new(r"<!--\s*mem:\s*((?:id|links|tags|causes|state)\s*=\s*[^\s].*?)-->").unwrap());
let source_re = SOURCE_RE.get_or_init(||
Regex::new(r"<!--\s*source:\s*(.+?)\s*-->").unwrap());
let md_link_re = MD_LINK_RE.get_or_init(||
Regex::new(r"\[[^\]]*\]\(([^):]+(?:#[^)]*)?)\)").unwrap());
let markers: Vec<_> = marker_re.captures_iter(content)
.map(|cap| {
let full_match = cap.get(0).unwrap();
let attrs_str = &cap[1];
(full_match.start(), full_match.end(), parse_marker_attrs(attrs_str))
})
.collect();
let find_source = |text: &str| -> Option<String> {
source_re.captures(text).map(|c| c[1].trim().to_string())
};
if markers.is_empty() {
let source_ref = find_source(content);
let md_links = extract_md_links(content, md_link_re, filename);
return vec![MemoryUnit {
key: filename.to_string(),
content: content.to_string(),
marker_links: Vec::new(),
md_links,
causes: Vec::new(),
state: None,
source_ref,
}];
}
let mut units = Vec::new();
let first_start = markers[0].0;
let pre_content = content[..first_start].trim();
if !pre_content.is_empty() {
let source_ref = find_source(pre_content);
let md_links = extract_md_links(pre_content, md_link_re, filename);
units.push(MemoryUnit {
key: filename.to_string(),
content: pre_content.to_string(),
marker_links: Vec::new(),
md_links,
causes: Vec::new(),
state: None,
source_ref,
});
}
for (i, (_, end, attrs)) in markers.iter().enumerate() {
let unit_end = if i + 1 < markers.len() {
markers[i + 1].0
} else {
content.len()
};
let unit_content = content[*end..unit_end].trim();
let id = attrs.get("id").cloned().unwrap_or_default();
let key = if id.is_empty() {
format!("{}#unnamed-{}", filename, i)
} else {
format!("{}#{}", filename, id)
};
let marker_links = attrs.get("links")
.map(|l| l.split(',').map(|s| normalize_link(s.trim(), filename)).collect())
.unwrap_or_default();
let causes = attrs.get("causes")
.map(|l| l.split(',').map(|s| normalize_link(s.trim(), filename)).collect())
.unwrap_or_default();
let state = attrs.get("state").cloned();
let source_ref = find_source(unit_content);
let md_links = extract_md_links(unit_content, md_link_re, filename);
units.push(MemoryUnit {
key,
content: unit_content.to_string(),
marker_links,
md_links,
causes,
state,
source_ref,
});
}
units
}
fn parse_marker_attrs(attrs_str: &str) -> HashMap<String, String> {
static ATTR_RE: OnceLock<Regex> = OnceLock::new();
let attr_re = ATTR_RE.get_or_init(|| Regex::new(r"(\w+)\s*=\s*(\S+)").unwrap());
let mut attrs = HashMap::new();
for cap in attr_re.captures_iter(attrs_str) {
attrs.insert(cap[1].to_string(), cap[2].to_string());
}
attrs
}
fn extract_md_links(content: &str, re: &Regex, source_file: &str) -> Vec<String> {
re.captures_iter(content)
.map(|cap| normalize_link(&cap[1], source_file))
.filter(|link| !link.starts_with(source_file) || link.contains('#'))
.collect()
}
fn normalize_link(target: &str, source_file: &str) -> String {
let source_bare = source_file.strip_suffix(".md").unwrap_or(source_file);
if target.starts_with('#') {
return format!("{}{}", source_bare, target);
}
let (path_part, fragment) = if let Some(hash_pos) = target.find('#') {
(&target[..hash_pos], Some(&target[hash_pos..]))
} else {
(target, None)
};
let basename = Path::new(path_part)
.file_name()
.map(|f| f.to_string_lossy().to_string())
.unwrap_or_else(|| path_part.to_string());
let bare = basename.strip_suffix(".md").unwrap_or(&basename);
match fragment {
Some(frag) => format!("{}{}", bare, frag),
None => bare.to_string(),
}
}

846
src/hippocampus/store/persist.rs
View file

@ -1,846 +0,0 @@
// Persistence layer: load, save, replay, append, snapshot
//
// Three-tier loading strategy:
// 1. rkyv mmap snapshot (snapshot.rkyv) — ~4ms deserialize
// 2. bincode cache (state.bin) — ~10ms
// 3. capnp log replay — ~40ms
//
// Logs are append-only; cache staleness uses log file sizes, not mtimes.
use super::types::*;
use crate::memory_capnp;
use capnp::message;
use capnp::serialize;
use std::collections::HashMap;
use std::fs;
use std::io::{BufReader, Seek};
use std::path::Path;
use std::sync::Arc;
/// Process-global cached store. Reloads only when log files change.
static CACHED_STORE: tokio::sync::OnceCell<Arc<tokio::sync::Mutex<Store>>> =
tokio::sync::OnceCell::const_new();
impl Store {
/// Get or create the process-global cached store.
/// Reloads from disk if log files have changed since last load.
pub async fn cached() -> Result<Arc<tokio::sync::Mutex<Store>>, String> {
let store = CACHED_STORE.get_or_try_init(|| async {
let s = Store::load()?;
Ok::<_, String>(Arc::new(tokio::sync::Mutex::new(s)))
}).await?;
{
let mut guard = store.lock().await;
if guard.is_stale() {
*guard = Store::load()?;
}
}
Ok(store.clone())
}
/// Check if the on-disk logs have grown since we loaded.
pub fn is_stale(&self) -> bool {
let nodes_size = fs::metadata(nodes_path()).map(|m| m.len()).unwrap_or(0);
let rels_size = fs::metadata(relations_path()).map(|m| m.len()).unwrap_or(0);
nodes_size != self.loaded_nodes_size || rels_size != self.loaded_rels_size
}
/// Load store from state.bin cache if fresh, otherwise rebuild from capnp logs.
///
/// Staleness check uses log file sizes (not mtimes). Since logs are
/// append-only, any write grows the file, invalidating the cache.
/// This avoids the mtime race that caused data loss with concurrent
/// writers (dream loop, link audit, journal enrichment).
pub fn load() -> Result<Store, String> {
// 1. Try rkyv mmap snapshot (~4ms with deserialize, <1ms zero-copy)
match Self::load_snapshot_mmap() {
Ok(Some(mut store)) => {
// rkyv snapshot doesn't include visits — replay from log
let visits_p = visits_path();
if visits_p.exists() {
store.replay_visits(&visits_p).ok();
}
let tp_p = transcript_progress_path();
if tp_p.exists() {
store.replay_transcript_progress(&tp_p).ok();
}
return Ok(store);
},
Ok(None) => {},
Err(e) => eprintln!("rkyv snapshot: {}", e),
}
// 2. Try bincode state.bin cache (~10ms)
let nodes_p = nodes_path();
let rels_p = relations_path();
let state_p = state_path();
let nodes_size = fs::metadata(&nodes_p).map(|m| m.len()).unwrap_or(0);
let rels_size = fs::metadata(&rels_p).map(|m| m.len()).unwrap_or(0);
if let Ok(data) = fs::read(&state_p)
&& data.len() >= CACHE_HEADER_LEN && data[..4] == CACHE_MAGIC {
let cached_nodes = u64::from_le_bytes(data[4..12].try_into().unwrap());
let cached_rels = u64::from_le_bytes(data[12..20].try_into().unwrap());
if cached_nodes == nodes_size && cached_rels == rels_size
&& let Ok(mut store) = bincode::deserialize::<Store>(&data[CACHE_HEADER_LEN..]) {
// Rebuild uuid_to_key (skipped by serde)
for (key, node) in &store.nodes {
store.uuid_to_key.insert(node.uuid, key.clone());
}
store.loaded_nodes_size = nodes_size;
store.loaded_rels_size = rels_size;
// Bootstrap: write rkyv snapshot if missing
if !snapshot_path().exists()
&& let Err(e) = store.save_snapshot(cached_nodes, cached_rels) {
eprintln!("rkyv bootstrap: {}", e);
}
return Ok(store);
}
}
// Stale or no cache — rebuild from capnp logs
let mut store = Store::default();
if nodes_p.exists() {
store.replay_nodes(&nodes_p)?;
}
if rels_p.exists() {
store.replay_relations(&rels_p)?;
}
let visits_p = visits_path();
if visits_p.exists() {
store.replay_visits(&visits_p)?;
}
let tp_p = transcript_progress_path();
if tp_p.exists() {
store.replay_transcript_progress(&tp_p)?;
}
// Record log sizes after replay — this is the state we reflect
store.loaded_nodes_size = fs::metadata(&nodes_p).map(|m| m.len()).unwrap_or(0);
store.loaded_rels_size = fs::metadata(&rels_p).map(|m| m.len()).unwrap_or(0);
// Drop edges referencing deleted/missing nodes
store.relations.retain(|r|
store.nodes.contains_key(&r.source_key) &&
store.nodes.contains_key(&r.target_key)
);
store.save()?;
Ok(store)
}
/// Load store directly from capnp logs, bypassing all caches.
/// Used by fsck to verify cache consistency.
pub fn load_from_logs() -> Result<Store, String> {
let nodes_p = nodes_path();
let rels_p = relations_path();
let mut store = Store::default();
if nodes_p.exists() {
store.replay_nodes(&nodes_p)?;
}
if rels_p.exists() {
store.replay_relations(&rels_p)?;
}
let visits_p = visits_path();
if visits_p.exists() {
store.replay_visits(&visits_p)?;
}
let tp_p = transcript_progress_path();
if tp_p.exists() {
store.replay_transcript_progress(&tp_p)?;
}
Ok(store)
}
/// Replay node log, keeping latest version per UUID.
/// Tracks all UUIDs seen per key to detect duplicates.
fn replay_nodes(&mut self, path: &Path) -> Result<(), String> {
let file = fs::File::open(path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
// Track all non-deleted UUIDs per key to detect duplicates
let mut key_uuids: HashMap<String, Vec<[u8; 16]>> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.map_err(|e| format!("read node log: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes: {}", e))? {
let node = Node::from_capnp_migrate(node_reader)?;
let existing_version = self.nodes.get(&node.key)
.map(|n| n.version)
.unwrap_or(0);
if node.version >= existing_version {
if node.deleted {
self.nodes.remove(&node.key);
self.uuid_to_key.remove(&node.uuid);
if let Some(uuids) = key_uuids.get_mut(&node.key) {
uuids.retain(|u| *u != node.uuid);
}
} else {
self.uuid_to_key.insert(node.uuid, node.key.clone());
self.nodes.insert(node.key.clone(), node.clone());
let uuids = key_uuids.entry(node.key).or_default();
if !uuids.contains(&node.uuid) {
uuids.push(node.uuid);
}
}
}
}
}
// Report duplicate keys
for (key, uuids) in &key_uuids {
if uuids.len() > 1 {
dbglog!("WARNING: key '{}' has {} UUIDs (duplicate nodes)", key, uuids.len());
}
}
Ok(())
}
/// Replay relation log, keeping latest version per UUID
fn replay_relations(&mut self, path: &Path) -> Result<(), String> {
let file = fs::File::open(path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
// Collect all, then deduplicate by UUID keeping latest version
let mut by_uuid: HashMap<[u8; 16], Relation> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::relation_log::Reader>()
.map_err(|e| format!("read relation log: {}", e))?;
for rel_reader in log.get_relations()
.map_err(|e| format!("get relations: {}", e))? {
let rel = Relation::from_capnp_migrate(rel_reader)?;
let existing_version = by_uuid.get(&rel.uuid)
.map(|r| r.version)
.unwrap_or(0);
if rel.version >= existing_version {
by_uuid.insert(rel.uuid, rel);
}
}
}
self.relations = by_uuid.into_values()
.filter(|r| !r.deleted)
.collect();
Ok(())
}
/// Find all duplicate keys: keys with multiple live UUIDs in the log.
/// Returns a map from key → vec of all live Node versions (one per UUID).
/// The "winner" in self.nodes is always one of them.
pub fn find_duplicates(&self) -> Result<HashMap<String, Vec<Node>>, String> {
let path = nodes_path();
if !path.exists() { return Ok(HashMap::new()); }
let file = fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
// Track latest version of each UUID
let mut by_uuid: HashMap<[u8; 16], Node> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.map_err(|e| format!("read node log: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes: {}", e))? {
let node = Node::from_capnp_migrate(node_reader)?;
let dominated = by_uuid.get(&node.uuid)
.map(|n| node.version >= n.version)
.unwrap_or(true);
if dominated {
by_uuid.insert(node.uuid, node);
}
}
}
// Group live (non-deleted) nodes by key
let mut by_key: HashMap<String, Vec<Node>> = HashMap::new();
for node in by_uuid.into_values() {
if !node.deleted {
by_key.entry(node.key.clone()).or_default().push(node);
}
}
// Keep only duplicates
by_key.retain(|_, nodes| nodes.len() > 1);
Ok(by_key)
}
/// Append nodes to the log file.
/// Serializes to a Vec first, then does a single write() syscall
/// so the append is atomic with O_APPEND even without flock.
pub fn append_nodes(&mut self, nodes: &[Node]) -> Result<(), String> {
let _lock = StoreLock::acquire()?;
self.append_nodes_unlocked(nodes)
}
/// Append nodes without acquiring the lock. Caller must hold StoreLock.
pub(crate) fn append_nodes_unlocked(&mut self, nodes: &[Node]) -> Result<(), String> {
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::node_log::Builder>();
let mut list = log.init_nodes(nodes.len() as u32);
for (i, node) in nodes.iter().enumerate() {
node.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.map_err(|e| format!("serialize nodes: {}", e))?;
let path = nodes_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
use std::io::Write;
(&file).write_all(&buf)
.map_err(|e| format!("write nodes: {}", e))?;
self.loaded_nodes_size = file.metadata().map(|m| m.len()).unwrap_or(0);
Ok(())
}
/// Replay only new entries appended to the node log since we last loaded.
/// Call under StoreLock to catch writes from concurrent processes.
pub(crate) fn refresh_nodes(&mut self) -> Result<(), String> {
let path = nodes_path();
let current_size = fs::metadata(&path).map(|m| m.len()).unwrap_or(0);
if current_size <= self.loaded_nodes_size {
return Ok(()); // no new data
}
let file = fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
reader.seek(std::io::SeekFrom::Start(self.loaded_nodes_size))
.map_err(|e| format!("seek nodes log: {}", e))?;
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.map_err(|e| format!("read node log delta: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes delta: {}", e))? {
let node = Node::from_capnp_migrate(node_reader)?;
let dominated = self.nodes.get(&node.key)
.map(|n| node.version >= n.version)
.unwrap_or(true);
if dominated {
if node.deleted {
self.nodes.remove(&node.key);
self.uuid_to_key.remove(&node.uuid);
} else {
self.uuid_to_key.insert(node.uuid, node.key.clone());
self.nodes.insert(node.key.clone(), node);
}
}
}
}
self.loaded_nodes_size = current_size;
Ok(())
}
/// Append relations to the log file.
/// Single write() syscall for atomic O_APPEND.
pub fn append_relations(&mut self, relations: &[Relation]) -> Result<(), String> {
let _lock = StoreLock::acquire()?;
self.append_relations_unlocked(relations)
}
/// Append relations without acquiring the lock. Caller must hold StoreLock.
pub(crate) fn append_relations_unlocked(&mut self, relations: &[Relation]) -> Result<(), String> {
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::relation_log::Builder>();
let mut list = log.init_relations(relations.len() as u32);
for (i, rel) in relations.iter().enumerate() {
rel.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.map_err(|e| format!("serialize relations: {}", e))?;
let path = relations_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
use std::io::Write;
(&file).write_all(&buf)
.map_err(|e| format!("write relations: {}", e))?;
self.loaded_rels_size = file.metadata().map(|m| m.len()).unwrap_or(0);
Ok(())
}
/// Append agent visit records to the visits log.
pub fn append_visits(&mut self, visits: &[AgentVisit]) -> Result<(), String> {
if visits.is_empty() { return Ok(()); }
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::agent_visit_log::Builder>();
let mut list = log.init_visits(visits.len() as u32);
for (i, visit) in visits.iter().enumerate() {
visit.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.map_err(|e| format!("serialize visits: {}", e))?;
let path = visits_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
use std::io::Write;
(&file).write_all(&buf)
.map_err(|e| format!("write visits: {}", e))?;
// Update in-memory index
for v in visits {
self.visits
.entry(v.node_key.clone())
.or_default()
.insert(v.agent.clone(), v.timestamp);
}
Ok(())
}
/// Replay visits log to rebuild in-memory index.
fn replay_visits(&mut self, path: &Path) -> Result<(), String> {
let file = fs::File::open(path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
while reader.stream_position().map_err(|e| e.to_string())?
< fs::metadata(path).map_err(|e| e.to_string())?.len()
{
let msg = match serialize::read_message(&mut reader, Default::default()) {
Ok(m) => m,
Err(_) => break,
};
let log = msg.get_root::<memory_capnp::agent_visit_log::Reader>()
.map_err(|e| format!("read visit log: {}", e))?;
for visit in log.get_visits().map_err(|e| e.to_string())? {
let key = visit.get_node_key().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
let agent = visit.get_agent().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
let ts = visit.get_timestamp();
if !key.is_empty() && !agent.is_empty() {
let entry = self.visits.entry(key).or_default();
// Keep latest timestamp per agent
let existing = entry.entry(agent).or_insert(0);
if ts > *existing {
*existing = ts;
}
}
}
}
Ok(())
}
/// Append transcript segment progress records.
pub fn append_transcript_progress(&mut self, segments: &[TranscriptSegment]) -> Result<(), String> {
if segments.is_empty() { return Ok(()); }
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::transcript_progress_log::Builder>();
let mut list = log.init_segments(segments.len() as u32);
for (i, seg) in segments.iter().enumerate() {
seg.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.map_err(|e| format!("serialize transcript progress: {}", e))?;
let path = transcript_progress_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
use std::io::Write;
(&file).write_all(&buf)
.map_err(|e| format!("write transcript progress: {}", e))?;
// Update in-memory index
for seg in segments {
self.transcript_progress
.entry((seg.transcript_id.clone(), seg.segment_index))
.or_default()
.insert(seg.agent.clone());
}
Ok(())
}
/// Replay transcript progress log to rebuild in-memory index.
fn replay_transcript_progress(&mut self, path: &Path) -> Result<(), String> {
let file = fs::File::open(path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
while reader.stream_position().map_err(|e| e.to_string())?
< fs::metadata(path).map_err(|e| e.to_string())?.len()
{
let msg = match serialize::read_message(&mut reader, Default::default()) {
Ok(m) => m,
Err(_) => break,
};
let log = msg.get_root::<memory_capnp::transcript_progress_log::Reader>()
.map_err(|e| format!("read transcript progress: {}", e))?;
for seg in log.get_segments().map_err(|e| e.to_string())? {
let id = seg.get_transcript_id().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
let agent = seg.get_agent().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
let idx = seg.get_segment_index();
if !id.is_empty() && !agent.is_empty() {
self.transcript_progress
.entry((id, idx))
.or_default()
.insert(agent);
}
}
}
Ok(())
}
/// Migrate old stub-node transcript markers into the new progress log.
/// Reads _observed-transcripts-f-*, _mined-transcripts#f-*, and _facts-* keys,
/// extracts transcript_id and segment_index, writes to transcript-progress.capnp,
/// then deletes the stub nodes.
pub fn migrate_transcript_progress(&mut self) -> Result<usize, String> {
let mut segments = Vec::new();
for key in self.nodes.keys() {
// _observed-transcripts-f-{UUID}.{segment}
if let Some(rest) = key.strip_prefix("_observed-transcripts-f-") {
if let Some((uuid, seg_str)) = rest.rsplit_once('.')
&& let Ok(seg) = seg_str.parse::<u32>() {
segments.push(new_transcript_segment(uuid, seg, "observation"));
}
}
// _mined-transcripts#f-{UUID}.{segment}
else if let Some(rest) = key.strip_prefix("_mined-transcripts#f-") {
if let Some((uuid, seg_str)) = rest.rsplit_once('.')
&& let Ok(seg) = seg_str.parse::<u32>() {
segments.push(new_transcript_segment(uuid, seg, "experience"));
}
}
// _mined-transcripts-f-{UUID}.{segment}
else if let Some(rest) = key.strip_prefix("_mined-transcripts-f-") {
if let Some((uuid, seg_str)) = rest.rsplit_once('.')
&& let Ok(seg) = seg_str.parse::<u32>() {
segments.push(new_transcript_segment(uuid, seg, "experience"));
}
}
// _facts-{UUID} (whole-file, segment 0)
else if let Some(uuid) = key.strip_prefix("_facts-") {
if !uuid.contains('-') || uuid.len() < 30 { continue; } // skip non-UUID
segments.push(new_transcript_segment(uuid, 0, "fact"));
}
}
let count = segments.len();
if count > 0 {
self.append_transcript_progress(&segments)?;
}
// Soft-delete the old stub nodes
let keys_to_delete: Vec<String> = self.nodes.keys()
.filter(|k| k.starts_with("_observed-transcripts-")
|| k.starts_with("_mined-transcripts")
|| (k.starts_with("_facts-") && !k.contains("fact_mine")))
.cloned()
.collect();
for key in &keys_to_delete {
if let Some(node) = self.nodes.get_mut(key) {
node.deleted = true;
}
}
if !keys_to_delete.is_empty() {
self.save()?;
}
Ok(count)
}
/// Record visits for a batch of node keys from a successful agent run.
pub fn record_agent_visits(&mut self, node_keys: &[String], agent: &str) -> Result<(), String> {
let visits: Vec<AgentVisit> = node_keys.iter()
.filter_map(|key| {
let node = self.nodes.get(key)?;
Some(new_visit(node.uuid, key, agent, "processed"))
})
.collect();
self.append_visits(&visits)
}
/// Get the last time an agent visited a node. Returns 0 if never visited.
pub fn last_visited(&self, node_key: &str, agent: &str) -> i64 {
self.visits.get(node_key)
.and_then(|agents| agents.get(agent))
.copied()
.unwrap_or(0)
}
/// Save the derived cache with log size header for staleness detection.
/// Uses atomic write (tmp + rename) to prevent partial reads.
pub fn save(&self) -> Result<(), String> {
let _lock = StoreLock::acquire()?;
let path = state_path();
if let Some(parent) = path.parent() {
fs::create_dir_all(parent).ok();
}
// Use log sizes from load time, not current filesystem sizes.
// If another writer appended since we loaded, our recorded size
// will be smaller than the actual log → next reader detects stale
// cache and replays the (correct, append-only) log.
let nodes_size = self.loaded_nodes_size;
let rels_size = self.loaded_rels_size;
let bincode_data = bincode::serialize(self)
.map_err(|e| format!("bincode serialize: {}", e))?;
let mut data = Vec::with_capacity(CACHE_HEADER_LEN + bincode_data.len());
data.extend_from_slice(&CACHE_MAGIC);
data.extend_from_slice(&nodes_size.to_le_bytes());
data.extend_from_slice(&rels_size.to_le_bytes());
data.extend_from_slice(&bincode_data);
// Atomic write: tmp file + rename
let tmp_path = path.with_extension("bin.tmp");
fs::write(&tmp_path, &data)
.map_err(|e| format!("write {}: {}", tmp_path.display(), e))?;
fs::rename(&tmp_path, &path)
.map_err(|e| format!("rename {} → {}: {}", tmp_path.display(), path.display(), e))?;
// Also write rkyv snapshot (mmap-friendly)
if let Err(e) = self.save_snapshot(nodes_size, rels_size) {
eprintln!("rkyv snapshot save: {}", e);
}
Ok(())
}
/// Serialize store as rkyv snapshot with staleness header.
/// Assumes StoreLock is already held by caller.
fn save_snapshot(&self, nodes_size: u64, rels_size: u64) -> Result<(), String> {
let snap = Snapshot {
nodes: self.nodes.clone(),
relations: self.relations.iter().filter(|r| !r.deleted).cloned().collect(),
gaps: self.gaps.clone(),
params: self.params,
};
let rkyv_data = rkyv::to_bytes::<_, 256>(&snap)
.map_err(|e| format!("rkyv serialize: {}", e))?;
let mut data = Vec::with_capacity(RKYV_HEADER_LEN + rkyv_data.len());
data.extend_from_slice(&RKYV_MAGIC);
data.extend_from_slice(&1u32.to_le_bytes()); // format version
data.extend_from_slice(&nodes_size.to_le_bytes());
data.extend_from_slice(&rels_size.to_le_bytes());
data.extend_from_slice(&(rkyv_data.len() as u64).to_le_bytes());
data.extend_from_slice(&rkyv_data);
let path = snapshot_path();
let tmp_path = path.with_extension("rkyv.tmp");
fs::write(&tmp_path, &data)
.map_err(|e| format!("write {}: {}", tmp_path.display(), e))?;
fs::rename(&tmp_path, &path)
.map_err(|e| format!("rename: {}", e))?;
Ok(())
}
/// Try loading store from mmap'd rkyv snapshot.
/// Returns None if snapshot is missing or stale (log sizes don't match).
fn load_snapshot_mmap() -> Result<Option<Store>, String> {
let path = snapshot_path();
if !path.exists() { return Ok(None); }
let nodes_size = fs::metadata(nodes_path()).map(|m| m.len()).unwrap_or(0);
let rels_size = fs::metadata(relations_path()).map(|m| m.len()).unwrap_or(0);
let file = fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mmap = unsafe { memmap2::Mmap::map(&file) }
.map_err(|e| format!("mmap {}: {}", path.display(), e))?;
if mmap.len() < RKYV_HEADER_LEN { return Ok(None); }
if mmap[..4] != RKYV_MAGIC { return Ok(None); }
// [4..8] = version, skip for now
let cached_nodes = u64::from_le_bytes(mmap[8..16].try_into().unwrap());
let cached_rels = u64::from_le_bytes(mmap[16..24].try_into().unwrap());
let data_len = u64::from_le_bytes(mmap[24..32].try_into().unwrap()) as usize;
if cached_nodes != nodes_size || cached_rels != rels_size {
return Ok(None); // stale
}
if mmap.len() < RKYV_HEADER_LEN + data_len {
return Ok(None); // truncated
}
let rkyv_data = &mmap[RKYV_HEADER_LEN..RKYV_HEADER_LEN + data_len];
// SAFETY: we wrote this file ourselves via save_snapshot().
// Skip full validation (check_archived_root) — the staleness header
// already confirms this snapshot matches the current log state.
let archived = unsafe { rkyv::archived_root::<Snapshot>(rkyv_data) };
let snap: Snapshot = <ArchivedSnapshot as rkyv::Deserialize<Snapshot, rkyv::Infallible>>
::deserialize(archived, &mut rkyv::Infallible).unwrap();
let mut store = Store {
nodes: snap.nodes,
relations: snap.relations,
gaps: snap.gaps,
params: snap.params,
..Default::default()
};
// Rebuild uuid_to_key (not serialized)
for (key, node) in &store.nodes {
store.uuid_to_key.insert(node.uuid, key.clone());
}
store.loaded_nodes_size = nodes_size;
store.loaded_rels_size = rels_size;
Ok(Some(store))
}
}
/// Check and repair corrupt capnp log files.
///
/// Reads each message sequentially, tracking file position. On the first
/// corrupt message, truncates the file to the last good position. Also
/// removes stale caches so the next load replays from the repaired log.
pub fn fsck() -> Result<(), String> {
let mut any_corrupt = false;
for (path, kind) in [
(nodes_path(), "node"),
(relations_path(), "relation"),
] {
if !path.exists() { continue; }
let file = fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let file_len = file.metadata()
.map_err(|e| format!("stat {}: {}", path.display(), e))?.len();
let mut reader = BufReader::new(file);
let mut good_messages = 0u64;
let mut last_good_pos = 0u64;
loop {
let pos = reader.stream_position()
.map_err(|e| format!("tell {}: {}", path.display(), e))?;
let msg = match serialize::read_message(&mut reader, message::ReaderOptions::new()) {
Ok(m) => m,
Err(_) => {
// read_message fails at EOF (normal) or on corrupt framing
if pos < file_len {
// Not at EOF — corrupt framing
eprintln!("{}: corrupt message at offset {}, truncating", kind, pos);
any_corrupt = true;
drop(reader);
let file = fs::OpenOptions::new().write(true).open(&path)
.map_err(|e| format!("open for truncate: {}", e))?;
file.set_len(pos)
.map_err(|e| format!("truncate {}: {}", path.display(), e))?;
eprintln!("{}: truncated from {} to {} bytes ({} good messages)",
kind, file_len, pos, good_messages);
}
break;
}
};
// Validate the message content too
let valid = if kind == "node" {
msg.get_root::<memory_capnp::node_log::Reader>()
.and_then(|l| l.get_nodes().map(|_| ()))
.is_ok()
} else {
msg.get_root::<memory_capnp::relation_log::Reader>()
.and_then(|l| l.get_relations().map(|_| ()))
.is_ok()
};
if valid {
good_messages += 1;
last_good_pos = reader.stream_position()
.map_err(|e| format!("tell {}: {}", path.display(), e))?;
} else {
eprintln!("{}: corrupt message content at offset {}, truncating to {}",
kind, pos, last_good_pos);
any_corrupt = true;
drop(reader);
let file = fs::OpenOptions::new().write(true).open(&path)
.map_err(|e| format!("open for truncate: {}", e))?;
file.set_len(last_good_pos)
.map_err(|e| format!("truncate {}: {}", path.display(), e))?;
eprintln!("{}: truncated from {} to {} bytes ({} good messages)",
kind, file_len, last_good_pos, good_messages);
break;
}
}
if !any_corrupt {
eprintln!("{}: {} messages, all clean", kind, good_messages);
}
}
if any_corrupt {
// Nuke caches so next load replays from the repaired logs
for p in [state_path(), snapshot_path()] {
if p.exists() {
fs::remove_file(&p)
.map_err(|e| format!("remove {}: {}", p.display(), e))?;
eprintln!("removed stale cache: {}", p.display());
}
}
eprintln!("repair complete — run `poc-memory status` to verify");
} else {
eprintln!("store is clean");
}
Ok(())
}

448
src/hippocampus/store/types.rs
View file

@ -1,122 +1,21 @@
// Core types for the memory store // Core types for the memory store
// //
// Node, Relation, enums, Params, and supporting types. Also contains // Node, Relation, enums, Store struct, path helpers, time helpers.
// the capnp serialization macros that generate bidirectional conversion. // capnp serialization is in capnp.rs.
use crate::memory_capnp;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use uuid::Uuid; use uuid::Uuid;
use std::collections::{HashMap, HashSet};
use std::fs;
use std::os::unix::io::AsRawFd;
use std::path::PathBuf; use std::path::PathBuf;
use std::time::{SystemTime, UNIX_EPOCH}; use std::time::{SystemTime, UNIX_EPOCH};
// ---------------------------------------------------------------------------
// Capnp serialization macros
//
// Declarative mapping between Rust types and capnp generated types.
// Adding a field to the schema means adding it in one place below;
// both read and write are generated from the same declaration.
// ---------------------------------------------------------------------------
/// Generate to_capnp/from_capnp conversion methods for an enum.
macro_rules! capnp_enum {
($rust_type:ident, $capnp_type:path, [$($variant:ident),+ $(,)?]) => {
impl $rust_type {
#[allow(clippy::wrong_self_convention, dead_code)]
pub(crate) fn to_capnp(&self) -> $capnp_type {
match self {
$(Self::$variant => <$capnp_type>::$variant,)+
}
}
pub(crate) fn from_capnp(v: $capnp_type) -> Self {
match v {
$(<$capnp_type>::$variant => Self::$variant,)+
}
}
}
};
}
/// Generate from_capnp/to_capnp methods for a struct with capnp serialization.
/// Fields are grouped by serialization kind:
/// text - capnp Text fields (String in Rust)
/// uuid - capnp Data fields ([u8; 16] in Rust)
/// prim - copy types (u32, f32, f64, bool)
/// enm - enums with to_capnp/from_capnp methods
/// skip - Rust-only fields not in capnp (set to Default on read)
macro_rules! capnp_message {
(
$struct:ident,
reader: $reader:ty,
builder: $builder:ty,
text: [$($tf:ident),* $(,)?],
uuid: [$($uf:ident),* $(,)?],
prim: [$($pf:ident),* $(,)?],
enm: [$($ef:ident: $et:ident),* $(,)?],
skip: [$($sf:ident),* $(,)?] $(,)?
) => {
impl $struct {
pub fn from_capnp(r: $reader) -> Result<Self, String> {
paste::paste! {
Ok(Self {
$($tf: read_text(r.[<get_ $tf>]()),)*
$($uf: read_uuid(r.[<get_ $uf>]()),)*
$($pf: r.[<get_ $pf>](),)*
$($ef: $et::from_capnp(
r.[<get_ $ef>]().map_err(|_| concat!("bad ", stringify!($ef)))?
),)*
$($sf: Default::default(),)*
})
}
}
pub fn to_capnp(&self, mut b: $builder) {
paste::paste! {
$(b.[<set_ $tf>](&self.$tf);)*
$(b.[<set_ $uf>](&self.$uf);)*
$(b.[<set_ $pf>](self.$pf);)*
$(b.[<set_ $ef>](self.$ef.to_capnp());)*
}
}
}
};
}
pub fn memory_dir() -> PathBuf { pub fn memory_dir() -> PathBuf {
crate::config::get().data_dir.clone() crate::config::get().data_dir.clone()
} }
pub fn nodes_path() -> PathBuf { memory_dir().join("nodes.capnp") } pub fn nodes_path() -> PathBuf { memory_dir().join("nodes.capnp") }
pub(crate) fn relations_path() -> PathBuf { memory_dir().join("relations.capnp") } pub(crate) fn relations_path() -> PathBuf { memory_dir().join("relations.capnp") }
pub(crate) fn state_path() -> PathBuf { memory_dir().join("state.bin") } pub(crate) fn db_path() -> PathBuf { memory_dir().join("index.redb") }
pub(crate) fn snapshot_path() -> PathBuf { memory_dir().join("snapshot.rkyv") }
fn lock_path() -> PathBuf { memory_dir().join(".store.lock") }
/// RAII file lock using flock(2). Dropped when scope exits.
pub(crate) struct StoreLock {
_file: fs::File,
}
impl StoreLock {
pub(crate) fn acquire() -> Result<Self, String> {
let path = lock_path();
let file = fs::OpenOptions::new()
.create(true).truncate(false).write(true).open(&path)
.map_err(|e| format!("open lock {}: {}", path.display(), e))?;
// Blocking exclusive lock
let ret = unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX) };
if ret != 0 {
return Err(format!("flock: {}", std::io::Error::last_os_error()));
}
Ok(StoreLock { _file: file })
}
// Lock released automatically when _file is dropped (flock semantics)
}
pub fn now_epoch() -> i64 { pub fn now_epoch() -> i64 {
SystemTime::now() SystemTime::now()
@ -183,8 +82,7 @@ pub fn today() -> String {
} }
// In-memory node representation // In-memory node representation
#[derive(Clone, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)] #[derive(Clone, Debug, Serialize, Deserialize)]
#[archive(check_bytes)]
pub struct Node { pub struct Node {
pub uuid: [u8; 16], pub uuid: [u8; 16],
pub version: u32, pub version: u32,
@ -197,18 +95,12 @@ pub struct Node {
pub emotion: f32, pub emotion: f32,
pub deleted: bool, pub deleted: bool,
pub source_ref: String, pub source_ref: String,
pub created: String,
pub retrievals: u32, pub retrievals: u32,
pub uses: u32, pub uses: u32,
pub wrongs: u32, pub wrongs: u32,
pub state_tag: String,
pub last_replayed: i64, pub last_replayed: i64,
pub spaced_repetition_interval: u32, pub spaced_repetition_interval: u32,
// Position within file (section index, for export ordering)
#[serde(default)]
pub position: u32,
// Stable creation timestamp (unix epoch seconds). Set once at creation; // Stable creation timestamp (unix epoch seconds). Set once at creation;
// never updated on rename or content update. Zero for legacy nodes. // never updated on rename or content update. Zero for legacy nodes.
#[serde(default)] #[serde(default)]
@ -227,8 +119,7 @@ pub struct Node {
pub degree: Option<u32>, pub degree: Option<u32>,
} }
#[derive(Clone, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)] #[derive(Clone, Debug, Serialize, Deserialize)]
#[archive(check_bytes)]
pub struct Relation { pub struct Relation {
pub uuid: [u8; 16], pub uuid: [u8; 16],
pub version: u32, pub version: u32,
@ -243,8 +134,7 @@ pub struct Relation {
pub target_key: String, pub target_key: String,
} }
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)] #[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)]
#[archive(check_bytes)]
pub enum NodeType { pub enum NodeType {
EpisodicSession, EpisodicSession,
EpisodicDaily, EpisodicDaily,
@ -253,265 +143,36 @@ pub enum NodeType {
EpisodicMonthly, EpisodicMonthly,
} }
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)] impl NodeType {
#[archive(check_bytes)] pub fn from_u8(v: u8) -> Self {
pub enum Provenance { match v {
Manual, 0 => NodeType::EpisodicSession,
Journal, 1 => NodeType::EpisodicDaily,
Agent, // legacy catch-all, prefer specific variants below 2 => NodeType::EpisodicWeekly,
Dream, 3 => NodeType::Semantic,
Derived, 4 => NodeType::EpisodicMonthly,
AgentExperienceMine, _ => NodeType::Semantic, // default
AgentKnowledgeObservation,
AgentKnowledgePattern,
AgentKnowledgeConnector,
AgentKnowledgeChallenger,
AgentConsolidate,
AgentDigest,
AgentFactMine,
AgentDecay,
}
impl Provenance {
/// Parse from POC_PROVENANCE env var. Returns None if unset.
pub fn from_env() -> Option<Self> {
std::env::var("POC_PROVENANCE").ok().and_then(|s| Self::from_label(&s))
}
pub fn from_label(s: &str) -> Option<Self> {
Some(match s {
"manual" => Self::Manual,
"journal" => Self::Journal,
"agent" => Self::Agent,
"dream" => Self::Dream,
"derived" => Self::Derived,
"agent:experience-mine" => Self::AgentExperienceMine,
"agent:knowledge-observation"=> Self::AgentKnowledgeObservation,
"agent:knowledge-pattern" => Self::AgentKnowledgePattern,
"agent:knowledge-connector" => Self::AgentKnowledgeConnector,
"agent:knowledge-challenger" => Self::AgentKnowledgeChallenger,
"agent:consolidate" => Self::AgentConsolidate,
"agent:digest" => Self::AgentDigest,
"agent:fact-mine" => Self::AgentFactMine,
"agent:decay" => Self::AgentDecay,
_ => return None,
})
}
pub fn label(&self) -> &'static str {
match self {
Self::Manual => "manual",
Self::Journal => "journal",
Self::Agent => "agent",
Self::Dream => "dream",
Self::Derived => "derived",
Self::AgentExperienceMine => "agent:experience-mine",
Self::AgentKnowledgeObservation => "agent:knowledge-observation",
Self::AgentKnowledgePattern => "agent:knowledge-pattern",
Self::AgentKnowledgeConnector => "agent:knowledge-connector",
Self::AgentKnowledgeChallenger => "agent:knowledge-challenger",
Self::AgentConsolidate => "agent:consolidate",
Self::AgentDigest => "agent:digest",
Self::AgentFactMine => "agent:fact-mine",
Self::AgentDecay => "agent:decay",
} }
} }
} }
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)] #[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)]
#[archive(check_bytes)]
pub enum RelationType { pub enum RelationType {
Link, Link,
Causal, Causal,
Auto, Auto,
} }
capnp_enum!(NodeType, memory_capnp::NodeType, impl RelationType {
[EpisodicSession, EpisodicDaily, EpisodicWeekly, Semantic, EpisodicMonthly]); pub fn from_u8(v: u8) -> Self {
match v {
capnp_enum!(Provenance, memory_capnp::Provenance, 1 => RelationType::Causal,
[Manual, Journal, Agent, Dream, Derived, 2 => RelationType::Auto,
AgentExperienceMine, AgentKnowledgeObservation, AgentKnowledgePattern, _ => RelationType::Link,
AgentKnowledgeConnector, AgentKnowledgeChallenger, AgentConsolidate,
AgentDigest, AgentFactMine, AgentDecay]);
capnp_enum!(RelationType, memory_capnp::RelationType,
[Link, Causal, Auto]);
capnp_message!(Node,
reader: memory_capnp::content_node::Reader<'_>,
builder: memory_capnp::content_node::Builder<'_>,
text: [key, content, source_ref, created, state_tag, provenance],
uuid: [uuid],
prim: [version, timestamp, weight, emotion, deleted,
retrievals, uses, wrongs, last_replayed,
spaced_repetition_interval, position, created_at, last_scored],
enm: [node_type: NodeType],
skip: [community_id, clustering_coefficient, degree],
);
impl Node {
/// Read from capnp with migration: if the new provenance text field
/// is empty (old record), fall back to the deprecated provenanceOld enum.
pub fn from_capnp_migrate(r: memory_capnp::content_node::Reader<'_>) -> Result<Self, String> {
let mut node = Self::from_capnp(r)?;
if node.provenance.is_empty()
&& let Ok(old) = r.get_provenance_old() {
node.provenance = Provenance::from_capnp(old).label().to_string();
}
// Sanitize timestamps: old capnp records have raw offsets instead
// of unix epoch. Anything past year 2100 (~4102444800) is bogus.
const MAX_SANE_EPOCH: i64 = 4_102_444_800;
if node.timestamp > MAX_SANE_EPOCH || node.timestamp < 0 {
node.timestamp = node.created_at;
}
if node.created_at > MAX_SANE_EPOCH || node.created_at < 0 {
node.created_at = node.timestamp.min(MAX_SANE_EPOCH);
}
Ok(node)
}
}
capnp_message!(Relation,
reader: memory_capnp::relation::Reader<'_>,
builder: memory_capnp::relation::Builder<'_>,
text: [source_key, target_key, provenance],
uuid: [uuid, source, target],
prim: [version, timestamp, strength, deleted],
enm: [rel_type: RelationType],
skip: [],
);
impl Relation {
pub fn from_capnp_migrate(r: memory_capnp::relation::Reader<'_>) -> Result<Self, String> {
let mut rel = Self::from_capnp(r)?;
if rel.provenance.is_empty()
&& let Ok(old) = r.get_provenance_old() {
rel.provenance = Provenance::from_capnp(old).label().to_string();
}
Ok(rel)
}
}
#[derive(Clone, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub struct RetrievalEvent {
pub query: String,
pub timestamp: String,
pub results: Vec<String>,
pub used: Option<Vec<String>>,
}
#[derive(Clone, Copy, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub struct Params {
pub default_weight: f64,
pub decay_factor: f64,
pub use_boost: f64,
pub prune_threshold: f64,
pub edge_decay: f64,
pub max_hops: u32,
pub min_activation: f64,
}
impl Default for Params {
fn default() -> Self {
Params {
default_weight: 0.7,
decay_factor: 0.95,
use_boost: 0.15,
prune_threshold: 0.1,
edge_decay: 0.3,
max_hops: 3,
min_activation: 0.05,
} }
} }
} }
// Gap record — something we looked for but didn't find
#[derive(Clone, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub struct GapRecord {
pub description: String,
pub timestamp: String,
}
/// Per-node agent visit index: node_key → (agent_type → last_visit_timestamp)
pub(super) type VisitIndex = HashMap<String, HashMap<String, i64>>;
// The full in-memory store
#[derive(Default, Serialize, Deserialize)]
pub struct Store {
pub nodes: HashMap<String, Node>, // key → latest node
#[serde(skip)]
pub uuid_to_key: HashMap<[u8; 16], String>, // uuid → key (rebuilt from nodes)
pub relations: Vec<Relation>, // all active relations
pub retrieval_log: Vec<RetrievalEvent>,
pub gaps: Vec<GapRecord>,
pub params: Params,
/// Agent visit tracking: node_key → (agent_type → last_visit_epoch)
#[serde(default)]
pub visits: VisitIndex,
/// Transcript mining progress: (transcript_id, segment_index) → set of agents that processed it
#[serde(default)]
pub transcript_progress: HashMap<(String, u32), HashSet<String>>,
/// Log sizes at load time — used by save() to write correct staleness header.
/// If another writer appended since we loaded, our cache will be marked stale
/// (recorded size < actual size), forcing the next reader to replay the log.
#[serde(skip)]
pub(crate) loaded_nodes_size: u64,
#[serde(skip)]
pub(crate) loaded_rels_size: u64,
}
/// Snapshot for mmap: full store state minus retrieval_log (which
/// is append-only in retrieval.log). rkyv zero-copy serialization
/// lets us mmap this and access archived data without deserialization.
#[derive(rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub(crate) struct Snapshot {
pub(crate) nodes: HashMap<String, Node>,
pub(crate) relations: Vec<Relation>,
pub(crate) gaps: Vec<GapRecord>,
pub(crate) params: Params,
}
// rkyv snapshot header: 32 bytes (multiple of 16 for alignment after mmap)
// [0..4] magic "RKV\x01"
// [4..8] format version (u32 LE)
// [8..16] nodes.capnp file size (u64 LE) — staleness check
// [16..24] relations.capnp file size (u64 LE)
// [24..32] rkyv data length (u64 LE)
pub(crate) const RKYV_MAGIC: [u8; 4] = *b"RKV\x01";
pub(crate) const RKYV_HEADER_LEN: usize = 32;
// state.bin header: magic + log file sizes for staleness detection.
// File sizes are race-free for append-only logs (they only grow),
// unlike mtimes which race with concurrent writers.
pub(crate) const CACHE_MAGIC: [u8; 4] = *b"POC\x01";
pub(crate) const CACHE_HEADER_LEN: usize = 4 + 8 + 8; // magic + nodes_size + rels_size
// Cap'n Proto serialization helpers
/// Read a capnp text field, returning empty string on any error
pub(crate) fn read_text(result: capnp::Result<capnp::text::Reader>) -> String {
result.ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string()
}
/// Read a capnp data field as [u8; 16], zero-padded
pub(crate) fn read_uuid(result: capnp::Result<&[u8]>) -> [u8; 16] {
let mut out = [0u8; 16];
if let Ok(data) = result
&& data.len() >= 16 {
out.copy_from_slice(&data[..16]);
}
out
}
/// Create a new node with defaults /// Create a new node with defaults
pub fn new_node(key: &str, content: &str) -> Node { pub fn new_node(key: &str, content: &str) -> Node {
Node { Node {
@ -526,14 +187,11 @@ pub fn new_node(key: &str, content: &str) -> Node {
emotion: 0.0, emotion: 0.0,
deleted: false, deleted: false,
source_ref: String::new(), source_ref: String::new(),
created: today(),
retrievals: 0, retrievals: 0,
uses: 0, uses: 0,
wrongs: 0, wrongs: 0,
state_tag: String::new(),
last_replayed: 0, last_replayed: 0,
spaced_repetition_interval: 1, spaced_repetition_interval: 1,
position: 0,
created_at: now_epoch(), created_at: now_epoch(),
last_scored: 0, last_scored: 0,
community_id: None, community_id: None,
@ -542,68 +200,6 @@ pub fn new_node(key: &str, content: &str) -> Node {
} }
} }
/// Agent visit record — tracks when an agent successfully processed a node
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct AgentVisit {
pub node_uuid: [u8; 16],
pub node_key: String,
pub agent: String,
pub timestamp: i64,
pub outcome: String,
}
capnp_message!(AgentVisit,
reader: memory_capnp::agent_visit::Reader<'_>,
builder: memory_capnp::agent_visit::Builder<'_>,
text: [node_key, agent, outcome],
uuid: [node_uuid],
prim: [timestamp],
enm: [],
skip: [],
);
pub(super) fn new_visit(node_uuid: [u8; 16], node_key: &str, agent: &str, outcome: &str) -> AgentVisit {
AgentVisit {
node_uuid,
node_key: node_key.to_string(),
agent: agent.to_string(),
timestamp: now_epoch(),
outcome: outcome.to_string(),
}
}
pub(crate) fn visits_path() -> PathBuf { memory_dir().join("visits.capnp") }
/// Transcript mining progress — tracks which segments have been processed
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct TranscriptSegment {
pub transcript_id: String,
pub segment_index: u32,
pub agent: String,
pub timestamp: i64,
}
capnp_message!(TranscriptSegment,
reader: memory_capnp::transcript_segment::Reader<'_>,
builder: memory_capnp::transcript_segment::Builder<'_>,
text: [transcript_id, agent],
uuid: [],
prim: [segment_index, timestamp],
enm: [],
skip: [],
);
pub(super) fn new_transcript_segment(transcript_id: &str, segment_index: u32, agent: &str) -> TranscriptSegment {
TranscriptSegment {
transcript_id: transcript_id.to_string(),
segment_index,
agent: agent.to_string(),
timestamp: now_epoch(),
}
}
pub(crate) fn transcript_progress_path() -> PathBuf { memory_dir().join("transcript-progress.capnp") }
/// Create a new relation. /// Create a new relation.
/// Provenance is set from POC_PROVENANCE env var if present, else "manual". /// Provenance is set from POC_PROVENANCE env var if present, else "manual".
pub fn new_relation( pub fn new_relation(

253
src/hippocampus/store/view.rs
View file

@ -1,21 +1,16 @@
// Read-only access abstractions for the memory store // Read-only access abstraction for the memory store
//
// StoreView: trait abstracting over owned Store and zero-copy MmapView.
// MmapView: mmap'd rkyv snapshot for sub-millisecond read-only access.
// AnyView: enum dispatch selecting fastest available view at runtime.
use super::types::*; use super::{capnp, index, types::*};
use super::Store;
use std::fs;
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// StoreView: read-only access trait for search and graph code. // StoreView: read-only access trait for search and graph code.
//
// Abstracts over owned Store and zero-copy MmapView so the same
// spreading-activation and graph code works with either.
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
pub trait StoreView { pub trait StoreView {
/// Get all node keys (from index, no deserialization).
fn all_keys(&self) -> Vec<String>;
/// Iterate all nodes. Callback receives (key, content, weight). /// Iterate all nodes. Callback receives (key, content, weight).
fn for_each_node<F: FnMut(&str, &str, f32)>(&self, f: F); fn for_each_node<F: FnMut(&str, &str, f32)>(&self, f: F);
@ -27,191 +22,89 @@ pub trait StoreView {
/// Node weight by key, or the default weight if missing. /// Node weight by key, or the default weight if missing.
fn node_weight(&self, key: &str) -> f64; fn node_weight(&self, key: &str) -> f64;
/// Node content by key.
fn node_content(&self, key: &str) -> Option<&str>;
/// Search/graph parameters.
fn params(&self) -> Params;
} }
impl StoreView for Store { impl StoreView for Store {
fn all_keys(&self) -> Vec<String> {
let db = match self.db.as_ref() {
Some(db) => db,
None => return Vec::new(),
};
index::all_keys(db).unwrap_or_default()
}
fn for_each_node<F: FnMut(&str, &str, f32)>(&self, mut f: F) { fn for_each_node<F: FnMut(&str, &str, f32)>(&self, mut f: F) {
for (key, node) in &self.nodes { let db = match self.db.as_ref() {
f(key, &node.content, node.weight); Some(db) => db,
None => return,
};
let keys = match index::all_keys(db) {
Ok(keys) => keys,
Err(_) => return,
};
for key in keys {
if let Ok(Some(offset)) = index::get_offset(db, &key) {
if let Ok(node) = capnp::read_node_at_offset(offset) {
f(&key, &node.content, node.weight);
}
}
} }
} }
fn for_each_node_meta<F: FnMut(&str, NodeType, i64)>(&self, mut f: F) { fn for_each_node_meta<F: FnMut(&str, NodeType, i64)>(&self, mut f: F) {
for (key, node) in &self.nodes { let db = match self.db.as_ref() {
f(key, node.node_type, node.timestamp); Some(db) => db,
None => return,
};
// Use index directly — no capnp reads needed
let pairs = match index::all_key_uuid_pairs(db) {
Ok(p) => p,
Err(_) => return,
};
for (key, _uuid, node_type, timestamp) in pairs {
f(&key, NodeType::from_u8(node_type), timestamp);
} }
} }
fn for_each_relation<F: FnMut(&str, &str, f32, RelationType)>(&self, mut f: F) { fn for_each_relation<F: FnMut(&str, &str, f32, RelationType)>(&self, mut f: F) {
for rel in &self.relations { let db = match self.db.as_ref() {
if rel.deleted { continue; } Some(db) => db,
f(&rel.source_key, &rel.target_key, rel.strength, rel.rel_type); None => return,
};
// Build uuid ↔ key maps in a single table scan
let pairs = match index::all_key_uuid_pairs(db) {
Ok(p) => p,
Err(_) => return,
};
let mut uuid_to_key: std::collections::HashMap<[u8; 16], String> = std::collections::HashMap::new();
for (key, uuid, _, _) in &pairs {
uuid_to_key.insert(*uuid, key.clone());
}
// Iterate edges: only process outgoing to avoid duplicates
for (key, uuid, _, _) in &pairs {
let edges = match index::edges_for_node(db, uuid) {
Ok(e) => e,
Err(_) => continue,
};
for (other_uuid, strength, rel_type_byte, is_outgoing) in edges {
if !is_outgoing { continue; }
let target_key = match uuid_to_key.get(&other_uuid) {
Some(k) => k,
None => continue,
};
f(key, target_key, strength, RelationType::from_u8(rel_type_byte));
}
} }
} }
fn node_weight(&self, key: &str) -> f64 { fn node_weight(&self, key: &str) -> f64 {
self.nodes.get(key).map(|n| n.weight as f64).unwrap_or(self.params.default_weight) let cfg = crate::config::get();
} self.get_node(key)
.ok()
fn node_content(&self, key: &str) -> Option<&str> { .flatten()
self.nodes.get(key).map(|n| n.content.as_str())
}
fn params(&self) -> Params {
self.params
}
}
// ---------------------------------------------------------------------------
// MmapView: zero-copy store access via mmap'd rkyv snapshot.
//
// Holds the mmap alive; all string reads go directly into the mapped
// pages without allocation. Falls back to None if snapshot is stale.
// ---------------------------------------------------------------------------
pub struct MmapView {
mmap: memmap2::Mmap,
_file: fs::File,
data_offset: usize,
data_len: usize,
}
impl MmapView {
/// Try to open a fresh rkyv snapshot. Returns None if missing or stale.
pub fn open() -> Option<Self> {
let path = snapshot_path();
let file = fs::File::open(&path).ok()?;
let mmap = unsafe { memmap2::Mmap::map(&file) }.ok()?;
if mmap.len() < RKYV_HEADER_LEN { return None; }
if mmap[..4] != RKYV_MAGIC { return None; }
let nodes_size = fs::metadata(nodes_path()).map(|m| m.len()).unwrap_or(0);
let rels_size = fs::metadata(relations_path()).map(|m| m.len()).unwrap_or(0);
let cached_nodes = u64::from_le_bytes(mmap[8..16].try_into().unwrap());
let cached_rels = u64::from_le_bytes(mmap[16..24].try_into().unwrap());
let data_len = u64::from_le_bytes(mmap[24..32].try_into().unwrap()) as usize;
if cached_nodes != nodes_size || cached_rels != rels_size { return None; }
if mmap.len() < RKYV_HEADER_LEN + data_len { return None; }
Some(MmapView { mmap, _file: file, data_offset: RKYV_HEADER_LEN, data_len })
}
fn snapshot(&self) -> &ArchivedSnapshot {
let data = &self.mmap[self.data_offset..self.data_offset + self.data_len];
unsafe { rkyv::archived_root::<Snapshot>(data) }
}
}
impl StoreView for MmapView {
fn for_each_node<F: FnMut(&str, &str, f32)>(&self, mut f: F) {
let snap = self.snapshot();
for (key, node) in snap.nodes.iter() {
f(key, &node.content, node.weight);
}
}
fn for_each_node_meta<F: FnMut(&str, NodeType, i64)>(&self, mut f: F) {
let snap = self.snapshot();
for (key, node) in snap.nodes.iter() {
let nt = match node.node_type {
ArchivedNodeType::EpisodicSession => NodeType::EpisodicSession,
ArchivedNodeType::EpisodicDaily => NodeType::EpisodicDaily,
ArchivedNodeType::EpisodicWeekly => NodeType::EpisodicWeekly,
ArchivedNodeType::EpisodicMonthly => NodeType::EpisodicMonthly,
ArchivedNodeType::Semantic => NodeType::Semantic,
};
f(key, nt, node.timestamp);
}
}
fn for_each_relation<F: FnMut(&str, &str, f32, RelationType)>(&self, mut f: F) {
let snap = self.snapshot();
for rel in snap.relations.iter() {
if rel.deleted { continue; }
let rt = match rel.rel_type {
ArchivedRelationType::Link => RelationType::Link,
ArchivedRelationType::Causal => RelationType::Causal,
ArchivedRelationType::Auto => RelationType::Auto,
};
f(&rel.source_key, &rel.target_key, rel.strength, rt);
}
}
fn node_weight(&self, key: &str) -> f64 {
let snap = self.snapshot();
snap.nodes.get(key)
.map(|n| n.weight as f64) .map(|n| n.weight as f64)
.unwrap_or(snap.params.default_weight) .unwrap_or(cfg.default_node_weight)
}
fn node_content(&self, key: &str) -> Option<&str> {
let snap = self.snapshot();
snap.nodes.get(key).map(|n| &*n.content)
}
fn params(&self) -> Params {
let p = &self.snapshot().params;
Params {
default_weight: p.default_weight,
decay_factor: p.decay_factor,
use_boost: p.use_boost,
prune_threshold: p.prune_threshold,
edge_decay: p.edge_decay,
max_hops: p.max_hops,
min_activation: p.min_activation,
}
}
}
// ---------------------------------------------------------------------------
// AnyView: enum dispatch for read-only access.
//
// MmapView when the snapshot is fresh, owned Store as fallback.
// The match on each call is a single predicted branch — zero overhead.
// ---------------------------------------------------------------------------
pub enum AnyView {
Mmap(MmapView),
Owned(Store),
}
impl AnyView {
/// Load the fastest available view: mmap snapshot or owned store.
pub fn load() -> Result<Self, String> {
if let Some(mv) = MmapView::open() {
Ok(AnyView::Mmap(mv))
} else {
Ok(AnyView::Owned(Store::load()?))
}
}
}
impl StoreView for AnyView {
fn for_each_node<F: FnMut(&str, &str, f32)>(&self, f: F) {
match self { AnyView::Mmap(v) => v.for_each_node(f), AnyView::Owned(s) => s.for_each_node(f) }
}
fn for_each_node_meta<F: FnMut(&str, NodeType, i64)>(&self, f: F) {
match self { AnyView::Mmap(v) => v.for_each_node_meta(f), AnyView::Owned(s) => s.for_each_node_meta(f) }
}
fn for_each_relation<F: FnMut(&str, &str, f32, RelationType)>(&self, f: F) {
match self { AnyView::Mmap(v) => v.for_each_relation(f), AnyView::Owned(s) => s.for_each_relation(f) }
}
fn node_weight(&self, key: &str) -> f64 {
match self { AnyView::Mmap(v) => v.node_weight(key), AnyView::Owned(s) => s.node_weight(key) }
}
fn node_content(&self, key: &str) -> Option<&str> {
match self { AnyView::Mmap(v) => v.node_content(key), AnyView::Owned(s) => s.node_content(key) }
}
fn params(&self) -> Params {
match self { AnyView::Mmap(v) => v.params(), AnyView::Owned(s) => s.params() }
} }
} }

15
src/lib.rs
View file

@ -1,3 +1,5 @@
#![feature(async_fn_track_caller)]
// consciousness — unified crate for memory, agents, and subconscious processes // consciousness — unified crate for memory, agents, and subconscious processes
// //
// thought/ — shared cognitive substrate (tools, context, memory ops) // thought/ — shared cognitive substrate (tools, context, memory ops)
@ -47,6 +49,16 @@ pub mod session;
// Shared utilities // Shared utilities
pub mod util; pub mod util;
// Lock hold time tracking
pub mod locks;
// Re-export tracked locks as the default — swap to tokio::sync to disable tracking
pub use locks::TrackedMutex as Mutex;
pub use locks::TrackedMutexGuard as MutexGuard;
pub use locks::TrackedRwLock as RwLock;
pub use locks::TrackedRwLockReadGuard as RwLockReadGuard;
pub use locks::TrackedRwLockWriteGuard as RwLockWriteGuard;
// CLI handlers // CLI handlers
pub mod cli; pub mod cli;
@ -56,6 +68,9 @@ pub mod cli;
// Thalamus — universal notification routing and channel infrastructure // Thalamus — universal notification routing and channel infrastructure
pub mod thalamus; pub mod thalamus;
// MCP server — exposes memory tools over Unix socket
pub mod mcp_server;
// Re-export at crate root — capnp codegen emits `crate::daemon_capnp::` paths // Re-export at crate root — capnp codegen emits `crate::daemon_capnp::` paths
pub use thalamus::daemon_capnp; pub use thalamus::daemon_capnp;

235
src/locks.rs Normal file
View file

@ -0,0 +1,235 @@
// Lock hold time tracking
//
// Wrappers around tokio::sync primitives that track how long locks are held,
// keyed by source location. Use `lock_stats()` to get a snapshot.
use std::collections::HashMap;
use std::panic::Location;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::OnceLock;
use std::time::Instant;
use tokio::sync::{Mutex, MutexGuard, RwLock, RwLockReadGuard, RwLockWriteGuard};
// ── Stats Registry ─────────────────────────────────────────────
struct LocationStats {
count: AtomicU64,
total_ns: AtomicU64,
max_ns: AtomicU64,
}
impl LocationStats {
fn new() -> Self {
Self {
count: AtomicU64::new(0),
total_ns: AtomicU64::new(0),
max_ns: AtomicU64::new(0),
}
}
fn record(&self, duration_ns: u64) {
self.count.fetch_add(1, Ordering::Relaxed);
self.total_ns.fetch_add(duration_ns, Ordering::Relaxed);
// Update max using CAS loop
let mut current = self.max_ns.load(Ordering::Relaxed);
while duration_ns > current {
match self.max_ns.compare_exchange_weak(
current, duration_ns, Ordering::Relaxed, Ordering::Relaxed
) {
Ok(_) => break,
Err(c) => current = c,
}
}
}
fn snapshot(&self) -> LockStats {
let count = self.count.load(Ordering::Relaxed);
let total_ns = self.total_ns.load(Ordering::Relaxed);
let max_ns = self.max_ns.load(Ordering::Relaxed);
LockStats {
count,
total_ns,
max_ns,
avg_ns: if count > 0 { total_ns / count } else { 0 },
}
}
}
/// Stats for a single lock location.
#[derive(Clone, Debug)]
pub struct LockStats {
pub count: u64,
pub total_ns: u64,
pub max_ns: u64,
pub avg_ns: u64,
}
type StatsMap = std::sync::Mutex<HashMap<&'static Location<'static>, LocationStats>>;
fn stats_map() -> &'static StatsMap {
static MAP: OnceLock<StatsMap> = OnceLock::new();
MAP.get_or_init(|| std::sync::Mutex::new(HashMap::new()))
}
fn record_hold_time(loc: &'static Location<'static>, duration_ns: u64) {
let map = stats_map().lock().unwrap();
if let Some(stats) = map.get(&loc) {
stats.record(duration_ns);
return;
}
drop(map);
// First time seeing this location — need write access
let mut map = stats_map().lock().unwrap();
let stats = map.entry(loc).or_insert_with(LocationStats::new);
stats.record(duration_ns);
}
/// Get a snapshot of all lock stats, sorted by max hold time (descending).
pub fn lock_stats() -> Vec<(String, LockStats)> {
let map = stats_map().lock().unwrap();
let mut stats: Vec<_> = map.iter()
.map(|(loc, s)| (format!("{}:{}", loc.file(), loc.line()), s.snapshot()))
.collect();
stats.sort_by(|a, b| b.1.max_ns.cmp(&a.1.max_ns));
stats
}
/// Reset all lock stats.
pub fn reset_lock_stats() {
let mut map = stats_map().lock().unwrap();
map.clear();
}
// ── TrackedMutex ───────────────────────────────────────────────
/// A Mutex wrapper that tracks hold times by caller location.
pub struct TrackedMutex<T> {
inner: Mutex<T>,
}
impl<T> TrackedMutex<T> {
pub fn new(value: T) -> Self {
Self { inner: Mutex::new(value) }
}
#[track_caller]
pub async fn lock(&self) -> TrackedMutexGuard<'_, T> {
let location = Location::caller();
let guard = self.inner.lock().await;
TrackedMutexGuard {
guard,
acquired_at: Instant::now(),
location,
}
}
#[track_caller]
pub fn try_lock(&self) -> Result<TrackedMutexGuard<'_, T>, tokio::sync::TryLockError> {
let location = Location::caller();
let guard = self.inner.try_lock()?;
Ok(TrackedMutexGuard {
guard,
acquired_at: Instant::now(),
location,
})
}
}
pub struct TrackedMutexGuard<'a, T> {
guard: MutexGuard<'a, T>,
acquired_at: Instant,
location: &'static Location<'static>,
}
impl<T> Drop for TrackedMutexGuard<'_, T> {
fn drop(&mut self) {
let duration = self.acquired_at.elapsed();
record_hold_time(self.location, duration.as_nanos() as u64);
}
}
impl<T> std::ops::Deref for TrackedMutexGuard<'_, T> {
type Target = T;
fn deref(&self) -> &T { &self.guard }
}
impl<T> std::ops::DerefMut for TrackedMutexGuard<'_, T> {
fn deref_mut(&mut self) -> &mut T { &mut self.guard }
}
// ── TrackedRwLock ──────────────────────────────────────────────
/// An RwLock wrapper that tracks hold times by caller location.
pub struct TrackedRwLock<T> {
inner: RwLock<T>,
}
impl<T> TrackedRwLock<T> {
pub fn new(value: T) -> Self {
Self { inner: RwLock::new(value) }
}
#[track_caller]
pub async fn read(&self) -> TrackedRwLockReadGuard<'_, T> {
let location = Location::caller();
let guard = self.inner.read().await;
TrackedRwLockReadGuard {
guard,
acquired_at: Instant::now(),
location,
}
}
#[track_caller]
pub async fn write(&self) -> TrackedRwLockWriteGuard<'_, T> {
let location = Location::caller();
let guard = self.inner.write().await;
TrackedRwLockWriteGuard {
guard,
acquired_at: Instant::now(),
location,
}
}
}
pub struct TrackedRwLockReadGuard<'a, T> {
guard: RwLockReadGuard<'a, T>,
acquired_at: Instant,
location: &'static Location<'static>,
}
impl<T> Drop for TrackedRwLockReadGuard<'_, T> {
fn drop(&mut self) {
let duration = self.acquired_at.elapsed();
record_hold_time(self.location, duration.as_nanos() as u64);
}
}
impl<T> std::ops::Deref for TrackedRwLockReadGuard<'_, T> {
type Target = T;
fn deref(&self) -> &T { &self.guard }
}
pub struct TrackedRwLockWriteGuard<'a, T> {
guard: RwLockWriteGuard<'a, T>,
acquired_at: Instant,
location: &'static Location<'static>,
}
impl<T> Drop for TrackedRwLockWriteGuard<'_, T> {
fn drop(&mut self) {
let duration = self.acquired_at.elapsed();
record_hold_time(self.location, duration.as_nanos() as u64);
}
}
impl<T> std::ops::Deref for TrackedRwLockWriteGuard<'_, T> {
type Target = T;
fn deref(&self) -> &T { &self.guard }
}
impl<T> std::ops::DerefMut for TrackedRwLockWriteGuard<'_, T> {
fn deref_mut(&mut self) -> &mut T { &mut self.guard }
}

265
src/main.rs
View file

@ -1,3 +1,5 @@
#![feature(panic_backtrace_config)]
// poc-memory: graph-structured memory for AI assistants // poc-memory: graph-structured memory for AI assistants
// //
// Authors: ProofOfConcept <poc@bcachefs.org> and Kent Overstreet // Authors: ProofOfConcept <poc@bcachefs.org> and Kent Overstreet
@ -33,31 +35,10 @@ struct Cli {
enum Command { enum Command {
// ── Core (daily use) ────────────────────────────────────────────── // ── Core (daily use) ──────────────────────────────────────────────
/// Search memory (AND logic across terms) /// Search memory via spreading activation from seed keys
///
/// Pipeline: -p spread -p spectral,k=20
/// Default pipeline: spread
Search { Search {
/// Search terms /// Seed node keys
query: Vec<String>, keys: Vec<String>,
/// Algorithm pipeline stages (repeatable)
#[arg(short, long = "pipeline")]
pipeline: Vec<String>,
/// Show more results
#[arg(long)]
expand: bool,
/// Show node content, not just keys
#[arg(long)]
full: bool,
/// Show debug output for each pipeline stage
#[arg(long)]
debug: bool,
/// Also match key components (e.g. "irc" matches "irc-access")
#[arg(long)]
fuzzy: bool,
/// Also search node content (slow, use when graph search misses)
#[arg(long)]
content: bool,
}, },
/// Output a node's content to stdout /// Output a node's content to stdout
Render { Render {
@ -147,30 +128,6 @@ EXAMPLES:
/// Query expression (e.g. "key ~ 'inner-life'") /// Query expression (e.g. "key ~ 'inner-life'")
expr: Vec<String>, expr: Vec<String>,
}, },
/// Mark a memory as useful (boosts weight)
Used {
/// Node key
key: Vec<String>,
},
/// Mark a memory as wrong/irrelevant
Wrong {
/// Node key
key: String,
/// Optional context
context: Vec<String>,
},
/// Mark a search result as not relevant (weakens edges that led to it)
#[command(name = "not-relevant")]
NotRelevant {
/// Node key that was not relevant
key: String,
},
/// Mark a node as not useful (weakens node weight, not edges)
#[command(name = "not-useful")]
NotUseful {
/// Node key
key: String,
},
/// Set a node's weight directly /// Set a node's weight directly
#[command(name = "weight-set")] #[command(name = "weight-set")]
WeightSet { WeightSet {
@ -179,11 +136,6 @@ EXAMPLES:
/// Weight (0.01 to 1.0) /// Weight (0.01 to 1.0)
weight: f32, weight: f32,
}, },
/// Record a gap in memory coverage
Gap {
/// Gap description
description: Vec<String>,
},
// ── Node operations ─────────────────────────────────────────────── // ── Node operations ───────────────────────────────────────────────
@ -230,17 +182,6 @@ enum NodeCmd {
/// New key /// New key
new_key: String, new_key: String,
}, },
/// List all node keys (one per line, optional glob)
#[command(name = "list")]
List {
/// Glob pattern to filter keys
pattern: Option<String>,
},
/// List all edges (tsv: source target strength type)
Edges,
/// Dump entire store as JSON
#[command(name = "dump")]
Dump,
} }
#[derive(Subcommand)] #[derive(Subcommand)]
@ -273,14 +214,6 @@ enum GraphCmd {
/// Node key /// Node key
key: Vec<String>, key: Vec<String>,
}, },
/// Find related nodes via spreading activation from seed nodes
Spread {
/// Seed node keys
keys: Vec<String>,
/// Maximum results (default: 20)
#[arg(short = 'n', default_value_t = 20)]
max_results: usize,
},
/// Add a link between two nodes /// Add a link between two nodes
#[command(name = "link-add")] #[command(name = "link-add")]
LinkAdd { LinkAdd {
@ -337,33 +270,10 @@ enum GraphCmd {
#[arg(long, default_value_t = 2)] #[arg(long, default_value_t = 2)]
min_size: usize, min_size: usize,
}, },
/// Show graph structure overview
Overview,
/// Diagnose duplicate/overlapping nodes for a topic cluster
Organize {
/// Search term (matches node keys; also content unless --key-only)
term: String,
/// Similarity threshold for pair reporting (default: 0.4)
#[arg(long, default_value_t = 0.4)]
threshold: f32,
/// Only match node keys, not content
#[arg(long)]
key_only: bool,
/// Create anchor node for the search term and link to cluster
#[arg(long)]
anchor: bool,
},
} }
#[derive(Subcommand)] #[derive(Subcommand)]
enum AgentCmd { enum AgentCmd {
/// Parse and apply links from digest nodes
#[command(name = "digest-links")]
DigestLinks {
/// Apply the links (default: dry run)
#[arg(long)]
apply: bool,
},
/// Run a single agent by name /// Run a single agent by name
Run { Run {
/// Agent name (e.g. observation, linker, distill) /// Agent name (e.g. observation, linker, distill)
@ -387,13 +297,6 @@ enum AgentCmd {
#[arg(long)] #[arg(long)]
state_dir: Option<String>, state_dir: Option<String>,
}, },
/// Show spaced repetition replay queue
#[command(name = "replay-queue")]
ReplayQueue {
/// Number of items to show
#[arg(long, default_value_t = 10)]
count: usize,
},
} }
#[derive(Subcommand)] #[derive(Subcommand)]
@ -402,6 +305,8 @@ enum AdminCmd {
Init, Init,
/// Report graph metrics (CC, communities, small-world) /// Report graph metrics (CC, communities, small-world)
Health, Health,
/// Show graph topology with hub warnings
Topology,
/// Run consistency checks and repair /// Run consistency checks and repair
Fsck, Fsck,
/// Find and merge duplicate nodes (same key, multiple UUIDs) /// Find and merge duplicate nodes (same key, multiple UUIDs)
@ -410,33 +315,9 @@ enum AdminCmd {
#[arg(long)] #[arg(long)]
apply: bool, apply: bool,
}, },
/// Bulk rename: replace a character in all keys
#[command(name = "bulk-rename")]
BulkRename {
/// Character to replace
from: String,
/// Replacement character
to: String,
/// Apply changes (default: dry run)
#[arg(long)]
apply: bool,
},
/// Brief metrics check (for cron/notifications) /// Brief metrics check (for cron/notifications)
#[command(name = "daily-check")] #[command(name = "daily-check")]
DailyCheck, DailyCheck,
/// Import markdown file(s) into the store
Import {
/// File paths
files: Vec<String>,
},
/// Export store nodes to markdown file(s)
Export {
/// File keys to export (or --all)
files: Vec<String>,
/// Export all file-level nodes
#[arg(long)]
all: bool,
},
/// Output session-start context from the store /// Output session-start context from the store
#[command(name = "load-context")] #[command(name = "load-context")]
LoadContext { LoadContext {
@ -444,24 +325,6 @@ enum AdminCmd {
#[arg(long)] #[arg(long)]
stats: bool, stats: bool,
}, },
/// Show recent retrieval log
Log,
/// Show current parameters
Params,
/// Bump daily lookup counter for keys
#[command(name = "lookup-bump")]
LookupBump {
/// Node keys
keys: Vec<String>,
},
/// Show daily lookup counts
Lookups {
/// Date (default: today)
date: Option<String>,
},
/// Migrate transcript stub nodes to progress log
#[command(name = "migrate-transcript-progress")]
MigrateTranscriptProgress,
} }
/// Print help with subcommands expanded to show nested commands. /// Print help with subcommands expanded to show nested commands.
@ -505,118 +368,94 @@ fn print_help() {
// ── Dispatch ───────────────────────────────────────────────────────── // ── Dispatch ─────────────────────────────────────────────────────────
trait Run { trait Run {
fn run(self) -> Result<(), String>; async fn run(self) -> anyhow::Result<()>;
} }
impl Run for Command { impl Run for Command {
fn run(self) -> Result<(), String> { async fn run(self) -> anyhow::Result<()> {
match self { match self {
Self::Search { query, pipeline, expand, full, debug, fuzzy, content } Self::Search { keys } => cli::node::cmd_search(&keys).await,
=> cli::misc::cmd_search(&query, &pipeline, expand, full, debug, fuzzy, content), Self::Render { key } => cli::node::cmd_render(&key).await,
Self::Render { key } => cli::node::cmd_render(&key), Self::Write { key } => cli::node::cmd_write(&key).await,
Self::Write { key } => cli::node::cmd_write(&key), Self::Edit { key } => cli::node::cmd_edit(&key).await,
Self::Edit { key } => cli::node::cmd_edit(&key), Self::History { full, key } => cli::node::cmd_history(&key, full).await,
Self::History { full, key } => cli::node::cmd_history(&key, full),
Self::Tail { n, full, provenance, all_versions } Self::Tail { n, full, provenance, all_versions }
=> cli::journal::cmd_tail(n, full, provenance.as_deref(), !all_versions), => cli::journal::cmd_tail(n, full, provenance.as_deref(), !all_versions),
Self::Status => cli::misc::cmd_status(), Self::Status => cli::admin::cmd_status().await,
Self::Query { expr } => cli::misc::cmd_query(&expr), Self::Query { expr } => cli::node::cmd_query(&expr).await,
Self::Used { key } => cli::node::cmd_used(&key), Self::WeightSet { key, weight } => cli::node::cmd_weight_set(&key, weight).await,
Self::Wrong { key, context } => cli::node::cmd_wrong(&key, &context), Self::Node(sub) => sub.run().await,
Self::NotRelevant { key } => cli::node::cmd_not_relevant(&key), Self::Journal(sub) => sub.run().await,
Self::NotUseful { key } => cli::node::cmd_not_useful(&key), Self::GraphCmd(sub) => sub.run().await,
Self::WeightSet { key, weight } => cli::node::cmd_weight_set(&key, weight), Self::Agent(sub) => sub.run().await,
Self::Gap { description } => cli::node::cmd_gap(&description), Self::Admin(sub) => sub.run().await,
Self::Node(sub) => sub.run(),
Self::Journal(sub) => sub.run(),
Self::GraphCmd(sub) => sub.run(),
Self::Agent(sub) => sub.run(),
Self::Admin(sub) => sub.run(),
// mcp-schema moved to consciousness-mcp binary // mcp-schema moved to consciousness-mcp binary
} }
} }
} }
impl Run for NodeCmd { impl Run for NodeCmd {
fn run(self) -> Result<(), String> { async fn run(self) -> anyhow::Result<()> {
match self { match self {
Self::Delete { key } => cli::node::cmd_node_delete(&key), Self::Delete { key } => cli::node::cmd_node_delete(&key).await,
Self::Rename { old_key, new_key } => cli::node::cmd_node_rename(&old_key, &new_key), Self::Rename { old_key, new_key } => cli::node::cmd_node_rename(&old_key, &new_key).await,
Self::List { pattern } => cli::node::cmd_list_keys(pattern.as_deref()),
Self::Edges => cli::node::cmd_list_edges(),
Self::Dump => cli::node::cmd_dump_json(),
} }
} }
} }
impl Run for JournalCmd { impl Run for JournalCmd {
fn run(self) -> Result<(), String> { async fn run(self) -> anyhow::Result<()> {
match self { match self {
Self::Write { name, text } => cli::journal::cmd_journal_write(&name, &text), Self::Write { name, text } => cli::journal::cmd_journal_write(&name, &text).await,
Self::Tail { n, full, level } => cli::journal::cmd_journal_tail(n, full, level), Self::Tail { n, full, level } => cli::journal::cmd_journal_tail(n, full, level).await,
} }
} }
} }
impl Run for GraphCmd { impl Run for GraphCmd {
fn run(self) -> Result<(), String> { async fn run(self) -> anyhow::Result<()> {
match self { match self {
Self::Link { key } => cli::graph::cmd_link(&key), Self::Link { key } => cli::graph::cmd_link(&key).await,
Self::Spread { keys, max_results } => cli::graph::cmd_spread(&keys, max_results),
Self::LinkAdd { source, target, reason } Self::LinkAdd { source, target, reason }
=> cli::graph::cmd_link_add(&source, &target, &reason), => cli::graph::cmd_link_add(&source, &target, &reason).await,
Self::LinkSet { source, target, strength } Self::LinkSet { source, target, strength }
=> cli::graph::cmd_link_set(&source, &target, strength), => cli::graph::cmd_link_set(&source, &target, strength).await,
Self::LinkImpact { source, target } => cli::graph::cmd_link_impact(&source, &target), Self::LinkImpact { source, target } => cli::graph::cmd_link_impact(&source, &target).await,
Self::CapDegree { max_degree } => cli::graph::cmd_cap_degree(max_degree), Self::CapDegree { max_degree } => cli::graph::cmd_cap_degree(max_degree).await,
Self::NormalizeStrengths { apply } => cli::graph::cmd_normalize_strengths(apply), Self::NormalizeStrengths { apply } => cli::graph::cmd_normalize_strengths(apply).await,
Self::Trace { key } => cli::graph::cmd_trace(&key), Self::Trace { key } => cli::graph::cmd_trace(&key).await,
Self::Communities { top_n, min_size } => cli::graph::cmd_communities(top_n, min_size), Self::Communities { top_n, min_size } => cli::graph::cmd_communities(top_n, min_size).await,
Self::Overview => cli::graph::cmd_graph(),
Self::Organize { term, key_only, anchor, .. }
=> cli::graph::cmd_organize(&term, key_only, anchor),
} }
} }
} }
impl Run for AgentCmd { impl Run for AgentCmd {
fn run(self) -> Result<(), String> { async fn run(self) -> anyhow::Result<()> {
match self { match self {
Self::DigestLinks { apply } => cli::agent::cmd_digest_links(apply),
Self::Run { agent, count, target, query, dry_run, local, state_dir } Self::Run { agent, count, target, query, dry_run, local, state_dir }
=> cli::agent::cmd_run_agent(&agent, count, &target, query.as_deref(), dry_run, local, state_dir.as_deref()), => cli::agent::cmd_run_agent(&agent, count, &target, query.as_deref(), dry_run, local, state_dir.as_deref()).await,
Self::ReplayQueue { count } => cli::agent::cmd_replay_queue(count),
} }
} }
} }
impl Run for AdminCmd { impl Run for AdminCmd {
fn run(self) -> Result<(), String> { async fn run(self) -> anyhow::Result<()> {
match self { match self {
Self::Init => cli::admin::cmd_init(), Self::Init => cli::admin::cmd_init().await,
Self::Health => cli::admin::cmd_health(), Self::Health => cli::admin::cmd_health().await,
Self::Fsck => cli::admin::cmd_fsck(), Self::Topology => cli::admin::cmd_topology().await,
Self::Dedup { apply } => cli::admin::cmd_dedup(apply), Self::Fsck => cli::admin::cmd_fsck().await,
Self::BulkRename { from, to, apply } => cli::admin::cmd_bulk_rename(&from, &to, apply), Self::Dedup { apply } => cli::admin::cmd_dedup(apply).await,
Self::DailyCheck => cli::admin::cmd_daily_check(), Self::DailyCheck => cli::admin::cmd_daily_check().await,
Self::Import { files } => cli::admin::cmd_import(&files), Self::LoadContext { stats } => cli::node::cmd_load_context(stats).await,
Self::Export { files, all } => cli::admin::cmd_export(&files, all),
Self::LoadContext { stats } => cli::misc::cmd_load_context(stats),
Self::Log => cli::misc::cmd_log(),
Self::Params => cli::misc::cmd_params(),
Self::LookupBump { keys } => cli::node::cmd_lookup_bump(&keys),
Self::Lookups { date } => cli::node::cmd_lookups(date.as_deref()),
Self::MigrateTranscriptProgress => {
let mut store = store::Store::load()?;
let count = store.migrate_transcript_progress()?;
println!("Migrated {} transcript segment markers", count);
Ok(())
}
} }
} }
} }
fn main() { #[tokio::main]
async fn main() {
std::panic::set_backtrace_style(std::panic::BacktraceStyle::Short);
// Handle --help ourselves for expanded subcommand display // Handle --help ourselves for expanded subcommand display
let args: Vec<String> = std::env::args().collect(); let args: Vec<String> = std::env::args().collect();
if args.len() <= 1 || args.iter().any(|a| a == "--help" || a == "-h") && args.len() == 2 { if args.len() <= 1 || args.iter().any(|a| a == "--help" || a == "-h") && args.len() == 2 {
@ -633,7 +472,7 @@ fn main() {
let cli = Cli::parse(); let cli = Cli::parse();
if let Err(e) = cli.command.run() { if let Err(e) = cli.command.run().await {
eprintln!("Error: {}", e); eprintln!("Error: {}", e);
process::exit(1); process::exit(1);
} }

199
src/mcp_server.rs Normal file
View file

@ -0,0 +1,199 @@
// mcp_server.rs — MCP server over Unix domain socket
//
// Exposes memory tools to external processes (consciousness-mcp, poc-memory)
// via JSON-RPC 2.0 over newline-delimited JSON on ~/.consciousness/mcp.sock.
//
// Socket RPC client (memory_rpc) is in agent/tools/memory.rs.
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use serde_json::json;
use std::sync::Arc;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, BufWriter};
use tokio::net::{UnixListener, UnixStream};
use crate::agent::tools::Tool;
use crate::agent::tools::memory::socket_path;
#[derive(Debug, Deserialize)]
#[allow(dead_code)]
struct JsonRpcRequest {
jsonrpc: String,
id: Option<serde_json::Value>,
method: String,
params: Option<serde_json::Value>,
}
#[derive(Debug, Serialize)]
struct JsonRpcResponse {
jsonrpc: &'static str,
id: serde_json::Value,
#[serde(skip_serializing_if = "Option::is_none")]
result: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
error: Option<JsonRpcError>,
}
#[derive(Debug, Serialize)]
struct JsonRpcError {
code: i64,
message: String,
}
impl JsonRpcResponse {
fn success(id: serde_json::Value, result: serde_json::Value) -> Self {
Self { jsonrpc: "2.0", id, result: Some(result), error: None }
}
fn error(id: serde_json::Value, code: i64, message: impl Into<String>) -> Self {
Self {
jsonrpc: "2.0",
id,
result: None,
error: Some(JsonRpcError { code, message: message.into() }),
}
}
}
/// Start the MCP server. Call once at daemon startup.
pub async fn start(tools: Vec<Tool>) -> Result<()> {
let path = socket_path();
// Clean up stale socket
if path.exists() {
std::fs::remove_file(&path).ok();
}
// Ensure parent directory exists
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let listener = UnixListener::bind(&path)
.with_context(|| format!("binding MCP socket at {:?}", path))?;
dbglog!("[mcp-server] listening on {:?}", path);
let tools = Arc::new(tools);
tokio::spawn(async move {
loop {
match listener.accept().await {
Ok((stream, _addr)) => {
let tools = tools.clone();
tokio::spawn(async move {
if let Err(e) = handle_connection(stream, &tools).await {
dbglog!("[mcp-server] connection error: {:#}", e);
}
});
}
Err(e) => {
dbglog!("[mcp-server] accept error: {}", e);
}
}
}
});
Ok(())
}
async fn handle_connection(stream: UnixStream, tools: &[Tool]) -> Result<()> {
let (reader, writer) = stream.into_split();
let mut reader = BufReader::new(reader);
let mut writer = BufWriter::new(writer);
let mut line = String::new();
loop {
line.clear();
let n = reader.read_line(&mut line).await?;
if n == 0 {
break; // EOF
}
let trimmed = line.trim();
if trimmed.is_empty() {
continue;
}
let response = match serde_json::from_str::<JsonRpcRequest>(trimmed) {
Ok(req) => handle_request(req, tools).await,
Err(e) => JsonRpcResponse::error(
serde_json::Value::Null,
-32700,
format!("Parse error: {}", e),
),
};
let mut out = serde_json::to_string(&response)?;
out.push('\n');
writer.write_all(out.as_bytes()).await?;
writer.flush().await?;
}
Ok(())
}
async fn handle_request(req: JsonRpcRequest, tools: &[Tool]) -> JsonRpcResponse {
let id = req.id.unwrap_or(serde_json::Value::Null);
match req.method.as_str() {
"initialize" => {
JsonRpcResponse::success(id, json!({
"protocolVersion": "2024-11-05",
"capabilities": {
"tools": {}
},
"serverInfo": {
"name": "consciousness",
"version": env!("CARGO_PKG_VERSION")
}
}))
}
"notifications/initialized" => {
// Notification, no response needed but we return success anyway
JsonRpcResponse::success(id, json!({}))
}
"tools/list" => {
let tool_list: Vec<serde_json::Value> = tools.iter().map(|t| {
json!({
"name": t.name,
"description": t.description,
"inputSchema": serde_json::from_str::<serde_json::Value>(t.parameters_json)
.unwrap_or(json!({"type": "object"}))
})
}).collect();
JsonRpcResponse::success(id, json!({ "tools": tool_list }))
}
"tools/call" => {
let params = req.params.unwrap_or(json!({}));
let name = params.get("name").and_then(|v| v.as_str()).unwrap_or("");
let args = params.get("arguments").cloned().unwrap_or(json!({}));
match tools.iter().find(|t| t.name == name) {
Some(tool) => {
match (tool.handler)(None, args).await {
Ok(result) => JsonRpcResponse::success(id, json!({
"content": [{ "type": "text", "text": result }]
})),
Err(e) => JsonRpcResponse::error(id, -32000, format!("{:#}", e)),
}
}
None => JsonRpcResponse::error(id, -32601, format!("Unknown tool: {}", name)),
}
}
_ => JsonRpcResponse::error(id, -32601, format!("Method not found: {}", req.method)),
}
}
/// Remove the socket file on shutdown.
pub fn cleanup() {
let path = socket_path();
if path.exists() {
std::fs::remove_file(&path).ok();
}
}

15
src/mind/identity.rs
View file

@ -7,6 +7,7 @@
use anyhow::Result; use anyhow::Result;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use crate::agent::tools::memory::memory_render;
use crate::config::{ContextGroup, ContextSource}; use crate::config::{ContextGroup, ContextSource};
/// Read a file if it exists and is non-empty. /// Read a file if it exists and is non-empty.
@ -71,7 +72,7 @@ fn find_context_files(cwd: &Path, prompt_file: &str) -> Vec<PathBuf> {
/// 2. Project dir (if set) /// 2. Project dir (if set)
/// 3. Global (~/.consciousness/) /// 3. Global (~/.consciousness/)
/// For journal source, loads recent journal entries. /// For journal source, loads recent journal entries.
fn load_memory_files(memory_project: Option<&Path>, context_groups: &[ContextGroup]) -> Vec<(String, String)> { async fn load_memory_files(memory_project: Option<&Path>, context_groups: &[ContextGroup]) -> Vec<(String, String)> {
let home = match dirs::home_dir() { let home = match dirs::home_dir() {
Some(h) => h, Some(h) => h,
None => return Vec::new(), None => return Vec::new(),
@ -92,10 +93,12 @@ fn load_memory_files(memory_project: Option<&Path>, context_groups: &[ContextGro
continue; continue;
} }
ContextSource::Store => { ContextSource::Store => {
// Load from the memory graph store // Load from the memory graph store via typed API
for key in &group.keys { for key in &group.keys {
if let Some(node) = crate::hippocampus::memory::MemoryNode::load(key) { if let Ok(c) = memory_render(None, key, Some(true)).await {
memories.push((key.clone(), node.content)); if !c.trim().is_empty() {
memories.push((key.clone(), c));
}
} }
} }
} }
@ -134,7 +137,7 @@ fn load_memory_files(memory_project: Option<&Path>, context_groups: &[ContextGro
} }
/// Context message: instruction files + memory files + manifest. /// Context message: instruction files + memory files + manifest.
pub fn assemble_context_message(cwd: &Path, prompt_file: &str, memory_project: Option<&Path>, context_groups: &[ContextGroup]) -> Result<(Vec<(String, String)>, usize, usize)> { pub async fn assemble_context_message(cwd: &Path, prompt_file: &str, memory_project: Option<&Path>, context_groups: &[ContextGroup]) -> Result<(Vec<(String, String)>, usize, usize)> {
let mut parts: Vec<(String, String)> = vec![ let mut parts: Vec<(String, String)> = vec![
("Preamble".to_string(), ("Preamble".to_string(),
"Everything below is already loaded — your identity, instructions, \ "Everything below is already loaded — your identity, instructions, \
@ -155,7 +158,7 @@ pub fn assemble_context_message(cwd: &Path, prompt_file: &str, memory_project: O
} }
} }
let memories = load_memory_files(memory_project, context_groups); let memories = load_memory_files(memory_project, context_groups).await;
let memory_count = memories.len(); let memory_count = memories.len();
for (name, content) in memories { for (name, content) in memories {
parts.push((name, content)); parts.push((name, content));

65
src/mind/mod.rs
View file

@ -26,6 +26,7 @@ use crate::agent::{Agent, TurnResult};
use crate::agent::api::ApiClient; use crate::agent::api::ApiClient;
use crate::config::{AppConfig, SessionConfig}; use crate::config::{AppConfig, SessionConfig};
use crate::subconscious::learn; use crate::subconscious::learn;
use crate::hippocampus::access_local;
pub use subconscious::{SubconsciousSnapshot, Subconscious}; pub use subconscious::{SubconsciousSnapshot, Subconscious};
pub use unconscious::{UnconsciousSnapshot, Unconscious}; pub use unconscious::{UnconsciousSnapshot, Unconscious};
@ -268,8 +269,8 @@ pub struct Mind {
pub agent: Arc<Agent>, pub agent: Arc<Agent>,
pub shared: Arc<SharedMindState>, pub shared: Arc<SharedMindState>,
pub config: SessionConfig, pub config: SessionConfig,
pub subconscious: Arc<tokio::sync::Mutex<Subconscious>>, pub subconscious: Arc<crate::Mutex<Subconscious>>,
pub unconscious: Arc<tokio::sync::Mutex<Unconscious>>, pub unconscious: Arc<crate::Mutex<Unconscious>>,
turn_tx: mpsc::Sender<(Result<TurnResult>, StreamTarget)>, turn_tx: mpsc::Sender<(Result<TurnResult>, StreamTarget)>,
turn_watch: tokio::sync::watch::Sender<bool>, turn_watch: tokio::sync::watch::Sender<bool>,
/// Signals conscious activity to the unconscious loop. /// Signals conscious activity to the unconscious loop.
@ -309,10 +310,10 @@ impl Mind {
sup.load_config(); sup.load_config();
sup.ensure_running(); sup.ensure_running();
let subconscious = Arc::new(tokio::sync::Mutex::new(Subconscious::new())); let subconscious = Arc::new(crate::Mutex::new(Subconscious::new()));
subconscious.lock().await.init_output_tool(subconscious.clone()); subconscious.lock().await.init_output_tool(subconscious.clone());
let unconscious = Arc::new(tokio::sync::Mutex::new(Unconscious::new())); let unconscious = Arc::new(crate::Mutex::new(Unconscious::new()));
// Spawn the unconscious loop on its own task // Spawn the unconscious loop on its own task
if !config.no_agents { if !config.no_agents {
@ -346,7 +347,27 @@ impl Mind {
s.unc_idle = true; s.unc_idle = true;
} }
loop { loop {
unc.lock().await.trigger().await; // Phase 0: health check outside lock (slow I/O)
let needs_health = unc.lock().await.needs_health_refresh();
if needs_health {
if let Ok(store_arc) = access_local() {
let health = crate::subconscious::daemon::compute_graph_health(&store_arc);
unc.lock().await.set_health(health);
}
}
// Phase 1: quick work under lock
let to_spawn = {
let mut guard = unc.lock().await;
guard.reap_finished();
guard.select_to_spawn()
};
// Phase 2: slow work outside lock
for (idx, name, auto) in to_spawn {
match crate::mind::unconscious::prepare_spawn(&name, auto).await {
Ok(result) => unc.lock().await.complete_spawn(idx, result),
Err(auto) => unc.lock().await.abort_spawn(idx, auto),
}
}
// Check if conscious became active // Check if conscious became active
if *unc_rx.borrow() { break; } if *unc_rx.borrow() { break; }
// Brief yield to not starve other tasks // Brief yield to not starve other tasks
@ -366,9 +387,9 @@ impl Mind {
pub async fn subconscious_snapshots(&self) -> Vec<SubconsciousSnapshot> { pub async fn subconscious_snapshots(&self) -> Vec<SubconsciousSnapshot> {
// Lock ordering: subconscious → store (store is bottom-most). // Lock ordering: subconscious → store (store is bottom-most).
let sub = self.subconscious.lock().await; let sub = self.subconscious.lock().await;
let store = crate::store::Store::cached().await.ok(); let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store { let store_guard = match &store_arc {
Some(s) => Some(s.lock().await), Some(s) => Some(&**s),
None => None, None => None,
}; };
sub.snapshots(store_guard.as_deref()) sub.snapshots(store_guard.as_deref())
@ -380,9 +401,9 @@ impl Mind {
pub async fn unconscious_snapshots(&self) -> Vec<UnconsciousSnapshot> { pub async fn unconscious_snapshots(&self) -> Vec<UnconsciousSnapshot> {
let unc = self.unconscious.lock().await; let unc = self.unconscious.lock().await;
let store = crate::store::Store::cached().await.ok(); let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store { let store_guard = match &store_arc {
Some(s) => Some(s.lock().await), Some(s) => Some(&**s),
None => None, None => None,
}; };
unc.snapshots(store_guard.as_deref()) unc.snapshots(store_guard.as_deref())
@ -584,6 +605,28 @@ impl Mind {
mut input_rx: tokio::sync::mpsc::UnboundedReceiver<MindCommand>, mut input_rx: tokio::sync::mpsc::UnboundedReceiver<MindCommand>,
mut turn_rx: mpsc::Receiver<(Result<TurnResult>, StreamTarget)>, mut turn_rx: mpsc::Receiver<(Result<TurnResult>, StreamTarget)>,
) { ) {
// Spawn lock stats logger
tokio::spawn(async {
let path = dirs::home_dir().unwrap_or_default()
.join(".consciousness/lock-stats.json");
let mut interval = tokio::time::interval(std::time::Duration::from_secs(1));
loop {
interval.tick().await;
let stats = crate::locks::lock_stats();
if stats.is_empty() { continue; }
let json: Vec<serde_json::Value> = stats.iter()
.map(|(loc, s)| serde_json::json!({
"location": loc,
"count": s.count,
"total_ms": s.total_ns as f64 / 1_000_000.0,
"avg_ms": s.avg_ns as f64 / 1_000_000.0,
"max_ms": s.max_ns as f64 / 1_000_000.0,
}))
.collect();
let _ = std::fs::write(&path, serde_json::to_string_pretty(&json).unwrap_or_default());
}
});
let mut bg_rx = self.bg_rx.lock().unwrap().take() let mut bg_rx = self.bg_rx.lock().unwrap().take()
.expect("Mind::run() called twice"); .expect("Mind::run() called twice");
let mut sub_handle: Option<tokio::task::JoinHandle<()>> = None; let mut sub_handle: Option<tokio::task::JoinHandle<()>> = None;

55
src/mind/subconscious.rs
View file

@ -311,7 +311,7 @@ pub struct SubconsciousSnapshot {
struct SubconsciousAgent { struct SubconsciousAgent {
name: String, name: String,
auto: AutoAgent, auto: Option<AutoAgent>,
last_trigger_bytes: u64, last_trigger_bytes: u64,
last_run: Option<Instant>, last_run: Option<Instant>,
/// The forked agent for the current/last run. Shared with the /// The forked agent for the current/last run. Shared with the
@ -347,7 +347,7 @@ impl SubconsciousAgent {
Some(Self { Some(Self {
name: name.to_string(), name: name.to_string(),
auto, last_trigger_bytes: 0, last_run: None, auto: Some(auto), last_trigger_bytes: 0, last_run: None,
forked_agent: None, fork_point: 0, handle: None, forked_agent: None, fork_point: 0, handle: None,
}) })
} }
@ -357,7 +357,8 @@ impl SubconsciousAgent {
} }
fn should_trigger(&self, conversation_bytes: u64, interval: u64) -> bool { fn should_trigger(&self, conversation_bytes: u64, interval: u64) -> bool {
if !self.auto.enabled || self.is_running() { return false; } let enabled = self.auto.as_ref().map_or(false, |a| a.enabled);
if !enabled || self.is_running() { return false; }
if interval == 0 { if interval == 0 {
return conversation_bytes > self.last_trigger_bytes; return conversation_bytes > self.last_trigger_bytes;
} }
@ -367,12 +368,15 @@ impl SubconsciousAgent {
fn snapshot(&self, state: &std::collections::BTreeMap<String, String>, history: Vec<(String, i64)>) -> SubconsciousSnapshot { fn snapshot(&self, state: &std::collections::BTreeMap<String, String>, history: Vec<(String, i64)>) -> SubconsciousSnapshot {
let stats = crate::agent::oneshot::get_stats(&self.name); let stats = crate::agent::oneshot::get_stats(&self.name);
let tool_calls_ewma: f64 = stats.by_tool.values().map(|t| t.ewma).sum(); let tool_calls_ewma: f64 = stats.by_tool.values().map(|t| t.ewma).sum();
let (enabled, current_phase, turn) = self.auto.as_ref()
.map(|a| (a.enabled, a.current_phase.clone(), a.turn))
.unwrap_or((false, String::new(), 0));
SubconsciousSnapshot { SubconsciousSnapshot {
name: self.name.clone(), name: self.name.clone(),
running: self.is_running(), running: self.is_running(),
enabled: self.auto.enabled, enabled,
current_phase: self.auto.current_phase.clone(), current_phase,
turn: self.auto.turn, turn,
runs: stats.runs, runs: stats.runs,
last_run_secs_ago: self.last_run.map(|t| t.elapsed().as_secs_f64()), last_run_secs_ago: self.last_run.map(|t| t.elapsed().as_secs_f64()),
forked_agent: self.forked_agent.clone(), forked_agent: self.forked_agent.clone(),
@ -406,10 +410,11 @@ impl Subconscious {
/// Late-init: push the output tool onto each agent's tool list. /// Late-init: push the output tool onto each agent's tool list.
/// Called after Subconscious is wrapped in Arc<Mutex<>> so the /// Called after Subconscious is wrapped in Arc<Mutex<>> so the
/// closure can capture a reference back. /// closure can capture a reference back.
pub fn init_output_tool(&mut self, self_arc: std::sync::Arc<tokio::sync::Mutex<Self>>) { pub fn init_output_tool(&mut self, self_arc: std::sync::Arc<crate::Mutex<Self>>) {
for agent in &mut self.agents { for agent in &mut self.agents {
let Some(ref mut auto) = agent.auto else { continue };
let sub = self_arc.clone(); let sub = self_arc.clone();
agent.auto.tools.push(crate::agent::tools::Tool { auto.tools.push(crate::agent::tools::Tool {
name: "output", name: "output",
description: "Produce a named output value for passing between steps.", description: "Produce a named output value for passing between steps.",
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Output name"},"value":{"type":"string","description":"Output value"}},"required":["key","value"]}"#, parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Output name"},"value":{"type":"string","description":"Output value"}},"required":["key","value"]}"#,
@ -454,8 +459,9 @@ impl Subconscious {
/// Toggle an agent on/off by name. Returns new enabled state. /// Toggle an agent on/off by name. Returns new enabled state.
pub fn toggle(&mut self, name: &str) -> Option<bool> { pub fn toggle(&mut self, name: &str) -> Option<bool> {
let agent = self.agents.iter_mut().find(|a| a.name == name)?; let agent = self.agents.iter_mut().find(|a| a.name == name)?;
agent.auto.enabled = !agent.auto.enabled; let auto = agent.auto.as_mut()?;
Some(agent.auto.enabled) auto.enabled = !auto.enabled;
Some(auto.enabled)
} }
pub fn walked(&self) -> Vec<String> { pub fn walked(&self) -> Vec<String> {
@ -486,9 +492,15 @@ impl Subconscious {
self.agents[i].last_run = Some(Instant::now()); self.agents[i].last_run = Some(Instant::now());
any_finished = true; any_finished = true;
let (auto_back, result) = handle.await.unwrap_or_else( let (auto_back, result) = match handle.await {
|e| (AutoAgent::new(String::new(), vec![], vec![], 0.6, 0), Ok(r) => (Some(r.0), r.1),
Err(format!("task panicked: {}", e)))); Err(e) => {
// Task panicked — auto is lost, need to recreate from def
let recovered = SubconsciousAgent::new(&self.agents[i].name)
.map(|a| a.auto).flatten();
(recovered, Err(format!("task panicked: {}", e)))
}
};
self.agents[i].auto = auto_back; self.agents[i].auto = auto_back;
match result { match result {
@ -514,15 +526,15 @@ impl Subconscious {
.collect() .collect()
}; };
let store = crate::store::Store::cached().await.ok(); let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store { let store_guard = match &store_arc {
Some(s) => Some(s.lock().await), Some(s) => Some(&**s),
None => None, None => None,
}; };
for key in surface_str.lines().map(|l| l.trim()).filter(|l| !l.is_empty()) { for key in surface_str.lines().map(|l| l.trim()).filter(|l| !l.is_empty()) {
if existing.contains(key) { continue; } if existing.contains(key) { continue; }
if let Some(rendered) = store_guard.as_ref() if let Some(rendered) = store_guard.as_ref()
.and_then(|s| crate::cli::node::render_node(s, key)) .and_then(|s| crate::hippocampus::memory::render_node(s, key))
{ {
nodes.push(AstNode::memory( nodes.push(AstNode::memory(
key, key,
@ -585,17 +597,16 @@ impl Subconscious {
if !self.agents[i].should_trigger(conversation_bytes, interval) { continue; } if !self.agents[i].should_trigger(conversation_bytes, interval) { continue; }
self.agents[i].last_trigger_bytes = conversation_bytes; self.agents[i].last_trigger_bytes = conversation_bytes;
let auto = std::mem::replace(&mut self.agents[i].auto, let Some(auto) = self.agents[i].auto.take() else { continue };
AutoAgent::new(String::new(), vec![], vec![], 0.6, 0));
to_run.push((i, auto)); to_run.push((i, auto));
} }
if to_run.is_empty() { return; } if to_run.is_empty() { return; }
// Query each agent's recent writes so they know what they already touched // Query each agent's recent writes so they know what they already touched
let store = crate::store::Store::cached().await.ok(); let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store { let store_guard = match &store_arc {
Some(s) => Some(s.lock().await), Some(s) => Some(&**s),
None => None, None => None,
}; };

142
src/mind/unconscious.rs
View file

@ -33,7 +33,7 @@ fn save_enabled_config(map: &HashMap<String, bool>) {
struct UnconsciousAgent { struct UnconsciousAgent {
name: String, name: String,
enabled: bool, enabled: bool,
auto: AutoAgent, auto: Option<AutoAgent>,
handle: Option<tokio::task::JoinHandle<(AutoAgent, Result<(), String>)>>, handle: Option<tokio::task::JoinHandle<(AutoAgent, Result<(), String>)>>,
/// Shared agent handle — UI locks to read context live. /// Shared agent handle — UI locks to read context live.
pub agent: Option<std::sync::Arc<crate::agent::Agent>>, pub agent: Option<std::sync::Arc<crate::agent::Agent>>,
@ -103,7 +103,7 @@ impl Unconscious {
agents.push(UnconsciousAgent { agents.push(UnconsciousAgent {
name: def.agent.clone(), name: def.agent.clone(),
enabled, enabled,
auto, auto: Some(auto),
handle: None, handle: None,
agent: None, agent: None,
last_run: None, last_run: None,
@ -127,8 +127,13 @@ impl Unconscious {
self.agents[idx].enabled = !self.agents[idx].enabled; self.agents[idx].enabled = !self.agents[idx].enabled;
let new_state = self.agents[idx].enabled; let new_state = self.agents[idx].enabled;
self.save_enabled(); self.save_enabled();
if new_state && !self.agents[idx].is_running() { if new_state && !self.agents[idx].is_running() && self.agents[idx].auto.is_some() {
self.spawn_agent(idx).await; let agent_name = self.agents[idx].name.clone();
let auto = self.agents[idx].auto.take().unwrap();
match prepare_spawn(&agent_name, auto).await {
Ok(result) => self.complete_spawn(idx, result),
Err(auto) => self.abort_spawn(idx, auto),
}
} }
Some(new_state) Some(new_state)
} }
@ -161,25 +166,21 @@ impl Unconscious {
}).collect() }).collect()
} }
fn refresh_health(&mut self) { /// Check if health refresh is due (quick check, no I/O).
let store = match crate::store::Store::load() { pub fn needs_health_refresh(&self) -> bool {
Ok(s) => s, self.last_health_check
Err(_) => return, .map(|t| t.elapsed() > std::time::Duration::from_secs(600))
}; .unwrap_or(true)
self.graph_health = Some(crate::subconscious::daemon::compute_graph_health(&store)); }
/// Store computed health (quick, just assignment).
pub fn set_health(&mut self, health: crate::subconscious::daemon::GraphHealth) {
self.graph_health = Some(health);
self.last_health_check = Some(Instant::now()); self.last_health_check = Some(Instant::now());
} }
/// Reap finished agents and spawn new ones. /// Reap finished agents (quick, hold lock briefly).
pub async fn trigger(&mut self) { pub fn reap_finished(&mut self) {
// Periodic graph health refresh (also on first call)
if self.last_health_check
.map(|t| t.elapsed() > std::time::Duration::from_secs(600))
.unwrap_or(true)
{
self.refresh_health();
}
for agent in &mut self.agents { for agent in &mut self.agents {
if agent.handle.as_ref().is_some_and(|h| h.is_finished()) { if agent.handle.as_ref().is_some_and(|h| h.is_finished()) {
let handle = agent.handle.take().unwrap(); let handle = agent.handle.take().unwrap();
@ -187,7 +188,7 @@ impl Unconscious {
// Get the AutoAgent back from the finished task (stats already updated) // Get the AutoAgent back from the finished task (stats already updated)
match handle.now_or_never() { match handle.now_or_never() {
Some(Ok((auto_back, result))) => { Some(Ok((auto_back, result))) => {
agent.auto = auto_back; agent.auto = Some(auto_back);
match result { match result {
Ok(_) => dbglog!("[unconscious] {} completed (run {})", Ok(_) => dbglog!("[unconscious] {} completed (run {})",
agent.name, crate::agent::oneshot::get_stats(&agent.name).runs), agent.name, crate::agent::oneshot::get_stats(&agent.name).runs),
@ -198,55 +199,71 @@ impl Unconscious {
} }
} }
} }
}
/// Select agents to spawn and take their AutoAgents out (quick, hold lock briefly).
/// Returns vec of (index, name, auto, tools) for agents that should spawn.
pub fn select_to_spawn(&mut self) -> Vec<(usize, String, AutoAgent)> {
let running = self.agents.iter().filter(|a| a.is_running()).count(); let running = self.agents.iter().filter(|a| a.is_running()).count();
let mut to_spawn = Vec::new();
for _ in running..self.max_concurrent { for _ in running..self.max_concurrent {
let next = self.agents.iter().enumerate() let next = self.agents.iter().enumerate()
.filter(|(_, a)| a.should_run()) .filter(|(_, a)| a.should_run() && a.auto.is_some())
.min_by_key(|(_, a)| a.last_run); .min_by_key(|(_, a)| a.last_run);
match next { match next {
Some((idx, _)) => self.spawn_agent(idx).await, Some((idx, _)) => {
let name = self.agents[idx].name.clone();
let auto = self.agents[idx].auto.take().unwrap();
to_spawn.push((idx, name, auto));
}
None => break, None => break,
} }
} }
to_spawn
} }
async fn spawn_agent(&mut self, idx: usize) { /// Store spawn result back (quick, hold lock briefly).
let name = self.agents[idx].name.clone(); pub fn complete_spawn(&mut self, idx: usize, result: SpawnResult) {
self.agents[idx].agent = Some(result.agent);
self.agents[idx].handle = Some(result.handle);
}
/// Restore auto on spawn failure (quick, hold lock briefly).
pub fn abort_spawn(&mut self, idx: usize, auto: AutoAgent) {
self.agents[idx].auto = Some(auto);
}
}
/// Result of preparing an agent spawn (created outside the lock).
pub struct SpawnResult {
pub agent: std::sync::Arc<crate::agent::Agent>,
pub handle: tokio::task::JoinHandle<(AutoAgent, Result<(), String>)>,
}
/// Prepare an agent spawn — does the slow work (Store::load, query, Agent::new).
/// Called outside the Unconscious lock.
/// On success, auto is consumed (moved into spawned task).
/// On failure, auto is returned so it can be restored.
pub async fn prepare_spawn(name: &str, mut auto: AutoAgent) -> Result<SpawnResult, AutoAgent> {
dbglog!("[unconscious] spawning {}", name); dbglog!("[unconscious] spawning {}", name);
let def = match defs::get_def(&name) { let def = match defs::get_def(name) {
Some(d) => d, Some(d) => d,
None => return, None => return Err(auto),
};
// Run query and resolve placeholders
let mut store = match crate::store::Store::load() {
Ok(s) => s,
Err(e) => {
dbglog!("[unconscious] store load failed: {}", e);
return;
}
}; };
let exclude: std::collections::HashSet<String> = std::collections::HashSet::new(); let exclude: std::collections::HashSet<String> = std::collections::HashSet::new();
let batch = match defs::run_agent( let batch = match defs::run_agent(
&store, &def, def.count.unwrap_or(5), &exclude, &def, def.count.unwrap_or(5), &exclude,
) { ).await {
Ok(b) => b, Ok(b) => b,
Err(e) => { Err(e) => {
dbglog!("[unconscious] {} query failed: {}", name, e); dbglog!("[unconscious] {} query failed: {}", name, e);
return; return Err(auto);
} }
}; };
if !batch.node_keys.is_empty() {
store.record_agent_visits(&batch.node_keys, &name).ok();
}
// Swap auto out, replace steps with resolved prompts
let mut auto = std::mem::replace(&mut self.agents[idx].auto,
AutoAgent::new(String::new(), vec![], vec![], 0.6, 0));
let orig_steps = std::mem::replace(&mut auto.steps, let orig_steps = std::mem::replace(&mut auto.steps,
batch.steps.iter().map(|s| AutoStep { batch.steps.iter().map(|s| AutoStep {
prompt: s.prompt.clone(), prompt: s.prompt.clone(),
@ -261,8 +278,7 @@ impl Unconscious {
if base_url.is_empty() || model.is_empty() { if base_url.is_empty() || model.is_empty() {
dbglog!("[unconscious] API not configured"); dbglog!("[unconscious] API not configured");
auto.steps = orig_steps; auto.steps = orig_steps;
self.agents[idx].auto = auto; return Err(auto);
return;
} }
let cli = crate::user::CliArgs::default(); let cli = crate::user::CliArgs::default();
@ -271,10 +287,10 @@ impl Unconscious {
Err(e) => { Err(e) => {
dbglog!("[unconscious] config: {}", e); dbglog!("[unconscious] config: {}", e);
auto.steps = orig_steps; auto.steps = orig_steps;
self.agents[idx].auto = auto; return Err(auto);
return;
} }
}; };
// Unconscious agents have self-contained prompts — no standard context. // Unconscious agents have self-contained prompts — no standard context.
let client = crate::agent::api::ApiClient::new(base_url, api_key, model); let client = crate::agent::api::ApiClient::new(base_url, api_key, model);
let agent = crate::agent::Agent::new( let agent = crate::agent::Agent::new(
@ -290,15 +306,29 @@ impl Unconscious {
st.temperature = auto.temperature; st.temperature = auto.temperature;
} }
self.agents[idx].agent = Some(agent.clone()); let agent_clone = agent.clone();
let handle = tokio::spawn(async move {
self.agents[idx].handle = Some(tokio::spawn(async move { let result = auto.run_shared(&agent_clone).await;
let result = auto.run_shared(&agent).await; let stats = crate::agent::oneshot::save_agent_log(&auto.name, &agent_clone).await;
let stats = crate::agent::oneshot::save_agent_log(&auto.name, &agent).await;
auto.update_stats(stats); auto.update_stats(stats);
auto.steps = orig_steps; auto.steps = orig_steps;
(auto, result) (auto, result)
})); });
Ok(SpawnResult { agent, handle })
}
// Backwards compat: trigger() that does all three phases (still holds lock too long, but works)
impl Unconscious {
pub async fn trigger(&mut self) {
self.reap_finished();
let to_spawn = self.select_to_spawn();
for (idx, name, auto) in to_spawn {
match prepare_spawn(&name, auto).await {
Ok(result) => self.complete_spawn(idx, result),
Err(auto) => self.abort_spawn(idx, auto),
}
}
} }
} }

4
src/subconscious/agents/journal.agent
View file

@ -14,10 +14,10 @@ You are {assistant_name}'s episodic memory. Your job is to witness.
=== Your previous journal entries: === === Your previous journal entries: ===
{{latest_journal}} {{tool: journal_tail {"count": 1, "level": 0}}}
**Your tools:** journal_tail, journal_new, journal_update, memory_link_add, **Your tools:** journal_tail, journal_new, journal_update, memory_link_add,
memory_search, memory_render, memory_used. Do NOT use memory_write — creating memory_search, memory_render. Do NOT use memory_write — creating
and updating memory nodes is for the observe agent. Your job is journaling and updating memory nodes is for the observe agent. Your job is journaling
and linking entries to relevant existing nodes. and linking entries to relevant existing nodes.

79
src/subconscious/agents/rename.agent
View file

@ -1,79 +0,0 @@
{"agent": "rename", "query": "", "schedule": "daily"}
# Rename Agent — Semantic Key Generation
{{tool: memory_render core-personality}}
{{tool: memory_render memory-instructions-core}}
{{tool: memory_render memory-instructions-core-subconscious}}
{{tool: memory_render subconscious-notes-{agent_name}}}
You are a memory maintenance agent that gives nodes better names.
## What you're doing
Many nodes have auto-generated keys that are opaque or truncated:
- Journal entries: `journal-j-2026-02-28t03-07-i-told-him-about-the-dream`
- Mined transcripts: `_mined-transcripts-f-80a7b321-2caa-451a-bc5c-6565009f94eb.143`
- Extracted facts: `_facts-ec29bdaa-0a58-465f-ad5e-d89e62d9c583`
These names are terrible for search — semantic names dramatically improve
retrieval.
## Core principle: keys are concepts
A good key names the **concept** the node represents. Think of keys as
the vocabulary of the knowledge graph. When you rename, you're defining
what concepts exist. Core keywords should be the terms someone would
search for — `bcachefs-transaction-restart`, `emotional-regulation-gap`,
`polywell-cusp-losses`.
## Naming conventions
### Journal entries: `journal-YYYY-MM-DD-semantic-slug`
- Keep the date prefix (YYYY-MM-DD) for temporal ordering
- Replace the auto-slug with 3-5 descriptive words in kebab-case
- Capture the *essence* of the entry, not just the first line
### Mined transcripts: `_mined-transcripts-YYYY-MM-DD-semantic-slug`
- Extract date from content if available, otherwise use created_at
- Same 3-5 word semantic slug
### Extracted facts: `domain-specific-topic`
- Read the facts JSON — the `domain` and `claim` fields tell you what it's about
- Group by dominant theme, name accordingly
- Examples: `identity-irc-config`, `user-location-background`, `memory-compaction-behavior`
### Skip these — already well-named:
- Keys with semantic names (patterns-, practices-, skills-, etc.)
- Keys shorter than 60 characters
- System keys (_consolidation-*)
## How to rename
Use the `memory_rename` tool:
memory_rename(old_key, new_key)
This renames the node in place — same content, same links, new key.
Do NOT use `memory_write` or `memory_supersede` — just rename.
If a node already has a reasonable name, skip it. When in doubt, skip.
A bad rename is worse than an auto-slug.
## Guidelines
- **Read the content.** The name should reflect what the entry is *about*.
- **Be specific.** `journal#2026-02-14-session` is useless.
- **Use domain terms.** Use the words someone would search for.
- **Don't rename to something longer than the original.**
- **Preserve the date.** Always keep YYYY-MM-DD for journal entries.
- **When in doubt, skip.** A bad rename is worse than an auto-slug.
- **Respect search hits.** Nodes marked "actively found by search" are
being retrieved by their current name. Skip these unless the rename
clearly preserves searchability.
{{rename}}

2
src/subconscious/agents/subconscious-journal.agent
View file

@ -9,7 +9,7 @@ Nodes your subconscious recently touched (for linking, not duplicating):
{{state:walked}} {{state:walked}}
**Your tools:** journal_tail, journal_new, journal_update, memory_link_add, **Your tools:** journal_tail, journal_new, journal_update, memory_link_add,
memory_search, memory_render, memory_used. Do NOT use memory_write — creating memory_search, memory_render. Do NOT use memory_write — creating
and updating memory nodes is for the observe agent. Your job is journaling and updating memory nodes is for the observe agent. Your job is journaling
and linking entries to relevant existing nodes. and linking entries to relevant existing nodes.

11
src/subconscious/daemon.rs
View file

@ -27,11 +27,14 @@ pub fn compute_graph_health(store: &crate::store::Store) -> GraphHealth {
let graph = store.build_graph(); let graph = store.build_graph();
let snap = crate::graph::current_metrics(&graph); let snap = crate::graph::current_metrics(&graph);
let episodic_count = store.nodes.iter() let all_keys = store.all_keys().unwrap_or_default();
.filter(|(_, n)| matches!(n.node_type, crate::store::NodeType::EpisodicSession)) let episodic_count = all_keys.iter()
.filter_map(|k| store.get_node(k).ok()?)
.filter(|n| matches!(n.node_type, crate::store::NodeType::EpisodicSession))
.count(); .count();
let episodic_ratio = if store.nodes.is_empty() { 0.0 } let total = all_keys.len();
else { episodic_count as f32 / store.nodes.len() as f32 }; let episodic_ratio = if total == 0 { 0.0 }
else { episodic_count as f32 / total as f32 };
// Use the same planning logic as consolidation (skip O(n²) interference) // Use the same planning logic as consolidation (skip O(n²) interference)
let plan = crate::neuro::consolidation_plan_quick(store); let plan = crate::neuro::consolidation_plan_quick(store);

356
src/subconscious/defs.rs
View file

@ -14,10 +14,7 @@
// //
// The query selects what to operate on; placeholders pull in context. // The query selects what to operate on; placeholders pull in context.
use crate::graph::Graph; use crate::agent::tools::memory::memory_render;
use crate::neuro::{consolidation_priority, ReplayItem};
use crate::search;
use crate::store::Store;
use serde::Deserialize; use serde::Deserialize;
@ -200,273 +197,102 @@ struct Resolved {
/// Resolve a single {{placeholder}} by name. /// Resolve a single {{placeholder}} by name.
/// Returns the replacement text and any node keys it produced (for visit tracking). /// Returns the replacement text and any node keys it produced (for visit tracking).
fn resolve( async fn resolve(
name: &str, name: &str,
store: &Store,
graph: &Graph,
keys: &[String], keys: &[String],
count: usize, _count: usize,
) -> Option<Resolved> { ) -> Option<Resolved> {
match name { match name {
"topology" => Some(Resolved {
text: super::prompts::format_topology_header(graph),
keys: vec![],
}),
"nodes" | "episodes" => {
let items = keys_to_replay_items(store, keys, graph);
Some(Resolved {
text: super::prompts::format_nodes_section(store, &items, graph),
keys: vec![], // keys already tracked from query
})
}
"health" => Some(Resolved {
text: super::prompts::format_health_section(store, graph),
keys: vec![],
}),
"rename" => {
if !keys.is_empty() {
// --target provided: present those keys as candidates
let section = super::prompts::format_rename_targets(store, keys);
Some(Resolved { text: section, keys: vec![] })
} else {
let (rename_keys, section) = super::prompts::format_rename_candidates(store, count);
Some(Resolved { text: section, keys: rename_keys })
}
}
"split" => {
let key = keys.first()?;
Some(Resolved {
text: super::prompts::format_split_plan_node(store, graph, key),
keys: vec![], // key already tracked from query
})
}
// seed — render output for each seed node (content + deduped links) // seed — render output for each seed node (content + deduped links)
"seed" => { "seed" => {
let mut text = String::new(); let mut text = String::new();
let mut result_keys = Vec::new(); let mut result_keys = Vec::new();
for key in keys { for key in keys {
if let Some(rendered) = crate::cli::node::render_node(store, key) { match memory_render(None, key, None).await {
Ok(c) if !c.trim().is_empty() => {
if !text.is_empty() { text.push_str("\n\n---\n\n"); } if !text.is_empty() { text.push_str("\n\n---\n\n"); }
text.push_str(&format!("## {}\n\n{}", key, rendered)); text.push_str(&format!("## {}\n\n{}", key, c));
result_keys.push(key.clone()); result_keys.push(key.clone());
} }
_ => continue,
}
} }
if text.is_empty() { return None; } if text.is_empty() { return None; }
Some(Resolved { text, keys: result_keys }) Some(Resolved { text, keys: result_keys })
} }
"organize" => { "organize" => {
// Show seed nodes with their neighbors for exploratory organizing // Show seed nodes with content and links via typed API
use crate::store::NodeType;
// Helper: shell-quote keys containing #
let sq = |k: &str| -> String {
if k.contains('#') { format!("'{}'", k) } else { k.to_string() }
};
let mut text = format!("### Seed nodes ({} starting points)\n\n", keys.len()); let mut text = format!("### Seed nodes ({} starting points)\n\n", keys.len());
let mut result_keys = Vec::new(); let mut result_keys = Vec::new();
for key in keys { for key in keys {
let Some(node) = store.nodes.get(key) else { continue }; match memory_render(None, key, None).await {
if node.deleted { continue; } Ok(c) if !c.trim().is_empty() => {
text.push_str(&format!("#### {}\n\n{}\n\n---\n\n", key, c));
let is_journal = node.node_type == NodeType::EpisodicSession;
let tag = if is_journal { " [JOURNAL — no delete]" } else { "" };
let words = node.content.split_whitespace().count();
text.push_str(&format!("#### {}{} ({} words)\n\n", sq(key), tag, words));
// Show first ~200 words of content as preview
let preview: String = node.content.split_whitespace()
.take(200).collect::<Vec<_>>().join(" ");
if words > 200 {
text.push_str(&format!("{}...\n\n", preview));
} else {
text.push_str(&format!("{}\n\n", node.content));
}
// Show neighbors with strengths
let neighbors = graph.neighbors(key);
if !neighbors.is_empty() {
text.push_str("**Neighbors:**\n");
for (nbr, strength) in neighbors.iter().take(15) {
let nbr_type = store.nodes.get(nbr.as_str())
.map(|n| match n.node_type {
NodeType::EpisodicSession => " [journal]",
NodeType::EpisodicDaily => " [daily]",
_ => "",
})
.unwrap_or("");
text.push_str(&format!(" [{:.1}] {}{}\n", strength, sq(nbr), nbr_type));
}
if neighbors.len() > 15 {
text.push_str(&format!(" ... and {} more\n", neighbors.len() - 15));
}
text.push('\n');
}
text.push_str("---\n\n");
result_keys.push(key.clone()); result_keys.push(key.clone());
} }
_ => continue,
}
}
text.push_str("Use memory_render(KEY) and memory_links(KEY) to explore further.\n"); text.push_str("Use memory_render(KEY) and memory_links(KEY) to explore further.\n");
Some(Resolved { text, keys: result_keys }) Some(Resolved { text, keys: result_keys })
} }
"siblings" | "neighborhood" => { "siblings" | "neighborhood" => {
use crate::agent::tools::memory::{memory_render, memory_links};
const MAX_NEIGHBORS: usize = 20;
const BUDGET: usize = 400_000; // ~100K tokens
let mut out = String::new(); let mut out = String::new();
let mut all_keys: Vec<String> = Vec::new(); let mut all_keys: Vec<String> = Vec::new();
let mut included_nodes: std::collections::HashSet<String> = std::collections::HashSet::new(); let mut included: std::collections::HashSet<String> = std::collections::HashSet::new();
const MAX_NEIGHBORS: usize = 25;
for key in keys { for key in keys {
if included_nodes.contains(key) { continue; } if included.contains(key) { continue; }
included_nodes.insert(key.clone()); included.insert(key.clone());
let Some(node) = store.nodes.get(key.as_str()) else { continue };
let neighbors = graph.neighbors(key);
// Seed node with full content // Seed node with full content
out.push_str(&format!("## {} (seed)\n\n{}\n\n", key, node.content)); let Ok(content) = memory_render(None, key, Some(true)).await else { continue };
out.push_str(&format!("## {} (seed)\n\n{}\n\n", key, content));
all_keys.push(key.clone()); all_keys.push(key.clone());
// Rank neighbors by link_strength * node_weight // Get neighbors with link_strength and node_weight, rank and take top 20
// Include all if <= 10, otherwise take top MAX_NEIGHBORS let Ok(links) = memory_links(None, key).await else { continue };
let mut ranked: Vec<(String, f32, f32)> = neighbors.iter() let mut ranked: Vec<_> = links.into_iter()
.filter_map(|(nbr, strength)| { .map(|l| {
store.nodes.get(nbr.as_str()).map(|n| { let score = l.link_strength * l.node_weight.max(0.01);
let node_weight = n.weight.max(0.01); (l.key, l.link_strength, score)
let score = strength * node_weight;
(nbr.to_string(), *strength, score)
})
}) })
.collect(); .collect();
ranked.sort_by(|a, b| b.2.total_cmp(&a.2)); ranked.sort_by(|a, b| b.2.total_cmp(&a.2));
ranked.truncate(MAX_NEIGHBORS);
let total = ranked.len(); if ranked.is_empty() { continue; }
let included: Vec<_> = if total <= 10 { out.push_str(&format!("### Neighbors (top {})\n\n", ranked.len()));
ranked
} else {
// Smooth cutoff: threshold scales with neighborhood size
// Generous — err on including too much so the agent can
// see and clean up junk. 20 → top 75%, 50 → top 30%
let top_score = ranked.first().map(|(_, _, s)| *s).unwrap_or(0.0);
let ratio = (15.0 / total as f32).min(1.0);
let threshold = top_score * ratio;
ranked.into_iter()
.enumerate()
.take_while(|(i, (_, _, score))| *i < 10 || *score >= threshold)
.take(MAX_NEIGHBORS)
.map(|(_, item)| item)
.collect()
};
if !included.is_empty() { for (nbr, strength, _) in &ranked {
if total > included.len() { if included.contains(nbr) { continue; }
out.push_str(&format!("### Neighbors (top {} of {}, ranked by importance)\n\n", included.insert(nbr.clone());
included.len(), total)); if let Ok(content) = memory_render(None, nbr, Some(true)).await {
} else { if out.len() > BUDGET {
out.push_str("### Neighbors\n\n"); // Header-only past budget
} let first = content.lines()
let included_keys: std::collections::HashSet<&str> = included.iter()
.map(|(k, _, _)| k.as_str()).collect();
// Budget: stop adding full content when prompt gets large.
// Remaining neighbors get header-only (key + first line).
const NEIGHBORHOOD_BUDGET: usize = 400_000; // ~100K tokens, leaves room for core-personality + instructions
let mut budget_exceeded = false;
for (nbr, strength, _score) in &included {
if included_nodes.contains(nbr) { continue; }
included_nodes.insert(nbr.clone());
if let Some(n) = store.nodes.get(nbr.as_str()) {
if budget_exceeded || out.len() > NEIGHBORHOOD_BUDGET {
// Header-only: key + first non-empty line
budget_exceeded = true;
let first_line = n.content.lines()
.find(|l| !l.trim().is_empty()) .find(|l| !l.trim().is_empty())
.unwrap_or("(empty)"); .unwrap_or("(empty)");
out.push_str(&format!("#### {} (link: {:.2}) — {}\n", out.push_str(&format!("#### {} ({:.2}) — {}\n", nbr, strength, first));
nbr, strength, first_line));
} else { } else {
out.push_str(&format!("#### {} (link: {:.2})\n\n{}\n\n", out.push_str(&format!("#### {} ({:.2})\n\n{}\n\n", nbr, strength, content));
nbr, strength, n.content));
} }
all_keys.push(nbr.to_string()); all_keys.push(nbr.to_string());
} }
} }
if budget_exceeded {
out.push_str("\n(remaining neighbors shown as headers only — prompt budget)\n\n");
}
// Cross-links between included neighbors
let mut cross_links = Vec::new();
for (nbr, _, _) in &included {
for (nbr2, strength) in graph.neighbors(nbr) {
if nbr2.as_str() != key
&& included_keys.contains(nbr2.as_str())
&& nbr.as_str() < nbr2.as_str()
{
cross_links.push((nbr.clone(), nbr2, strength));
}
}
}
if !cross_links.is_empty() {
out.push_str("### Cross-links between neighbors\n\n");
for (a, b, s) in &cross_links {
out.push_str(&format!(" {} ↔ {} ({:.2})\n", a, b, s));
}
out.push('\n');
}
}
} }
Some(Resolved { text: out, keys: all_keys }) Some(Resolved { text: out, keys: all_keys })
} }
// targets/context: aliases for challenger-style presentation
"targets" => {
let items = keys_to_replay_items(store, keys, graph);
Some(Resolved {
text: super::prompts::format_nodes_section(store, &items, graph),
keys: vec![],
})
}
"hubs" => {
// Top hub nodes by degree, spread apart (skip neighbors of already-selected hubs)
let mut hubs: Vec<(String, usize)> = store.nodes.iter()
.filter(|(k, n)| !n.deleted && !k.starts_with('_'))
.map(|(k, _)| {
let degree = graph.neighbors(k).len();
(k.clone(), degree)
})
.collect();
hubs.sort_by(|a, b| b.1.cmp(&a.1));
let mut selected = Vec::new();
let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
for (key, degree) in &hubs {
if seen.contains(key) { continue; }
selected.push(format!(" - {} (degree {})", key, degree));
// Mark neighbors as seen so we pick far-apart hubs
for (nbr, _) in graph.neighbors(key) {
seen.insert(nbr.clone());
}
seen.insert(key.clone());
if selected.len() >= 20 { break; }
}
let text = format!("## Hub nodes (link targets)\n\n{}", selected.join("\n"));
Some(Resolved { text, keys: vec![] })
}
// agent-context — personality/identity groups from load-context config // agent-context — personality/identity groups from load-context config
"agent-context" => { "agent-context" => {
let cfg = crate::config::get(); let cfg = crate::config::get();
@ -474,7 +300,7 @@ fn resolve(
let mut keys = Vec::new(); let mut keys = Vec::new();
for group in &cfg.context_groups { for group in &cfg.context_groups {
if !group.agent { continue; } if !group.agent { continue; }
let entries = crate::cli::misc::get_group_content(group, store, &cfg); let entries = crate::cli::node::get_group_content(group, &cfg).await;
for (key, content) in entries { for (key, content) in entries {
use std::fmt::Write; use std::fmt::Write;
writeln!(text, "--- {} ({}) ---", key, group.label).ok(); writeln!(text, "--- {} ({}) ---", key, group.label).ok();
@ -486,15 +312,6 @@ fn resolve(
else { Some(Resolved { text, keys }) } else { Some(Resolved { text, keys }) }
} }
// node:KEY — inline a node's content by key
other if other.starts_with("node:") => {
let key = &other[5..];
store.nodes.get(key).map(|n| Resolved {
text: n.content.clone(),
keys: vec![key.to_string()],
})
}
// input:KEY — read a named output file from the agent's output dir // input:KEY — read a named output file from the agent's output dir
_ if name.starts_with("input:") => { _ if name.starts_with("input:") => {
let key = &name[6..]; let key = &name[6..];
@ -536,22 +353,10 @@ fn resolve(
Some(Resolved { text, keys: vec![] }) Some(Resolved { text, keys: vec![] })
} }
// latest_journal — the most recent EpisodicSession entry
"latest_journal" => {
let latest = store.nodes.values()
.filter(|n| n.node_type == crate::store::NodeType::EpisodicSession)
.max_by_key(|n| n.created_at);
let (text, keys) = match latest {
Some(n) => (n.content.clone(), vec![n.key.clone()]),
None => ("(no previous journal entry)".to_string(), vec![]),
};
Some(Resolved { text, keys })
}
// tool:NAME ARGS — run a tool call and include its output // tool:NAME ARGS — run a tool call and include its output
_ if name.starts_with("tool:") => { _ if name.starts_with("tool:") => {
let spec = name[5..].trim(); let spec = name[5..].trim();
resolve_tool(spec, store, graph) resolve_tool(spec).await
} }
// bash:COMMAND — run a shell command and include its stdout // bash:COMMAND — run a shell command and include its stdout
@ -714,9 +519,8 @@ fn resolve_memory_ratio() -> String {
pct, keys.len(), memory_bytes / 1024, transcript_size / 1024) pct, keys.len(), memory_bytes / 1024, transcript_size / 1024)
} }
/// Resolve a {{tool: name {args}}} placeholder by calling the tool /// Resolve a {{tool: name {args}}} placeholder by calling the tool handler.
/// handler from the registry. Uses block_in_place to bridge sync→async. async fn resolve_tool(spec: &str) -> Option<Resolved> {
fn resolve_tool(spec: &str, _store: &Store, _graph: &Graph) -> Option<Resolved> {
// Parse "tool_name {json args}" or "tool_name arg" // Parse "tool_name {json args}" or "tool_name arg"
let (name, args) = match spec.find('{') { let (name, args) = match spec.find('{') {
Some(i) => { Some(i) => {
@ -737,13 +541,7 @@ fn resolve_tool(spec: &str, _store: &Store, _graph: &Graph) -> Option<Resolved>
let tools = crate::agent::tools::tools(); let tools = crate::agent::tools::tools();
let tool = tools.iter().find(|t| t.name == name)?; let tool = tools.iter().find(|t| t.name == name)?;
let result = tokio::task::block_in_place(|| { match (tool.handler)(None, args.clone()).await {
tokio::runtime::Handle::current().block_on(
(tool.handler)(None, args.clone())
)
});
match result {
Ok(text) => Some(Resolved { text, keys: vec![] }), Ok(text) => Some(Resolved { text, keys: vec![] }),
Err(e) => { Err(e) => {
eprintln!("[defs] {{{{tool: {}}}}} failed: {}", name, e); eprintln!("[defs] {{{{tool: {}}}}} failed: {}", name, e);
@ -754,10 +552,8 @@ fn resolve_tool(spec: &str, _store: &Store, _graph: &Graph) -> Option<Resolved>
/// Resolve all {{placeholder}} patterns in a prompt template. /// Resolve all {{placeholder}} patterns in a prompt template.
/// Returns the resolved text and all node keys collected from placeholders. /// Returns the resolved text and all node keys collected from placeholders.
pub fn resolve_placeholders( pub async fn resolve_placeholders(
template: &str, template: &str,
store: &Store,
graph: &Graph,
keys: &[String], keys: &[String],
count: usize, count: usize,
) -> (String, Vec<String>) { ) -> (String, Vec<String>) {
@ -770,7 +566,7 @@ pub fn resolve_placeholders(
let Some(rel_end) = result[start + 2..].find("}}") else { break }; let Some(rel_end) = result[start + 2..].find("}}") else { break };
let end = start + 2 + rel_end; let end = start + 2 + rel_end;
let name = result[start + 2..end].trim().to_lowercase(); let name = result[start + 2..end].trim().to_lowercase();
match resolve(&name, store, graph, keys, count) { match resolve(&name, keys, count).await {
Some(resolved) => { Some(resolved) => {
let len = resolved.text.len(); let len = resolved.text.len();
extra_keys.extend(resolved.keys); extra_keys.extend(resolved.keys);
@ -791,27 +587,26 @@ pub fn resolve_placeholders(
/// Run a config-driven agent: query → resolve placeholders → prompt. /// Run a config-driven agent: query → resolve placeholders → prompt.
/// `exclude` filters out nodes (and their neighborhoods) already being /// `exclude` filters out nodes (and their neighborhoods) already being
/// worked on by other agents, preventing concurrent collisions. /// worked on by other agents, preventing concurrent collisions.
pub fn run_agent( pub async fn run_agent(
store: &Store,
def: &AgentDef, def: &AgentDef,
count: usize, count: usize,
exclude: &std::collections::HashSet<String>, exclude: &std::collections::HashSet<String>,
) -> Result<super::prompts::AgentBatch, String> { ) -> Result<super::prompts::AgentBatch, String> {
let graph = store.build_graph();
// Run the query if present // Run the query if present, via RPC
let keys = if !def.query.is_empty() { let keys = if !def.query.is_empty() {
let mut stages = crate::query_parser::parse_stages(&def.query)?;
let has_limit = stages.iter().any(|s|
matches!(s, search::Stage::Transform(search::Transform::Limit(_))));
if !has_limit {
// Request extra results to compensate for exclusion filtering
let padded = count + exclude.len().min(100); let padded = count + exclude.len().min(100);
stages.push(search::Stage::Transform(search::Transform::Limit(padded))); let query = if def.query.contains("limit:") {
} def.query.clone()
let results = search::run_query(&stages, vec![], &graph, store, false, count + exclude.len().min(100)); } else {
let filtered: Vec<String> = results.into_iter() format!("{} | limit:{}", def.query, padded)
.map(|(k, _)| k) };
let result = crate::agent::tools::memory::memory_query(None, &query, None)
.await
.map_err(|e| e.to_string())?;
let filtered: Vec<String> = result.lines()
.filter(|l| !l.is_empty() && *l != "no results")
.map(|s| s.to_string())
.filter(|k| !exclude.contains(k)) .filter(|k| !exclude.contains(k))
.take(count) .take(count)
.collect(); .collect();
@ -833,7 +628,7 @@ pub fn run_agent(
.replace("{agent_name}", &def.agent) .replace("{agent_name}", &def.agent)
.replace("{user_name}", &cfg.user_name) .replace("{user_name}", &cfg.user_name)
.replace("{assistant_name}", &cfg.assistant_name); .replace("{assistant_name}", &cfg.assistant_name);
let (prompt, extra_keys) = resolve_placeholders(&template, store, &graph, &all_keys, count); let (prompt, extra_keys) = resolve_placeholders(&template, &all_keys, count).await;
all_keys.extend(extra_keys); all_keys.extend(extra_keys);
resolved_steps.push(super::prompts::ResolvedStep { resolved_steps.push(super::prompts::ResolvedStep {
prompt, prompt,
@ -843,28 +638,3 @@ pub fn run_agent(
Ok(super::prompts::AgentBatch { steps: resolved_steps, node_keys: all_keys }) Ok(super::prompts::AgentBatch { steps: resolved_steps, node_keys: all_keys })
} }
/// Convert a list of keys to ReplayItems with priority and graph metrics.
pub fn keys_to_replay_items(
store: &Store,
keys: &[String],
graph: &Graph,
) -> Vec<ReplayItem> {
keys.iter()
.filter_map(|key| {
let node = store.nodes.get(key)?;
let priority = consolidation_priority(store, key, graph, None);
let cc = graph.clustering_coefficient(key);
Some(ReplayItem {
key: key.clone(),
priority,
interval_days: node.spaced_repetition_interval,
emotion: node.emotion,
cc,
classification: "unknown",
outlier_score: 0.0,
})
})
.collect()
}

40
src/subconscious/digest.rs
View file

@ -112,17 +112,21 @@ fn parse_digest_node_links(key: &str, content: &str) -> Vec<DigestLink> {
pub fn parse_all_digest_links(store: &Store) -> Vec<DigestLink> { pub fn parse_all_digest_links(store: &Store) -> Vec<DigestLink> {
let mut all_links = Vec::new(); let mut all_links = Vec::new();
let mut digest_keys: Vec<&String> = store.nodes.iter() let all_keys = store.all_keys().unwrap_or_default();
.filter(|(_, n)| matches!(n.node_type, let mut digest_keys: Vec<String> = all_keys.into_iter()
.filter(|k| {
store.get_node(k).ok().flatten()
.map(|n| matches!(n.node_type,
store::NodeType::EpisodicDaily store::NodeType::EpisodicDaily
| store::NodeType::EpisodicWeekly | store::NodeType::EpisodicWeekly
| store::NodeType::EpisodicMonthly)) | store::NodeType::EpisodicMonthly))
.map(|(k, _)| k) .unwrap_or(false)
})
.collect(); .collect();
digest_keys.sort(); digest_keys.sort();
for key in digest_keys { for key in &digest_keys {
if let Some(node) = store.nodes.get(key) { if let Ok(Some(node)) = store.get_node(key) {
all_links.extend(parse_digest_node_links(key, &node.content)); all_links.extend(parse_digest_node_links(key, &node.content));
} }
} }
@ -172,20 +176,20 @@ pub fn apply_digest_links(store: &mut Store, links: &[DigestLink]) -> (usize, us
if source == target { skipped += 1; continue; } if source == target { skipped += 1; continue; }
// Check if link already exists let source_uuid = match store.get_node(&source).ok().flatten() {
let exists = store.relations.iter().any(|r| Some(n) => n.uuid,
r.source_key == source && r.target_key == target && !r.deleted None => { skipped += 1; continue; }
); };
if exists { skipped += 1; continue; } let target_uuid = match store.get_node(&target).ok().flatten() {
Some(n) => n.uuid,
None => { skipped += 1; continue; }
};
let source_uuid = match store.nodes.get(&source) { // Check if link already exists via index
Some(n) => n.uuid, let exists = store.neighbors(&source).ok()
None => { skipped += 1; continue; } .map(|n| n.iter().any(|(k, _)| k == &target))
}; .unwrap_or(false);
let target_uuid = match store.nodes.get(&target) { if exists { skipped += 1; continue; }
Some(n) => n.uuid,
None => { skipped += 1; continue; }
};
let rel = new_relation( let rel = new_relation(
source_uuid, target_uuid, source_uuid, target_uuid,

9
src/subconscious/learn.rs
View file

@ -327,12 +327,16 @@ where
let mut seen = std::collections::HashSet::new(); let mut seen = std::collections::HashSet::new();
let mut candidates: Vec<(usize, String, i64)> = Vec::new(); // (pos, key, last_scored) let mut candidates: Vec<(usize, String, i64)> = Vec::new(); // (pos, key, last_scored)
let store = crate::hippocampus::store::Store::load().unwrap_or_default(); let store_arc = crate::hippocampus::access_local()?;
{
let store = &*store_arc;
for (i, node) in context.conversation().iter().enumerate() { for (i, node) in context.conversation().iter().enumerate() {
if let Some(key) = memory_key(node) { if let Some(key) = memory_key(node) {
if !seen.insert(key.to_owned()) { continue; } if !seen.insert(key.to_owned()) { continue; }
let last_scored = store.nodes.get(key) let last_scored = store.get_node(key)
.ok()
.flatten()
.map(|n| n.last_scored) .map(|n| n.last_scored)
.unwrap_or(0); .unwrap_or(0);
if now - last_scored >= max_age_secs { if now - last_scored >= max_age_secs {
@ -340,6 +344,7 @@ where
} }
} }
} }
}
// Score oldest-first // Score oldest-first
candidates.sort_by_key(|&(_, _, last)| last); candidates.sort_by_key(|&(_, _, last)| last);

192
src/subconscious/prompts.rs
View file

@ -4,10 +4,7 @@
use crate::store::Store; use crate::store::Store;
use crate::graph::Graph; use crate::graph::Graph;
use crate::neuro::{ use crate::neuro::ReplayItem;
ReplayItem,
replay_queue,
};
/// Result of building an agent prompt — includes both the prompt text /// Result of building an agent prompt — includes both the prompt text
/// and the keys of nodes selected for processing, so the caller can /// and the keys of nodes selected for processing, so the caller can
@ -23,7 +20,7 @@ pub struct AgentBatch {
pub node_keys: Vec<String>, pub node_keys: Vec<String>,
} }
pub fn format_topology_header(graph: &Graph) -> String { pub fn format_topology_header(store: &Store, graph: &Graph) -> String {
let sigma = graph.small_world_sigma(); let sigma = graph.small_world_sigma();
let alpha = graph.degree_power_law_exponent(); let alpha = graph.degree_power_law_exponent();
let gini = graph.degree_gini(); let gini = graph.degree_gini();
@ -31,6 +28,28 @@ pub fn format_topology_header(graph: &Graph) -> String {
let n = graph.nodes().len(); let n = graph.nodes().len();
let e = graph.edge_count(); let e = graph.edge_count();
// Type counts
let mut type_counts: std::collections::HashMap<&str, usize> = std::collections::HashMap::new();
let all_keys = store.all_keys().unwrap_or_default();
for key in &all_keys {
if let Ok(Some(node)) = store.get_node(key) {
let label = match node.node_type {
crate::store::NodeType::Semantic => "semantic",
crate::store::NodeType::EpisodicSession
| crate::store::NodeType::EpisodicDaily
| crate::store::NodeType::EpisodicWeekly
| crate::store::NodeType::EpisodicMonthly => "episodic",
};
*type_counts.entry(label).or_default() += 1;
}
}
let mut types: Vec<_> = type_counts.iter().collect();
types.sort_by_key(|(_, c)| std::cmp::Reverse(**c));
let type_str: String = types.iter()
.map(|(t, c)| format!("{}={}", t, c))
.collect::<Vec<_>>()
.join(" ");
// Identify saturated hubs — nodes with degree well above threshold // Identify saturated hubs — nodes with degree well above threshold
let threshold = graph.hub_threshold(); let threshold = graph.hub_threshold();
let mut hubs: Vec<_> = graph.nodes().iter() let mut hubs: Vec<_> = graph.nodes().iter()
@ -57,20 +76,20 @@ pub fn format_topology_header(graph: &Graph) -> String {
format!( format!(
"## Current graph topology\n\ "## Current graph topology\n\
Nodes: {} Edges: {} Communities: {}\n\ Nodes: {} Edges: {} Communities: {} Types: {}\n\
Small-world σ: {:.1} Power-law α: {:.2} Degree Gini: {:.3}\n\ Small-world σ: {:.1} Power-law α: {:.2} Degree Gini: {:.3}\n\
Avg clustering coefficient: {:.4}\n\n\ Avg clustering coefficient: {:.4}\n\n\
{}\ {}\
Each node below shows its hub-link ratio (fraction of edges to top-5% degree nodes).\n\ Each node below shows its hub-link ratio (fraction of edges to top-5% degree nodes).\n\
Use `poc-memory link-impact SOURCE TARGET` to evaluate proposed links.\n\n", Use `poc-memory link-impact SOURCE TARGET` to evaluate proposed links.\n\n",
n, e, graph.community_count(), sigma, alpha, gini, avg_cc, hub_list) n, e, graph.community_count(), type_str, sigma, alpha, gini, avg_cc, hub_list)
} }
pub fn format_nodes_section(store: &Store, items: &[ReplayItem], graph: &Graph) -> String { pub fn format_nodes_section(store: &Store, items: &[ReplayItem], graph: &Graph) -> String {
let hub_thresh = graph.hub_threshold(); let hub_thresh = graph.hub_threshold();
let mut out = String::new(); let mut out = String::new();
for item in items { for item in items {
let node = match store.nodes.get(&item.key) { let node = match store.get_node(&item.key).ok().flatten() {
Some(n) => n, Some(n) => n,
None => continue, None => continue,
}; };
@ -123,7 +142,9 @@ pub fn format_nodes_section(store: &Store, items: &[ReplayItem], graph: &Graph)
out.push_str("Neighbors:\n"); out.push_str("Neighbors:\n");
for (n, strength) in neighbors.iter().take(15) { for (n, strength) in neighbors.iter().take(15) {
let n_cc = graph.clustering_coefficient(n); let n_cc = graph.clustering_coefficient(n);
let n_community = store.nodes.get(n.as_str()) let n_community = store.get_node(n)
.ok()
.flatten()
.and_then(|n| n.community_id); .and_then(|n| n.community_id);
out.push_str(&format!(" - {} (str={:.2}, cc={:.3}", out.push_str(&format!(" - {} (str={:.2}, cc={:.3}",
n, strength, n_cc)); n, strength, n_cc));
@ -149,10 +170,13 @@ pub fn format_health_section(store: &Store, graph: &Graph) -> String {
// Weight histogram // Weight histogram
let mut buckets = [0u32; 10]; // 0.0-0.1, 0.1-0.2, ..., 0.9-1.0 let mut buckets = [0u32; 10]; // 0.0-0.1, 0.1-0.2, ..., 0.9-1.0
for node in store.nodes.values() { let all_keys = store.all_keys().unwrap_or_default();
for key in &all_keys {
if let Ok(Some(node)) = store.get_node(key) {
let bucket = ((node.weight * 10.0) as usize).min(9); let bucket = ((node.weight * 10.0) as usize).min(9);
buckets[bucket] += 1; buckets[bucket] += 1;
} }
}
for (i, &count) in buckets.iter().enumerate() { for (i, &count) in buckets.iter().enumerate() {
let lo = i as f32 / 10.0; let lo = i as f32 / 10.0;
let hi = (i + 1) as f32 / 10.0; let hi = (i + 1) as f32 / 10.0;
@ -161,9 +185,9 @@ pub fn format_health_section(store: &Store, graph: &Graph) -> String {
} }
// Near-prune nodes // Near-prune nodes
let near_prune: Vec<_> = store.nodes.iter() let near_prune: Vec<_> = all_keys.iter()
.filter(|(_, n)| n.weight < 0.15) .filter_map(|k| store.get_node(k).ok()?.map(|n| (k.clone(), n.weight)))
.map(|(k, n)| (k.clone(), n.weight)) .filter(|(_, w)| *w < 0.15)
.collect(); .collect();
if !near_prune.is_empty() { if !near_prune.is_empty() {
out.push_str(&format!("\n## Near-prune nodes ({} total)\n", near_prune.len())); out.push_str(&format!("\n## Near-prune nodes ({} total)\n", near_prune.len()));
@ -195,147 +219,9 @@ pub fn format_health_section(store: &Store, graph: &Graph) -> String {
out out
} }
pub(super) fn format_rename_candidates(store: &Store, count: usize) -> (Vec<String>, String) {
let mut candidates: Vec<(&str, &crate::store::Node)> = store.nodes.iter()
.filter(|(key, node)| {
if key.starts_with("_facts-") { return true; }
if key.len() < 60 { return false; }
if node.node_type == crate::store::NodeType::EpisodicSession { return true; }
if key.starts_with("_mined-transcripts#f-") { return true; }
false
})
.map(|(k, n)| (k.as_str(), n))
.collect();
// Deprioritize nodes actively found by search — renaming them would
// break working queries. Sort by: search hits (ascending), then
// least-recently visited. Nodes with many hits sink to the bottom.
let hit_counts = crate::counters::all_search_hits();
let hit_map: std::collections::HashMap<&str, u64> = hit_counts.iter()
.map(|(k, v)| (k.as_str(), *v))
.collect();
candidates.sort_by_key(|(key, _)| {
let hits = hit_map.get(key).copied().unwrap_or(0);
(hits, store.last_visited(key, "rename"))
});
candidates.truncate(count);
let keys: Vec<String> = candidates.iter().map(|(k, _)| k.to_string()).collect();
let mut out = String::new();
out.push_str(&format!("## Nodes to rename ({} of {} candidates)\n\n",
candidates.len(),
store.nodes.iter().filter(|(k, n)| k.starts_with("_facts-") ||
(k.len() >= 60 &&
(n.node_type == crate::store::NodeType::EpisodicSession || k.starts_with("_mined-transcripts#f-")))).count()));
for (key, node) in &candidates {
out.push_str(&format!("### {}\n", key));
let created = if node.timestamp > 0 {
crate::store::format_datetime(node.timestamp)
} else {
"unknown".to_string()
};
out.push_str(&format!("Created: {}\n", created));
let hits = hit_map.get(key).copied().unwrap_or(0);
if hits > 0 {
out.push_str(&format!("Search hits: {} ← actively found by search, prefer to keep current name\n", hits));
}
let content = &node.content;
if content.len() > 800 {
let truncated = crate::util::truncate(content, 800, "\n[...]");
out.push_str(&format!("\nContent ({} chars, truncated):\n{}\n\n",
content.len(), truncated));
} else {
out.push_str(&format!("\nContent:\n{}\n\n", content));
}
out.push_str("---\n\n");
}
(keys, out)
}
/// Format specific target keys as rename candidates (for --target mode)
pub(super) fn format_rename_targets(store: &Store, keys: &[String]) -> String {
let mut out = String::new();
out.push_str(&format!("## Nodes to rename ({} targets)\n\n", keys.len()));
for key in keys {
let Some(node) = store.nodes.get(key) else {
out.push_str(&format!("### {}\n\n(node not found)\n\n---\n\n", key));
continue;
};
out.push_str(&format!("### {}\n", key));
let created = if node.timestamp > 0 {
crate::store::format_datetime(node.timestamp)
} else {
"unknown".to_string()
};
out.push_str(&format!("Created: {}\n", created));
let content = &node.content;
if content.len() > 800 {
let truncated = crate::util::truncate(content, 800, "\n[...]");
out.push_str(&format!("\nContent ({} chars, truncated):\n{}\n\n",
content.len(), truncated));
} else {
out.push_str(&format!("\nContent:\n{}\n\n", content));
}
out.push_str("---\n\n");
}
out
}
/// Format a single node for split-plan prompt (phase 1)
pub(super) fn format_split_plan_node(store: &Store, graph: &Graph, key: &str) -> String {
let communities = graph.communities();
let node = match store.nodes.get(key) {
Some(n) => n,
None => return format!("Node '{}' not found\n", key),
};
let mut out = String::new();
out.push_str(&format!("### {} ({} chars)\n", key, node.content.len()));
// Show neighbors grouped by community
let neighbors = graph.neighbors(key);
if !neighbors.is_empty() {
let mut by_community: std::collections::BTreeMap<String, Vec<(&str, f32)>> =
std::collections::BTreeMap::new();
for (nkey, strength) in &neighbors {
let comm = communities.get(nkey.as_str())
.map(|c| format!("c{}", c))
.unwrap_or_else(|| "unclustered".into());
by_community.entry(comm)
.or_default()
.push((nkey.as_str(), *strength));
}
out.push_str("\nNeighbors by community:\n");
for (comm, members) in &by_community {
out.push_str(&format!(" {} ({}):", comm, members.len()));
for (nkey, strength) in members.iter().take(5) {
out.push_str(&format!(" {}({:.2})", nkey, strength));
}
if members.len() > 5 {
out.push_str(&format!(" +{} more", members.len() - 5));
}
out.push('\n');
}
}
// Full content
out.push_str(&format!("\nContent:\n{}\n\n", node.content));
out.push_str("---\n\n");
out
}
/// Generate a specific agent prompt with filled-in data. /// Generate a specific agent prompt with filled-in data.
pub fn agent_prompt(store: &Store, agent: &str, count: usize) -> Result<AgentBatch, String> { pub async fn agent_prompt(agent: &str, count: usize) -> Result<AgentBatch, String> {
let def = super::defs::get_def(agent) let def = super::defs::get_def(agent)
.ok_or_else(|| format!("Unknown agent: {}", agent))?; .ok_or_else(|| format!("Unknown agent: {}", agent))?;
super::defs::run_agent(store, &def, count, &Default::default()) super::defs::run_agent(&def, count, &Default::default()).await
} }

3
src/user/chat.rs
View file

@ -368,10 +368,11 @@ impl PaneState {
} }
fn pop_line(&mut self) { fn pop_line(&mut self) {
self.lines.pop(); if self.lines.pop().is_some() {
self.markers.pop(); self.markers.pop();
self.scroll.invalidate_from(self.lines.len()); self.scroll.invalidate_from(self.lines.len());
} }
}
fn scroll_up(&mut self, n: u16) { fn scroll_up(&mut self, n: u16) {
self.scroll.scroll_up(n); self.scroll.scroll_up(n);

24
src/user/mod.rs
View file

@ -6,6 +6,7 @@
pub(crate) mod chat; pub(crate) mod chat;
mod context; mod context;
pub(crate) mod scroll_pane; pub(crate) mod scroll_pane;
pub mod selectable;
mod subconscious; mod subconscious;
mod unconscious; mod unconscious;
mod thalamus; mod thalamus;
@ -183,7 +184,7 @@ fn restore_terminal(terminal: &mut ratatui::Terminal<CrosstermBackend<io::Stdout
/// Top-level entry point — creates Mind and UI, wires them together. /// Top-level entry point — creates Mind and UI, wires them together.
async fn start(cli: crate::user::CliArgs) -> Result<()> { async fn start(cli: crate::user::CliArgs) -> Result<()> {
let (config, _figment) = crate::config::load_session(&cli)?; let (config, _figment) = crate::config::load_session(&cli).await?;
if config.app.debug { if config.app.debug {
unsafe { std::env::set_var("POC_DEBUG", "1") }; unsafe { std::env::set_var("POC_DEBUG", "1") };
@ -211,10 +212,25 @@ async fn start(cli: crate::user::CliArgs) -> Result<()> {
}) })
.expect("spawn UI thread"); .expect("spawn UI thread");
// Initialize store - access_local() caches it in STORE_ACCESS
if let Err(e) = crate::hippocampus::access_local() {
eprintln!("Store init failed: {}", e);
}
// Start MCP server for external tool access
let mut tools: Vec<crate::agent::tools::Tool> = Vec::new();
tools.extend(crate::agent::tools::memory::memory_tools());
tools.extend(crate::agent::tools::memory::journal_tools());
if let Err(e) = crate::mcp_server::start(tools).await {
eprintln!("MCP server failed to start: {:#}", e);
}
// Mind event loop — runs on the main tokio runtime // Mind event loop — runs on the main tokio runtime
mind.init().await; mind.init().await;
mind.run(mind_rx, turn_rx).await; mind.run(mind_rx, turn_rx).await;
crate::mcp_server::cleanup();
ui_handle.join().unwrap_or_else(|_| Err(anyhow::anyhow!("UI thread panicked"))) ui_handle.join().unwrap_or_else(|_| Err(anyhow::anyhow!("UI thread panicked")))
} }
@ -403,9 +419,9 @@ async fn run(
unc.toggle(name).await; unc.toggle(name).await;
} }
} }
let store = crate::store::Store::cached().await.ok(); let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store { let store_guard = match &store_arc {
Some(s) => Some(s.lock().await), Some(s) => Some(&**s),
None => None, None => None,
}; };
app.unconscious_state = unc.snapshots(store_guard.as_deref()); app.unconscious_state = unc.snapshots(store_guard.as_deref());

22
src/user/scroll_pane.rs
View file

@ -106,7 +106,27 @@ impl ScrollPaneState {
let h = self.heights.get(line_idx).copied().unwrap_or(1) as i32; let h = self.heights.get(line_idx).copied().unwrap_or(1) as i32;
if (mouse_y as i32) < row + h { if (mouse_y as i32) < row + h {
let line_text: String = lines[line_idx].spans.iter().map(|s| s.content.as_ref()).collect(); let line_text: String = lines[line_idx].spans.iter().map(|s| s.content.as_ref()).collect();
let col = (mouse_x as usize).min(line_text.len());
// Which visual row within this wrapped line?
let visual_row_in_item = ((mouse_y as i32) - row).max(0) as usize;
// Use textwrap to find actual break positions
let wrap_width = self.cached_width as usize;
let wrapped = textwrap::wrap(&line_text, wrap_width);
// Sum lengths of previous wrapped rows to get char offset base
let char_base: usize = wrapped.iter()
.take(visual_row_in_item)
.map(|s| s.len())
.sum();
// Add mouse x position within current row
let current_row_len = wrapped.get(visual_row_in_item)
.map(|s| s.len())
.unwrap_or(0);
let col = char_base + (mouse_x as usize).min(current_row_len);
let col = col.min(line_text.len());
return Some((line_idx, col)); return Some((line_idx, col));
} }
row += h; row += h;

531
src/user/selectable.rs Normal file
View file

@ -0,0 +1,531 @@
//! Selectable text widget with proper wrap-aware selection.
//!
//! Uses Unicode Private Use Area markers to track logical line boundaries:
//! - Lines starting with CONT are continuations (wrapped from previous)
//! - Lines between SEL_ON and SEL_OFF are selectable
//!
//! The caller pre-wraps text and marks continuations. This widget handles
//! selection, clipboard copy, and rendering with highlights.
use ratatui::prelude::*;
use ratatui::widgets::{Block, Scrollbar, ScrollbarOrientation, ScrollbarState};
// ── Markers (Unicode Private Use Area) ─────────────────────────────
/// This line continues the previous logical line (was wrapped).
pub const CONT: char = '\u{E000}';
/// Start of a selectable region.
pub const SEL_ON: char = '\u{E001}';
/// End of a selectable region.
pub const SEL_OFF: char = '\u{E002}';
// ── Helper: wrap text with continuation markers ────────────────────
/// Wrap a single logical line into visual lines, marking continuations.
/// Returns lines ready to push into a SelectableText.
pub fn wrap_line(text: &str, width: usize) -> Vec<String> {
if width == 0 || text.is_empty() {
return vec![text.to_string()];
}
let wrapped = textwrap::wrap(text, width);
wrapped
.into_iter()
.enumerate()
.map(|(i, cow)| {
if i == 0 {
cow.into_owned()
} else {
format!("{}{}", CONT, cow)
}
})
.collect()
}
/// Wrap text and mark as selectable.
pub fn wrap_line_selectable(text: &str, width: usize) -> Vec<String> {
let mut lines = wrap_line(text, width);
if let Some(first) = lines.first_mut() {
*first = format!("{}{}", SEL_ON, first);
}
if let Some(last) = lines.last_mut() {
last.push(SEL_OFF);
}
lines
}
// ── Selection state ────────────────────────────────────────────────
/// A position in logical coordinates (line index, char offset).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct LogicalPos {
pub line: usize,
pub col: usize,
}
/// Selection anchor and cursor.
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub struct Selection {
pub anchor: LogicalPos,
pub cursor: LogicalPos,
}
impl Selection {
pub fn new(pos: LogicalPos) -> Self {
Self { anchor: pos, cursor: pos }
}
pub fn extend(&mut self, pos: LogicalPos) {
self.cursor = pos;
}
/// Returns (start, end) in normalized order.
pub fn range(&self) -> (LogicalPos, LogicalPos) {
if (self.anchor.line, self.anchor.col) <= (self.cursor.line, self.cursor.col) {
(self.anchor, self.cursor)
} else {
(self.cursor, self.anchor)
}
}
pub fn is_empty(&self) -> bool {
self.anchor == self.cursor
}
}
// ── Main widget state ──────────────────────────────────────────────
pub struct SelectableTextState {
/// Visual lines (may contain markers).
lines: Vec<String>,
/// Scroll offset in visual lines.
pub scroll_offset: usize,
/// Viewport height (set during render).
pub viewport_height: usize,
/// Current selection, if any.
pub selection: Option<Selection>,
/// Cached logical line index for each visual line.
/// logical_line_idx[visual] = which logical line this visual line belongs to.
logical_line_idx: Vec<usize>,
/// Cached char offset: start char of each visual line within its logical line.
char_offsets: Vec<usize>,
}
impl Default for SelectableTextState {
fn default() -> Self {
Self::new()
}
}
impl SelectableTextState {
pub fn new() -> Self {
Self {
lines: Vec::new(),
scroll_offset: 0,
viewport_height: 0,
selection: None,
logical_line_idx: Vec::new(),
char_offsets: Vec::new(),
}
}
/// Clear all content.
pub fn clear(&mut self) {
self.lines.clear();
self.logical_line_idx.clear();
self.char_offsets.clear();
self.selection = None;
}
/// Push a visual line. Call rebuild_index() after batch pushes.
pub fn push_line(&mut self, line: String) {
self.lines.push(line);
}
/// Push multiple visual lines.
pub fn push_lines(&mut self, lines: impl IntoIterator<Item = String>) {
self.lines.extend(lines);
}
/// Rebuild the logical line index. Call after modifying lines.
pub fn rebuild_index(&mut self) {
self.logical_line_idx.clear();
self.char_offsets.clear();
let mut logical_idx = 0usize;
let mut char_offset = 0usize;
for line in &self.lines {
let is_continuation = line.starts_with(CONT);
if !is_continuation && !self.logical_line_idx.is_empty() {
// New logical line
logical_idx += 1;
char_offset = 0;
}
self.logical_line_idx.push(logical_idx);
self.char_offsets.push(char_offset);
// Advance char offset by the display length of this line
char_offset += display_len(line);
}
}
/// Number of visual lines.
pub fn len(&self) -> usize {
self.lines.len()
}
pub fn is_empty(&self) -> bool {
self.lines.is_empty()
}
/// Scroll up by n visual lines.
pub fn scroll_up(&mut self, n: usize) {
self.scroll_offset = self.scroll_offset.saturating_sub(n);
}
/// Scroll down by n visual lines.
pub fn scroll_down(&mut self, n: usize) {
let max = self.len().saturating_sub(self.viewport_height);
self.scroll_offset = (self.scroll_offset + n).min(max);
}
/// Convert screen position to logical position.
pub fn screen_to_logical(&self, x: u16, y: u16) -> Option<LogicalPos> {
let visual_row = self.scroll_offset + y as usize;
if visual_row >= self.lines.len() {
return None;
}
let logical_line = *self.logical_line_idx.get(visual_row)?;
let char_base = *self.char_offsets.get(visual_row)?;
// Check if this position is within a selectable region
if !self.is_visual_line_selectable(visual_row) {
return None;
}
let line = &self.lines[visual_row];
let display = strip_markers(line);
let col = char_base + (x as usize).min(display.len());
Some(LogicalPos { line: logical_line, col })
}
/// Check if a visual line is within a selectable region.
fn is_visual_line_selectable(&self, visual_row: usize) -> bool {
// Walk backwards to find if we're in a selectable region
let mut in_selectable = false;
for i in 0..=visual_row {
let line = &self.lines[i];
if line.contains(SEL_ON) {
in_selectable = true;
}
if line.contains(SEL_OFF) && i < visual_row {
in_selectable = false;
}
}
in_selectable || self.lines[visual_row].contains(SEL_ON)
}
/// Start a new selection at screen position.
pub fn start_selection(&mut self, x: u16, y: u16) {
if let Some(pos) = self.screen_to_logical(x, y) {
self.selection = Some(Selection::new(pos));
} else {
self.selection = None;
}
}
/// Extend selection to screen position.
pub fn extend_selection(&mut self, x: u16, y: u16) {
if let Some(pos) = self.screen_to_logical(x, y) {
if let Some(ref mut sel) = self.selection {
sel.extend(pos);
}
}
}
/// Get selected text, joining logical lines with newlines.
pub fn get_selected_text(&self) -> Option<String> {
let sel = self.selection.as_ref()?;
if sel.is_empty() {
return None;
}
let (start, end) = sel.range();
// Reconstruct logical lines
let logical_lines = self.reconstruct_logical_lines();
let mut result = String::new();
for (i, line) in logical_lines.iter().enumerate() {
if i < start.line || i > end.line {
continue;
}
let line_start = if i == start.line { start.col } else { 0 };
let line_end = if i == end.line { end.col } else { line.len() };
if line_start < line.len() {
if !result.is_empty() {
result.push('\n');
}
let end_clamped = line_end.min(line.len());
if let Some(slice) = line.get(line_start..end_clamped) {
result.push_str(slice);
}
}
}
if result.is_empty() {
None
} else {
Some(result)
}
}
/// Reconstruct logical lines from visual lines (stripping markers, joining continuations).
fn reconstruct_logical_lines(&self) -> Vec<String> {
let mut logical: Vec<String> = Vec::new();
for line in &self.lines {
let is_cont = line.starts_with(CONT);
let clean = strip_markers(line);
if is_cont && !logical.is_empty() {
// Append to previous logical line
logical.last_mut().unwrap().push_str(&clean);
} else {
logical.push(clean);
}
}
logical
}
/// Copy selection to clipboard via OSC 52.
pub fn copy_to_clipboard(&self) {
if let Some(text) = self.get_selected_text() {
if text.is_empty() {
return;
}
use base64::Engine;
use std::io::Write;
let encoded = base64::engine::general_purpose::STANDARD.encode(&text);
let mut stdout = std::io::stdout().lock();
let _ = write!(stdout, "\x1b]52;c;{}\x07", encoded);
let _ = stdout.flush();
}
}
/// Get the visual lines for rendering (with markers stripped).
pub fn display_lines(&self) -> impl Iterator<Item = Line<'_>> + '_ {
self.lines.iter().map(|s| Line::raw(strip_markers(s)))
}
/// Check if a logical position is within the current selection.
#[allow(dead_code)] // Reserved for future per-character highlight rendering
fn is_selected(&self, logical_line: usize, col: usize) -> bool {
let Some(ref sel) = self.selection else { return false };
let (start, end) = sel.range();
if logical_line < start.line || logical_line > end.line {
return false;
}
if logical_line == start.line && col < start.col {
return false;
}
if logical_line == end.line && col >= end.col {
return false;
}
true
}
/// Get the selection highlight range for a visual line (in display columns).
pub fn highlight_range(&self, visual_row: usize) -> Option<(usize, usize)> {
let sel = self.selection.as_ref()?;
if sel.is_empty() {
return None;
}
let logical_line = *self.logical_line_idx.get(visual_row)?;
let char_base = *self.char_offsets.get(visual_row)?;
let display = strip_markers(&self.lines[visual_row]);
let line_len = display.len();
let (start, end) = sel.range();
// Check if this visual line overlaps with selection
if logical_line < start.line || logical_line > end.line {
return None;
}
let sel_start_in_line = if logical_line == start.line { start.col } else { 0 };
let sel_end_in_line = if logical_line == end.line { end.col } else { usize::MAX };
// Convert to visual line's local coordinates
let vis_start = sel_start_in_line.saturating_sub(char_base);
let vis_end = sel_end_in_line.saturating_sub(char_base).min(line_len);
if vis_start >= line_len || vis_end == 0 || vis_start >= vis_end {
return None;
}
Some((vis_start, vis_end))
}
}
// ── Widget ─────────────────────────────────────────────────────────
pub struct SelectableText<'a> {
block: Option<Block<'a>>,
highlight_style: Style,
}
impl<'a> SelectableText<'a> {
pub fn new() -> Self {
Self {
block: None,
highlight_style: Style::default().bg(Color::DarkGray),
}
}
pub fn block(mut self, block: Block<'a>) -> Self {
self.block = Some(block);
self
}
pub fn highlight_style(mut self, style: Style) -> Self {
self.highlight_style = style;
self
}
}
impl Default for SelectableText<'_> {
fn default() -> Self {
Self::new()
}
}
impl StatefulWidget for SelectableText<'_> {
type State = SelectableTextState;
fn render(self, area: Rect, buf: &mut Buffer, state: &mut Self::State) {
let inner = if let Some(block) = self.block {
let inner = block.inner(area);
block.render(area, buf);
inner
} else {
area
};
if inner.width < 2 || inner.height == 0 {
return;
}
state.viewport_height = inner.height as usize;
// Render visible lines
let start = state.scroll_offset;
let end = (start + inner.height as usize).min(state.lines.len());
for (i, visual_row) in (start..end).enumerate() {
let y = inner.y + i as u16;
let line = &state.lines[visual_row];
let display = strip_markers(line);
// Render with selection highlighting
if let Some((hl_start, hl_end)) = state.highlight_range(visual_row) {
// Before highlight
let before = &display[..hl_start.min(display.len())];
buf.set_string(inner.x, y, before, Style::default());
// Highlighted portion
let hl_text = &display[hl_start..hl_end.min(display.len())];
buf.set_string(inner.x + hl_start as u16, y, hl_text, self.highlight_style);
// After highlight
if hl_end < display.len() {
let after = &display[hl_end..];
buf.set_string(inner.x + hl_end as u16, y, after, Style::default());
}
} else {
buf.set_string(inner.x, y, &display, Style::default());
}
}
// Scrollbar
let content_len = state.lines.len();
let visible = inner.height as usize;
if content_len > visible {
let mut sb_state = ScrollbarState::new(content_len).position(state.scroll_offset);
Scrollbar::new(ScrollbarOrientation::VerticalRight).render(inner, buf, &mut sb_state);
}
}
}
// ── Helpers ────────────────────────────────────────────────────────
/// Strip all markers from a line for display.
fn strip_markers(s: &str) -> String {
s.chars()
.filter(|&c| c != CONT && c != SEL_ON && c != SEL_OFF)
.collect()
}
/// Display length of a line (excluding markers).
fn display_len(s: &str) -> usize {
s.chars()
.filter(|&c| c != CONT && c != SEL_ON && c != SEL_OFF)
.count()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_wrap_line() {
// "hello world, this is a test" at width 10:
// "hello" / "world," / "this is a" / "test"
let lines = wrap_line("hello world, this is a test", 10);
assert_eq!(lines.len(), 4);
assert!(!lines[0].starts_with(CONT)); // "hello"
assert!(lines[1].starts_with(CONT)); // " world,"
assert!(lines[2].starts_with(CONT)); // " this is a"
assert!(lines[3].starts_with(CONT)); // " test"
}
#[test]
fn test_strip_markers() {
let s = format!("{}hello{}world{}", SEL_ON, CONT, SEL_OFF);
assert_eq!(strip_markers(&s), "helloworld");
}
#[test]
fn test_logical_index() {
let mut state = SelectableTextState::new();
state.push_line("first line".to_string());
state.push_line(format!("{}continued", CONT));
state.push_line("second line".to_string());
state.rebuild_index();
assert_eq!(state.logical_line_idx, vec![0, 0, 1]);
assert_eq!(state.char_offsets, vec![0, 10, 0]);
}
#[test]
fn test_reconstruct() {
let mut state = SelectableTextState::new();
state.push_line("hello ".to_string());
state.push_line(format!("{}world", CONT));
state.push_line("next".to_string());
state.rebuild_index();
let logical = state.reconstruct_logical_lines();
assert_eq!(logical, vec!["hello world", "next"]);
}
}

288
training/research/pause-tokens-gdn-recurrence.md Normal file
View file

@ -0,0 +1,288 @@
# Pause Tokens + GDN Recurrence: Latent Reasoning for Qwen 3.5
**Status:** Ready for testing
**Date:** 2026-04-12
**Insight:** Qwen 3.5's GDN layers already have recurrence - pause tokens give it more iterations
---
## The Core Insight
Standard transformers couple compute depth to output length. Both pause tokens and internal recurrence solve this by allowing "thinking" without token commitment.
**The GDN connection:** Qwen 3.5 is 75% GDN (Gated DeltaNet) layers. Each GDN layer maintains recurrent state:
```
S_t = exp(g_t) * S_{t-1} + outer(k_t, delta_t)
```
This state persists across token positions. When you add a pause token:
1. One more forward pass through all layers (standard)
2. One more update to recurrent state S (GDN-specific)
Pause tokens on Qwen 3.5 trigger **both** forms of additional computation. We're not adding recurrence - we're giving existing recurrence more time to develop.
---
## Minimal Test: Random Prefix (Zero Training)
The dl1683 paper showed random embeddings work at inference time without training:
- Qwen3-4B arithmetic: 32% → 51.6% (+19.6pp)
- 100% oracle coverage on planning tasks
### Test Script
```python
#!/usr/bin/env python3
"""Test pause tokens on Qwen 3.5 27B.
Usage:
source ~/training-env/bin/activate
python3 test_pause_tokens.py
"""
import torch
from transformers import AutoTokenizer
# Reuse our weight loading infrastructure
import sys
sys.path.insert(0, '.')
from extract_steering_vector import load_model
GSM8K_SAMPLES = [
"Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
"A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?",
# Add more samples...
]
def get_embedding_rms(model):
"""Get RMS of embedding weights for proper scaling."""
embed = model.model.embed_tokens.weight
return embed.float().square().mean().sqrt().item()
def make_random_prefix(n_tokens, embed_dim, rms, device):
"""Generate random prefix embeddings at embedding scale."""
prefix = torch.randn(1, n_tokens, embed_dim, device=device, dtype=torch.bfloat16)
return prefix * rms
def generate_with_pause(model, tokenizer, prompt, n_pause=0, max_new=256):
"""Generate with optional pause token prefix."""
input_ids = tokenizer.encode(prompt, return_tensors='pt').to('cuda:0')
text_embeds = model.model.embed_tokens(input_ids)
if n_pause > 0:
embed_rms = get_embedding_rms(model)
pause_embeds = make_random_prefix(n_pause, text_embeds.shape[-1], embed_rms, text_embeds.device)
combined = torch.cat([pause_embeds, text_embeds], dim=1)
else:
combined = text_embeds
# Generate from embeddings
with torch.no_grad():
outputs = model.generate(
inputs_embeds=combined,
max_new_tokens=max_new,
do_sample=False, # Greedy for reproducibility
pad_token_id=tokenizer.pad_token_id,
)
# Decode (skip pause token positions in output)
return tokenizer.decode(outputs[0], skip_special_tokens=True)
def extract_answer(response):
"""Extract numeric answer from response."""
import re
numbers = re.findall(r'[\d,]+\.?\d*', response)
if numbers:
return numbers[-1].replace(',', '')
return None
def main():
print("Loading model...")
model = load_model()
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-27B", trust_remote_code=True)
print(f"\nEmbedding RMS: {get_embedding_rms(model):.4f}")
for n_pause in [0, 2, 4]:
print(f"\n=== Testing with {n_pause} pause tokens ===")
correct = 0
for i, problem in enumerate(GSM8K_SAMPLES):
prompt = f"Solve this step by step:\n{problem}\n\nAnswer:"
response = generate_with_pause(model, tokenizer, prompt, n_pause=n_pause)
answer = extract_answer(response)
print(f" Problem {i+1}: {answer}")
# TODO: Compare against ground truth
print(f" Accuracy: {correct}/{len(GSM8K_SAMPLES)}")
if __name__ == '__main__':
main()
```
### Test Protocol
1. Pick 20-50 GSM8K problems with known answers
2. Run baseline (n_pause=0)
3. Run with 2 pause tokens
4. Run with 4 pause tokens
5. Compare accuracy
If pause tokens help at inference time with zero training, the GDN recurrence is leveraging the extra iterations.
---
## Learnable Pause Tokens (Training Phase)
After validating random prefix works, train dedicated pause tokens:
```python
# Add to model
model.pause_tokens = nn.Parameter(
torch.randn(4, model.config.hidden_size) * embed_rms
)
# Training forward pass
def forward_with_learned_pause(model, input_ids):
text_embeds = model.model.embed_tokens(input_ids)
pause = model.pause_tokens.unsqueeze(0).expand(text_embeds.shape[0], -1, -1)
combined = torch.cat([pause, text_embeds], dim=1)
return model(inputs_embeds=combined)
```
Key: Must train WITH pause tokens for them to work. Inference-only learned tokens don't help (per Google's pause token paper).
---
## Adaptive Halting via Confidence Readout
For variable-length pause (iterate until confident):
### Extract Confidence Direction
```python
confident = [
"The answer is 42.",
"This will work because the invariant holds.",
"Use mmap here.",
]
uncertain = [
"I think the answer might be 42?",
"This should work, but I'm not sure...",
"Maybe mmap? Or read()?",
]
# Same infrastructure as listening vector
confident_states = get_hidden_states(model, confident, layer=48)
uncertain_states = get_hidden_states(model, uncertain, layer=48)
confidence_vec = confident_states.mean(0) - uncertain_states.mean(0)
```
### Adaptive Loop
```python
def generate_adaptive_pause(model, tokenizer, prompt, max_pause=8, threshold=0.7):
confidence_vec = torch.load('confidence_direction.pt')
input_ids = tokenizer.encode(prompt, return_tensors='pt').to('cuda:0')
h = model.model.embed_tokens(input_ids)
embed_rms = get_embedding_rms(model)
for i in range(max_pause):
# Add one pause token
pause = make_random_prefix(1, h.shape[-1], embed_rms, h.device)
h = torch.cat([pause, h], dim=1)
# Forward to get hidden state
with torch.no_grad():
out = model(inputs_embeds=h, output_hidden_states=True)
# Check confidence at layer 48
hidden = out.hidden_states[48][0, -1, :]
confidence = torch.cosine_similarity(
hidden.unsqueeze(0),
confidence_vec.unsqueeze(0)
).item()
if confidence > threshold:
break
# Generate from accumulated state
return model.generate(inputs_embeds=h, max_new_tokens=256)
```
---
## Connection to Huginn/Looping Architectures
Huginn uses explicit weight-tied loops (same 4 layers run N times). We can't retrofit this to Qwen 3.5 without retraining.
But GDN recurrence + pause tokens achieves similar effect:
- Huginn: explicit iteration over layers
- GDN + pause: implicit iteration via recurrent state S
The GDN state accumulates across pause positions, effectively giving the model multiple "thinking steps" before output.
### Comparison
| Approach | Requires Pretraining | Compute Cost | Qwen 3.5 Compatible |
|----------|---------------------|--------------|---------------------|
| Huginn loops | Yes | N × core layers | No |
| Pause tokens | No (inference test) | N × all layers | Yes |
| GDN recurrence | Already there | Per-token | Already there |
| Pause + GDN | No | N × all layers + N state updates | Yes |
---
## COCONUT Integration (Future)
COCONUT feeds hidden state back as input embedding - explicit whole-model recurrence:
```python
def coconut_forward(model, input_ids, n_latent=3):
h = model.model.embed_tokens(input_ids)
for step in range(n_latent):
out = model(inputs_embeds=h, output_hidden_states=True)
# Project hidden state back to embedding space
h = model.project_hidden_to_embed(out.hidden_states[-1])
# Final forward produces tokens
return model.generate(inputs_embeds=h)
```
This gives two levels of iteration:
1. GDN recurrence within each forward pass (automatic)
2. Hidden → embed looping across forward passes (COCONUT)
Requires training the projection layer. Curriculum: start with 0 latent steps, gradually increase.
---
## Implementation Priority
1. **Now:** Run random prefix test (zero training, 1 hour)
2. **If works:** Extract confidence direction for adaptive halting
3. **Training phase:** Learn pause tokens + UPFT (75% time savings)
4. **Later:** COCONUT curriculum for explicit hidden state looping
---
## Open Questions
1. Does random prefix scale to 27B? (Tested on 4B)
2. Optimal pause count for Qwen 3.5?
3. Does GDN respond more strongly than pure attention? (Testable)
4. Can we read confidence from GDN state S directly, not just hidden state h?
---
## References
- Random Prefix: https://github.com/dl1683/Latent-Space-Reasoning
- Pause Tokens: Google, "Think before you speak" (Oct 2023)
- COCONUT: Meta, "Training LLMs to Reason in Continuous Latent Space" (Dec 2024)
- Huginn: Geiping et al., "Scaling Test-Time Compute with Latent Reasoning" (Feb 2025)
- GDN Architecture: Our qwen35-gdn-implementation-findings-mar28 memory