Index functions now take &WriteTransaction instead of &Database,
allowing callers to batch multiple index operations in a single
transaction. Store mutations (upsert, delete, rename, etc.) now
begin_write/commit their own transactions, ensuring atomicity.
- replay_relations uses single txn for all relation indexing
- Store::db() exposes Database for callers needing txn control
- Convenience wrappers open their own txn for simple cases
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The relations Vec is gone from Store. dedup now iterates via
edges_for_uuid() instead of mutating in-memory Vec — removes/re-adds
edges through the index directly.
Removed load_relations_vec() and clear_relations() — no longer needed.
Added helper methods: edges_for_uuid, index_relation, remove_relation_from_index.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
- Add index::clear_relations() to drop and recreate RELS table
- Add Store::reindex_relations() to rebuild index from Vec
- Call reindex_relations() at end of dedup command
This ensures index stays in sync with Vec after complex mutations
like UUID redirection in dedup. Vec mutations remain for now but
index is correctly updated afterward.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
Complete redb schema with bidirectional relation indexing:
- RELS multimap: uuid → packed(other_uuid, strength, rel_type, is_outgoing)
- Each edge stored twice (once per endpoint) with direction bit
- pack_rel/unpack_rel for 22-byte packed format
Wired up:
- replay_relations indexes all relations on load
- add_relation indexes new relations
- for_each_relation reads from index (graph building)
- add_link uses index for existence check
- set_link_strength finds/updates edges via index
- cap_degree uses index for degree counting and pruning
- rename_node finds edges by uuid
Vec<Relation> still maintained for remaining uses (normalize_strengths,
graph_health diagnostics). To be removed in follow-up.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
- capnp.rs: remove reference to removed self.nodes field
- parser.rs: comment out tests for not-yet-implemented features
(not-visited filter, recency() in composite sorts)
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
All node access now goes through index → capnp:
- scoring.rs: consolidation_priority, replay_queue, consolidation_plan
- admin.rs: cmd_init, cmd_fsck, cmd_dedup
- engine.rs: run_generator, eval_filter, run_transform
- parser.rs: resolve_field, execute_query
Added Store::remove_from_index() for dedup cleanup.
The relations Vec remains for now (used for graph building).
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
- Add get_node() and contains_key() methods that read via redb index
- Migrate all store/ reads to use index lookup
- Remove HashMap cache updates from mutations (write-through to capnp+index only)
- Remove replay_nodes() - load no longer builds HashMap
- Update db_is_healthy to validate by spot-checking offsets
- Fix set_weight bug: now persists weight changes to capnp
Store.nodes HashMap still exists for code outside store/ module,
but store/ itself no longer uses it.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Consolidate capnp serialization in one place:
- capnp_enum! and capnp_message! macros
- read_text/read_uuid helpers
- Type-to-capnp mappings
- from_capnp_migrate migration impls
types.rs now only has pure Rust types and helpers.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
With singleton Store (one daemon, RPC for clients), there's no concurrent
writers to capnp log. The file-based flock and incremental refresh logic
was for multi-process coordination we no longer need.
-110 lines of dead concurrency code.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Restructure store module with clearer file names:
- persist.rs → capnp.rs (capnp log IO)
- db.rs → index.rs (redb index operations)
redb now stores key → offset mapping, not serialized nodes.
Mutations record the offset after appending to capnp log.
rebuild_index scans capnp log to reconstruct the index.
The HashMap still exists for now; next step is to use the
index for lookups and remove it.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Mutations (upsert_node, upsert_provenance, delete_node, rename_node)
now update redb indices atomically with capnp log appends, under the
same StoreLock.
Also removes dead cmd_import command and the parse.rs module it depended on.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Remove AgentVisit, TranscriptSegment, and all related visit tracking code.
Provenance is what we've been using to track agent interaction with nodes.
Also removes dead fields from Node (state_tag, created).
-349 lines.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
- Add db: Option<Database> field to Store
- Store::load() opens redb after replaying capnp logs
- Health check compares node count + spot checks keys
- Rebuilds automatically if db is missing, corrupt, or stale
- Make table definitions public for cross-module access
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
- Remove CACHED_STORE, cached(), is_stale(), set_store() - redundant
- Convert all Store::cached() callers to use access_local()
- Single Store::load() call remains in access() fallback path
All store access now goes through hippocampus::access() / access_local(),
which handles socket connection or local fallback with caching.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Convert cmd_fsck to async and use access_local() for the cached store.
Still uses Store::load_from_logs() for fresh comparison.
Remove unused AnyView::load() method - was never called.
Remaining Store::load() calls are all internal caching infrastructure:
- persist.rs cached() for CACHED_STORE
- mod.rs access() fallback for STORE_ACCESS
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Replace Result<_, String> with anyhow::Result throughout:
- hippocampus/store module (persist, ops, types, view, mod)
- CLI modules (admin, agent, graph, journal, node)
- Run trait in main.rs
Use .context() and .with_context() instead of .map_err(|e| format!(...))
patterns. Add bail!() for early error returns.
Add access_local() helper in hippocampus/mod.rs that returns
Result<Arc<Mutex<Store>>> for direct local store access.
Fix store access patterns to properly lock Arc<Mutex<Store>> before
accessing fields in mind/unconscious.rs, mind/mod.rs, subconscious/learn.rs,
and hippocampus/memory.rs.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
These were early experiments with manual feedback signals that
never worked well. The scoring system will handle this properly.
Removed:
- CLI: used, wrong, not-relevant, not-useful, gap
- MCP: memory_used
- Store: mark_used, mark_wrong, record_gap, modify_node
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
TrackedMutex and TrackedRwLock wrappers that record hold durations
by source location using #[track_caller]. Stats written to
~/.consciousness/lock-stats.json every second, sorted by max hold time.
Re-exported as crate::Mutex so all locks are instrumented. To disable,
swap the re-export back to tokio::sync::Mutex.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Duplicate key warnings fire on every store load and were writing to
stderr, corrupting the TUI display. Log write warnings and MCP
server failures are similarly routine. Route these to dbglog.
Serious errors (rkyv snapshot failures, store corruption) remain on
stderr — those are real problems the user needs to see.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
- Agent state (outputs) persists across runs in subconscious-state.json,
loaded on startup, saved after each run completes
- Merge semantics: each run's outputs accumulate into persistent_state
rather than replacing
- Walked keys restored from surface agent state on load
- Store::recent_by_provenance() queries nodes by agent provenance for
the store activity view
- Switch outputs from HashMap to BTreeMap for stable display ordering
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Store::cached() returns a process-global Arc<tokio::sync::Mutex<Store>>
that loads once and reloads only when log files change (is_stale()
checks file sizes). All memory and journal tools use cached_store()
instead of Store::load() per invocation.
Fixes CPU saturation from HashMap hashing when multiple subconscious
agents make concurrent tool calls.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Memory scoring now uses the graph as source of truth:
- last_scored timestamp on each node (new capnp field @22)
- Nodes scored when older than scoring_interval_secs (default 1hr)
- Oldest-scored-first ordering
- Window: scoring_response_window assistant responses (default 100)
- First-quarter memories scored even without full window
- Per-response normalization (raw divergence / response count)
- Asymmetric weight update: alpha=0.5 up, alpha=0.1 down
(responds fast to importance, decays slowly — memories stay
surfaced even if only useful 1/4 of the time)
Graph writes disabled pending normalization calibration.
Also: configurable scoring_interval_secs and scoring_response_window.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Delete subconscious/transcript.rs (94 lines), is_segment_mined,
mark_segment_mined — all orphaned by the extraction pipeline removal.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
pub → pub(crate) for SseReader methods (used across child modules).
pub → pub(super) for openai::stream_events, tool definitions, store
helpers. pub → private for normalize_link and differentiate_hub_with_graph
(only used within their own files).
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Provenance now flows as a function parameter through the entire tool
dispatch chain: thought::dispatch → memory::dispatch → store methods.
Removed task_local (TASK_AGENT), thread_local (TASK_PHASE), and env
var (POC_PROVENANCE) from the tool dispatch path. The env var remains
only as a fallback for non-tool paths (CLI commands, digest).
Phase names are passed from knowledge.rs → llm.rs → api.rs, and
api.rs updates the provenance string between steps. No globals needed.
Split TASK_PROVENANCE into TASK_AGENT (task_local, set once per agent
run) and TASK_PHASE (thread_local, updated between steps). Provenance
now reports "agent:surface-observe:observe" instead of just
"agent:surface-observe", making it possible to identify which pipeline
phase created a node.
Priority: task_local agent + thread_local phase > POC_PROVENANCE env
var > "manual".
Also includes memory_search catchup throttle and pipelining fixes
from the surface-observe refactor.
- journal_new: key is slugified title (agent names things properly)
- journal_tail: sort by created_at (immutable), not timestamp (mutable)
- journal_update: find latest by created_at
- {{latest_journal}}: query by NodeType::EpisodicSession, not "journal" key
- poc-memory journal write: requires a name argument
- Removed all journal#j-{timestamp}-{slug} patterns from:
- prompts.rs (rename candidates)
- graph.rs (date extraction, organize skip list)
- cursor.rs (date extraction)
- store/mod.rs (doc comment)
- graph.rs organize: filter by NodeType::Semantic instead of key prefix
- cursor.rs: use created_at for date extraction instead of key parsing
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
Creates the link if it doesn't exist, avoiding wasted agent turns
from the link_set/link_add confusion.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
MemoryNode moved from agent/memory.rs to hippocampus/memory.rs — it's
a view over hippocampus data, not agent-specific.
Store operations (set_weight, set_link_strength, add_link) moved into
store/ops.rs. CLI code (cli/graph.rs, cli/node.rs) and agent tools
both call the same store methods now. render_node() delegates to
MemoryNode::from_store().render() — 3 lines instead of 40.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
hippocampus/ — memory storage, retrieval, and consolidation:
store, graph, query, similarity, spectral, neuro, counters,
config, transcript, memory_search, lookups, cursor, migrate
subconscious/ — autonomous agents that process without being asked:
reflect, surface, consolidate, digest, audit, etc.
All existing crate::X paths preserved via re-exports in lib.rs.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
