Commit graph

1150 commits

Author SHA1 Message Date
Kent Overstreet
b649a11645 hours_since_last_dream: return 0 if dream in progress
The function was reading from dream-log.jsonl which only updates
when dreams complete. If a dream session was started but not yet
ended, it would show stale hours. Now checks for active dream
state first.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 21:58:03 -04:00
Kent Overstreet
81e0632cf3 DMN: wire dream hours reminder into Foraging state
The hours_since_last_dream() function existed but wasn't called
after refactoring moved the DMN prompts from hooks to Rust.
Now shows "You haven't dreamed in X hours" when >= 18h since
last dream session.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 21:52:20 -04:00
Kent Overstreet
4603947506 Display memory scores in status column
Move score display from name (via label()) to status column for cleaner
layout. Score now appears right of tokens for all memory nodes.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 06:08:27 -04:00
Kent Overstreet
7046e63b9d Include identity nodes in memory scoring
Identity memory nodes now participate in importance scoring alongside
conversation memories. Score loading/saving handles both sections, and
the conscious screen uses node.label() consistently for memory display.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 05:59:58 -04:00
Kent Overstreet
ba4e01b6f3 store: add weight to index, index-only key matching
- KEY_TO_UUID now stores weight (30 bytes: uuid+type+ts+deleted+weight)
- UUID_OFFSETS changed to composite key for O(log n) max-offset lookup
- Add NODES_BY_TYPE index for efficient type+date range queries
- Add for_each_key_weight() to StoreView for index-only iteration
- match_seeds uses index-only path when content not needed
- Fix transaction consistency in ops (single txn for related updates)
- rebuild() now records all uuid→offset mappings for version history
- Backwards compatible: old index formats decoded with default weight

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2026-04-15 05:14:04 -04:00
Kent Overstreet
fc978e2f2e Remove find_context_files — identity comes from memory nodes
Deleted the directory-walking CLAUDE.md/POC.md loader. Identity now
comes entirely from personality_nodes in the memory graph.

Simplified:
- assemble_context_message() takes just personality_nodes
- Removed config_file_count/memory_file_count tracking
- reload_for_model() → reload_context() (no longer model-specific)

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
2026-04-15 03:11:27 -04:00
Kent Overstreet
e847a313b4 memory_render: default to no links footer
Links clutter context windows. Use memory_links() to see links.
Pass raw=false explicitly if you want the footer.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-04-15 03:00:17 -04:00
Kent Overstreet
82eeb9807e Add -tool exclusion syntax, exclude delete/restore for agents
memory_delete and memory_restore are now in memory_tools() (available
via MCP for CLI). Agent tool lists support "-tool_name" to exclude.
Agents automatically exclude memory_delete and memory_restore.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-04-15 02:44:13 -04:00
Kent Overstreet
a88428d642 Simplify context config: personality_nodes and agent_nodes
Replace complex context_groups (with ContextGroup struct, ContextSource
enum, labels, keys arrays) with simple string lists:
- personality_nodes: loaded into main session context
- agent_nodes: loaded into subconscious agent context

Removed ~200 lines of code. The distinction between session and agent
context is now just which list you're in, not a per-group flag.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-04-15 02:37:49 -04:00
Kent Overstreet
688e8dbc3e Remove ContextSource::File — all identity in store now
Identity files migrated to memory nodes:
- identity, core-personality, reflections, where-am-i

Removed:
- ContextSource::File enum variant
- File source parsing and handling
- load_memory_file helper function

Config now only supports Store and Journal sources.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 02:21:07 -04:00
Kent Overstreet
e8462af505 Remove .md suffix stripping from key lookups
The strip_md_suffix function was removed but its usages remained,
causing lookups like `identity.md` to fail (stripped to `identity`
which didn't exist). Now keys are used as-is.

Renamed 4 nodes that had .md suffixes to canonical form:
- identity.md → identity
- promotion-work-queue.md-* → promotion-work-queue-*
- patterns.md#* → patterns-*
- practices.md#* → practices-*

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 02:08:35 -04:00
Kent Overstreet
6c28eebb3f TUI: redirect stderr to log file and display in UI
Raw terminal mode swallows stderr output, making debugging difficult.
Now redirects stderr through a pipe to:
1. Log file at ~/.consciousness/logs/tui-stderr.log (persistent)
2. Channel polled by UI thread (shown as notifications)

The reader thread ensures both destinations see every line. Original
stderr is restored on exit so post-session errors reach the terminal.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 02:01:41 -04:00
Kent Overstreet
4b710eb7a7 logs: assert non-empty agent names, fix debug.log path
- save_agent_log: assert name is not empty (panic to find the bug)
- AutoAgent:🆕 assert name is not empty
- dbglog: write to daemon/ subdir instead of toplevel logs/

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 01:52:31 -04:00
Kent Overstreet
bf5def4871 logs: write debug.log to daemon/ subdir
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 01:47:15 -04:00
Kent Overstreet
90e68d6081 deps: add tempfile for fsck index comparison
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 01:40:44 -04:00
Kent Overstreet
2a7b0daea1 agent: remove memory_delete from tools, supersede transfers links
- memory_delete no longer exposed to agents - use supersede instead
- memory_supersede now transfers all edges from old node to new node
  (keeps whichever strength is higher if new node already has the link)
  This preserves graph structure during consolidation.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 01:40:34 -04:00
Kent Overstreet
6a5b840db3 cli: add 'node restore' command for undeleting nodes
Restores a deleted node to its last non-deleted content with proper
version continuity (version number continues from absolute latest,
content from last live version).

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 01:40:29 -04:00
Kent Overstreet
290505fc51 store: fsck improvements, fix index rebuild and batch offset bug
- Add fsck_full(): compares current index with rebuilt, reports zombies/missing
- Add repair_index(): rebuilds index from capnp log
- Index rebuild now uses timestamp (not version) for "latest" detection
  Fixes tombstones shadowing restored nodes when version numbers reset
- Add read_node_at_offset_for_key() to handle batch writes correctly
  When multiple nodes share an offset, filter by key to get the right one
- Add find_latest_by_key() and find_last_live_version() for restore support

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 01:40:24 -04:00
Kent Overstreet
6ec7fcb777 store: protected nodes, explicit provenance in mutations
- Add protected_nodes config list - blocks delete/rename of core nodes
- Remove current_provenance() env var lookup, pass provenance explicitly
- delete_node, rename_node, set_link_strength now take provenance param
- Fix new_relation calls in admin.rs to pass "system" provenance

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 01:40:18 -04:00
Kent Overstreet
cc29cd2225 provenance: new_relation takes explicit provenance parameter
Remove POC_PROVENANCE env var lookup from new_relation - callers
now pass provenance explicitly. This fixes tracking when the env
var wasn't set correctly.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-15 01:39:58 -04:00
Kent Overstreet
5d6e663b60 thalamus: add thinking mode toggles (native + tool)
Two independent toggles on the thalamus screen:
- 't' toggles native Qwen <think> tags (adds <think>\n to generation prompt)
- 'T' toggles think tool (Anthropic-style structured reasoning tool)

Both can be enabled simultaneously. Native thinking is on by default.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-14 18:25:00 -04:00
Kent Overstreet
be909028a7 update tokenizers 2026-04-13 22:39:50 -04:00
Kent Overstreet
4d22a28794 unconscious: event-driven loop via tokio::select!
Replace yield_now() polling with proper event-driven wakeups:
- Add wake: Arc<Notify> to Unconscious struct
- Spawned agents call wake.notify_one() on completion
- Loop uses select! on: unc_rx.changed(), wake.notified(), health timer

Eliminates spinning (was 27.9M iterations per interval).

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 22:38:01 -04:00
Kent Overstreet
19789b7e74 index: add NODES_BY_PROVENANCE with timestamp-sorted values
- Store [negated_timestamp:8][key] as value for descending sort
- recent_by_provenance uses index directly, no capnp reads
- Eliminates 24k×5 capnp reads from subconscious snapshots

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 22:25:12 -04:00
Kent Overstreet
a966dd9d5d kill rusqlite dep 2026-04-13 22:16:09 -04:00
Kent Overstreet
faad14dc95 graph: use index for bulk reads, skip capnp deserialization
- Add all_keys() to StoreView, use in build_adjacency instead of
  for_each_node (which was ignoring content/weight anyway)
- Add all_key_uuid_pairs() for single-pass uuid mapping
- Extend KEY_TO_UUID to store [uuid:16][node_type:1][timestamp:8]
- for_each_node_meta now reads from index, no capnp needed
- Add NodeType::from_u8() for unpacking

Graph health: 7s → 2s (3.5x faster)

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 22:15:13 -04:00
Kent Overstreet
b3d0a3ab25 store: internal locking, remove Arc<Mutex<Store>> wrapper
Store now has internal Mutex for capnp appends and AtomicU64 for
size tracking. All methods take &self. The external Arc<Mutex<Store>>
is replaced with Arc<Store>.

- Store::append_lock protects file appends
- local.rs functions take &Store (not &mut Store)
- access_local() returns Arc<Store>
- All .lock().await calls removed from callers

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 21:49:54 -04:00
Kent Overstreet
4696bb8b7d store: index ops take WriteTransaction, mutations batch properly
Index functions now take &WriteTransaction instead of &Database,
allowing callers to batch multiple index operations in a single
transaction. Store mutations (upsert, delete, rename, etc.) now
begin_write/commit their own transactions, ensuring atomicity.

- replay_relations uses single txn for all relation indexing
- Store::db() exposes Database for callers needing txn control
- Convenience wrappers open their own txn for simple cases

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 21:44:20 -04:00
Kent Overstreet
2548ca059d store: remove Vec<Relation>, dedup uses index iteration
The relations Vec is gone from Store. dedup now iterates via
edges_for_uuid() instead of mutating in-memory Vec — removes/re-adds
edges through the index directly.

Removed load_relations_vec() and clear_relations() — no longer needed.
Added helper methods: edges_for_uuid, index_relation, remove_relation_from_index.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 21:32:48 -04:00
Kent Overstreet
c2de14dcab store: add reindex_relations for after Vec mutations
- Add index::clear_relations() to drop and recreate RELS table
- Add Store::reindex_relations() to rebuild index from Vec
- Call reindex_relations() at end of dedup command

This ensures index stays in sync with Vec after complex mutations
like UUID redirection in dedup. Vec mutations remain for now but
index is correctly updated afterward.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-04-13 21:24:49 -04:00
Kent Overstreet
58b0947625 admin: convert fsck and dedup reads to use index
- fsck: use for_each_relation for dangling edge detection
  (pruning deferred - needs delete_edge operation)
- dedup: use for_each_relation for edge counting

Remaining Vec uses in dedup mutation section need new index ops:
- redirect_edge: change source/target UUID
- delete_edge_by_uuid: tombstone by UUID

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-04-13 21:22:52 -04:00
Kent Overstreet
5832e57970 store: convert more callers to use RELS index
Convert remaining Vec users to index-based access:
- memory.rs: MemoryNode::from_store uses Store::neighbors()
- graph.rs: orphan detection uses for_each_relation
- local.rs: normalize_strengths uses for_each_relation + set_link_strength

Add Store::neighbors() method and index::get_offsets_for_uuid().

Cleanup:
- for_each_relation: build both uuid↔key maps in one pass
- cap_degree: consolidate key/uuid/degree collection

Remaining Vec uses: admin.rs (fsck, dedup), capnp.rs (load path).

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-04-13 21:20:27 -04:00
Kent Overstreet
5fe51fbfda store: wire up RELS index for relations
Complete redb schema with bidirectional relation indexing:
- RELS multimap: uuid → packed(other_uuid, strength, rel_type, is_outgoing)
- Each edge stored twice (once per endpoint) with direction bit
- pack_rel/unpack_rel for 22-byte packed format

Wired up:
- replay_relations indexes all relations on load
- add_relation indexes new relations
- for_each_relation reads from index (graph building)
- add_link uses index for existence check
- set_link_strength finds/updates edges via index
- cap_degree uses index for degree counting and pruning
- rename_node finds edges by uuid

Vec<Relation> still maintained for remaining uses (normalize_strengths,
graph_health diagnostics). To be removed in follow-up.

Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-04-13 21:12:47 -04:00
Kent Overstreet
8cfe9a4d70 fix stale comment and skip unimplemented query tests
- capnp.rs: remove reference to removed self.nodes field
- parser.rs: comment out tests for not-yet-implemented features
  (not-visited filter, recency() in composite sorts)

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 20:20:44 -04:00
Kent Overstreet
c9b51c941e store/index: remove unused get_key_by_uuid and node_count
Speculative helpers that were never called. Easy to re-add if needed.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 20:12:54 -04:00
Kent Overstreet
5877fd857a store: remove nodes and uuid_to_key HashMaps
All node access now goes through index → capnp:
- scoring.rs: consolidation_priority, replay_queue, consolidation_plan
- admin.rs: cmd_init, cmd_fsck, cmd_dedup
- engine.rs: run_generator, eval_filter, run_transform
- parser.rs: resolve_field, execute_query

Added Store::remove_from_index() for dedup cleanup.

The relations Vec remains for now (used for graph building).

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 19:49:09 -04:00
Kent Overstreet
af3e41f1d9 migrate more files to use index-based node access
- learn.rs, daemon.rs, graph.rs, digest.rs, prompts.rs
- Convert store.nodes.get() → store.get_node()
- Convert store.nodes.contains_key() → store.contains_key()
- Convert store.nodes.values/iter() → all_keys + get_node

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 19:37:11 -04:00
Kent Overstreet
fe6450223c migrate local.rs and memory.rs to use index
- Add Store::all_keys() method for iteration
- Convert store.nodes.get() → store.get_node()
- Convert store.nodes.contains_key() → store.contains_key()
- Convert store.nodes.values() iteration → all_keys + get_node

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 19:34:45 -04:00
Kent Overstreet
7eb86656d4 store: read nodes via index instead of HashMap
- Add get_node() and contains_key() methods that read via redb index
- Migrate all store/ reads to use index lookup
- Remove HashMap cache updates from mutations (write-through to capnp+index only)
- Remove replay_nodes() - load no longer builds HashMap
- Update db_is_healthy to validate by spot-checking offsets
- Fix set_weight bug: now persists weight changes to capnp

Store.nodes HashMap still exists for code outside store/ module,
but store/ itself no longer uses it.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 19:31:28 -04:00
Kent Overstreet
ba53597cf2 store: move all capnp code to capnp.rs
Consolidate capnp serialization in one place:
- capnp_enum! and capnp_message! macros
- read_text/read_uuid helpers
- Type-to-capnp mappings
- from_capnp_migrate migration impls

types.rs now only has pure Rust types and helpers.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 19:21:56 -04:00
Kent Overstreet
e48ca2ecad store: remove StoreLock and refresh_nodes
With singleton Store (one daemon, RPC for clients), there's no concurrent
writers to capnp log. The file-based flock and incremental refresh logic
was for multi-process coordination we no longer need.

-110 lines of dead concurrency code.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 19:13:25 -04:00
Kent Overstreet
f413a853d8 store: redb indexes offsets into capnp log, not full nodes
Restructure store module with clearer file names:
- persist.rs → capnp.rs (capnp log IO)
- db.rs → index.rs (redb index operations)

redb now stores key → offset mapping, not serialized nodes.
Mutations record the offset after appending to capnp log.
rebuild_index scans capnp log to reconstruct the index.

The HashMap still exists for now; next step is to use the
index for lookups and remove it.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 19:10:08 -04:00
Kent Overstreet
9309de68fc store: wire up redb updates on mutations
Mutations (upsert_node, upsert_provenance, delete_node, rename_node)
now update redb indices atomically with capnp log appends, under the
same StoreLock.

Also removes dead cmd_import command and the parse.rs module it depended on.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 19:03:09 -04:00
Kent Overstreet
a1accc7cd4 store: remove visit tracking infrastructure
Remove AgentVisit, TranscriptSegment, and all related visit tracking code.
Provenance is what we've been using to track agent interaction with nodes.

Also removes dead fields from Node (state_tag, created).

-349 lines.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 18:57:12 -04:00
Kent Overstreet
7d49f29fde store: remove dead code and move params to config
Remove:
- score_weight() - never called
- position field on Node - never read (was for export)
- Provenance enum - inline helper for capnp migration
- migrate_transcript_progress + CLI command
- init_from_markdown, import_file, ingest_units
- export command and export_to_markdown
- RetrievalEvent, GapRecord types
- classify_filename, new_transcript_segment

Move spreading activation params to Config:
- default_node_weight, edge_decay, max_hops, min_activation
- Remove Params struct and StoreView::params()

Simplify cmd_init to just seed identity via upsert().
Simplify cmd_import to use parse_units + upsert directly.

-576 lines

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 18:50:21 -04:00
Kent Overstreet
6104c63890 Integrate redb into Store::load() with health check
- Add db: Option<Database> field to Store
- Store::load() opens redb after replaying capnp logs
- Health check compares node count + spot checks keys
- Rebuilds automatically if db is missing, corrupt, or stale
- Make table definitions public for cross-module access

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 18:33:47 -04:00
Kent Overstreet
2caccf875d Replace rkyv/bincode caching with redb indices
Remove three-tier loading (rkyv snapshot, bincode cache, capnp replay)
in favor of direct capnp log replay + redb for indexed access.

- Remove all rkyv derives from types (Node, Relation, enums, etc.)
- Remove Snapshot struct, RKYV_MAGIC, CACHE_MAGIC constants
- Remove load_snapshot_mmap(), save(), save_snapshot()
- Remove MmapView, AnyView from view.rs (keep StoreView trait)
- Simplify Store::load() to just replay capnp logs
- Add db.rs with redb schema: nodes, uuid_to_key, visits, transcript_progress
- Simplify cmd_fsck to just check capnp integrity + graph health

capnp logs remain source of truth; redb indices will be rebuilt on demand.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 18:30:58 -04:00
Kent Overstreet
1d88293ccf Remove Store::cached(), consolidate on access_local()
- Remove CACHED_STORE, cached(), is_stale(), set_store() - redundant
- Convert all Store::cached() callers to use access_local()
- Single Store::load() call remains in access() fallback path

All store access now goes through hippocampus::access() / access_local(),
which handles socket connection or local fallback with caching.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 18:11:58 -04:00
Kent Overstreet
09b30d64f2 cmd_fsck: use access_local(), remove dead AnyView::load()
Convert cmd_fsck to async and use access_local() for the cached store.
Still uses Store::load_from_logs() for fresh comparison.

Remove unused AnyView::load() method - was never called.

Remaining Store::load() calls are all internal caching infrastructure:
- persist.rs cached() for CACHED_STORE
- mod.rs access() fallback for STORE_ACCESS

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 18:09:02 -04:00
Kent Overstreet
f6f330b07b Convert cmd_import, cmd_export, MigrateTranscriptProgress to access_local()
These were the last Store::load() calls that should use the shared store.
Remaining calls are intentional: fsck (needs both cached and fresh),
persist.rs cached() infrastructure, view.rs read-only fallback, and
access() bootstrap path.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 18:06:58 -04:00