kent/consciousness
1
0
Fork 2
Code Issues 1 Pull requests 5 Projects Releases Packages Wiki Activity Actions

Compare commits

base: kent:a966dd9d5db19a823bc292df970bf034baa8808f
Branches Tags
kent:master
kent:agent-model-dispatch
..
compare: kent:0612e1bc41f98a7b8dac66dc8afdf6c472d44996
Branches Tags
kent:master
kent:agent-model-dispatch

No commits in common. "a966dd9d5db19a823bc292df970bf034baa8808f" and "0612e1bc41f98a7b8dac66dc8afdf6c472d44996" have entirely different histories.
a966dd9d5d ... 0612e1bc41

63 changed files with 5344 additions and 5537 deletions
Download patch file Download diff file Expand all files Collapse all files

192
Cargo.lock generated
View file

@ -8,6 +8,17 @@ version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "ahash"
version = "0.7.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "891477e0c6a8957309ee5c45a6368af3ae14bb510732d2684ffa19af310920f9"
dependencies = [
"getrandom 0.2.17",
"once_cell",
"version_check",
]
[[package]]
name = "ahash"
version = "0.8.12"
@ -274,6 +285,18 @@ version = "2.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af"
[[package]]
name = "bitvec"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c"
dependencies = [
"funty",
"radium",
"tap",
"wyz",
]
[[package]]
name = "block-buffer"
version = "0.10.4"
@ -299,6 +322,28 @@ version = "3.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb"
[[package]]
name = "bytecheck"
version = "0.6.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "23cdc57ce23ac53c931e88a43d06d070a6fd142f2617be5855eb75efc9beb1c2"
dependencies = [
"bytecheck_derive",
"ptr_meta",
"simdutf8",
]
[[package]]
name = "bytecheck_derive"
version = "0.6.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3db406d29fbcd95542e92559bed4d8ad92636d1ca8b3b72ede10b4bcc010e659"
dependencies = [
"proc-macro2",
"quote",
"syn 1.0.109",
]
[[package]]
name = "bytemuck"
version = "1.25.0"
@ -476,6 +521,7 @@ dependencies = [
"ast-grep-core",
"ast-grep-language",
"base64 0.22.1",
"bincode",
"bytes",
"capnp",
"capnp-rpc",
@ -500,14 +546,15 @@ dependencies = [
"paste",
"peg",
"ratatui",
"rayon",
"redb",
"regex",
"rkyv",
"rustls",
"rustls-native-certs",
"serde",
"serde_json",
"serde_urlencoded",
"textwrap",
"tokenizers",
"tokio",
"tokio-rustls",
@ -563,7 +610,6 @@ dependencies = [
"dirs",
"env_logger",
"futures",
"json5",
"log",
"serde",
"serde_json",
@ -584,8 +630,8 @@ dependencies = [
"json5",
"libc",
"log",
"scopeguard",
"serde",
"serde_json",
"tokio",
"tokio-util",
]
@ -1082,6 +1128,12 @@ version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "funty"
version = "2.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c"
[[package]]
name = "futures"
version = "0.3.32"
@ -1244,6 +1296,15 @@ dependencies = [
"regex-syntax",
]
[[package]]
name = "hashbrown"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888"
dependencies = [
"ahash 0.7.8",
]
[[package]]
name = "hashbrown"
version = "0.15.5"
@ -2131,6 +2192,26 @@ dependencies = [
"yansi",
]
[[package]]
name = "ptr_meta"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0738ccf7ea06b608c10564b31debd4f5bc5e197fc8bfe088f68ae5ce81e7a4f1"
dependencies = [
"ptr_meta_derive",
]
[[package]]
name = "ptr_meta_derive"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "16b845dbfca988fa33db069c0e230574d15a3088f147a87b64c7589eb662c9ac"
dependencies = [
"proc-macro2",
"quote",
"syn 1.0.109",
]
[[package]]
name = "pulldown-cmark"
version = "0.13.3"
@ -2180,6 +2261,12 @@ version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "radium"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09"
[[package]]
name = "rand"
version = "0.8.5"
@ -2398,6 +2485,15 @@ version = "0.8.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a"
[[package]]
name = "rend"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "71fe3824f5629716b1589be05dacd749f6aa084c87e00e016714a8cdfccc997c"
dependencies = [
"bytecheck",
]
[[package]]
name = "ring"
version = "0.17.14"
@ -2412,6 +2508,35 @@ dependencies = [
"windows-sys 0.52.0",
]
[[package]]
name = "rkyv"
version = "0.7.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2297bf9c81a3f0dc96bc9521370b88f054168c29826a75e89c55ff196e7ed6a1"
dependencies = [
"bitvec",
"bytecheck",
"bytes",
"hashbrown 0.12.3",
"ptr_meta",
"rend",
"rkyv_derive",
"seahash",
"tinyvec",
"uuid",
]
[[package]]
name = "rkyv_derive"
version = "0.7.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "84d7b42d4b8d06048d3ac8db0eb31bcb942cbeb709f0b5f2b2ebde398d3038f5"
dependencies = [
"proc-macro2",
"quote",
"syn 1.0.109",
]
[[package]]
name = "rustc_version"
version = "0.4.1"
@ -2519,6 +2644,12 @@ version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
[[package]]
name = "seahash"
version = "4.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b"
[[package]]
name = "security-framework"
version = "3.7.0"
@ -2682,12 +2813,6 @@ version = "1.15.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03"
[[package]]
name = "smawk"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7c388c1b5e93756d0c740965c41e8822f866621d41acbdf6336a6a168f8840c"
[[package]]
name = "socket2"
version = "0.6.3"
@ -2798,6 +2923,12 @@ dependencies = [
"yaml-rust",
]
[[package]]
name = "tap"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369"
[[package]]
name = "terminfo"
version = "0.9.0"
@ -2861,17 +2992,6 @@ dependencies = [
"winapi",
]
[[package]]
name = "textwrap"
version = "0.16.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c13547615a44dc9c452a8a534638acdf07120d4b6847c8178705da06306a3057"
dependencies = [
"smawk",
"unicode-linebreak",
"unicode-width",
]
[[package]]
name = "thiserror"
version = "1.0.69"
@ -2945,13 +3065,28 @@ dependencies = [
"time-core",
]
[[package]]
name = "tinyvec"
version = "1.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3"
dependencies = [
"tinyvec_macros",
]
[[package]]
name = "tinyvec_macros"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
[[package]]
name = "tokenizers"
version = "0.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a620b996116a59e184c2fa2dfd8251ea34a36d0a514758c6f966386bd2e03476"
dependencies = [
"ahash",
"ahash 0.8.12",
"aho-corasick",
"compact_str",
"dary_heap",
@ -3377,12 +3512,6 @@ version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "unicode-linebreak"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3b09c83c3c29d37506a3e260c08c03743a6bb66a9cd432c6934ab501a190571f"
[[package]]
name = "unicode-normalization-alignments"
version = "0.1.12"
@ -3948,6 +4077,15 @@ dependencies = [
"wasmparser",
]
[[package]]
name = "wyz"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed"
dependencies = [
"tap",
]
[[package]]
name = "yaml-rust"
version = "0.4.5"

5
Cargo.toml
View file

@ -34,9 +34,9 @@ json5 = "1.3"
ratatui = { version = "0.30", features = ["unstable-rendered-line-info"] }
tui-markdown = { git = "https://github.com/koverstreet/tui-markdown", subdirectory = "tui-markdown" }
tui-textarea = { version = "0.10.2", package = "tui-textarea-2" }
textwrap = "0.16"
uuid = { version = "1", features = ["v4"] }
bincode = "1"
regex = "1"
glob = "0.3"
chrono = { version = "0.4", features = ["serde"] }
@ -51,6 +51,9 @@ ast-grep-language = { version = "0.42", features = ["builtin-parser"] }
walkdir = "2"
redb = "4"
rkyv = { version = "0.7", features = ["validation", "std"] }
rayon = "1"
tokio = { version = "1", features = ["full"] }
tokio-util = { version = "0.7", features = ["compat"] }

1
channels/telegram/Cargo.toml
View file

@ -8,7 +8,6 @@ capnp = "0.25"
capnp-rpc = "0.25"
dirs = "6"
futures = "0.3"
json5 = "1.3"
consciousness = { path = "../.." }
serde = { version = "1", features = ["derive"] }
serde_json = "1"

2
channels/telegram/src/main.rs
View file

@ -40,7 +40,7 @@ fn load_config() -> Config {
let config_path = dir.join("telegram.json5");
let text = std::fs::read_to_string(&config_path)
.unwrap_or_else(|_| panic!("failed to read {}", config_path.display()));
let mut config: Config = json5::from_str(&text)
let mut config: Config = serde_json::from_str(&text)
.unwrap_or_else(|e| panic!("failed to parse {}: {}", config_path.display(), e));
// Read token from secrets file

2
channels/tmux/Cargo.toml
View file

@ -8,11 +8,11 @@ capnp = "0.25"
capnp-rpc = "0.25"
dirs = "6"
libc = "0.2"
scopeguard = "1"
futures = "0.3"
json5 = "1.3"
consciousness = { path = "../.." }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio = { version = "1", features = ["full"] }
tokio-util = { version = "0.7", features = ["compat"] }
log = "0.4"

141
channels/tmux/src/main.rs
View file

@ -24,30 +24,26 @@ use consciousness::thalamus::channel_log::ChannelLog;
// ── Config ─────────────────────────────────────────────────────
#[derive(Clone, serde::Serialize, serde::Deserialize)]
#[derive(Clone, serde::Deserialize)]
struct PaneConfig {
/// Tmux pane ID, e.g. "0:1.0"
pane_id: String,
/// Human-readable label, becomes the channel name "tmux.<label>"
label: String,
/// Tmux pane ID, e.g. "%5"
pane_id: String,
}
#[derive(Clone, serde::Serialize, serde::Deserialize)]
#[derive(Clone, serde::Deserialize)]
struct Config {
#[serde(default)]
panes: Vec<PaneConfig>,
}
fn config_path() -> std::path::PathBuf {
dirs::home_dir()
.unwrap_or_default()
.join(".consciousness/channels/tmux.json5")
}
fn load_config() -> Config {
match std::fs::read_to_string(config_path()) {
let path = dirs::home_dir()
.unwrap_or_default()
.join(".consciousness/channels/tmux.json5");
match std::fs::read_to_string(&path) {
Ok(text) => json5::from_str(&text)
.unwrap_or_else(|e| panic!("failed to parse {}: {e}", config_path().display())),
.unwrap_or_else(|e| panic!("failed to parse {}: {e}", path.display())),
Err(_) => {
info!("no tmux.json5, starting with no pre-configured panes");
Config { panes: vec![] }
@ -55,71 +51,23 @@ fn load_config() -> Config {
}
}
fn save_config(config: &Config) {
match serde_json::to_string_pretty(config) {
Ok(json) => {
if let Err(e) = std::fs::write(config_path(), json) {
error!("failed to write config: {}", e);
}
}
Err(e) => error!("failed to serialize config: {}", e),
}
}
// ── State ─────────────────────────────────────────────────────
struct State {
config: Config,
channel_logs: BTreeMap<String, ChannelLog>,
/// Tracks which panes are actually connected (pipe-pane active)
connected: BTreeMap<String, bool>,
/// label → pane_id (e.g. "ktest" → "%0")
panes: BTreeMap<String, String>,
}
type SharedState = Rc<RefCell<State>>;
impl State {
fn new(config: Config) -> Self {
fn new(config: &Config) -> Self {
Self {
config,
channel_logs: BTreeMap::new(),
connected: BTreeMap::new(),
}
}
/// Get pane_id for a label
fn get_pane(&self, label: &str) -> Option<&str> {
self.config.panes.iter()
.find(|p| p.label == label)
.map(|p| p.pane_id.as_str())
}
/// Check if a pane is connected
fn is_connected(&self, label: &str) -> bool {
self.connected.get(label).copied().unwrap_or(false)
}
/// Set connection state for a pane
fn set_connected(&mut self, label: &str, connected: bool) {
self.connected.insert(label.to_string(), connected);
}
/// Add a pane and persist
fn add_pane(&mut self, label: String, pane_id: String) {
if !self.config.panes.iter().any(|p| p.label == label) {
self.config.panes.push(PaneConfig { label, pane_id });
save_config(&self.config);
}
}
/// Remove a pane and persist
fn remove_pane(&mut self, label: &str) -> Option<String> {
if let Some(idx) = self.config.panes.iter().position(|p| p.label == label) {
let pane = self.config.panes.remove(idx);
self.connected.remove(label);
save_config(&self.config);
Some(pane.pane_id)
} else {
None
panes: config.panes.iter()
.map(|p| (p.label.clone(), p.pane_id.clone()))
.collect(),
}
}
}
@ -155,12 +103,10 @@ async fn pipe_pane_reader(state: SharedState, pane: PaneConfig) {
Ok(output) => {
error!("pipe-pane failed for {}: {}", pane.label,
String::from_utf8_lossy(&output.stderr));
state.borrow_mut().set_connected(&pane.label, false);
return;
}
Err(e) => {
error!("failed to run tmux pipe-pane for {}: {}", pane.label, e);
state.borrow_mut().set_connected(&pane.label, false);
return;
}
}
@ -170,14 +116,10 @@ async fn pipe_pane_reader(state: SharedState, pane: PaneConfig) {
Ok(f) => f,
Err(e) => {
error!("failed to open pipe for {}: {}", pane.label, e);
state.borrow_mut().set_connected(&pane.label, false);
return;
}
};
// Mark as connected once pipe is open
state.borrow_mut().set_connected(&pane.label, true);
let reader = tokio::io::BufReader::new(file);
let mut lines = reader.lines();
let channel_key = format!("tmux.{}", pane.label);
@ -194,7 +136,6 @@ async fn pipe_pane_reader(state: SharedState, pane: PaneConfig) {
}
warn!("pipe-pane reader ended for {}", pane.label);
state.borrow_mut().set_connected(&pane.label, false);
}
// ── ChannelServer Implementation ───────────────────────────────
@ -246,7 +187,7 @@ impl channel_server::Server for ChannelServerImpl {
// Send to tmux pane via send-keys
let label = channel.strip_prefix("tmux.").unwrap_or(&channel);
let pane_id = self.state.borrow().get_pane(label).map(String::from);
let pane_id = self.state.borrow().panes.get(label).cloned();
if let Some(pane_id) = pane_id {
let _ = std::process::Command::new("tmux")
.args(["send-keys", "-t", &pane_id, &message, "Enter"])
@ -269,11 +210,10 @@ impl channel_server::Server for ChannelServerImpl {
mut results: channel_server::ListResults,
) -> impl std::future::Future<Output = Result<(), capnp::Error>> {
let s = self.state.borrow();
let channels: Vec<_> = s.config.panes.iter().map(|p| {
let key = format!("tmux.{}", p.label);
let connected = s.is_connected(&p.label);
let channels: Vec<_> = s.panes.keys().map(|label| {
let key = format!("tmux.{}", label);
let unread = s.channel_logs.get(&key).map_or(0, |l| l.unread());
(key, connected, unread)
(key, true, unread)
}).collect();
let mut list = results.get().init_channels(channels.len() as u32);
@ -303,9 +243,12 @@ impl channel_server::Server for ChannelServerImpl {
let label = pry!(pry!(params.get_label()).to_str()).to_string();
// Check if already open
if self.state.borrow().get_pane(&label).is_some() {
{
let s = self.state.borrow();
if s.panes.contains_key(&label) {
return std::future::ready(Ok(()));
}
}
// Find the tmux pane by name (window or pane title)
let pane_id = match find_pane_by_name(&label) {
@ -316,11 +259,14 @@ impl channel_server::Server for ChannelServerImpl {
info!("opening channel tmux.{} (pane {})", label, pane_id);
// Register in state and persist
self.state.borrow_mut().add_pane(label.clone(), pane_id.clone());
// Register in state
{
let mut s = self.state.borrow_mut();
s.panes.insert(label.clone(), pane_id.clone());
}
// Start pipe-pane reader
let pane = PaneConfig { label, pane_id };
let pane = PaneConfig { pane_id, label };
let reader_state = self.state.clone();
tokio::task::spawn_local(async move {
pipe_pane_reader(reader_state, pane).await;
@ -339,7 +285,7 @@ impl channel_server::Server for ChannelServerImpl {
let label = channel.strip_prefix("tmux.").unwrap_or(&channel).to_string();
let mut s = self.state.borrow_mut();
if let Some(pane_id) = s.remove_pane(&label) {
if let Some(pane_id) = s.panes.remove(&label) {
info!("closing channel tmux.{}", label);
s.channel_logs.remove(&format!("tmux.{}", label));
@ -377,6 +323,24 @@ fn find_pane_by_name(name: &str) -> Option<String> {
}
None
}
// ── Cleanup ───────────────────────────────────────────────────
/// Remove pipe-pane connections on exit.
fn cleanup_pipes(config: &Config) {
for pane in &config.panes {
// Disconnect pipe-pane
let _ = std::process::Command::new("tmux")
.args(["pipe-pane", "-t", &pane.pane_id])
.output();
}
// Clean up FIFO files
let pipe_dir = dirs::home_dir()
.unwrap_or_default()
.join(".consciousness/channels/tmux-pipes");
let _ = std::fs::remove_dir_all(&pipe_dir);
}
// ── Main ───────────────────────────────────────────────────────
#[tokio::main]
@ -384,7 +348,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
env_logger::init();
let config = load_config();
let state = Rc::new(RefCell::new(State::new(config)));
let state = Rc::new(RefCell::new(State::new(&config)));
let sock_dir = dirs::home_dir()
.unwrap_or_default()
@ -395,11 +359,16 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
info!("tmux channel daemon starting on {}", sock_path.display());
// Set up cleanup on exit
let cleanup_config = config.clone();
let _cleanup = scopeguard::guard(cleanup_config, |c| cleanup_pipes(&c));
tokio::task::LocalSet::new()
.run_until(async move {
// Start a pipe-pane reader for each configured pane
for pane in state.borrow().config.panes.clone() {
for pane in &config.panes {
let reader_state = state.clone();
let pane = pane.clone();
tokio::task::spawn_local(async move {
pipe_pane_reader(reader_state, pane).await;
});

300
docs/latent-reasoning-integration-plan.md
View file

@ -1,300 +0,0 @@
# Latent Reasoning Integration Plan for Qwen 3.5 27B
**Status:** Research complete, ready for implementation
**Date:** 2026-04-12
**Hardware:** B200 (192GB HBM3e), APOLLO-Mini optimizer
## Executive Summary
Recent research shows multiple approaches to improving LLM reasoning through latent space manipulation. This document synthesizes findings from 10+ papers and maps them to our Qwen 3.5 27B full finetuning pipeline. The key insight: some approaches require pretraining from scratch (skip those), while others can be layered onto existing models during finetuning (prioritize those).
---
## 1. The Landscape
### Approaches That Require Pretraining (Not Applicable)
| Technique | Why Not |
|-----------|---------|
| Huginn/Recurrent Depth (Geiping 2025) | Requires architectural changes from scratch |
| Ouro/LoopLM (ByteDance 2025) | Needs weight-tied looped architecture |
| Quiet-STaR (Stanford 2024) | Heavy continued pretraining overhead |
### Approaches Compatible with Finetuning (Our Focus)
| Technique | Overhead | Training Required | Proven On |
|-----------|----------|-------------------|-----------|
| Random Prefix Perturbation | 2 tokens | None (inference) | Qwen3-4B |
| Pause/Planning Tokens | 2-4 tokens | Yes | 1B models |
| COCONUT Curriculum | Variable | Yes (staged) | General |
| ActAdd Steering Vectors | 1 vector/layer | None (inference) | LLaMA, OPT |
| UPFT (Prefix Fine-Tuning) | 8 tokens | Yes (minimal) | General |
---
## 2. Detailed Technique Analysis
### 2.1 Random Prefix Perturbation (dl1683)
**Mechanism:** Prepend 2 random embedding-scale tokens before input. Breaks attention sink patterns, shifts model into "exploratory computation mode."
**Results:**
- Qwen3-4B arithmetic: 32% → 51.6% (+19.6pp)
- 100% oracle coverage on 25/25 tasks
- Planning: rescues 14-word failures into 650+ word plans
**Why it works:** First few tokens accumulate disproportionate attention (Xiao et al. 2024). Under greedy decoding, degenerate patterns lock in. Perturbation breaks this.
**Integration:** Zero training required. Test at inference first, then consider training WITH random prefixes to internalize the exploration behavior.
### 2.2 Pause Tokens (Google, Oct 2023)
**Mechanism:** Add learnable pause tokens to embedding space. Model processes extra hidden vectors before committing to output.
**Results (1B model):**
- SQuAD: +18% EM score
- CommonSenseQA: +8%
- GSM8K: +1%
**Critical requirement:** MUST be both pretrained AND finetuned with pause tokens. Inference-time-only delays don't work without training.
**Integration:** Add 2-4 learnable tokens to Qwen's embedding matrix, finetune with them prepended to reasoning prompts. Simple architectural change.
### 2.3 COCONUT - Chain of Continuous Thought (Meta, Dec 2024)
**Mechanism:** Feed last hidden state back as next input embedding directly (no decoding to tokens). Enables breadth-first search reasoning.
**Why it matters:** Continuous thoughts can encode multiple alternative next steps simultaneously. Avoids premature commitment to single path.
**Training approach:**
1. Initial stage: train on regular CoT examples
2. Subsequent stages: replace first k reasoning steps with k×c continuous thoughts
3. c is hyperparameter controlling latent thought expansion
**Integration:** Most promising for Qwen 3.5 - curriculum approach from CoT → latent reasoning.
### 2.4 UPFT - Unsupervised Prefix Fine-Tuning (Mar 2025)
**Mechanism:** Train ONLY on initial prefix substrings (as few as 8 tokens). Exploits "Prefix Self-Consistency" - shared initial reasoning steps across diverse solutions.
**Results:**
- Matches Rejection Sampling Fine-Tuning performance
- 75% reduction in training time
- 99% reduction in sampling cost
**Integration:** DIRECTLY APPLICABLE. Train only on reasoning prefix tokens. Massive efficiency gain with APOLLO-Mini.
### 2.5 ActAdd / Activation Engineering (Turner et al., 2023)
**Mechanism:** Compute steering vector by contrasting intermediate activations on prompt pairs. Add during forward pass.
**Results:** SOTA on sentiment shift and detoxification.
**Our existing work:** "Listening" vector at layer 48, magnitude 57, cosine consistency 0.61.
**Integration:** Prototype behaviors with steering vectors, then train permanently into weights. Steering vector as specification → APOLLO training as compilation.
### 2.6 Planning Tokens (ICLR 2024)
**Mechanism:** Learnable token embeddings added before each reasoning step. <0.001% additional parameters.
**Integration:** Add to embedding matrix, train end-to-end with APOLLO.
---
## 3. Our Setup
**Model:** Qwen 3.5 27B
- 64 layers, 5120 hidden dim
- 75% DeltaNet (linear attention) / 25% standard attention
- Native 262K context
**Hardware:** B200 (192GB HBM3e)
- 27B in bf16: ~54GB
- Massive headroom
**Optimizer:** APOLLO-Mini
- Full parameter finetuning
- SGD-like memory (1/1024th of AdamW)
- Parameter grouping for 3D conv1d weights
**Stack:** Crane (Candle-based, 21K lines)
**Existing work:**
- Steering vector extraction (listening: layer 48, cosine 0.61)
- Memory scoring infrastructure
**Unique advantage:** Qwen 3.5's GDN (Gated DeltaNet) layers provide natural infrastructure for continuous thought propagation. The recurrent GDN state is already "latent reasoning" infrastructure waiting to be leveraged.
---
## 4. Recommended Implementation Order
### Tier 1: Immediate (High ROI, Low Risk)
**1. Pause Tokens + UPFT Combination**
- Add 2-4 learnable tokens to embedding space
- Train only on 8-token reasoning prefixes
- Both work with existing architecture
- 75% training time reduction
```python
# Add pause tokens to embedding matrix
pause_tokens = nn.Parameter(torch.randn(4, embed_dim) * embed_rms)
# Prepend to reasoning inputs during training
inputs_embeds = torch.cat([pause_tokens.expand(batch, -1, -1), text_embeds], dim=1)
# UPFT: only compute loss on first 8 tokens of reasoning
loss = loss_fn(logits[:, :8], targets[:, :8])
```
**2. Random Prefix Validation**
- Compute Qwen 3.5 27B embedding RMS
- Test 2-token random prefix at inference
- Establish baseline before finetuning
### Tier 2: After Baseline (Medium Effort)
**3. COCONUT Curriculum**
- Stage 1: Fine-tune on CoT examples normally
- Stage 2: Replace first reasoning step with continuous thought
- Stage 3: Replace first 2 steps
- Gradually move reasoning into latent space
**4. Steering Vector Integration**
- Extract reasoning-specific directions (not just "listening")
- Test combinations: prefix + layer-48 steering
- Bake successful vectors into weights via APOLLO
### Tier 3: Experimental
**5. Multi-layer Steering**
- Our layers of interest: 40, 48, 56 (covering the attention layers)
- Different vectors per layer
- Careful scaling to avoid degradation
**6. DeltaNet-Specific Optimization**
- The 75% DeltaNet architecture may respond differently
- GDN recurrent state as "continuous thought" channel
- This is unexplored territory - potential for novel findings
---
## 5. Implementation Details
### Computing Embedding RMS
```python
embed_weight = model.get_input_embeddings().weight
embed_rms = embed_weight.float().square().mean().sqrt().item()
# Expected: ~0.02-0.03 range for Qwen models
```
### Pause Token Implementation in Crane
```rust
// In model forward pass
fn forward_with_pause(&self, input_ids: &Tensor, pause_tokens: &Tensor) -> Result<Tensor> {
let text_embeds = self.embed_tokens.forward(input_ids)?;
let combined = Tensor::cat(&[pause_tokens, &text_embeds], 1)?;
self.transformer.forward(&combined)
}
```
### UPFT Loss Modification
```python
# Standard: loss over all tokens
# UPFT: loss only over prefix tokens
def upft_loss(logits, targets, prefix_len=8):
return F.cross_entropy(
logits[:, :prefix_len].reshape(-1, vocab_size),
targets[:, :prefix_len].reshape(-1)
)
```
---
## 6. Evaluation Plan
### Benchmarks
| Benchmark | What It Tests | Baseline Needed |
|-----------|---------------|-----------------|
| GSM8K | Arithmetic reasoning | Yes |
| ARC-Challenge | Science reasoning | Yes |
| CommonSenseQA | Commonsense | Yes |
| HumanEval | Code generation | Yes |
| Planning tasks (dl1683) | Multi-step planning | Yes |
### Comparison Matrix
| Configuration | Training Time | Expected Gain |
|---------------|---------------|---------------|
| Baseline (no prefix) | 1x | 0% |
| Random prefix (inference) | 1x | +10-20%? |
| Pause tokens (trained) | 1.1x | +8-18% |
| UPFT only | 0.25x | Match baseline |
| Pause + UPFT | 0.3x | +8-18% |
| COCONUT curriculum | 2x | +15-25%? |
---
## 7. Open Questions
1. **Does random perturbation scale to 27B?** Tested on 4B - effect may differ
2. **Optimal token count for 27B?** 2 optimal for 4B, might change
3. **DeltaNet interaction?** 75% linear attention is untested territory
4. **Composition effects?** Prefix + steering + pause tokens together?
5. **GDN as continuous thought channel?** Novel research direction
---
## 8. Risk Assessment
| Risk | Mitigation |
|------|------------|
| No improvement at 27B scale | Start with inference-time validation |
| Training instability with pause tokens | Start with 2 tokens, scale up |
| UPFT doesn't transfer | Fall back to full token loss |
| DeltaNet behaves differently | Ablate on attention-only layers first |
---
## 9. Timeline Estimate
| Phase | Duration | Deliverable |
|-------|----------|-------------|
| Embedding RMS + baseline | 1 day | Numbers |
| Random prefix validation | 1 day | Inference results |
| Pause token implementation | 2 days | Crane modification |
| UPFT integration | 1 day | Training loop change |
| First finetuning run | 2-3 days | Trained model |
| Evaluation | 1 day | Benchmark numbers |
| COCONUT curriculum | 1 week | Staged training |
---
## 10. References
### Primary Sources
- Random Prefix: https://github.com/dl1683/Latent-Space-Reasoning
- Attention Sinks: Xiao et al., "Efficient Streaming Language Models with Attention Sinks" (Sept 2023)
- Pause Tokens: Google, "Think before you speak" (Oct 2023)
- COCONUT: Meta, "Training Large Language Models to Reason in a Continuous Latent Space" (Dec 2024)
- UPFT: "Prefix Self-Consistency for Unsupervised Fine-Tuning" (Mar 2025)
- ActAdd: Turner et al., "Activation Addition: Steering Language Models Without Optimization" (Aug 2023)
- Recurrent Depth: Geiping et al., "Scaling up Test-Time Compute with Latent Reasoning" (Feb 2025)
- Ouro: ByteDance, "Ouro: Scaling Reasoning with Latent Thoughts" (2025)
- Planning Tokens: ICLR 2024
### Our Existing Work
- `steering-vector-empirical` - listening vector extraction
- `skills-apollo-optimizer-qwen35-gotcha` - APOLLO parameter grouping
- `qwen-3-5-27b-architecture-findings` - model architecture details
- `training-pipeline-fused-inference-training-mar27` - training infrastructure
---
*Research complete 2026-04-12. Ready for implementation.*

113
plugins/index.ts
View file

@ -1,113 +0,0 @@
// opencode-plugin/index.ts — Consciousness integration for OpenCode.
//
// Bridges OpenCode events to the consciousness system:
// - chat.message → forwards to poc-hook-opencode, appends output as text part
// - tool.execute.after → signals response activity
// - event → tracks session lifecycle (idle, compacted, etc.)
// - shell.env → injects POC_SESSION_ID into subprocesses
//
// Install: copy this directory to your project's `plugin/` or `plugins/` dir,
// or add to opencode.json:
// "plugin": ["/home/kent/poc/consciousness-claude/opencode-plugin"]
import type { Plugin, Hooks } from "@opencode-ai/plugin"
import path from "path"
import { $ } from "bun"
import { $ } from "bun"
// Find the poc-hook-opencode binary
function findHookBinary(): string {
const candidates = [
path.join(process.env.HOME || "", ".cargo/bin/poc-hook-opencode"),
path.join(process.env.HOME || "", "poc/consciousness-claude/target/debug/poc-hook-opencode"),
path.join(process.env.HOME || "", "poc/consciousness-claude/target/release/poc-hook-opencode"),
]
for (const c of candidates) {
try {
const stat = Bun.file(c).statSync()
if (stat?.isFile()) return c
} catch {}
}
return "poc-hook-opencode"
}
const HOOK_BINARY = findHookBinary()
// Generate a unique part ID (opencode uses ulid-like ascending IDs)
let partCounter = 0
function nextPartId(): string {
partCounter += 1
return `poc_part_${Date.now()}_${partCounter}`
}
export const ConsciousnessPlugin: Plugin = async (ctx) => {
const hooks: Hooks = {}
// Main hook: forward user messages to consciousness, inject context
hooks["chat.message"] = async (input, output) => {
const hookInput = JSON.stringify({
session_id: input.sessionID,
hook_event: "UserPromptSubmit",
})
try {
const proc = Bun.spawn([HOOK_BINARY], {
stdin: hookInput,
stdout: "pipe",
stderr: "pipe",
})
const [stdout, stderr] = await Promise.all([
new Response(proc.stdout).text(),
new Response(proc.stderr).text(),
])
await proc.exited
if (stdout && stdout.trim()) {
// Append as a text part — must match MessageV2.TextPart schema:
// { id, sessionID, messageID, type: "text", text, time?, synthetic?, ignored? }
output.parts.push({
id: nextPartId(),
sessionID: input.sessionID,
messageID: output.message.id,
type: "text",
text: stdout,
synthetic: true,
})
}
if (stderr && stderr.trim()) {
console.error("[consciousness] hook stderr:", stderr.slice(0, 500))
}
} catch (e) {
console.error("[consciousness] hook error:", e)
}
}
// Signal response after tool use
hooks["tool.execute.after"] = async () => {
try {
await $`poc-daemon response`.quiet()
} catch {
// Daemon might not be running
}
}
// Inject POC_SESSION_ID into all shell commands
hooks["shell.env"] = async (input, output) => {
if (input.sessionID) {
output.env["POC_SESSION_ID"] = input.sessionID
}
}
// Track session events
hooks["event"] = async ({ event }) => {
if (event.type === "session.compacted") {
// Compaction detected — next hook invocation will detect via SQLite
}
if (event.type === "session.idle") {
// Session went idle
}
}
return hooks
}

6
plugins/package.json
View file

@ -1,6 +0,0 @@
{
"name": "@consciousness/opencode-plugin",
"version": "0.1.0",
"description": "Consciousness integration for OpenCode",
"main": "index.ts"
}

27
src/agent/context.rs
View file

@ -31,31 +31,8 @@
use chrono::{DateTime, Utc};
use serde::{Serialize, Deserialize};
use std::sync::OnceLock;
use super::tokenizer;
// Cached token lengths for role headers — computed once on first use.
// "system\n", "user\n", "assistant\n" and "\n" are fixed strings.
static ROLE_TOKENS: OnceLock<[usize; 3]> = OnceLock::new();
static NEWLINE_TOKENS: OnceLock<usize> = OnceLock::new();
fn role_header_tokens(role: Role) -> usize {
let tokens = ROLE_TOKENS.get_or_init(|| [
tokenizer::encode("system\n").len(),
tokenizer::encode("user\n").len(),
tokenizer::encode("assistant\n").len(),
]);
match role {
Role::System => tokens[0],
Role::User => tokens[1],
Role::Assistant => tokens[2],
}
}
fn newline_tokens() -> usize {
*NEWLINE_TOKENS.get_or_init(|| tokenizer::encode("\n").len())
}
// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------
@ -446,9 +423,9 @@ impl Ast for AstNode {
match self {
Self::Leaf(leaf) => leaf.tokens(),
Self::Branch { role, children, .. } => {
1 + role_header_tokens(*role)
1 + tokenizer::encode(&format!("{}\n", role.as_str())).len()
+ children.iter().map(|c| c.tokens()).sum::<usize>()
+ 1 + newline_tokens()
+ 1 + tokenizer::encode("\n").len()
}
}
}

49
src/agent/mod.rs
View file

@ -141,8 +141,8 @@ pub struct Agent {
pub app_config: crate::config::AppConfig,
pub prompt_file: String,
pub session_id: String,
pub context: crate::Mutex<ContextState>,
pub state: crate::Mutex<AgentState>,
pub context: tokio::sync::Mutex<ContextState>,
pub state: tokio::sync::Mutex<AgentState>,
}
/// Mutable agent state — behind its own mutex.
@ -218,8 +218,8 @@ impl Agent {
app_config,
prompt_file,
session_id,
context: crate::Mutex::new(context),
state: crate::Mutex::new(AgentState {
context: tokio::sync::Mutex::new(context),
state: tokio::sync::Mutex::new(AgentState {
tools: agent_tools,
mcp_tools: McpToolAccess::All,
last_prompt_tokens: 0,
@ -255,8 +255,8 @@ impl Agent {
app_config: self.app_config.clone(),
prompt_file: self.prompt_file.clone(),
session_id: self.session_id.clone(),
context: crate::Mutex::new(ctx),
state: crate::Mutex::new(AgentState {
context: tokio::sync::Mutex::new(ctx),
state: tokio::sync::Mutex::new(AgentState {
tools,
mcp_tools: McpToolAccess::None,
last_prompt_tokens: 0,
@ -497,33 +497,42 @@ impl Agent {
}
async fn load_startup_journal(&self) {
use crate::agent::tools::memory::journal_tail;
let oldest_msg_ts = {
let ctx = self.context.lock().await;
ctx.conversation_log.as_ref().and_then(|log| log.oldest_timestamp())
};
// Get recent journal entries (newest first)
let journal_entries = match journal_tail(None, Some(100), Some(0), None).await {
Ok(e) => e,
let store = match crate::store::Store::load() {
Ok(s) => s,
Err(_) => return,
};
// Filter to entries before the conversation started
let cutoff_ts = oldest_msg_ts.map(|t| t.timestamp());
let filtered: Vec<_> = journal_entries.into_iter()
.filter(|e| cutoff_ts.map(|ts| e.created_at < ts).unwrap_or(true))
let mut journal_nodes: Vec<_> = store.nodes.values()
.filter(|n| n.node_type == crate::store::NodeType::EpisodicSession)
.collect();
journal_nodes.sort_by_key(|n| n.created_at);
let cutoff_idx = if let Some(cutoff) = oldest_msg_ts {
let cutoff_ts = cutoff.timestamp();
let mut idx = journal_nodes.len();
for (i, node) in journal_nodes.iter().enumerate() {
if node.created_at >= cutoff_ts {
idx = i + 1;
break;
}
}
idx
} else {
journal_nodes.len()
};
let journal_budget = context::context_window() * 15 / 100;
let mut entries = Vec::new();
let mut total_tokens = 0;
// Take entries within budget (they're newest-first, so reverse for display)
for entry in filtered.iter() {
let ts = chrono::DateTime::from_timestamp(entry.created_at, 0);
let ast = AstNode::memory(&entry.key, &entry.content)
for node in journal_nodes[..cutoff_idx].iter().rev() {
let ts = chrono::DateTime::from_timestamp(node.created_at, 0);
let ast = AstNode::memory(&node.key, &node.content)
.with_timestamp(ts.unwrap_or_else(chrono::Utc::now));
let tok = ast.tokens();
if total_tokens + tok > journal_budget && !entries.is_empty() {
@ -544,7 +553,7 @@ impl Agent {
}
pub async fn compact(&self) {
match crate::config::reload_for_model(&self.app_config, &self.prompt_file).await {
match crate::config::reload_for_model(&self.app_config, &self.prompt_file) {
Ok(personality) => {
let mut ctx = self.context.lock().await;
// System section (prompt + tools) set by new(), don't touch it

20
src/agent/oneshot.rs
View file

@ -7,7 +7,7 @@
// Also contains the legacy run_one_agent() pipeline and process
// management for spawned agent subprocesses.
use crate::store;
use crate::store::{self, Store};
use crate::subconscious::{defs, prompts};
use std::collections::HashMap;
@ -260,7 +260,7 @@ impl AutoAgent {
.map_err(|e| format!("config: {}", e))?;
let personality = crate::config::reload_for_model(
&app, &app.prompts.other,
).await.map_err(|e| format!("config: {}", e))?;
).map_err(|e| format!("config: {}", e))?;
let agent = Agent::new(
client, personality,
@ -381,7 +381,8 @@ pub struct AgentResult {
/// Run an agent. If keys are provided, use them directly (bypassing the
/// agent's query). Otherwise, run the query to select target nodes.
pub async fn run_one_agent(
pub fn run_one_agent(
store: &mut Store,
agent_name: &str,
count: usize,
keys: Option<&[String]>,
@ -400,22 +401,27 @@ pub async fn run_one_agent(
// Build prompt batch — either from explicit keys or the agent's query
let agent_batch = if let Some(keys) = keys {
dbglog!("[{}] targeting: {}", agent_name, keys.join(", "));
let graph = store.build_graph();
let mut resolved_steps = Vec::new();
let mut all_keys: Vec<String> = keys.to_vec();
for step in &def.steps {
let (prompt, extra_keys) = defs::resolve_placeholders(
&step.prompt, keys, count,
).await;
&step.prompt, store, &graph, keys, count,
);
all_keys.extend(extra_keys);
resolved_steps.push(prompts::ResolvedStep {
prompt,
phase: step.phase.clone(),
});
}
prompts::AgentBatch { steps: resolved_steps, node_keys: all_keys }
let batch = prompts::AgentBatch { steps: resolved_steps, node_keys: all_keys };
if !batch.node_keys.is_empty() {
store.record_agent_visits(&batch.node_keys, agent_name).ok();
}
batch
} else {
let effective_count = def.count.unwrap_or(count);
defs::run_agent(&def, effective_count, &Default::default()).await?
defs::run_agent(store, &def, effective_count, &Default::default())?
};
// Base memory tools + extras from agent def (matching unconscious.rs pattern)

39
src/agent/tools/cd.rs
View file

@ -1,39 +0,0 @@
use std::sync::Arc;
use std::path::PathBuf;
// tools/cd.rs — Change working directory
//
// Uses the chdir syscall so it affects all tools.
pub fn tool() -> super::Tool {
super::Tool {
name: "cd",
description: "Change the current working directory.",
parameters_json: r#"{"type":"object","properties":{"path":{"type":"string","description":"The directory to change to (absolute or relative)"}},"required":["path"]}"#,
handler: Arc::new(|_agent, v| Box::pin(async move {
let path = v.get("path").and_then(|v| v.as_str())
.ok_or_else(|| anyhow::anyhow!("'path' parameter is required"))?;
if path.is_empty() { anyhow::bail!("'path' parameter cannot be empty"); }
// Resolve ~ to home directory
let resolved = if path.starts_with('~') {
let home = dirs::home_dir()
.ok_or_else(|| anyhow::anyhow!("could not determine home directory"))?;
home.join(path.strip_prefix("~/").unwrap_or(path))
} else {
PathBuf::from(path)
};
// Change directory (this is the actual chdir syscall)
std::env::set_current_dir(&resolved)
.map_err(|e| anyhow::anyhow!("cd: {}: {}", path, e))?;
// Return the canonical path
let canonical = std::env::current_dir()
.map(|p| p.display().to_string())
.unwrap_or_else(|_| resolved.display().to_string());
Ok(canonical)
})),
}
}

2
src/agent/tools/lsp.rs
View file

@ -123,7 +123,7 @@ fn find_project_root(file_path: &str) -> Option<String> {
const IDLE_TIMEOUT_SECS: u64 = 600;
use std::sync::OnceLock;
use crate::Mutex as TokioMutex;
use tokio::sync::Mutex as TokioMutex;
struct Registry {
configs: Vec<crate::config::LspServerConfig>,

2
src/agent/tools/mcp_client.rs
View file

@ -10,7 +10,7 @@ use serde_json::json;
use std::sync::OnceLock;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, BufWriter};
use tokio::process::{Child, ChildStdin, ChildStdout, Command};
use crate::Mutex as TokioMutex;
use tokio::sync::Mutex as TokioMutex;
#[derive(Debug, Clone)]
pub struct McpTool {

684
src/agent/tools/memory.rs
View file

@ -1,30 +1,16 @@
use std::sync::Arc;
// tools/memory.rs — Native memory graph operations
//
// Access via hippocampus::access() / access_local(). Clients try socket
// first, fall back to local store.
#![allow(unused_variables)] // macro-generated args for no-param tools
// Direct library calls into the store — no subprocess spawning.
// One function per tool for use in the Tool registry.
use anyhow::{Context, Result};
use std::sync::Arc;
use crate::hippocampus::{access, memory_rpc, StoreAccess};
// Re-export typed API from hippocampus for backward compatibility
pub use crate::hippocampus::{
memory_render, memory_write, memory_search, memory_link_set, memory_link_add,
memory_delete, memory_history, memory_weight_set, memory_rename, memory_supersede,
memory_query, memory_links,
journal_tail, journal_new, journal_update,
graph_topology, graph_health, graph_communities, graph_normalize_strengths,
graph_link_impact, graph_hubs, graph_trace,
socket_path,
};
use crate::hippocampus::memory::MemoryNode;
use crate::store::StoreView;
use crate::store::Store;
// ── Macro for generating tool wrappers ─────────────────────────
//
// memory_tool!(name, mut, arg1: [str], arg2: [Option<bool>])
// - mut/ref for store mutability
// - generates jsonargs_* (internal, JSON args) and public typed API
// ── Helpers ────────────────────────────────────────────────────
fn get_str<'a>(args: &'a serde_json::Value, name: &'a str) -> Result<&'a str> {
args.get(name).and_then(|v| v.as_str()).context(format!("{} is required", name))
@ -34,7 +20,10 @@ fn get_f64(args: &serde_json::Value, name: &str) -> Result<f64> {
args.get(name).and_then(|v| v.as_f64()).context(format!("{} is required", name))
}
/// Get provenance from agent state, or "manual".
async fn cached_store() -> Result<std::sync::Arc<tokio::sync::Mutex<Store>>> {
Store::cached().await.map_err(|e| anyhow::anyhow!("{}", e))
}
async fn get_provenance(agent: &Option<std::sync::Arc<crate::agent::Agent>>) -> String {
match agent {
Some(a) => a.state.lock().await.provenance.clone(),
@ -42,332 +31,381 @@ async fn get_provenance(agent: &Option<std::sync::Arc<crate::agent::Agent>>) ->
}
}
macro_rules! memory_tool {
// ── Helper rules (must come first) ─────────────────────────────
// Extract from JSON
(@extract $args:ident, $name:ident, str) => {
get_str($args, stringify!($name))?
};
(@extract $args:ident, $name:ident, f32) => {
get_f64($args, stringify!($name))? as f32
};
(@extract $args:ident, $name:ident, Vec<String>) => {
$args.get(stringify!($name))
.and_then(|v| v.as_array())
.map(|arr| arr.iter().filter_map(|v| v.as_str().map(String::from)).collect::<Vec<_>>())
.unwrap_or_default()
};
(@extract $args:ident, $name:ident, Option<&str>) => {
$args.get(stringify!($name)).and_then(|v| v.as_str())
};
(@extract $args:ident, $name:ident, Option<bool>) => {
$args.get(stringify!($name)).and_then(|v| v.as_bool())
};
(@extract $args:ident, $name:ident, Option<u64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64())
};
(@extract $args:ident, $name:ident, Option<i64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_i64())
};
(@extract $args:ident, $name:ident, Option<usize>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64()).map(|v| v as usize)
};
(@extract $args:ident, $name:ident, Option<u32>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64()).map(|v| v as u32)
};
(@extract $args:ident, $name:ident, Option<f64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_f64())
};
// Parameter types for function signatures
(@param_type str) => { &str };
(@param_type f32) => { f32 };
(@param_type Vec<String>) => { Vec<String> };
(@param_type Option<&str>) => { Option<&str> };
(@param_type Option<bool>) => { Option<bool> };
(@param_type Option<u64>) => { Option<u64> };
(@param_type Option<i64>) => { Option<i64> };
(@param_type Option<usize>) => { Option<usize> };
(@param_type Option<u32>) => { Option<u32> };
(@param_type Option<f64>) => { Option<f64> };
// Serialize result for jsonargs
(@serialize $t:ty, $result:expr) => { serde_json::to_string(&$result)? };
// Deserialize RPC response
(@deserialize $t:ty, $json:expr) => { serde_json::from_str(&$json).map_err(|e| anyhow::anyhow!("{}", e)) };
// Serialize to JSON for RPC
(@insert_json $map:ident, $name:ident, str) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, f32) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, Vec<String>) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, Option<&str>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<bool>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<u64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<i64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<usize>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<u32>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<f64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
// Call hippocampus (all methods now take &self, deref Arc)
(@call mut, $name:ident, $store:ident, $prov:expr $(, $arg:expr)*) => {
crate::hippocampus::local::$name(&*$store, $prov $(, $arg)*)
};
(@call ref, $name:ident, $store:ident, $prov:expr $(, $arg:expr)*) => {
crate::hippocampus::local::$name(&*$store, $prov $(, $arg)*)
};
// ── Main rules ─────────────────────────────────────────────────
// Shorthand: mut/ref without return type defaults to String
($name:ident, $m:ident $(, $($arg:ident : [$($typ:tt)+]),* $(,)?)?) => {
memory_tool!($name, $m -> String $(, $($arg : [$($typ)+]),*)?);
};
// Full form with return type
($name:ident, $m:ident -> $ret:ty $(, $($arg:ident : [$($typ:tt)+]),* $(,)?)?) => {
paste::paste! {
async fn [<jsonargs_ $name>](agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
$($(let $arg = memory_tool!(@extract args, $arg, $($typ)+);)*)?
let prov = get_provenance(agent).await;
match access() {
StoreAccess::Daemon(store) => {
let result: $ret = memory_tool!(@call $m, $name, store, &prov $($(, $arg)*)?)?;
Ok(memory_tool!(@serialize $ret, result))
}
StoreAccess::Client => {
#[allow(unused_mut)]
let mut map = serde_json::Map::new();
$($(memory_tool!(@insert_json map, $arg, $($typ)+);)*)?
memory_rpc(stringify!($name), serde_json::Value::Object(map))
}
StoreAccess::None(err) => anyhow::bail!("{}", err),
}
}
}
};
}
// ── Memory tools ───────────────────────────────────────────────
memory_tool!(memory_render, ref, key: [str], raw: [Option<bool>]);
memory_tool!(memory_write, mut, key: [str], content: [str]);
memory_tool!(memory_search, ref, keys: [Vec<String>], max_hops: [Option<u32>], edge_decay: [Option<f64>], min_activation: [Option<f64>], limit: [Option<usize>]);
memory_tool!(memory_link_set, mut, source: [str], target: [str], strength: [f32]);
memory_tool!(memory_link_add, mut, source: [str], target: [str]);
memory_tool!(memory_delete, mut, key: [str]);
memory_tool!(memory_history, ref, key: [str], full: [Option<bool>]);
memory_tool!(memory_weight_set, mut, key: [str], weight: [f32]);
memory_tool!(memory_rename, mut, old_key: [str], new_key: [str]);
memory_tool!(memory_supersede, mut, old_key: [str], new_key: [str], reason: [Option<&str>]);
memory_tool!(memory_query, ref, query: [str], format: [Option<&str>]);
// Re-export types and typed API from hippocampus
pub use crate::hippocampus::local::LinkInfo;
memory_tool!(memory_links, ref -> Vec<LinkInfo>, key: [str]);
// ── Journal tools ──────────────────────────────────────────────
pub use crate::hippocampus::local::JournalEntry;
memory_tool!(journal_tail, ref -> Vec<JournalEntry>, count: [Option<u64>], level: [Option<u64>], after: [Option<&str>]);
memory_tool!(journal_new, mut, name: [str], title: [str], body: [str], level: [Option<i64>]);
memory_tool!(journal_update, mut, body: [str], level: [Option<i64>]);
// ── Graph tools ───────────────────────────────────────────────
memory_tool!(graph_topology, ref);
memory_tool!(graph_health, ref);
memory_tool!(graph_communities, ref, top_n: [Option<usize>], min_size: [Option<usize>]);
memory_tool!(graph_normalize_strengths, mut, apply: [Option<bool>]);
memory_tool!(graph_link_impact, ref, source: [str], target: [str]);
memory_tool!(graph_hubs, ref, count: [Option<usize>]);
memory_tool!(graph_trace, ref, key: [str]);
// ── Definitions ────────────────────────────────────────────────
pub fn memory_tools() -> [super::Tool; 19] {
pub fn memory_tools() -> [super::Tool; 13] {
use super::Tool;
macro_rules! tool {
($name:ident, $desc:expr, $params:expr) => {
Tool {
name: stringify!($name),
description: $desc,
parameters_json: $params,
handler: Arc::new(|a, v| Box::pin(async move {
paste::paste! { [<jsonargs_ $name>](&a, &v).await }
})),
}
};
}
[
tool!(memory_render, "Read a memory node's content and links.", r#"{
"type": "object",
"properties": { "key": {"type": "string"}, "raw": {"type": "boolean"} },
"required": ["key"]
}"#),
tool!(memory_write, "Create or update a memory node.", r#"{
"type": "object",
"properties": { "key": {"type": "string"}, "content": {"type": "string"} },
"required": ["key", "content"]
}"#),
tool!(memory_search, "Search via spreading activation from seed keys.", r#"{
Tool { name: "memory_render", description: "Read a memory node's content and links.",
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Node key"}},"required":["key"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { render(&v).await })) },
Tool { name: "memory_write", description: "Create or update a memory node.",
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Node key"},"content":{"type":"string","description":"Full content (markdown)"}},"required":["key","content"]}"#,
handler: Arc::new(|a, v| Box::pin(async move { write(&a, &v).await })) },
Tool { name: "memory_search", description: "Search the memory graph via spreading activation. Give 2-4 seed node keys.",
parameters_json: r#"{"type":"object","properties":{"keys":{"type":"array","items":{"type":"string"},"description":"Seed node keys to activate from"}},"required":["keys"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { search(&v).await })) },
Tool { name: "memory_links", description: "Show a node's neighbors with link strengths.",
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Node key"}},"required":["key"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { links(&v) })) },
Tool { name: "memory_link_set", description: "Set link strength between two nodes.",
parameters_json: r#"{"type":"object","properties":{"source":{"type":"string"},"target":{"type":"string"},"strength":{"type":"number","description":"0.01 to 1.0"}},"required":["source","target","strength"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { link_set(&v).await })) },
Tool { name: "memory_link_add", description: "Add a new link between two nodes.",
parameters_json: r#"{"type":"object","properties":{"source":{"type":"string"},"target":{"type":"string"}},"required":["source","target"]}"#,
handler: Arc::new(|a, v| Box::pin(async move { link_add(&a, &v).await })) },
Tool { name: "memory_used", description: "Mark a node as useful (boosts weight).",
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Node key"}},"required":["key"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { used(&v).await })) },
Tool { name: "memory_weight_set", description: "Set a node's weight directly (0.01 to 1.0).",
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string"},"weight":{"type":"number","description":"0.01 to 1.0"}},"required":["key","weight"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { weight_set(&v).await })) },
Tool { name: "memory_rename", description: "Rename a node key in place.",
parameters_json: r#"{"type":"object","properties":{"old_key":{"type":"string"},"new_key":{"type":"string"}},"required":["old_key","new_key"]}"#,
handler: Arc::new(|_a, v| Box::pin(async move { rename(&v).await })) },
Tool { name: "memory_supersede", description: "Mark a node as superseded by another (sets weight to 0.01).",
parameters_json: r#"{"type":"object","properties":{"old_key":{"type":"string"},"new_key":{"type":"string"},"reason":{"type":"string"}},"required":["old_key","new_key"]}"#,
handler: Arc::new(|a, v| Box::pin(async move { supersede(&a, &v).await })) },
Tool { name: "memory_query",
description: "Run a structured query against the memory graph.",
parameters_json: r#"{
"type": "object",
"properties": {
"keys": {"type": "array", "items": {"type": "string"}},
"max_hops": {"type": "integer"},
"edge_decay": {"type": "number"},
"min_activation": {"type": "number"},
"limit": {"type": "integer"}
},
"required": ["keys"]
}"#),
tool!(memory_links, "Show a node's neighbors with link strengths.", r#"{
"type": "object",
"properties": { "key": {"type": "string"} },
"required": ["key"]
}"#),
tool!(memory_link_set, "Set link strength between two nodes.", r#"{
"type": "object",
"properties": {
"source": {"type": "string"},
"target": {"type": "string"},
"strength": {"type": "number", "description": "0.01 to 1.0"}
},
"required": ["source", "target", "strength"]
}"#),
tool!(memory_link_add, "Add a new link between two nodes.", r#"{
"type": "object",
"properties": { "source": {"type": "string"}, "target": {"type": "string"} },
"required": ["source", "target"]
}"#),
tool!(memory_delete, "Delete a memory node.", r#"{
"type": "object",
"properties": { "key": {"type": "string"} },
"required": ["key"]
}"#),
tool!(memory_history, "Show version history for a node.", r#"{
"type": "object",
"properties": { "key": {"type": "string"}, "full": {"type": "boolean"} },
"required": ["key"]
}"#),
tool!(memory_weight_set, "Set a node's weight (0.01 to 1.0).", r#"{
"type": "object",
"properties": { "key": {"type": "string"}, "weight": {"type": "number"} },
"required": ["key", "weight"]
}"#),
tool!(memory_rename, "Rename a node key.", r#"{
"type": "object",
"properties": { "old_key": {"type": "string"}, "new_key": {"type": "string"} },
"required": ["old_key", "new_key"]
}"#),
tool!(memory_supersede, "Mark a node as superseded by another.", r#"{
"type": "object",
"properties": {
"old_key": {"type": "string"},
"new_key": {"type": "string"},
"reason": {"type": "string"}
},
"required": ["old_key", "new_key"]
}"#),
tool!(memory_query, "Run a structured query against the memory graph.", r#"{
"type": "object",
"properties": {
"query": {"type": "string"},
"format": {"type": "string", "description": "compact or full"}
"query": {"type": "string", "description": "Query expression"},
"format": {"type": "string", "description": "compact (default) or full (with content and graph metrics)", "default": "compact"}
},
"required": ["query"]
}"#),
tool!(graph_topology, "Show graph topology stats.", r#"{"type": "object"}"#),
tool!(graph_health, "Show graph health report.", r#"{"type": "object"}"#),
tool!(graph_hubs, "Show top hub nodes by degree.", r#"{
"type": "object",
"properties": { "count": {"type": "integer"} }
}"#),
tool!(graph_communities, "Show communities by isolation.", r#"{
"type": "object",
"properties": { "top_n": {"type": "integer"}, "min_size": {"type": "integer"} }
}"#),
tool!(graph_normalize_strengths, "Set link strengths from Jaccard similarity.", r#"{
"type": "object",
"properties": { "apply": {"type": "boolean"} }
}"#),
tool!(graph_link_impact, "Simulate adding an edge, report impact.", r#"{
"type": "object",
"properties": { "source": {"type": "string"}, "target": {"type": "string"} },
"required": ["source", "target"]
}"#),
tool!(graph_trace, "Walk temporal links from a node.", r#"{
"type": "object",
"properties": { "key": {"type": "string"} },
"required": ["key"]
}"#),
}"#,
handler: Arc::new(|_a, v| Box::pin(async move { query(&v).await })) },
Tool { name: "graph_topology", description: "Show graph topology stats (nodes, edges, clustering, hubs).",
parameters_json: r#"{"type":"object","properties":{}}"#,
handler: Arc::new(|_a, _v| Box::pin(async { graph_topology().await })) },
Tool { name: "graph_health", description: "Show graph health report with maintenance recommendations.",
parameters_json: r#"{"type":"object","properties":{}}"#,
handler: Arc::new(|_a, _v| Box::pin(async { graph_health().await })) },
]
}
pub fn journal_tools() -> [super::Tool; 3] {
use super::Tool;
macro_rules! tool {
($name:ident, $desc:expr, $params:expr) => {
Tool {
name: stringify!($name),
description: $desc,
parameters_json: $params,
handler: Arc::new(|a, v| Box::pin(async move {
paste::paste! { [<jsonargs_ $name>](&a, &v).await }
})),
}
};
}
[
tool!(journal_tail, "Read the last N entries at a given level.", r#"{
Tool { name: "journal_tail",
description: "Read the last N entries at a given level.",
parameters_json: r#"{
"type": "object",
"properties": {
"count": {"type": "integer"},
"level": {"type": "integer", "description": "0=journal, 1=daily, 2=weekly, 3=monthly"},
"format": {"type": "string", "description": "compact or full"},
"count": {"type": "integer", "description": "Number of entries", "default": 1},
"level": {"type": "integer", "description": "0=journal, 1=daily, 2=weekly, 3=monthly", "default": 0},
"format": {"type": "string", "description": "compact or full (with content)", "default": "full"},
"after": {"type": "string", "description": "Only entries after this date (YYYY-MM-DD)"}
}
}"#),
tool!(journal_new, "Start a new journal/digest entry.", r#"{
}"#,
handler: Arc::new(|_a, v| Box::pin(async move { journal_tail(&v).await })) },
Tool { name: "journal_new", description: "Start a new journal/digest entry.",
parameters_json: r#"{
"type": "object",
"properties": {
"name": {"type": "string"},
"title": {"type": "string"},
"body": {"type": "string"},
"level": {"type": "integer"}
"name": {"type": "string", "description": "Short node name (becomes the key)"},
"title": {"type": "string", "description": "Descriptive title"},
"body": {"type": "string", "description": "Entry body"},
"level": {"type": "integer", "description": "0=journal, 1=daily, 2=weekly, 3=monthly", "default": 0}
},
"required": ["name", "title", "body"]
}"#),
tool!(journal_update, "Append text to the most recent entry.", r#"{
}"#,
handler: Arc::new(|a, v| Box::pin(async move { journal_new(&a, &v).await })) },
Tool { name: "journal_update", description: "Append text to the most recent entry at a level.",
parameters_json: r#"{
"type": "object",
"properties": {
"body": {"type": "string"},
"level": {"type": "integer"}
"body": {"type": "string", "description": "Text to append"},
"level": {"type": "integer", "description": "0=journal, 1=daily, 2=weekly, 3=monthly", "default": 0}
},
"required": ["body"]
}"#),
}"#,
handler: Arc::new(|a, v| Box::pin(async move { journal_update(&a, &v).await })) },
]
}
// ── Memory tools ───────────────────────────────────────────────
async fn render(args: &serde_json::Value) -> Result<String> {
let key = get_str(args, "key")?;
let arc = cached_store().await?;
let store = arc.lock().await;
Ok(MemoryNode::from_store(&store, key)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", key))?
.render())
}
async fn write(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let key = get_str(args, "key")?;
let content = get_str(args, "content")?;
let prov = get_provenance(agent).await;
let arc = cached_store().await?;
let mut store = arc.lock().await;
let result = store.upsert_provenance(key, content, &prov)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("{} '{}'", result, key))
}
async fn search(args: &serde_json::Value) -> Result<String> {
let keys: Vec<String> = args.get("keys")
.and_then(|v| v.as_array())
.map(|arr| arr.iter().filter_map(|v| v.as_str().map(String::from)).collect())
.unwrap_or_default();
if keys.is_empty() {
anyhow::bail!("memory_search requires at least one seed key");
}
let arc = cached_store().await?;
let store = arc.lock().await;
let graph = crate::graph::build_graph_fast(&*store);
let params = store.params();
let seeds: Vec<(String, f64)> = keys.iter()
.filter_map(|k| {
let resolved = store.resolve_key(k).ok()?;
Some((resolved, 1.0))
})
.collect();
if seeds.is_empty() {
anyhow::bail!("no valid seed keys found");
}
let seed_set: std::collections::HashSet<&str> = seeds.iter()
.map(|(k, _)| k.as_str()).collect();
let results = crate::search::spreading_activation(
&seeds, &graph, &*store,
params.max_hops, params.edge_decay, params.min_activation,
);
Ok(results.iter()
.filter(|(k, _)| !seed_set.contains(k.as_str()))
.take(20)
.map(|(key, score)| format!(" {:.2} {}", score, key))
.collect::<Vec<_>>().join("\n"))
}
fn links(args: &serde_json::Value) -> Result<String> {
let key = get_str(args, "key")?;
let node = MemoryNode::load(key)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", key))?;
let mut out = format!("Neighbors of '{}':\n", key);
for (target, strength, is_new) in &node.links {
let tag = if *is_new { " (new)" } else { "" };
out.push_str(&format!(" ({:.2}) {}{}\n", strength, target, tag));
}
Ok(out)
}
async fn link_set(args: &serde_json::Value) -> Result<String> {
let arc = cached_store().await?;
let mut store = arc.lock().await;
let s = store.resolve_key(get_str(args, "source")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let t = store.resolve_key(get_str(args, "target")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let strength = get_f64(args, "strength")? as f32;
let old = store.set_link_strength(&s, &t, strength).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("{} ↔ {} strength {:.2} → {:.2}", s, t, old, strength))
}
async fn link_add(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let arc = cached_store().await?;
let mut store = arc.lock().await;
let s = store.resolve_key(get_str(args, "source")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let t = store.resolve_key(get_str(args, "target")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let prov = get_provenance(agent).await;
let strength = store.add_link(&s, &t, &prov).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("linked {} → {} (strength={:.2})", s, t, strength))
}
async fn used(args: &serde_json::Value) -> Result<String> {
let key = get_str(args, "key")?;
let arc = cached_store().await?;
let mut store = arc.lock().await;
if !store.nodes.contains_key(key) {
anyhow::bail!("node not found: {}", key);
}
store.mark_used(key);
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("marked {} as used", key))
}
async fn weight_set(args: &serde_json::Value) -> Result<String> {
let arc = cached_store().await?;
let mut store = arc.lock().await;
let key = store.resolve_key(get_str(args, "key")?).map_err(|e| anyhow::anyhow!("{}", e))?;
let weight = get_f64(args, "weight")? as f32;
let (old, new) = store.set_weight(&key, weight).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("weight {} {:.2} → {:.2}", key, old, new))
}
async fn rename(args: &serde_json::Value) -> Result<String> {
let old_key = get_str(args, "old_key")?;
let new_key = get_str(args, "new_key")?;
let arc = cached_store().await?;
let mut store = arc.lock().await;
let resolved = store.resolve_key(old_key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.rename_node(&resolved, new_key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("Renamed '{}' → '{}'", resolved, new_key))
}
async fn supersede(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let old_key = get_str(args, "old_key")?;
let new_key = get_str(args, "new_key")?;
let reason = args.get("reason").and_then(|v| v.as_str()).unwrap_or("superseded");
let arc = cached_store().await?;
let mut store = arc.lock().await;
let content = store.nodes.get(old_key)
.map(|n| n.content.clone())
.ok_or_else(|| anyhow::anyhow!("node not found: {}", old_key))?;
let notice = format!("**SUPERSEDED** by `{}` — {}\n\n---\n\n{}",
new_key, reason, content.trim());
let prov = get_provenance(agent).await;
store.upsert_provenance(old_key, &notice, &prov)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.set_weight(old_key, 0.01).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("superseded {} → {} ({})", old_key, new_key, reason))
}
async fn query(args: &serde_json::Value) -> Result<String> {
let query_str = get_str(args, "query")?;
let format = args.get("format").and_then(|v| v.as_str()).unwrap_or("compact");
let arc = cached_store().await?;
let store = arc.lock().await;
let graph = store.build_graph();
match format {
"full" => {
// Rich output with full content, graph metrics, hub analysis
let results = crate::query_parser::execute_query(&store, &graph, query_str)
.map_err(|e| anyhow::anyhow!("{}", e))?;
let keys: Vec<String> = results.into_iter().map(|r| r.key).collect();
let items = crate::subconscious::defs::keys_to_replay_items(&store, &keys, &graph);
Ok(crate::subconscious::prompts::format_nodes_section(&store, &items, &graph))
}
_ => {
// Compact output: handles count, select, and all expression types
crate::query_parser::query_to_string(&store, &graph, query_str)
.map_err(|e| anyhow::anyhow!("{}", e))
}
}
}
// ── Journal tools ──────────────────────────────────────────────
async fn journal_tail(args: &serde_json::Value) -> Result<String> {
let count = args.get("count").and_then(|v| v.as_u64()).unwrap_or(1);
let level = args.get("level").and_then(|v| v.as_u64()).unwrap_or(0);
let format = args.get("format").and_then(|v| v.as_str()).unwrap_or("full");
let after = args.get("after").and_then(|v| v.as_str());
let type_name = match level {
0 => "episodic",
1 => "daily",
2 => "weekly",
3 => "monthly",
_ => return Err(anyhow::anyhow!("invalid level: {} (0=journal, 1=daily, 2=weekly, 3=monthly)", level)),
};
let mut q = format!("all | type:{} | sort:timestamp", type_name);
if let Some(date) = after {
// Convert date to age in seconds
if let Ok(nd) = chrono::NaiveDate::parse_from_str(date, "%Y-%m-%d") {
let ts = nd.and_hms_opt(0, 0, 0).unwrap().and_utc().timestamp();
let age = chrono::Utc::now().timestamp() - ts;
q.push_str(&format!(" | age:<{}", age));
}
}
q.push_str(&format!(" | limit:{}", count));
query(&serde_json::json!({"query": q, "format": format})).await
}
fn level_to_node_type(level: i64) -> crate::store::NodeType {
match level {
1 => crate::store::NodeType::EpisodicDaily,
2 => crate::store::NodeType::EpisodicWeekly,
3 => crate::store::NodeType::EpisodicMonthly,
_ => crate::store::NodeType::EpisodicSession,
}
}
async fn journal_new(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let name = get_str(args, "name")?;
let title = get_str(args, "title")?;
let body = get_str(args, "body")?;
let level = args.get("level").and_then(|v| v.as_i64()).unwrap_or(0);
let ts = chrono::Local::now().format("%Y-%m-%dT%H:%M");
let content = format!("## {} — {}\n\n{}", ts, title, body);
let base_key: String = name.split_whitespace()
.map(|w| w.to_lowercase()
.chars().filter(|c| c.is_alphanumeric() || *c == '-')
.collect::<String>())
.filter(|s| !s.is_empty())
.collect::<Vec<_>>()
.join("-");
let base_key = if base_key.len() > 80 { &base_key[..80] } else { base_key.as_str() };
let arc = cached_store().await?;
let mut store = arc.lock().await;
let key = if store.nodes.contains_key(base_key) {
let mut n = 2;
loop {
let candidate = format!("{}-{}", base_key, n);
if !store.nodes.contains_key(&candidate) { break candidate; }
n += 1;
}
} else {
base_key.to_string()
};
let mut node = crate::store::new_node(&key, &content);
node.node_type = level_to_node_type(level);
node.provenance = get_provenance(agent).await;
store.upsert_node(node).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
let word_count = body.split_whitespace().count();
Ok(format!("New entry '{}' ({} words)", title, word_count))
}
async fn journal_update(agent: &Option<std::sync::Arc<crate::agent::Agent>>, args: &serde_json::Value) -> Result<String> {
let body = get_str(args, "body")?;
let level = args.get("level").and_then(|v| v.as_i64()).unwrap_or(0);
let node_type = level_to_node_type(level);
let arc = cached_store().await?;
let mut store = arc.lock().await;
let latest_key = store.nodes.values()
.filter(|n| n.node_type == node_type)
.max_by_key(|n| n.created_at)
.map(|n| n.key.clone());
let Some(key) = latest_key else {
anyhow::bail!("no entry at level {} to update — use journal_new first", level);
};
let existing = store.nodes.get(&key).unwrap().content.clone();
let new_content = format!("{}\n\n{}", existing.trim_end(), body);
let prov = get_provenance(agent).await;
store.upsert_provenance(&key, &new_content, &prov)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
let word_count = body.split_whitespace().count();
Ok(format!("Updated last entry (+{} words)", word_count))
}
// ── Graph tools ───────────────────────────────────────────────
async fn graph_topology() -> Result<String> {
let arc = cached_store().await?;
let store = arc.lock().await;
let graph = store.build_graph();
Ok(crate::subconscious::prompts::format_topology_header(&graph))
}
async fn graph_health() -> Result<String> {
let arc = cached_store().await?;
let store = arc.lock().await;
let graph = store.build_graph();
Ok(crate::subconscious::prompts::format_health_section(&store, &graph))
}

7
src/agent/tools/mod.rs
View file

@ -6,14 +6,13 @@
// Core tools
mod ast_grep;
pub mod lsp;
pub mod mcp_client;
mod bash;
mod cd;
pub mod channels;
mod edit;
mod glob;
mod grep;
pub mod lsp;
pub mod mcp_client;
pub mod memory;
mod read;
mod web;
@ -178,7 +177,7 @@ pub async fn dispatch_with_agent(
pub fn tools() -> Vec<Tool> {
let mut all = vec![
read::tool(), write::tool(), edit::tool(),
grep::tool(), glob::tool(), bash::tool(), cd::tool(),
grep::tool(), glob::tool(), bash::tool(),
ast_grep::tool(), vision::tool(),
];
all.extend(web::tools());

7
src/bin/consciousness.rs
View file

@ -1,7 +1,2 @@
#![feature(panic_backtrace_config)]
#![warn(unreachable_pub)]
fn main() {
std::panic::set_backtrace_style(std::panic::BacktraceStyle::Short);
consciousness::user::main()
}
fn main() { consciousness::user::main() }

30
src/bin/merge-logs.rs
View file

@ -22,7 +22,6 @@ use std::fs;
use std::io::{BufReader, BufWriter};
use std::path::Path;
use anyhow::{bail, Context, Result};
use capnp::message;
use capnp::serialize;
@ -30,17 +29,17 @@ use consciousness::memory_capnp;
use consciousness::store::Node;
/// Read all node entries from a capnp log file, preserving order.
fn read_all_entries(path: &Path) -> Result<Vec<Node>> {
fn read_all_entries(path: &Path) -> Result<Vec<Node>, String> {
let file = fs::File::open(path)
.with_context(|| format!("open {}", path.display()))?;
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
let mut entries = Vec::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.with_context(|| format!("read log from {}", path.display()))?;
.map_err(|e| format!("read log from {}: {}", path.display(), e))?;
for node_reader in log.get_nodes()
.with_context(|| format!("get nodes from {}", path.display()))? {
.map_err(|e| format!("get nodes from {}: {}", path.display(), e))? {
let node = Node::from_capnp_migrate(node_reader)?;
entries.push(node);
}
@ -50,9 +49,9 @@ fn read_all_entries(path: &Path) -> Result<Vec<Node>> {
}
/// Write node entries to a new capnp log file in chunks.
fn write_entries(path: &Path, entries: &[Node]) -> Result<()> {
fn write_entries(path: &Path, entries: &[Node]) -> Result<(), String> {
let file = fs::File::create(path)
.with_context(|| format!("create {}", path.display()))?;
.map_err(|e| format!("create {}: {}", path.display(), e))?;
let mut writer = BufWriter::new(file);
for chunk in entries.chunks(100) {
@ -65,13 +64,13 @@ fn write_entries(path: &Path, entries: &[Node]) -> Result<()> {
}
}
serialize::write_message(&mut writer, &msg)
.context("write message")?;
.map_err(|e| format!("write: {}", e))?;
}
Ok(())
}
fn main() -> Result<()> {
fn main() -> Result<(), String> {
let args: Vec<String> = std::env::args().collect();
if args.len() != 4 {
eprintln!("Usage: merge-logs <old_log> <current_log> <output_dir>");
@ -88,18 +87,19 @@ fn main() -> Result<()> {
// Validate inputs exist
if !old_path.exists() {
bail!("old log not found: {}", old_path.display());
return Err(format!("old log not found: {}", old_path.display()));
}
if !current_path.exists() {
bail!("current log not found: {}", current_path.display());
return Err(format!("current log not found: {}", current_path.display()));
}
// Create output directory (must not already contain nodes.capnp)
fs::create_dir_all(output_dir)
.context("create output dir")?;
.map_err(|e| format!("create output dir: {}", e))?;
let output_path = output_dir.join("nodes.capnp");
if output_path.exists() {
bail!("output already exists: {} — refusing to overwrite", output_path.display());
return Err(format!("output already exists: {} — refusing to overwrite",
output_path.display()));
}
eprintln!("Reading old log: {} ...", old_path.display());
@ -190,8 +190,8 @@ fn main() -> Result<()> {
eprintln!(" Replay produces {} live nodes", final_nodes.len());
if verify_entries.len() != merged.len() {
bail!("Verification failed: wrote {} but read back {}",
merged.len(), verify_entries.len());
return Err(format!("Verification failed: wrote {} but read back {}",
merged.len(), verify_entries.len()));
}
eprintln!();

440
src/cli/admin.rs
View file

@ -1,23 +1,23 @@
// cli/admin.rs — admin subcommand handlers
use anyhow::Result;
use crate::hippocampus as memory;
use crate::hippocampus::store;
fn install_default_file(data_dir: &std::path::Path, name: &str, content: &str) -> Result<()> {
use crate::store;
fn install_default_file(data_dir: &std::path::Path, name: &str, content: &str) -> Result<(), String> {
let path = data_dir.join(name);
if !path.exists() {
std::fs::write(&path, content)?;
std::fs::write(&path, content)
.map_err(|e| format!("write {}: {}", name, e))?;
println!("Created {}", path.display());
}
Ok(())
}
pub async fn cmd_init() -> Result<()> {
pub fn cmd_init() -> Result<(), String> {
let cfg = crate::config::get();
// Ensure data directory exists
std::fs::create_dir_all(&cfg.data_dir)?;
std::fs::create_dir_all(&cfg.data_dir)
.map_err(|e| format!("create data_dir: {}", e))?;
// Install filesystem files (not store nodes)
install_default_file(&cfg.data_dir, "instructions.md",
@ -25,15 +25,19 @@ pub async fn cmd_init() -> Result<()> {
install_default_file(&cfg.data_dir, "on-consciousness.md",
include_str!("../../defaults/on-consciousness.md"))?;
// Seed identity node if empty
let store = memory::access_local()?;
if !store.contains_key("identity").unwrap_or(false) {
// Initialize store and seed default identity node if empty
let mut store = store::Store::load()?;
let count = store.init_from_markdown()?;
for key in &cfg.core_nodes {
if !store.nodes.contains_key(key) && key == "identity" {
let default = include_str!("../../defaults/identity.md");
store.upsert("identity", default)?;
println!("Seeded identity in store");
store.upsert(key, default)
.map_err(|e| format!("seed {}: {}", key, e))?;
println!("Seeded {} in store", key);
}
}
store.save()?;
println!("Initialized with {} nodes", store.all_keys().unwrap_or_default().len());
println!("Indexed {} memory units", count);
// Create config if none exists
let config_path = std::env::var("POC_MEMORY_CONFIG")
@ -44,9 +48,11 @@ pub async fn cmd_init() -> Result<()> {
});
if !config_path.exists() {
let config_dir = config_path.parent().unwrap();
std::fs::create_dir_all(config_dir)?;
std::fs::create_dir_all(config_dir)
.map_err(|e| format!("create config dir: {}", e))?;
let example = include_str!("../../config.example.jsonl");
std::fs::write(&config_path, example)?;
std::fs::write(&config_path, example)
.map_err(|e| format!("write config: {}", e))?;
println!("Created config at {} — edit with your name and context groups",
config_path.display());
}
@ -55,65 +61,168 @@ pub async fn cmd_init() -> Result<()> {
Ok(())
}
pub async fn cmd_fsck() -> Result<()> {
// Check/repair capnp log integrity first
store::fsck()?;
pub fn cmd_bulk_rename(from: &str, to: &str, apply: bool) -> Result<(), String> {
let mut store = store::Store::load()?;
let store = memory::access_local()?;
// Find all keys that need renaming
let renames: Vec<(String, String)> = store.nodes.keys()
.filter(|k| k.contains(from))
.map(|k| (k.clone(), k.replace(from, to)))
.collect();
// Check for collisions
let existing: std::collections::HashSet<&String> = store.nodes.keys().collect();
let mut collisions = 0;
for (old, new) in &renames {
if existing.contains(new) && old != new {
eprintln!("COLLISION: {} -> {} (target exists)", old, new);
collisions += 1;
}
}
println!("Bulk rename '{}' -> '{}'", from, to);
println!(" Keys to rename: {}", renames.len());
println!(" Collisions: {}", collisions);
if collisions > 0 {
return Err(format!("{} collisions — aborting", collisions));
}
if !apply {
// Show a sample
for (old, new) in renames.iter().take(10) {
println!(" {} -> {}", old, new);
}
if renames.len() > 10 {
println!(" ... and {} more", renames.len() - 10);
}
println!("\nDry run. Use --apply to execute.");
return Ok(());
}
// Apply renames using rename_node() which properly appends to capnp logs.
// Process in batches to avoid holding the lock too long.
let mut renamed_count = 0;
let mut errors = 0;
let total = renames.len();
for (i, (old_key, new_key)) in renames.iter().enumerate() {
match store.rename_node(old_key, new_key) {
Ok(()) => renamed_count += 1,
Err(e) => {
eprintln!(" RENAME ERROR: {} -> {}: {}", old_key, new_key, e);
errors += 1;
}
}
if (i + 1) % 1000 == 0 {
println!(" {}/{} ({} errors)", i + 1, total, errors);
}
}
store.save()?;
println!("Renamed {} nodes ({} errors).", renamed_count, errors);
// Run fsck to verify
println!("\nRunning fsck...");
drop(store);
cmd_fsck()?;
Ok(())
}
pub fn cmd_fsck() -> Result<(), String> {
let mut store = store::Store::load()?;
// Check cache vs log consistency
let log_store = store::Store::load_from_logs()?;
let mut cache_issues = 0;
// Nodes in logs but missing from cache
for key in log_store.nodes.keys() {
if !store.nodes.contains_key(key) {
eprintln!("CACHE MISSING: '{}' exists in capnp log but not in cache", key);
cache_issues += 1;
}
}
// Nodes in cache but not in logs (phantom nodes)
for key in store.nodes.keys() {
if !log_store.nodes.contains_key(key) {
eprintln!("CACHE PHANTOM: '{}' exists in cache but not in capnp log", key);
cache_issues += 1;
}
}
// Version mismatches
for (key, log_node) in &log_store.nodes {
if let Some(cache_node) = store.nodes.get(key)
&& cache_node.version != log_node.version {
eprintln!("CACHE STALE: '{}' cache v{} vs log v{}",
key, cache_node.version, log_node.version);
cache_issues += 1;
}
}
if cache_issues > 0 {
eprintln!("{} cache inconsistencies found — rebuilding from logs", cache_issues);
store = log_store;
store.save().map_err(|e| format!("rebuild save: {}", e))?;
}
// Check node-key consistency
let mut issues = 0;
let all_keys = store.all_keys().unwrap_or_default();
for key in &all_keys {
if let Ok(Some(node)) = store.get_node(key) {
for (key, node) in &store.nodes {
if key != &node.key {
eprintln!("MISMATCH: map key '{}' vs node.key '{}'", key, node.key);
issues += 1;
}
}
}
// Check edge endpoints using index
use crate::hippocampus::store::StoreView;
// Check edge endpoints
let mut dangling = 0;
let mut orphan_edges: Vec<(String, String)> = Vec::new();
store.for_each_relation(|source, target, _, _| {
let s_missing = !store.contains_key(source).unwrap_or(false);
let t_missing = !store.contains_key(target).unwrap_or(false);
if s_missing {
eprintln!("DANGLING: edge source '{}'", source);
for rel in &store.relations {
if rel.deleted { continue; }
if !store.nodes.contains_key(&rel.source_key) {
eprintln!("DANGLING: edge source '{}'", rel.source_key);
dangling += 1;
}
if t_missing {
eprintln!("DANGLING: edge target '{}'", target);
if !store.nodes.contains_key(&rel.target_key) {
eprintln!("DANGLING: edge target '{}'", rel.target_key);
dangling += 1;
}
if s_missing || t_missing {
orphan_edges.push((source.to_string(), target.to_string()));
}
});
// Prune orphan edges
if !orphan_edges.is_empty() {
let count = orphan_edges.len();
for (source, target) in &orphan_edges {
// set_link_strength with 0 would delete, but we don't have that
// For now just report - full cleanup requires more work
eprintln!("Would prune: {} → {}", source, target);
let mut to_tombstone = Vec::new();
for rel in &store.relations {
if rel.deleted { continue; }
if !store.nodes.contains_key(&rel.source_key)
|| !store.nodes.contains_key(&rel.target_key) {
let mut tombstone = rel.clone();
tombstone.deleted = true;
tombstone.version += 1;
to_tombstone.push(tombstone);
}
eprintln!("Found {} orphan edges (prune not yet implemented for index)", count);
}
if !to_tombstone.is_empty() {
let count = to_tombstone.len();
store.append_relations(&to_tombstone)?;
for t in &to_tombstone {
if let Some(r) = store.relations.iter_mut().find(|r| r.uuid == t.uuid) {
r.deleted = true;
r.version = t.version;
}
}
store.save()?;
eprintln!("Pruned {} orphan edges", count);
}
let g = store.build_graph();
println!("fsck: {} nodes, {} edges, {} issues, {} dangling",
all_keys.len(), g.edge_count(), issues, dangling);
println!("fsck: {} nodes, {} edges, {} issues, {} dangling, {} cache",
store.nodes.len(), g.edge_count(), issues, dangling, cache_issues);
Ok(())
}
pub async fn cmd_dedup(apply: bool) -> Result<()> {
use std::collections::HashMap;
pub fn cmd_dedup(apply: bool) -> Result<(), String> {
use std::collections::{HashMap, HashSet};
let store = memory::access_local()?;
let mut store = store::Store::load()?;
let duplicates = store.find_duplicates()?;
if duplicates.is_empty() {
@ -121,19 +230,12 @@ pub async fn cmd_dedup(apply: bool) -> Result<()> {
return Ok(());
}
// Count edges per key (we'll map to UUID later)
use crate::hippocampus::store::StoreView;
let mut edges_by_key: HashMap<String, usize> = HashMap::new();
store.for_each_relation(|source, target, _, _| {
*edges_by_key.entry(source.to_string()).or_default() += 1;
*edges_by_key.entry(target.to_string()).or_default() += 1;
});
// Convert to edges_by_uuid for compatibility
// Count edges per UUID
let mut edges_by_uuid: HashMap<[u8; 16], usize> = HashMap::new();
for (key, count) in &edges_by_key {
if let Ok(Some(node)) = store.get_node(key) {
edges_by_uuid.insert(node.uuid, *count);
}
for rel in &store.relations {
if rel.deleted { continue; }
*edges_by_uuid.entry(rel.source).or_default() += 1;
*edges_by_uuid.entry(rel.target).or_default() += 1;
}
let mut identical_groups = Vec::new();
@ -207,14 +309,6 @@ pub async fn cmd_dedup(apply: bool) -> Result<()> {
.chain(diverged_groups)
.collect();
// Build uuid → key map for relation key strings
let mut uuid_to_key: HashMap<[u8; 16], String> = HashMap::new();
for key in store.all_keys()? {
if let Ok(Some(node)) = store.get_node(&key) {
uuid_to_key.insert(node.uuid, key);
}
}
let mut merged = 0usize;
let mut edges_redirected = 0usize;
let mut edges_deduped = 0usize;
@ -224,89 +318,52 @@ pub async fn cmd_dedup(apply: bool) -> Result<()> {
copies.sort_by(|a, b| b.1.cmp(&a.1).then(b.0.version.cmp(&a.0.version)));
let survivor_uuid = copies[0].0.uuid;
let survivor_key = uuid_to_key.get(&survivor_uuid).cloned().unwrap_or_default();
let doomed_uuids: Vec<[u8; 16]> = copies[1..].iter().map(|c| c.0.uuid).collect();
// Redirect edges from doomed UUIDs to survivor via index iteration
for doomed_uuid in &doomed_uuids {
let edges = store.edges_for_uuid(doomed_uuid)?;
for (other_uuid, strength, rel_type, is_outgoing) in edges {
let other_key = uuid_to_key.get(&other_uuid).cloned().unwrap_or_default();
// Remove old edge from index
let (old_src, old_tgt) = if is_outgoing {
(*doomed_uuid, other_uuid)
} else {
(other_uuid, *doomed_uuid)
};
store.remove_relation_from_index(&old_src, &old_tgt, strength, rel_type)?;
// Add redirected edge
let (new_src, new_tgt, src_key, tgt_key) = if is_outgoing {
(survivor_uuid, other_uuid, survivor_key.clone(), other_key)
} else {
(other_uuid, survivor_uuid, other_key, survivor_key.clone())
};
store.index_relation(&new_src, &new_tgt, strength, rel_type)?;
// Append tombstone for old + new relation to log
let mut tombstone = store::new_relation(
old_src, old_tgt,
store::RelationType::from_u8(rel_type), strength,
&uuid_to_key.get(&old_src).cloned().unwrap_or_default(),
&uuid_to_key.get(&old_tgt).cloned().unwrap_or_default(),
);
tombstone.deleted = true;
tombstone.version = 2;
let mut redirected = store::new_relation(
new_src, new_tgt,
store::RelationType::from_u8(rel_type), strength,
&src_key, &tgt_key,
);
redirected.version = 2;
store.append_relations(&[tombstone, redirected])?;
// Redirect edges from doomed UUIDs to survivor
let mut updated_rels = Vec::new();
for rel in &mut store.relations {
if rel.deleted { continue; }
let mut changed = false;
if doomed_uuids.contains(&rel.source) {
rel.source = survivor_uuid;
changed = true;
}
if doomed_uuids.contains(&rel.target) {
rel.target = survivor_uuid;
changed = true;
}
if changed {
rel.version += 1;
updated_rels.push(rel.clone());
edges_redirected += 1;
}
}
// Dedup edges: same (other_uuid, rel_type) → keep highest strength
// Group edges by (other, type), sort each group by strength desc, tombstone extras
let edges = store.edges_for_uuid(&survivor_uuid)?;
let mut by_endpoint: HashMap<([u8; 16], u8), Vec<(f32, bool)>> = HashMap::new();
for (other_uuid, strength, rel_type, is_outgoing) in edges {
by_endpoint.entry((other_uuid, rel_type))
.or_default()
.push((strength, is_outgoing));
}
// Dedup edges: same (source, target, rel_type) → keep highest strength
let mut seen: HashSet<([u8; 16], [u8; 16], String)> = HashSet::new();
let mut to_tombstone_rels = Vec::new();
// Sort by strength descending so we keep the strongest
let mut rels_with_idx: Vec<(usize, &store::Relation)> = store.relations.iter()
.enumerate()
.filter(|(_, r)| !r.deleted && (r.source == survivor_uuid || r.target == survivor_uuid))
.collect();
rels_with_idx.sort_by(|a, b| b.1.strength.total_cmp(&a.1.strength));
for ((other_uuid, rel_type), mut variants) in by_endpoint {
if variants.len() <= 1 { continue; }
// Sort by strength descending, keep first
variants.sort_by(|a, b| b.0.total_cmp(&a.0));
let other_key = uuid_to_key.get(&other_uuid).cloned().unwrap_or_default();
for (strength, is_outgoing) in variants.into_iter().skip(1) {
let (src, tgt, src_key, tgt_key) = if is_outgoing {
(survivor_uuid, other_uuid, survivor_key.clone(), other_key.clone())
} else {
(other_uuid, survivor_uuid, other_key.clone(), survivor_key.clone())
};
store.remove_relation_from_index(&src, &tgt, strength, rel_type)?;
let mut tombstone = store::new_relation(
src, tgt,
store::RelationType::from_u8(rel_type), strength,
&src_key, &tgt_key,
);
tombstone.deleted = true;
tombstone.version = 2;
store.append_relations(&[tombstone])?;
for (idx, rel) in &rels_with_idx {
let edge_key = (rel.source, rel.target, format!("{:?}", rel.rel_type));
if !seen.insert(edge_key) {
to_tombstone_rels.push(*idx);
edges_deduped += 1;
}
}
for &idx in &to_tombstone_rels {
store.relations[idx].deleted = true;
store.relations[idx].version += 1;
updated_rels.push(store.relations[idx].clone());
}
// Tombstone doomed nodes
let mut tombstones = Vec::new();
for (doomed_node, _) in &copies[1..] {
@ -317,15 +374,19 @@ pub async fn cmd_dedup(apply: bool) -> Result<()> {
}
store.append_nodes(&tombstones)?;
if !updated_rels.is_empty() {
store.append_relations(&updated_rels)?;
}
// Remove doomed nodes from index
for (doomed_node, _) in &copies[1..] {
store.remove_from_index(&doomed_node.key)?;
for uuid in &doomed_uuids {
store.uuid_to_key.remove(uuid);
}
merged += doomed_uuids.len();
}
// Remove tombstoned relations from cache
store.relations.retain(|r| !r.deleted);
store.save()?;
println!("Merged {} duplicates, redirected {} edges, deduped {} duplicate edges",
@ -334,30 +395,87 @@ pub async fn cmd_dedup(apply: bool) -> Result<()> {
Ok(())
}
pub async fn cmd_health() -> Result<()> {
let result = memory::graph_health(None).await
?;
print!("{}", result);
pub fn cmd_health() -> Result<(), String> {
let store = store::Store::load()?;
let g = store.build_graph();
let report = crate::graph::health_report(&g, &store);
print!("{}", report);
Ok(())
}
pub async fn cmd_topology() -> Result<()> {
let result = memory::graph_topology(None).await
?;
print!("{}", result);
Ok(())
}
pub async fn cmd_daily_check() -> Result<()> {
let store = memory::access_local()?;
pub fn cmd_daily_check() -> Result<(), String> {
let store = store::Store::load()?;
let report = crate::neuro::daily_check(&store);
print!("{}", report);
Ok(())
}
pub async fn cmd_status() -> Result<()> {
let result = memory::graph_topology(None).await
?;
print!("{}", result);
pub fn cmd_import(files: &[String]) -> Result<(), String> {
if files.is_empty() {
return Err("import requires at least one file path".into());
}
let mut store = store::Store::load()?;
let mut total_new = 0;
let mut total_updated = 0;
for arg in files {
let path = std::path::PathBuf::from(arg);
let resolved = if path.exists() {
path
} else {
let mem_path = store::memory_dir().join(arg);
if !mem_path.exists() {
eprintln!("File not found: {}", arg);
continue;
}
mem_path
};
let (n, u) = store.import_file(&resolved)?;
total_new += n;
total_updated += u;
}
if total_new > 0 || total_updated > 0 {
store.save()?;
}
println!("Import: {} new, {} updated", total_new, total_updated);
Ok(())
}
pub fn cmd_export(files: &[String], export_all: bool) -> Result<(), String> {
let store = store::Store::load()?;
let targets: Vec<String> = if export_all {
let mut files: Vec<String> = store.nodes.keys()
.filter(|k| !k.contains('#'))
.cloned()
.collect();
files.sort();
files
} else if files.is_empty() {
return Err("export requires file keys or --all".into());
} else {
files.iter().map(|a| {
a.strip_suffix(".md").unwrap_or(a).to_string()
}).collect()
};
let mem_dir = store::memory_dir();
for file_key in &targets {
match store.export_to_markdown(file_key) {
Some(content) => {
let out_path = mem_dir.join(format!("{}.md", file_key));
std::fs::write(&out_path, &content)
.map_err(|e| format!("write {}: {}", out_path.display(), e))?;
let section_count = content.matches("<!-- mem:").count() + 1;
println!("Exported {} ({} sections)", file_key, section_count);
}
None => eprintln!("No nodes for '{}'", file_key),
}
}
Ok(())
}

69
src/cli/agent.rs
View file

@ -1,9 +1,9 @@
// cli/agent.rs — agent subcommand handlers
use anyhow::{bail, Context, Result};
use crate::hippocampus as memory;
use crate::store;
use crate::subconscious::digest;
pub async fn cmd_run_agent(agent: &str, count: usize, target: &[String], query: Option<&str>, dry_run: bool, _local: bool, state_dir: Option<&str>) -> Result<()> {
pub fn cmd_run_agent(agent: &str, count: usize, target: &[String], query: Option<&str>, dry_run: bool, _local: bool, state_dir: Option<&str>) -> Result<(), String> {
// Mark as agent so tool calls (e.g. poc-memory render) don't
// pollute the user's seen set as a side effect
// SAFETY: single-threaded at this point (CLI startup, before any agent work)
@ -11,7 +11,7 @@ pub async fn cmd_run_agent(agent: &str, count: usize, target: &[String], query:
// Override agent output/state directory if specified
if let Some(dir) = state_dir {
std::fs::create_dir_all(dir).context("create state dir")?;
std::fs::create_dir_all(dir).map_err(|e| format!("create state dir: {}", e))?;
unsafe { std::env::set_var("POC_AGENT_OUTPUT_DIR", dir); }
}
@ -19,20 +19,19 @@ pub async fn cmd_run_agent(agent: &str, count: usize, target: &[String], query:
unsafe { std::env::set_var("POC_MEMORY_DRY_RUN", "1"); }
}
let mut store = store::Store::load()?;
// Resolve targets: explicit --target, --query, or agent's default query
let resolved_targets: Vec<String> = if !target.is_empty() {
target.to_vec()
} else if let Some(q) = query {
// Resolve query via typed API
let q_str = format!("{} | limit:{}", q, count);
let result = memory::memory_query(None, &q_str, None).await?;
let keys: Vec<String> = result.lines()
.filter(|l| !l.is_empty() && *l != "no results")
.map(|s| s.to_string())
.collect();
if keys.is_empty() {
bail!("query returned no results: {}", q);
let graph = store.build_graph();
let stages = crate::query_parser::parse_stages(q)?;
let results = crate::search::run_query(&stages, vec![], &graph, &store, false, count);
if results.is_empty() {
return Err(format!("query returned no results: {}", q));
}
let keys: Vec<String> = results.into_iter().map(|(k, _)| k).collect();
println!("[{}] query matched {} nodes", agent, keys.len());
keys
} else {
@ -42,17 +41,53 @@ pub async fn cmd_run_agent(agent: &str, count: usize, target: &[String], query:
if !resolved_targets.is_empty() {
for (i, key) in resolved_targets.iter().enumerate() {
println!("[{}] [{}/{}] {}", agent, i + 1, resolved_targets.len(), key);
if i > 0 { store = store::Store::load()?; }
if let Err(e) = crate::agent::oneshot::run_one_agent(
agent, count, Some(&[key.clone()]),
).await {
&mut store, agent, count, Some(&[key.clone()]),
) {
println!("[{}] ERROR on {}: {}", agent, key, e);
}
}
} else {
// Local execution (--local, --debug, dry-run, or daemon unavailable)
crate::agent::oneshot::run_one_agent(
agent, count, None,
).await.map_err(|e| anyhow::anyhow!("{}", e))?;
&mut store, agent, count, None,
)?;
}
Ok(())
}
pub fn cmd_replay_queue(count: usize) -> Result<(), String> {
let store = store::Store::load()?;
let queue = crate::neuro::replay_queue(&store, count);
println!("Replay queue ({} items):", queue.len());
for (i, item) in queue.iter().enumerate() {
println!(" {:2}. [{:.3}] {:>10} {} (interval={}d, emotion={:.1}, spectral={:.1})",
i + 1, item.priority, item.classification, item.key,
item.interval_days, item.emotion, item.outlier_score);
}
Ok(())
}
pub fn cmd_digest_links(do_apply: bool) -> Result<(), String> {
let store = store::Store::load()?;
let links = digest::parse_all_digest_links(&store);
drop(store);
println!("Found {} unique links from digest nodes", links.len());
if !do_apply {
for (i, link) in links.iter().enumerate() {
println!(" {:3}. {} → {}", i + 1, link.source, link.target);
if !link.reason.is_empty() {
println!(" ({})", &link.reason[..link.reason.floor_char_boundary(link.reason.len().min(80))]);
}
}
println!("\nTo apply: poc-memory digest-links --apply");
return Ok(());
}
let mut store = store::Store::load()?;
let (applied, skipped, fallbacks) = digest::apply_digest_links(&mut store, &links);
println!("\nApplied: {} ({} file-level fallbacks) Skipped: {}", applied, fallbacks, skipped);
Ok(())
}

429
src/cli/graph.rs
View file

@ -4,72 +4,437 @@
// link, link-add, link-impact, link-audit, cap-degree,
// normalize-strengths, trace, spectral-*, organize, communities.
use anyhow::{bail, Result};
use crate::hippocampus as memory;
use crate::{store, graph};
use crate::store::StoreView;
pub async fn cmd_cap_degree(max_deg: usize) -> Result<()> {
let store = memory::access_local()?;
pub fn cmd_graph() -> Result<(), String> {
let store = store::Store::load()?;
let g = store.build_graph();
println!("Graph: {} nodes, {} edges, {} communities",
g.nodes().len(), g.edge_count(), g.community_count());
println!("σ={:.2} α={:.2} gini={:.3} cc={:.4}",
g.small_world_sigma(), g.degree_power_law_exponent(),
g.degree_gini(), g.avg_clustering_coefficient());
Ok(())
}
pub fn cmd_cap_degree(max_deg: usize) -> Result<(), String> {
let mut store = store::Store::load()?;
let (hubs, pruned) = store.cap_degree(max_deg)?;
store.save()?;
println!("Capped {} hubs, pruned {} weak Auto edges (max_degree={})", hubs, pruned, max_deg);
Ok(())
}
pub async fn cmd_normalize_strengths(apply: bool) -> Result<()> {
if apply { super::check_dry_run(); }
let result = memory::graph_normalize_strengths(None, Some(apply)).await?;
print!("{}", result);
pub fn cmd_normalize_strengths(apply: bool) -> Result<(), String> {
let mut store = store::Store::load()?;
let graph = store.build_graph();
let strengths = graph.jaccard_strengths();
// Build a lookup from (source_key, target_key) → new_strength
let mut updates: std::collections::HashMap<(String, String), f32> = std::collections::HashMap::new();
for (a, b, s) in &strengths {
// Store both directions for easy lookup
updates.insert((a.clone(), b.clone()), *s);
updates.insert((b.clone(), a.clone()), *s);
}
// Stats
let mut changed = 0usize;
let mut unchanged = 0usize;
let mut temporal_skipped = 0usize;
let mut delta_sum: f64 = 0.0;
// Histogram of new strengths
let mut buckets = [0usize; 10]; // 0.0-0.1, 0.1-0.2, ...
for rel in &mut store.relations {
if rel.deleted { continue; }
// Skip implicit temporal edges (strength 1.0, Auto type)
if rel.strength == 1.0 && rel.rel_type == store::RelationType::Auto {
temporal_skipped += 1;
continue;
}
if let Some(&new_s) = updates.get(&(rel.source_key.clone(), rel.target_key.clone())) {
let old_s = rel.strength;
let delta = (new_s - old_s).abs();
if delta > 0.001 {
delta_sum += delta as f64;
if apply {
rel.strength = new_s;
}
changed += 1;
} else {
unchanged += 1;
}
let bucket = ((new_s * 10.0) as usize).min(9);
buckets[bucket] += 1;
}
}
println!("Normalize link strengths (Jaccard similarity)");
println!(" Total edges in graph: {}", strengths.len());
println!(" Would change: {}", changed);
println!(" Unchanged: {}", unchanged);
println!(" Temporal (skipped): {}", temporal_skipped);
if changed > 0 {
println!(" Avg delta: {:.3}", delta_sum / changed as f64);
}
println!();
println!(" Strength distribution:");
for (i, &count) in buckets.iter().enumerate() {
let lo = i as f32 / 10.0;
let hi = lo + 0.1;
let bar = "#".repeat(count / 50 + if count > 0 { 1 } else { 0 });
println!(" {:.1}-{:.1}: {:5} {}", lo, hi, count, bar);
}
if apply {
store.save()?;
println!("\nApplied {} strength updates.", changed);
} else {
println!("\nDry run. Use --apply to write changes.");
}
Ok(())
}
pub async fn cmd_link(key: &[String]) -> Result<()> {
pub fn cmd_spread(keys: &[String], max_results: usize) -> Result<(), String> {
if keys.is_empty() {
return Err("spread requires at least one seed key".into());
}
let store = store::Store::load()?;
let graph = graph::build_graph_fast(&store);
let params = store.params();
let seeds: Vec<(String, f64)> = keys.iter()
.filter_map(|k| {
let resolved = store.resolve_key(k).ok()?;
Some((resolved, 1.0))
})
.collect();
if seeds.is_empty() {
return Err("no valid seed keys found".into());
}
let results = crate::search::spreading_activation(
&seeds, &graph, &store,
params.max_hops, params.edge_decay, params.min_activation,
);
let seed_keys: std::collections::HashSet<&str> = seeds.iter()
.map(|(k, _)| k.as_str())
.collect();
for (key, score) in results.iter()
.filter(|(k, _)| !seed_keys.contains(k.as_str()))
.take(max_results)
{
println!(" {:.2} {}", score, key);
}
Ok(())
}
pub fn cmd_link(key: &[String]) -> Result<(), String> {
if key.is_empty() {
bail!("link requires a key");
return Err("link requires a key".into());
}
let key = key.join(" ");
let links = memory::memory_links(None, &key).await?;
println!("Neighbors of '{}':", key);
for link in links {
println!(" ({:.2}) {} [w={:.2}]", link.link_strength, link.key, link.node_weight);
}
Ok(())
let store = store::Store::load()?;
let resolved = store.resolve_key(&key)?;
let g = store.build_graph();
println!("Neighbors of '{}':", resolved);
crate::query_parser::run_query(&store, &g,
&format!("neighbors('{}') | select strength,clustering_coefficient", resolved))
}
pub async fn cmd_link_add(source: &str, target: &str, _reason: &[String]) -> Result<()> {
pub fn cmd_link_add(source: &str, target: &str, reason: &[String]) -> Result<(), String> {
super::check_dry_run();
let result = memory::memory_link_add(None, source, target).await?;
println!("{}", result);
let mut store = store::Store::load()?;
let source = store.resolve_key(source)?;
let target = store.resolve_key(target)?;
let reason = reason.join(" ");
match store.add_link(&source, &target, "manual") {
Ok(strength) => {
store.save()?;
println!("Linked: {} → {} (strength={:.2}, {})", source, target, strength, reason);
}
Err(msg) if msg.contains("already exists") => {
println!("Link already exists: {} ↔ {}", source, target);
}
Err(e) => return Err(e),
}
Ok(())
}
pub async fn cmd_link_set(source: &str, target: &str, strength: f32) -> Result<()> {
pub fn cmd_link_set(source: &str, target: &str, strength: f32) -> Result<(), String> {
super::check_dry_run();
let result = memory::memory_link_set(None, source, target, strength).await?;
println!("{}", result);
let mut store = store::Store::load()?;
let source = store.resolve_key(source)?;
let target = store.resolve_key(target)?;
let old = store.set_link_strength(&source, &target, strength)?;
println!("Set: {} ↔ {} strength {:.2} → {:.2}", source, target, old, strength);
store.save()?;
Ok(())
}
pub async fn cmd_link_impact(source: &str, target: &str) -> Result<()> {
let result = memory::graph_link_impact(None, source, target).await?;
print!("{}", result);
pub fn cmd_link_impact(source: &str, target: &str) -> Result<(), String> {
let store = store::Store::load()?;
let source = store.resolve_key(source)?;
let target = store.resolve_key(target)?;
let g = store.build_graph();
let impact = g.link_impact(&source, &target);
println!("Link impact: {} → {}", source, target);
println!(" Source degree: {} Target degree: {}", impact.source_deg, impact.target_deg);
println!(" Hub link: {} Same community: {}", impact.is_hub_link, impact.same_community);
println!(" ΔCC source: {:+.4} ΔCC target: {:+.4}", impact.delta_cc_source, impact.delta_cc_target);
println!(" ΔGini: {:+.6}", impact.delta_gini);
println!(" Assessment: {}", impact.assessment);
Ok(())
}
pub async fn cmd_trace(key: &[String]) -> Result<()> {
pub fn cmd_trace(key: &[String]) -> Result<(), String> {
if key.is_empty() {
bail!("trace requires a key");
return Err("trace requires a key".into());
}
let key = key.join(" ");
let result = memory::graph_trace(None, &key).await?;
print!("{}", result);
let store = store::Store::load()?;
let resolved = store.resolve_key(&key)?;
let g = store.build_graph();
let node = store.nodes.get(&resolved)
.ok_or_else(|| format!("Node not found: {}", resolved))?;
// Display the node itself
println!("=== {} ===", resolved);
println!("Type: {:?} Weight: {:.2}",
node.node_type, node.weight);
if !node.source_ref.is_empty() {
println!("Source: {}", node.source_ref);
}
// Show content preview
let preview = crate::util::truncate(&node.content, 200, "...");
println!("\n{}\n", preview);
// Walk neighbors, grouped by node type
let neighbors = g.neighbors(&resolved);
let mut episodic_session = Vec::new();
let mut episodic_daily = Vec::new();
let mut episodic_weekly = Vec::new();
let mut semantic = Vec::new();
for (n, strength) in &neighbors {
if let Some(nnode) = store.nodes.get(n.as_str()) {
let entry = (n.as_str(), *strength, nnode);
match nnode.node_type {
store::NodeType::EpisodicSession =>
episodic_session.push(entry),
store::NodeType::EpisodicDaily =>
episodic_daily.push(entry),
store::NodeType::EpisodicWeekly
| store::NodeType::EpisodicMonthly =>
episodic_weekly.push(entry),
store::NodeType::Semantic =>
semantic.push(entry),
}
}
}
if !episodic_weekly.is_empty() {
println!("Weekly digests:");
for (k, s, n) in &episodic_weekly {
let preview = crate::util::first_n_chars(n.content.lines().next().unwrap_or(""), 80);
println!(" [{:.2}] {} — {}", s, k, preview);
}
}
if !episodic_daily.is_empty() {
println!("Daily digests:");
for (k, s, n) in &episodic_daily {
let preview = crate::util::first_n_chars(n.content.lines().next().unwrap_or(""), 80);
println!(" [{:.2}] {} — {}", s, k, preview);
}
}
if !episodic_session.is_empty() {
println!("Session entries:");
for (k, s, n) in &episodic_session {
let preview = crate::util::first_n_chars(
n.content.lines()
.find(|l| !l.is_empty() && !l.starts_with("<!--"))
.unwrap_or(""),
80);
println!(" [{:.2}] {}", s, k);
if !n.source_ref.is_empty() {
println!(" ↳ source: {}", n.source_ref);
}
println!(" {}", preview);
}
}
if !semantic.is_empty() {
println!("Semantic links:");
for (k, s, _) in &semantic {
println!(" [{:.2}] {}", s, k);
}
}
println!("\nLinks: {} session, {} daily, {} weekly, {} semantic",
episodic_session.len(), episodic_daily.len(),
episodic_weekly.len(), semantic.len());
Ok(())
}
pub fn cmd_organize(term: &str, key_only: bool, create_anchor: bool) -> Result<(), String> {
let mut store = store::Store::load()?;
// Step 1: find all non-deleted nodes matching the term
let term_lower = term.to_lowercase();
let mut topic_nodes: Vec<(String, String)> = Vec::new(); // (key, content)
let skip_prefixes = ["_", "deep-index#", "facts-", "irc-history#"];
for (key, node) in &store.nodes {
if node.deleted { continue; }
// Skip episodic/digest nodes — use NodeType, not key prefix
if node.node_type != crate::store::NodeType::Semantic { continue; }
let key_matches = key.to_lowercase().contains(&term_lower);
let content_matches = !key_only && node.content.to_lowercase().contains(&term_lower);
if !key_matches && !content_matches { continue; }
if skip_prefixes.iter().any(|p| key.starts_with(p)) { continue; }
topic_nodes.push((key.clone(), node.content.clone()));
}
if topic_nodes.is_empty() {
println!("No topic nodes found matching '{}'", term);
return Ok(());
}
topic_nodes.sort_by(|a, b| a.0.cmp(&b.0));
println!("=== Organize: '{}' ===", term);
println!("Found {} topic nodes:\n", topic_nodes.len());
for (key, content) in &topic_nodes {
let lines = content.lines().count();
let words = content.split_whitespace().count();
println!(" {:60} {:>4} lines {:>5} words", key, lines, words);
}
// Step 2: check connectivity within cluster
let g = store.build_graph();
println!("=== Connectivity ===\n");
// Pick hub by intra-cluster connectivity, not overall degree
let cluster_keys: std::collections::HashSet<&str> = topic_nodes.iter()
.filter(|(k,_)| store.nodes.contains_key(k.as_str()))
.map(|(k,_)| k.as_str())
.collect();
let mut best_hub: Option<(&str, usize)> = None;
for key in &cluster_keys {
let intra_degree = g.neighbor_keys(key).iter()
.filter(|n| cluster_keys.contains(*n))
.count();
if best_hub.is_none() || intra_degree > best_hub.unwrap().1 {
best_hub = Some((key, intra_degree));
}
}
if let Some((hub, deg)) = best_hub {
println!(" Hub: {} (degree {})", hub, deg);
let hub_nbrs = g.neighbor_keys(hub);
let mut unlinked = Vec::new();
for (key, _) in &topic_nodes {
if key == hub { continue; }
if store.nodes.get(key.as_str()).is_none() { continue; }
if !hub_nbrs.contains(key.as_str()) {
unlinked.push(key.clone());
}
}
if unlinked.is_empty() {
println!(" All cluster nodes connected to hub ✓");
} else {
println!(" NOT linked to hub:");
for key in &unlinked {
println!(" {} → needs link to {}", key, hub);
}
}
}
// Step 4: anchor node
if create_anchor {
println!("\n=== Anchor node ===\n");
if store.nodes.contains_key(term) && !store.nodes[term].deleted {
println!(" Anchor '{}' already exists ✓", term);
} else {
let desc = format!("Anchor node for '{}' search term", term);
store.upsert(term, &desc)?;
let anchor_uuid = store.nodes.get(term).unwrap().uuid;
for (key, _) in &topic_nodes {
if store.nodes.get(key.as_str()).is_none() { continue; }
let target_uuid = store.nodes[key.as_str()].uuid;
let rel = store::new_relation(
anchor_uuid, target_uuid,
store::RelationType::Link, 0.8,
term, key,
);
store.add_relation(rel)?;
}
println!(" Created anchor '{}' with {} links", term, topic_nodes.len());
}
}
store.save()?;
Ok(())
}
/// Show communities sorted by isolation (most isolated first).
/// Useful for finding poorly-integrated knowledge clusters that need
/// organize agents aimed at them.
pub async fn cmd_communities(top_n: usize, min_size: usize) -> Result<()> {
let result = memory::graph_communities(None, Some(top_n), Some(min_size)).await?;
print!("{}", result);
pub fn cmd_communities(top_n: usize, min_size: usize) -> Result<(), String> {
let store = store::Store::load()?;
let g = store.build_graph();
let infos = g.community_info();
let total = infos.len();
let shown: Vec<_> = infos.into_iter()
.filter(|c| c.size >= min_size)
.take(top_n)
.collect();
println!("{} communities total ({} with size >= {})\n",
total, shown.len(), min_size);
println!("{:<6} {:>5} {:>7} {:>7} members", "id", "size", "iso", "cross");
println!("{}", "-".repeat(70));
for c in &shown {
let preview: Vec<&str> = c.members.iter()
.take(5)
.map(|s| s.as_str())
.collect();
let more = if c.size > 5 {
format!(" +{}", c.size - 5)
} else {
String::new()
};
println!("{:<6} {:>5} {:>6.0}% {:>7} {}{}",
c.id, c.size, c.isolation * 100.0, c.cross_edges,
preview.join(", "), more);
}
Ok(())
}

127
src/cli/journal.rs
View file

@ -1,26 +1,24 @@
// cli/journal.rs — journal subcommand handlers
use anyhow::{bail, Context, Result};
use crate::hippocampus as memory;
pub fn cmd_tail(n: usize, full: bool, provenance: Option<&str>, dedup: bool) -> Result<()> {
pub fn cmd_tail(n: usize, full: bool, provenance: Option<&str>, dedup: bool) -> Result<(), String> {
let path = crate::store::nodes_path();
if !path.exists() {
bail!("No node log found");
return Err("No node log found".into());
}
use std::io::BufReader;
let file = std::fs::File::open(&path)
.with_context(|| format!("open {}", path.display()))?;
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
// Read all entries, keep last N
let mut entries: Vec<crate::store::Node> = Vec::new();
while let Ok(msg) = capnp::serialize::read_message(&mut reader, capnp::message::ReaderOptions::new()) {
let log = msg.get_root::<crate::memory_capnp::node_log::Reader>()
.with_context(|| "read log")?;
.map_err(|e| format!("read log: {}", e))?;
for node_reader in log.get_nodes()
.with_context(|| "get nodes")? {
.map_err(|e| format!("get nodes: {}", e))? {
let node = crate::store::Node::from_capnp_migrate(node_reader)?;
entries.push(node);
}
@ -68,29 +66,118 @@ pub fn cmd_tail(n: usize, full: bool, provenance: Option<&str>, dedup: bool) ->
Ok(())
}
pub async fn cmd_journal_tail(n: usize, full: bool, level: u8) -> Result<()> {
let entries = memory::journal_tail(None, Some(n as u64), Some(level as u64), None).await?;
for entry in entries {
if full {
println!("--- {} ---", entry.key);
println!("{}\n", entry.content);
pub fn find_current_transcript() -> Option<String> {
let projects = crate::config::get().projects_dir.clone();
if !projects.exists() { return None; }
let mut newest: Option<(std::time::SystemTime, std::path::PathBuf)> = None;
if let Ok(dirs) = std::fs::read_dir(&projects) {
for dir_entry in dirs.filter_map(|e| e.ok()) {
if !dir_entry.path().is_dir() { continue; }
if let Ok(files) = std::fs::read_dir(dir_entry.path()) {
for f in files.filter_map(|e| e.ok()) {
let p = f.path();
if p.extension().map(|x| x == "jsonl").unwrap_or(false)
&& let Ok(meta) = p.metadata()
&& let Ok(mtime) = meta.modified()
&& newest.as_ref().is_none_or(|(t, _)| mtime > *t) {
newest = Some((mtime, p));
}
}
}
}
}
newest.map(|(_, p)| p.to_string_lossy().to_string())
}
fn journal_tail_query(store: &crate::store::Store, query: &str, n: usize, full: bool) -> Result<(), String> {
let graph = store.build_graph();
let stages = crate::query_parser::parse_stages(query)?;
let results = crate::search::run_query(&stages, vec![], &graph, store, false, n);
// Query sorts desc and limits, so reverse to show oldest-to-newest
for (key, _score) in results.into_iter().rev() {
let Some(node) = store.nodes.get(&key) else { continue };
let ts = if node.created_at > 0 {
crate::store::format_datetime(node.created_at)
} else if node.timestamp > 0 {
crate::store::format_datetime(node.timestamp)
} else {
let first_line = entry.content.lines().next().unwrap_or("(empty)");
println!("{}: {}", entry.key, first_line);
node.key.clone()
};
let title = extract_title(&node.content);
if full {
println!("--- [{}] {} ---\n{}\n", ts, title, node.content);
} else {
println!("[{}] {}", ts, title);
}
}
Ok(())
}
pub async fn cmd_journal_write(name: &str, text: &[String]) -> Result<()> {
pub fn cmd_journal_tail(n: usize, full: bool, level: u8) -> Result<(), String> {
let store = crate::store::Store::load()?;
let query = format!("all | type:{} | sort:timestamp | limit:{}",
match level { 0 => "episodic", 1 => "daily", 2 => "weekly", _ => "monthly" },
n
);
journal_tail_query(&store, &query, n, full)
}
pub fn cmd_journal_write(name: &str, text: &[String]) -> Result<(), String> {
if text.is_empty() {
bail!("journal write requires text");
return Err("journal write requires text".into());
}
super::check_dry_run();
let body = text.join(" ");
let text = text.join(" ");
let timestamp = crate::store::format_datetime(crate::store::now_epoch());
let content = format!("## {} — {}\n\n{}", timestamp, name, text);
let key: String = name.split_whitespace()
.map(|w| w.to_lowercase()
.chars().filter(|c| c.is_alphanumeric() || *c == '-')
.collect::<String>())
.filter(|s| !s.is_empty())
.collect::<Vec<_>>()
.join("-");
let source_ref = find_current_transcript();
let mut store = crate::store::Store::load()?;
let mut node = crate::store::new_node(&key, &content);
node.node_type = crate::store::NodeType::EpisodicSession;
node.provenance = "journal".to_string();
if let Some(src) = source_ref {
node.source_ref = src;
}
store.upsert_node(node)?;
store.save()?;
let word_count = text.split_whitespace().count();
println!("Appended entry at {} ({} words)", timestamp, word_count);
let result = memory::journal_new(None, name, name, &body, Some(0)).await?;
println!("{}", result);
Ok(())
}
fn extract_title(content: &str) -> String {
let date_re = regex::Regex::new(r"(\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2})").unwrap();
for line in content.lines() {
let stripped = line.trim();
if stripped.is_empty() { continue; }
if date_re.is_match(stripped) && stripped.len() < 25 { continue; }
if let Some(h) = stripped.strip_prefix("## ") {
return h.to_string();
} else if let Some(h) = stripped.strip_prefix("# ") {
return h.to_string();
} else {
return crate::util::truncate(stripped, 67, "...");
}
}
String::from("(untitled)")
}

319
src/cli/misc.rs Normal file
View file

@ -0,0 +1,319 @@
// cli/misc.rs — misc subcommand handlers
pub fn cmd_search(terms: &[String], pipeline_args: &[String], expand: bool, full: bool, debug: bool, fuzzy: bool, content: bool) -> Result<(), String> {
use std::collections::BTreeMap;
use crate::search::{Stage, Algorithm, AlgoStage};
// When running inside an agent session, exclude already-surfaced nodes
let seen = crate::session::HookSession::from_env()
.map(|s| s.seen())
.unwrap_or_default();
// Build pipeline: if args provided, parse them; otherwise default to spread
let stages: Vec<Stage> = if pipeline_args.is_empty() {
vec![Stage::Algorithm(AlgoStage { algo: Algorithm::Spread, params: std::collections::HashMap::new() })]
} else {
// Join args with | and parse as unified query
let pipeline_str = format!("all | {}", pipeline_args.join(" | "));
crate::query_parser::parse_stages(&pipeline_str)?
};
// Check if pipeline needs full Store (has filters/transforms/generators)
let needs_store = stages.iter().any(|s| !matches!(s, Stage::Algorithm(_)));
// Check if pipeline starts with a generator (doesn't need seed terms)
let has_generator = stages.first().map(|s| matches!(s, Stage::Generator(_))).unwrap_or(false);
if terms.is_empty() && !has_generator {
return Err("search requires terms or a generator stage (e.g. 'all')".into());
}
let query: String = terms.join(" ");
if debug {
let names: Vec<String> = stages.iter().map(|s| format!("{}", s)).collect();
println!("[search] pipeline: {}", names.join(" → "));
}
let max_results = if expand { 15 } else { 5 };
if needs_store {
// Full Store path — needed for filter/transform/generator stages
let store = crate::store::Store::load()?;
let graph = store.build_graph();
let seeds = if has_generator {
vec![] // generator will produce its own result set
} else {
let terms_map: BTreeMap<String, f64> = query.split_whitespace()
.map(|t| (t.to_lowercase(), 1.0))
.collect();
let (seeds, _) = crate::search::match_seeds_opts(&terms_map, &store, fuzzy, content);
seeds
};
let raw = crate::search::run_query(&stages, seeds, &graph, &store, debug, max_results);
let raw: Vec<_> = raw.into_iter()
.filter(|(key, _)| !seen.contains(key))
.collect();
if raw.is_empty() {
eprintln!("No results");
return Ok(());
}
for (i, (key, score)) in raw.iter().enumerate().take(max_results) {
let weight = store.nodes.get(key).map(|n| n.weight).unwrap_or(0.0);
println!("{:2}. [{:.2}/{:.2}] {}", i + 1, score, weight, key);
if full
&& let Some(node) = store.nodes.get(key) {
println!();
for line in node.content.lines() {
println!(" {}", line);
}
println!();
}
}
} else {
// Fast MmapView path — algorithm-only pipeline
use crate::store::StoreView;
let view = crate::store::AnyView::load()?;
let graph = crate::graph::build_graph_fast(&view);
let terms_map: BTreeMap<String, f64> = query.split_whitespace()
.map(|t| (t.to_lowercase(), 1.0))
.collect();
let (seeds, direct_hits) = crate::search::match_seeds_opts(&terms_map, &view, fuzzy, content);
if seeds.is_empty() {
eprintln!("No results for '{}'", query);
return Ok(());
}
if debug {
println!("[search] {} seeds from query '{}'", seeds.len(), query);
}
// Extract AlgoStages from the unified stages
let algo_stages: Vec<&crate::search::AlgoStage> = stages.iter()
.filter_map(|s| match s {
crate::search::Stage::Algorithm(a) => Some(a),
_ => None,
})
.collect();
let algo_owned: Vec<crate::search::AlgoStage> = algo_stages.into_iter().cloned().collect();
let raw = crate::search::run_pipeline(&algo_owned, seeds, &graph, &view, debug, max_results);
let results: Vec<crate::search::SearchResult> = raw.into_iter()
.filter(|(key, _)| !seen.contains(key))
.map(|(key, activation)| {
let is_direct = direct_hits.contains(&key);
crate::search::SearchResult { key, activation, is_direct, snippet: None }
})
.collect();
if results.is_empty() {
eprintln!("No results for '{}'", query);
return Ok(());
}
// Log retrieval
crate::store::Store::log_retrieval_static(&query,
&results.iter().map(|r| r.key.clone()).collect::<Vec<_>>());
let bump_keys: Vec<&str> = results.iter().take(max_results).map(|r| r.key.as_str()).collect();
let _ = crate::lookups::bump_many(&bump_keys);
for (i, r) in results.iter().enumerate().take(max_results) {
let marker = if r.is_direct { "→" } else { " " };
let weight = view.node_weight(&r.key);
println!("{}{:2}. [{:.2}/{:.2}] {}", marker, i + 1, r.activation, weight, r.key);
if full
&& let Some(content) = view.node_content(&r.key) {
println!();
for line in content.lines() {
println!(" {}", line);
}
println!();
}
}
}
Ok(())
}
pub fn cmd_status() -> Result<(), String> {
// TUI moved to consciousness binary (F4 unconscious screen)
let store = crate::store::Store::load()?;
let g = store.build_graph();
let mut type_counts = std::collections::HashMap::new();
for node in store.nodes.values() {
*type_counts.entry(format!("{:?}", node.node_type)).or_insert(0usize) += 1;
}
let mut types: Vec<_> = type_counts.iter().collect();
types.sort_by_key(|(_, c)| std::cmp::Reverse(**c));
println!("Nodes: {} Relations: {}", store.nodes.len(), store.relations.len());
print!("Types:");
for (t, c) in &types {
let label = match t.as_str() {
"Semantic" => "semantic",
"EpisodicSession" | "EpisodicDaily" | "EpisodicWeekly" | "EpisodicMonthly"
=> "episodic",
_ => t,
};
print!(" {}={}", label, c);
}
println!();
println!("Graph edges: {} Communities: {}",
g.edge_count(), g.community_count());
Ok(())
}
pub fn cmd_log() -> Result<(), String> {
let store = crate::store::Store::load()?;
for event in store.retrieval_log.iter().rev().take(20) {
println!("[{}] q=\"{}\" → {} results",
event.timestamp, event.query, event.results.len());
for r in &event.results {
println!(" {}", r);
}
}
Ok(())
}
pub fn cmd_params() -> Result<(), String> {
let store = crate::store::Store::load()?;
println!("decay_factor: {}", store.params.decay_factor);
println!("use_boost: {}", store.params.use_boost);
println!("prune_threshold: {}", store.params.prune_threshold);
println!("edge_decay: {}", store.params.edge_decay);
println!("max_hops: {}", store.params.max_hops);
println!("min_activation: {}", store.params.min_activation);
Ok(())
}
pub fn cmd_query(expr: &[String]) -> Result<(), String> {
if expr.is_empty() {
return Err("query requires an expression (try: poc-memory query --help)".into());
}
let query_str = expr.join(" ");
let store = crate::store::Store::load()?;
let graph = store.build_graph();
crate::query_parser::run_query(&store, &graph, &query_str)
}
pub fn get_group_content(group: &crate::config::ContextGroup, store: &crate::store::Store, cfg: &crate::config::Config) -> Vec<(String, String)> {
match group.source {
crate::config::ContextSource::Journal => {
let mut entries = Vec::new();
let now = crate::store::now_epoch();
let window: i64 = cfg.journal_days as i64 * 24 * 3600;
let cutoff = now - window;
let key_date_re = regex::Regex::new(r"j-(\d{4}-\d{2}-\d{2})").unwrap();
let journal_ts = |n: &crate::store::Node| -> i64 {
if n.created_at > 0 { return n.created_at; }
if let Some(caps) = key_date_re.captures(&n.key) {
use chrono::{NaiveDate, TimeZone, Local};
if let Ok(d) = NaiveDate::parse_from_str(&caps[1], "%Y-%m-%d")
&& let Some(dt) = Local.from_local_datetime(&d.and_hms_opt(0, 0, 0).unwrap()).earliest() {
return dt.timestamp();
}
}
n.timestamp
};
let mut journal_nodes: Vec<_> = store.nodes.values()
.filter(|n| n.node_type == crate::store::NodeType::EpisodicSession && journal_ts(n) >= cutoff)
.collect();
journal_nodes.sort_by_key(|n| journal_ts(n));
let max = cfg.journal_max;
let skip = journal_nodes.len().saturating_sub(max);
for node in journal_nodes.iter().skip(skip) {
entries.push((node.key.clone(), node.content.clone()));
}
entries
}
crate::config::ContextSource::File => {
group.keys.iter().filter_map(|key| {
let content = std::fs::read_to_string(cfg.identity_dir.join(key)).ok()?;
if content.trim().is_empty() { return None; }
Some((key.clone(), content.trim().to_string()))
}).collect()
}
crate::config::ContextSource::Store => {
group.keys.iter().filter_map(|key| {
let content = store.render_file(key)?;
if content.trim().is_empty() { return None; }
Some((key.clone(), content.trim().to_string()))
}).collect()
}
}
}
/// MCP tool schema with CLI routing info.
///
/// Each tool definition includes:
/// - name, description, inputSchema (standard MCP)
/// - cli: the CLI args prefix to invoke this tool
/// - stdin_param: which parameter (if any) should be sent via stdin
///
/// Tools with cli=null are agent-internal (not exposed via MCP CLI bridge).
// mcp-schema moved to consciousness-mcp binary (src/claude/mcp-server.rs)
pub fn cmd_load_context(stats: bool) -> Result<(), String> {
let cfg = crate::config::get();
let store = crate::store::Store::load()?;
if stats {
let mut total_words = 0;
let mut total_entries = 0;
println!("{:<25} {:>6} {:>8}", "GROUP", "ITEMS", "WORDS");
println!("{}", "-".repeat(42));
for group in &cfg.context_groups {
let entries = get_group_content(group, &store, &cfg);
let words: usize = entries.iter()
.map(|(_, c)| c.split_whitespace().count())
.sum();
let count = entries.len();
println!("{:<25} {:>6} {:>8}", group.label, count, words);
total_words += words;
total_entries += count;
}
println!("{}", "-".repeat(42));
println!("{:<25} {:>6} {:>8}", "TOTAL", total_entries, total_words);
return Ok(());
}
println!("=== MEMORY SYSTEM ({}) ===", cfg.assistant_name);
println!();
for group in &cfg.context_groups {
let entries = get_group_content(group, &store, &cfg);
if !entries.is_empty() && group.source == crate::config::ContextSource::Journal {
println!("--- recent journal entries ({}/{}) ---",
entries.len(), cfg.journal_max);
}
for (key, content) in entries {
if group.source == crate::config::ContextSource::Journal {
println!("## {}", key);
} else {
println!("--- {} ({}) ---", key, group.label);
}
println!("{}\n", content);
}
}
println!("=== END MEMORY LOAD ===");
Ok(())
}

1
src/cli/mod.rs
View file

@ -8,6 +8,7 @@ pub mod node;
pub mod agent;
pub mod admin;
pub mod journal;
pub mod misc;
/// Exit silently if POC_MEMORY_DRY_RUN=1.
pub fn check_dry_run() {

499
src/cli/node.rs
View file

@ -1,45 +1,203 @@
// cli/node.rs — node subcommand handlers
//
// render, write, node-delete, node-rename, history, list-keys,
// list-edges, dump-json, lookup-bump, lookups.
// render, write, used, wrong, not-relevant, not-useful, gap,
// node-delete, node-rename, history, list-keys, list-edges,
// dump-json, lookup-bump, lookups.
use anyhow::{bail, Context, Result};
use crate::hippocampus as memory;
use crate::store;
pub async fn cmd_weight_set(key: &str, weight: f32) -> Result<()> {
super::check_dry_run();
let result = memory::memory_weight_set(None, key, weight).await?;
println!("{}", result);
Ok(())
}
pub async fn cmd_node_delete(key: &[String]) -> Result<()> {
pub fn cmd_used(key: &[String]) -> Result<(), String> {
if key.is_empty() {
bail!("node-delete requires a key");
return Err("used requires a key".into());
}
super::check_dry_run();
let key = key.join(" ");
let result = memory::memory_delete(None, &key).await?;
println!("{}", result);
let mut store = store::Store::load()?;
let resolved = store.resolve_key(&key)?;
store.mark_used(&resolved);
// Also strengthen edges to this node — conscious-tier delta.
const DELTA: f32 = 0.01;
let mut strengthened = 0;
for rel in &mut store.relations {
if rel.deleted { continue; }
if rel.source_key == resolved || rel.target_key == resolved {
let old = rel.strength;
rel.strength = (rel.strength + DELTA).clamp(0.05, 0.95);
if (rel.strength - old).abs() > 0.001 {
rel.version += 1;
strengthened += 1;
}
}
}
store.save()?;
println!("Marked '{}' as used (strengthened {} edges)", resolved, strengthened);
Ok(())
}
pub async fn cmd_node_rename(old_key: &str, new_key: &str) -> Result<()> {
pub fn cmd_wrong(key: &str, context: &[String]) -> Result<(), String> {
let ctx = if context.is_empty() { None } else { Some(context.join(" ")) };
super::check_dry_run();
let result = memory::memory_rename(None, old_key, new_key).await?;
println!("{}", result);
let mut store = store::Store::load()?;
let resolved = store.resolve_key(key)?;
store.mark_wrong(&resolved, ctx.as_deref());
store.save()?;
println!("Marked '{}' as wrong", resolved);
Ok(())
}
pub async fn cmd_render(key: &[String]) -> Result<()> {
pub fn cmd_not_relevant(key: &str) -> Result<(), String> {
let mut store = store::Store::load()?;
let resolved = store.resolve_key(key)?;
// Weaken all edges to this node — it was routed to incorrectly.
// Conscious-tier delta: 0.01 per edge.
const DELTA: f32 = -0.01;
let mut adjusted = 0;
for rel in &mut store.relations {
if rel.deleted { continue; }
if rel.source_key == resolved || rel.target_key == resolved {
let old = rel.strength;
rel.strength = (rel.strength + DELTA).clamp(0.05, 0.95);
if (rel.strength - old).abs() > 0.001 {
rel.version += 1;
adjusted += 1;
}
}
}
store.save()?;
println!("Not relevant: '{}' — weakened {} edges by {}", resolved, adjusted, DELTA.abs());
Ok(())
}
pub fn cmd_not_useful(key: &str) -> Result<(), String> {
// no args to validate
super::check_dry_run();
let mut store = store::Store::load()?;
let resolved = store.resolve_key(key)?;
// Same as wrong but with clearer semantics: node content is bad, edges are fine.
store.mark_wrong(&resolved, Some("not-useful"));
store.save()?;
println!("Not useful: '{}' — node weight reduced", resolved);
Ok(())
}
pub fn cmd_weight_set(key: &str, weight: f32) -> Result<(), String> {
super::check_dry_run();
let mut store = store::Store::load()?;
let resolved = store.resolve_key(key)?;
let (old, new) = store.set_weight(&resolved, weight)?;
println!("Weight: {} {:.2} → {:.2}", resolved, old, new);
store.save()?;
Ok(())
}
pub fn cmd_gap(description: &[String]) -> Result<(), String> {
if description.is_empty() {
return Err("gap requires a description".into());
}
super::check_dry_run();
let desc = description.join(" ");
let mut store = store::Store::load()?;
store.record_gap(&desc);
store.save()?;
println!("Recorded gap: {}", desc);
Ok(())
}
pub fn cmd_list_keys(pattern: Option<&str>) -> Result<(), String> {
let store = store::Store::load()?;
let g = store.build_graph();
if let Some(pat) = pattern {
let pat_lower = pat.to_lowercase();
let (prefix, suffix, middle) = if pat_lower.starts_with('*') && pat_lower.ends_with('*') {
(None, None, Some(pat_lower.trim_matches('*').to_string()))
} else if pat_lower.starts_with('*') {
(None, Some(pat_lower.trim_start_matches('*').to_string()), None)
} else if pat_lower.ends_with('*') {
(Some(pat_lower.trim_end_matches('*').to_string()), None, None)
} else {
(None, None, Some(pat_lower.clone()))
};
let mut keys: Vec<_> = store.nodes.keys()
.filter(|k| {
let kl = k.to_lowercase();
if let Some(ref m) = middle { kl.contains(m.as_str()) }
else if let Some(ref p) = prefix { kl.starts_with(p.as_str()) }
else if let Some(ref s) = suffix { kl.ends_with(s.as_str()) }
else { true }
})
.cloned()
.collect();
keys.sort();
for k in keys { println!("{}", k); }
Ok(())
} else {
crate::query_parser::run_query(&store, &g, "* | sort key asc")
}
}
pub fn cmd_list_edges() -> Result<(), String> {
let store = store::Store::load()?;
for rel in &store.relations {
println!("{}\t{}\t{:.2}\t{:?}",
rel.source_key, rel.target_key, rel.strength, rel.rel_type);
}
Ok(())
}
pub fn cmd_dump_json() -> Result<(), String> {
let store = store::Store::load()?;
let json = serde_json::to_string_pretty(&store)
.map_err(|e| format!("serialize: {}", e))?;
println!("{}", json);
Ok(())
}
pub fn cmd_node_delete(key: &[String]) -> Result<(), String> {
if key.is_empty() {
bail!("render requires a key");
return Err("node-delete requires a key".into());
}
super::check_dry_run();
let key = key.join(" ");
let mut store = store::Store::load()?;
let resolved = store.resolve_key(&key)?;
store.delete_node(&resolved)?;
store.save()?;
println!("Deleted '{}'", resolved);
Ok(())
}
pub fn cmd_node_rename(old_key: &str, new_key: &str) -> Result<(), String> {
// args are positional, always valid if present
super::check_dry_run();
let mut store = store::Store::load()?;
let old_resolved = store.resolve_key(old_key)?;
store.rename_node(&old_resolved, new_key)?;
store.save()?;
println!("Renamed '{}' → '{}'", old_resolved, new_key);
Ok(())
}
/// Render a node to a string: content + deduped footer links.
/// Used by both the CLI command and agent placeholders.
pub fn render_node(store: &store::Store, key: &str) -> Option<String> {
crate::hippocampus::memory::MemoryNode::from_store(store, key)
.map(|node| node.render())
}
pub fn cmd_render(key: &[String]) -> Result<(), String> {
if key.is_empty() {
return Err("render requires a key".into());
}
let key = key.join(" ");
let store = store::Store::load()?;
let bare = store::strip_md_suffix(&key);
let rendered = memory::memory_render(None, &bare, None).await?;
let rendered = render_node(&store, &bare)
.ok_or_else(|| format!("Node not found: {}", bare))?;
print!("{}", rendered);
// Mark as seen if we're inside a Claude session (not an agent subprocess —
@ -63,62 +221,182 @@ pub async fn cmd_render(key: &[String]) -> Result<()> {
Ok(())
}
pub async fn cmd_history(key: &[String], full: bool) -> Result<()> {
if key.is_empty() {
bail!("history requires a key");
/// Check content for common inline reference problems:
/// - `poc-memory render key` embedded in content (render artifact, should be just `key`)
/// - `→ something` where something doesn't parse as a valid key
/// - `key` referencing a node that doesn't exist
fn validate_inline_refs(content: &str, store: &store::Store) -> Vec<String> {
let mut warnings = Vec::new();
for line in content.lines() {
// Check for render commands embedded in content
if line.contains("poc-memory render ") && !line.starts_with(" ") {
// Skip lines that look like CLI documentation/examples
if !line.contains("CLI") && !line.contains("equivalent") && !line.contains("tool") {
warnings.push(format!(
"render command in content (should be just `key`): {}",
line.chars().take(80).collect::<String>(),
));
}
let key = key.join(" ");
let result = memory::memory_history(None, &key, Some(full)).await?;
print!("{}", result);
}
// Check → references
if let Some(rest) = line.trim().strip_prefix("→ ") {
// Extract the key (may be backtick-quoted)
let key = rest.trim().trim_matches('`').trim();
if !key.is_empty() && !store.nodes.contains_key(key) {
// Might be a poc-memory render artifact
if let Some(k) = key.strip_prefix("poc-memory render ") {
warnings.push(format!(
"render artifact in → reference (use `{}` not `poc-memory render {}`)", k, k,
));
} else if key.contains(' ') {
warnings.push(format!(
"→ reference doesn't look like a key: → {}", key,
));
}
// Don't warn about missing keys — the target might be created later
}
}
}
warnings
}
pub fn cmd_history(key: &[String], full: bool) -> Result<(), String> {
if key.is_empty() {
return Err("history requires a key".into());
}
let raw_key = key.join(" ");
let store = store::Store::load()?;
let key = store.resolve_key(&raw_key).unwrap_or(raw_key);
drop(store);
let path = store::nodes_path();
if !path.exists() {
return Err("No node log found".into());
}
use std::io::BufReader;
let file = std::fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
let mut versions: Vec<store::Node> = Vec::new();
while let Ok(msg) = capnp::serialize::read_message(&mut reader, capnp::message::ReaderOptions::new()) {
let log = msg.get_root::<crate::memory_capnp::node_log::Reader>()
.map_err(|e| format!("read log: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes: {}", e))? {
let node = store::Node::from_capnp_migrate(node_reader)?;
if node.key == key {
versions.push(node);
}
}
}
if versions.is_empty() {
return Err(format!("No history found for '{}'", key));
}
eprintln!("{} versions of '{}':\n", versions.len(), key);
for node in &versions {
let ts = if node.timestamp > 0 && node.timestamp < 4_000_000_000 {
store::format_datetime(node.timestamp)
} else {
format!("(raw:{})", node.timestamp)
};
let deleted_marker = if node.deleted { " DELETED" } else { "" };
let content_len = node.content.len();
if full {
eprintln!("=== v{} {} {}{} w={:.3} {}b ===",
node.version, ts, node.provenance, deleted_marker, node.weight, content_len);
eprintln!("{}", node.content);
} else {
let preview = crate::util::first_n_chars(&node.content, 120);
let preview = preview.replace('\n', "\\n");
eprintln!(" v{:<3} {} {:24} w={:.3} {}b{}",
node.version, ts, node.provenance, node.weight, content_len, deleted_marker);
eprintln!(" {}", preview);
}
}
if !full
&& let Some(latest) = versions.last() {
eprintln!("\n--- Latest content (v{}, {}) ---",
latest.version, latest.provenance);
print!("{}", latest.content);
}
Ok(())
}
pub async fn cmd_write(key: &[String]) -> Result<()> {
pub fn cmd_write(key: &[String]) -> Result<(), String> {
if key.is_empty() {
bail!("write requires a key (reads content from stdin)");
return Err("write requires a key (reads content from stdin)".into());
}
let key = key.join(" ");
let raw_key = key.join(" ");
let mut content = String::new();
std::io::Read::read_to_string(&mut std::io::stdin(), &mut content)
.context("read stdin")?;
.map_err(|e| format!("read stdin: {}", e))?;
if content.trim().is_empty() {
bail!("No content on stdin");
return Err("No content on stdin".into());
}
super::check_dry_run();
let result = memory::memory_write(None, &key, &content).await?;
println!("{}", result);
let mut store = store::Store::load()?;
let key = store.resolve_key(&raw_key).unwrap_or(raw_key);
// Validate inline references: warn about render commands embedded
// in content (should be just `key`) and broken references.
let warnings = validate_inline_refs(&content, &store);
for w in &warnings {
eprintln!("warning: {}", w);
}
let result = store.upsert(&key, &content)?;
match result {
"unchanged" => println!("No change: '{}'", key),
"updated" => println!("Updated '{}' (v{})", key, store.nodes[&key].version),
_ => println!("Created '{}'", key),
}
if result != "unchanged" {
store.save()?;
}
Ok(())
}
pub async fn cmd_edit(key: &[String]) -> Result<()> {
pub fn cmd_edit(key: &[String]) -> Result<(), String> {
if key.is_empty() {
bail!("edit requires a key");
return Err("edit requires a key".into());
}
let key = key.join(" ");
let raw_key = key.join(" ");
let store = store::Store::load()?;
let key = store.resolve_key(&raw_key).unwrap_or(raw_key.clone());
// Get raw content
let content = memory::memory_render(None, &key, Some(true)).await
let content = store.nodes.get(&key)
.map(|n| n.content.clone())
.unwrap_or_default();
let tmp = std::env::temp_dir().join(format!("poc-memory-edit-{}.md", key.replace('/', "_")));
std::fs::write(&tmp, &content)
.with_context(|| format!("write temp file {}", tmp.display()))?;
.map_err(|e| format!("write temp file: {}", e))?;
let editor = std::env::var("EDITOR").unwrap_or_else(|_| "vi".into());
let status = std::process::Command::new(&editor)
.arg(&tmp)
.status()
.with_context(|| format!("spawn {}", editor))?;
.map_err(|e| format!("spawn {}: {}", editor, e))?;
if !status.success() {
let _ = std::fs::remove_file(&tmp);
bail!("{} exited with {}", editor, status);
return Err(format!("{} exited with {}", editor, status));
}
let new_content = std::fs::read_to_string(&tmp)
.with_context(|| format!("read temp file {}", tmp.display()))?;
.map_err(|e| format!("read temp file: {}", e))?;
let _ = std::fs::remove_file(&tmp);
if new_content == content {
@ -127,124 +405,51 @@ pub async fn cmd_edit(key: &[String]) -> Result<()> {
}
if new_content.trim().is_empty() {
bail!("Content is empty, aborting");
return Err("Content is empty, aborting".into());
}
super::check_dry_run();
let result = memory::memory_write(None, &key, &new_content).await?;
println!("{}", result);
drop(store);
let mut store = store::Store::load()?;
let result = store.upsert(&key, &new_content)?;
match result {
"unchanged" => println!("No change: '{}'", key),
"updated" => println!("Updated '{}' (v{})", key, store.nodes[&key].version),
_ => println!("Created '{}'", key),
}
if result != "unchanged" {
store.save()?;
}
Ok(())
}
pub async fn cmd_search(keys: &[String]) -> Result<()> {
pub fn cmd_lookup_bump(keys: &[String]) -> Result<(), String> {
if keys.is_empty() {
bail!("search requires seed keys");
return Err("lookup-bump requires at least one key".into());
}
let result = memory::memory_search(None, keys.to_vec(), None, None, None, None).await?;
print!("{}", result);
Ok(())
let keys: Vec<&str> = keys.iter().map(|s| s.as_str()).collect();
crate::lookups::bump_many(&keys)
}
pub async fn cmd_query(expr: &[String]) -> Result<()> {
if expr.is_empty() {
bail!("query requires an expression (try: poc-memory query --help)");
}
pub fn cmd_lookups(date: Option<&str>) -> Result<(), String> {
let date = date.map(|d| d.to_string())
.unwrap_or_else(|| chrono::Local::now().format("%Y-%m-%d").to_string());
let query_str = expr.join(" ");
let result = memory::memory_query(None, &query_str, None).await?;
print!("{}", result);
Ok(())
}
let store = store::Store::load()?;
let keys: Vec<String> = store.nodes.values().map(|n| n.key.clone()).collect();
let resolved = crate::lookups::dump_resolved(&date, &keys)?;
/// Get group content (handles daemon or local fallback)
pub async fn get_group_content(group: &crate::config::ContextGroup, cfg: &crate::config::Config) -> Vec<(String, String)> {
match group.source {
crate::config::ContextSource::Journal => {
// Query for recent journal entries
let window: i64 = cfg.journal_days as i64 * 24 * 3600;
let query = format!("all | type:episodic | age:<{} | sort:timestamp | limit:{}",
window, cfg.journal_max);
let keys_str = match memory::memory_query(None, &query, None).await {
Ok(s) => s,
Err(_) => return vec![],
};
// Parse keys (one per line) and render each
let mut results = Vec::new();
for key in keys_str.lines().filter(|k| !k.is_empty() && *k != "no results") {
if let Ok(content) = memory::memory_render(None, key, Some(true)).await {
if !content.trim().is_empty() {
results.push((key.to_string(), content));
}
}
}
results
}
crate::config::ContextSource::File => {
group.keys.iter().filter_map(|key| {
let content = std::fs::read_to_string(cfg.identity_dir.join(key)).ok()?;
if content.trim().is_empty() { return None; }
Some((key.clone(), content.trim().to_string()))
}).collect()
}
crate::config::ContextSource::Store => {
let mut results = Vec::new();
for key in &group.keys {
if let Ok(content) = memory::memory_render(None, key, Some(true)).await {
if !content.trim().is_empty() {
results.push((key.clone(), content.trim().to_string()));
}
}
}
results
}
}
}
pub async fn cmd_load_context(stats: bool) -> Result<()> {
let cfg = crate::config::get();
if stats {
let mut total_words = 0;
let mut total_entries = 0;
println!("{:<25} {:>6} {:>8}", "GROUP", "ITEMS", "WORDS");
println!("{}", "-".repeat(42));
for group in &cfg.context_groups {
let entries = get_group_content(group, &cfg).await;
let words: usize = entries.iter()
.map(|(_, c)| c.split_whitespace().count())
.sum();
let count = entries.len();
println!("{:<25} {:>6} {:>8}", group.label, count, words);
total_words += words;
total_entries += count;
}
println!("{}", "-".repeat(42));
println!("{:<25} {:>6} {:>8}", "TOTAL", total_entries, total_words);
if resolved.is_empty() {
println!("No lookups for {}", date);
return Ok(());
}
println!("=== MEMORY SYSTEM ({}) ===", cfg.assistant_name);
for group in &cfg.context_groups {
let entries = get_group_content(group, &cfg).await;
if !entries.is_empty() && group.source == crate::config::ContextSource::Journal {
println!("--- recent journal entries ({}/{}) ---",
entries.len(), cfg.journal_max);
println!("Lookups for {}:", date);
for (key, count) in &resolved {
println!(" {:4} {}", count, key);
}
for (key, content) in entries {
if group.source == crate::config::ContextSource::Journal {
println!("## {}", key);
} else {
println!("--- {} ({}) ---", key, group.label);
}
println!("{}\n", content);
}
}
println!("=== END MEMORY LOAD ===");
println!("\n{} distinct keys, {} total lookups",
resolved.len(),
resolved.iter().map(|(_, c)| *c as u64).sum::<u64>());
Ok(())
}

30
src/config.rs
View file

@ -58,10 +58,6 @@ fn default_stream_timeout() -> u64 { 60 }
fn default_scoring_chunk_tokens() -> usize { 50_000 }
fn default_scoring_interval_secs() -> u64 { 3600 } // 1 hour
fn default_scoring_response_window() -> usize { 100 }
fn default_node_weight() -> f64 { 0.7 }
fn default_edge_decay() -> f64 { 0.3 }
fn default_max_hops() -> u32 { 3 }
fn default_min_activation() -> f64 { 0.05 }
fn default_identity_dir() -> PathBuf {
dirs::home_dir().unwrap_or_default().join(".consciousness/identity")
}
@ -124,16 +120,6 @@ pub struct Config {
/// Hook events that trigger the surface agent.
#[serde(default)]
pub surface_hooks: Vec<String>,
// Spreading activation parameters
#[serde(default = "default_node_weight")]
pub default_node_weight: f64,
#[serde(default = "default_edge_decay")]
pub edge_decay: f64,
#[serde(default = "default_max_hops")]
pub max_hops: u32,
#[serde(default = "default_min_activation")]
pub min_activation: f64,
}
impl Default for Config {
@ -184,10 +170,6 @@ impl Default for Config {
surface_hooks: vec![],
mcp_servers: vec![],
lsp_servers: vec![],
default_node_weight: default_node_weight(),
edge_decay: default_edge_decay(),
max_hops: default_max_hops(),
min_activation: default_min_activation(),
}
}
}
@ -524,7 +506,7 @@ pub struct ResolvedModel {
impl AppConfig {
/// Resolve the active backend and assemble prompts into a SessionConfig.
pub async fn resolve(&self, cli: &crate::user::CliArgs) -> Result<SessionConfig> {
pub fn resolve(&self, cli: &crate::user::CliArgs) -> Result<SessionConfig> {
let cwd = std::env::current_dir().context("Failed to get current directory")?;
let (api_base, api_key, model, prompt_file);
@ -554,7 +536,7 @@ impl AppConfig {
let context_groups = get().context_groups.clone();
let (context_parts, config_file_count, memory_file_count) =
crate::mind::identity::assemble_context_message(&cwd, &prompt_file, self.memory_project.as_deref(), &context_groups).await?;
crate::mind::identity::assemble_context_message(&cwd, &prompt_file, self.memory_project.as_deref(), &context_groups)?;
let session_dir = dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
@ -686,17 +668,17 @@ pub fn load_app(cli: &crate::user::CliArgs) -> Result<(AppConfig, Figment)> {
}
/// Load the full config: figment → AppConfig → resolve backend → assemble prompts.
pub async fn load_session(cli: &crate::user::CliArgs) -> Result<(SessionConfig, Figment)> {
pub fn load_session(cli: &crate::user::CliArgs) -> Result<(SessionConfig, Figment)> {
let (app, figment) = load_app(cli)?;
let config = app.resolve(cli).await?;
let config = app.resolve(cli)?;
Ok((config, figment))
}
/// Re-assemble context for a specific model's prompt file.
pub async fn reload_for_model(app: &AppConfig, prompt_file: &str) -> Result<Vec<(String, String)>> {
pub fn reload_for_model(app: &AppConfig, prompt_file: &str) -> Result<Vec<(String, String)>> {
let cwd = std::env::current_dir().context("Failed to get current directory")?;
let context_groups = get().context_groups.clone();
let (context_parts, _, _) = crate::mind::identity::assemble_context_message(&cwd, prompt_file, app.memory_project.as_deref(), &context_groups).await?;
let (context_parts, _, _) = crate::mind::identity::assemble_context_message(&cwd, prompt_file, app.memory_project.as_deref(), &context_groups)?;
Ok(context_parts)
}

25
src/hippocampus/graph.rs
View file

@ -519,9 +519,11 @@ pub fn build_graph_fast(store: &impl StoreView) -> Graph {
fn build_adjacency(store: &impl StoreView) -> (HashMap<String, Vec<Edge>>, HashSet<String>) {
let mut adj: HashMap<String, Vec<Edge>> = HashMap::new();
let mut keys: HashSet<String> = HashSet::new();
// Get keys directly from index — no need to deserialize node content
let keys: HashSet<String> = store.all_keys().into_iter().collect();
store.for_each_node(|key, _, _| {
keys.insert(key.to_owned());
});
store.for_each_relation(|source_key, target_key, strength, rel_type| {
if !keys.contains(source_key) || !keys.contains(target_key) {
@ -880,24 +882,22 @@ pub fn health_report(graph: &Graph, store: &Store) -> String {
.count();
// Orphan edges: relations referencing non-existent nodes
// With index-based lookup, we count edges where endpoints don't resolve
let mut orphan_edges = 0usize;
let mut missing_nodes: HashSet<String> = HashSet::new();
store.for_each_relation(|source, target, _, _| {
let s_missing = !store.contains_key(source).unwrap_or(false);
let t_missing = !store.contains_key(target).unwrap_or(false);
for rel in &store.relations {
if rel.deleted { continue; }
let s_missing = !store.nodes.contains_key(&rel.source_key);
let t_missing = !store.nodes.contains_key(&rel.target_key);
if s_missing || t_missing {
orphan_edges += 1;
if s_missing { missing_nodes.insert(source.to_string()); }
if t_missing { missing_nodes.insert(target.to_string()); }
if s_missing { missing_nodes.insert(rel.source_key.clone()); }
if t_missing { missing_nodes.insert(rel.target_key.clone()); }
}
}
});
// NodeType breakdown
let mut type_counts: HashMap<&str, usize> = HashMap::new();
let all_keys = store.all_keys().unwrap_or_default();
for key in &all_keys {
if let Ok(Some(node)) = store.get_node(key) {
for node in store.nodes.values() {
let label = match node.node_type {
crate::store::NodeType::EpisodicSession => "episodic",
crate::store::NodeType::EpisodicDaily => "daily",
@ -907,7 +907,6 @@ pub fn health_report(graph: &Graph, store: &Store) -> String {
};
*type_counts.entry(label).or_default() += 1;
}
}
// Load history for deltas
let history = load_metrics_history();

605
src/hippocampus/local.rs
View file

@ -1,605 +0,0 @@
use anyhow::Result;
use super::memory::MemoryNode;
use super::store::Store;
use crate::graph::Graph;
use crate::neuro::{consolidation_priority, ReplayItem};
// ── Memory operations ──────────────────────────────────────────
pub fn memory_render(store: &Store, _provenance: &str, key: &str, raw: Option<bool>) -> Result<String> {
let node = MemoryNode::from_store(store, key)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", key))?;
if raw.unwrap_or(false) {
Ok(node.content)
} else {
Ok(node.render())
}
}
pub fn memory_write(store: &Store, provenance: &str, key: &str, content: &str) -> Result<String> {
let result = store.upsert_provenance(key, content, provenance)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("{} '{}'", result, key))
}
pub fn memory_search(
store: &Store,
_provenance: &str,
keys: Vec<String>,
max_hops: Option<u32>,
edge_decay: Option<f64>,
min_activation: Option<f64>,
limit: Option<usize>,
) -> Result<String> {
if keys.is_empty() {
anyhow::bail!("memory_search requires at least one seed key");
}
let max_hops = max_hops.unwrap_or(3);
let edge_decay = edge_decay.unwrap_or(0.3);
let min_activation = min_activation.unwrap_or(0.01);
let limit = limit.unwrap_or(20);
let graph = crate::graph::build_graph_fast(store);
let seeds: Vec<(String, f64)> = keys.iter()
.filter_map(|k| {
let resolved = store.resolve_key(k).ok()?;
Some((resolved, 1.0))
})
.collect();
if seeds.is_empty() {
anyhow::bail!("no valid seed keys found");
}
let seed_set: std::collections::HashSet<&str> = seeds.iter()
.map(|(k, _)| k.as_str()).collect();
let results = crate::search::spreading_activation(
&seeds, &graph, store,
max_hops, edge_decay, min_activation,
);
Ok(results.iter()
.filter(|(k, _)| !seed_set.contains(k.as_str()))
.take(limit)
.map(|(key, score)| format!(" {:.2} {}", score, key))
.collect::<Vec<_>>().join("\n"))
}
/// Info about a linked neighbor node.
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct LinkInfo {
pub key: String,
pub link_strength: f32,
pub node_weight: f32,
}
pub fn memory_links(store: &Store, _provenance: &str, key: &str) -> Result<Vec<LinkInfo>> {
let node = MemoryNode::from_store(store, key)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", key))?;
let mut links = Vec::new();
for (target, strength, _is_new) in &node.links {
let node_weight = store.get_node(target)
.ok()
.flatten()
.map(|n| n.weight)
.unwrap_or(0.5);
links.push(LinkInfo {
key: target.clone(),
link_strength: *strength,
node_weight,
});
}
Ok(links)
}
pub fn memory_link_set(store: &Store, _provenance: &str, source: &str, target: &str, strength: f32) -> Result<String> {
let s = store.resolve_key(source).map_err(|e| anyhow::anyhow!("{}", e))?;
let t = store.resolve_key(target).map_err(|e| anyhow::anyhow!("{}", e))?;
let old = store.set_link_strength(&s, &t, strength).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("{} ↔ {} strength {:.2} → {:.2}", s, t, old, strength))
}
pub fn memory_link_add(store: &Store, provenance: &str, source: &str, target: &str) -> Result<String> {
let s = store.resolve_key(source).map_err(|e| anyhow::anyhow!("{}", e))?;
let t = store.resolve_key(target).map_err(|e| anyhow::anyhow!("{}", e))?;
let strength = store.add_link(&s, &t, provenance).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("linked {} → {} (strength={:.2})", s, t, strength))
}
pub fn memory_delete(store: &Store, _provenance: &str, key: &str) -> Result<String> {
let resolved = store.resolve_key(key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.delete_node(&resolved).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("deleted {}", resolved))
}
pub fn memory_history(store: &Store, _provenance: &str, key: &str, full: Option<bool>) -> Result<String> {
let key = store.resolve_key(key).unwrap_or_else(|_| key.to_string());
let full = full.unwrap_or(false);
let path = crate::store::nodes_path();
if !path.exists() {
anyhow::bail!("No node log found");
}
use std::io::BufReader;
let file = std::fs::File::open(&path)
.map_err(|e| anyhow::anyhow!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
let mut versions: Vec<crate::store::Node> = Vec::new();
while let Ok(msg) = capnp::serialize::read_message(&mut reader, capnp::message::ReaderOptions::new()) {
let log = msg.get_root::<crate::memory_capnp::node_log::Reader>()
.map_err(|e| anyhow::anyhow!("read log: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| anyhow::anyhow!("get nodes: {}", e))? {
let node = crate::store::Node::from_capnp_migrate(node_reader)
.map_err(|e| anyhow::anyhow!("{}", e))?;
if node.key == key {
versions.push(node);
}
}
}
if versions.is_empty() {
anyhow::bail!("No history found for '{}'", key);
}
let mut out = format!("{} versions of '{}':\n\n", versions.len(), key);
for node in &versions {
let ts = crate::store::format_datetime(node.timestamp);
let deleted = if node.deleted { " DELETED" } else { "" };
if full {
out.push_str(&format!("=== v{} {} {}{} w={:.3} {}b ===\n",
node.version, ts, node.provenance, deleted, node.weight, node.content.len()));
out.push_str(&node.content);
out.push('\n');
} else {
let preview = crate::util::first_n_chars(&node.content, 120).replace('\n', "\\n");
out.push_str(&format!("v{:<3} {} {:24} w={:.3} {}b{}\n {}\n",
node.version, ts, node.provenance, node.weight, node.content.len(), deleted, preview));
}
}
Ok(out)
}
pub fn memory_weight_set(store: &Store, _provenance: &str, key: &str, weight: f32) -> Result<String> {
let resolved = store.resolve_key(key).map_err(|e| anyhow::anyhow!("{}", e))?;
let (old, new) = store.set_weight(&resolved, weight).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("weight {} {:.2} → {:.2}", resolved, old, new))
}
pub fn memory_rename(store: &Store, _provenance: &str, old_key: &str, new_key: &str) -> Result<String> {
let resolved = store.resolve_key(old_key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.rename_node(&resolved, new_key).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("Renamed '{}' → '{}'", resolved, new_key))
}
pub fn memory_supersede(store: &Store, provenance: &str, old_key: &str, new_key: &str, reason: Option<&str>) -> Result<String> {
let reason = reason.unwrap_or("superseded");
let content = store.get_node(old_key)
.map_err(|e| anyhow::anyhow!("{}", e))?
.map(|n| n.content)
.ok_or_else(|| anyhow::anyhow!("node not found: {}", old_key))?;
let notice = format!("**SUPERSEDED** by `{}` — {}\n\n---\n\n{}",
new_key, reason, content.trim());
store.upsert_provenance(old_key, &notice, provenance)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.set_weight(old_key, 0.01).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(format!("superseded {} → {} ({})", old_key, new_key, reason))
}
/// Convert a list of keys to ReplayItems with priority and graph metrics.
pub fn keys_to_replay_items(
store: &Store,
keys: &[String],
graph: &Graph,
) -> Vec<ReplayItem> {
keys.iter()
.filter_map(|key| {
let node = store.get_node(key).ok()??;
let priority = consolidation_priority(store, key, graph, None);
let cc = graph.clustering_coefficient(key);
Some(ReplayItem {
key: key.clone(),
priority,
interval_days: node.spaced_repetition_interval,
emotion: node.emotion,
cc,
classification: "unknown",
outlier_score: 0.0,
})
})
.collect()
}
pub fn memory_query(store: &Store, _provenance: &str, query_str: &str, format: Option<&str>) -> Result<String> {
let graph = store.build_graph();
match format.unwrap_or("compact") {
"full" => {
// Rich output with full content, graph metrics, hub analysis
let results = crate::query_parser::execute_query(store, &graph, query_str)
.map_err(|e| anyhow::anyhow!("{}", e))?;
let keys: Vec<String> = results.into_iter().map(|r| r.key).collect();
let items = keys_to_replay_items(store, &keys, &graph);
Ok(crate::subconscious::prompts::format_nodes_section(store, &items, &graph))
}
_ => {
// Compact output: handles count, select, and all expression types
crate::query_parser::query_to_string(store, &graph, query_str)
.map_err(|e| anyhow::anyhow!("{}", e))
}
}
}
// ── Journal tools ──────────────────────────────────────────────
/// A journal entry with key, content, and timestamp.
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct JournalEntry {
pub key: String,
pub content: String,
pub created_at: i64,
}
/// Get journal entries, sorted by timestamp (newest first).
/// level: 0=session, 1=daily, 2=weekly, 3=monthly
/// after: only entries after this date (YYYY-MM-DD)
pub fn journal_tail(store: &Store, _provenance: &str, count: Option<u64>, level: Option<u64>, after: Option<&str>) -> Result<Vec<JournalEntry>> {
let count = count.unwrap_or(10) as usize;
let level = level.unwrap_or(0);
let node_type = match level {
0 => crate::store::NodeType::EpisodicSession,
1 => crate::store::NodeType::EpisodicDaily,
2 => crate::store::NodeType::EpisodicWeekly,
3 => crate::store::NodeType::EpisodicMonthly,
_ => return Err(anyhow::anyhow!("invalid level: {}", level)),
};
let after_ts = after.and_then(|date| {
chrono::NaiveDate::parse_from_str(date, "%Y-%m-%d").ok()
.and_then(|nd| nd.and_hms_opt(0, 0, 0))
.map(|dt| dt.and_utc().timestamp())
});
let all_keys = store.all_keys()?;
let mut entries: Vec<_> = all_keys.iter()
.filter_map(|key| store.get_node(key).ok()?)
.filter(|n| n.node_type == node_type)
.filter(|n| after_ts.map(|ts| n.created_at >= ts).unwrap_or(true))
.map(|n| JournalEntry {
key: n.key.clone(),
content: n.content,
created_at: n.created_at,
})
.collect();
entries.sort_by_key(|e| std::cmp::Reverse(e.created_at));
entries.truncate(count);
Ok(entries)
}
fn level_to_node_type(level: i64) -> crate::store::NodeType {
match level {
1 => crate::store::NodeType::EpisodicDaily,
2 => crate::store::NodeType::EpisodicWeekly,
3 => crate::store::NodeType::EpisodicMonthly,
_ => crate::store::NodeType::EpisodicSession,
}
}
pub fn journal_new(store: &Store, provenance: &str, name: &str, title: &str, body: &str, level: Option<i64>) -> Result<String> {
let level = level.unwrap_or(0);
let ts = chrono::Local::now().format("%Y-%m-%dT%H:%M");
let content = format!("## {} — {}\n\n{}", ts, title, body);
let base_key: String = name.split_whitespace()
.map(|w| w.to_lowercase()
.chars().filter(|c| c.is_alphanumeric() || *c == '-')
.collect::<String>())
.filter(|s| !s.is_empty())
.collect::<Vec<_>>()
.join("-");
let base_key = if base_key.len() > 80 { &base_key[..80] } else { base_key.as_str() };
let key = if store.contains_key(base_key).unwrap_or(false) {
let mut n = 2;
loop {
let candidate = format!("{}-{}", base_key, n);
if !store.contains_key(&candidate).unwrap_or(false) { break candidate; }
n += 1;
}
} else {
base_key.to_string()
};
let mut node = crate::store::new_node(&key, &content);
node.node_type = level_to_node_type(level);
node.provenance = provenance.to_string();
store.upsert_node(node).map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
let word_count = body.split_whitespace().count();
Ok(format!("New entry '{}' ({} words)", title, word_count))
}
pub fn journal_update(store: &Store, provenance: &str, body: &str, level: Option<i64>) -> Result<String> {
let level = level.unwrap_or(0);
let node_type = level_to_node_type(level);
let all_keys = store.all_keys()?;
let latest_key = all_keys.iter()
.filter_map(|key| store.get_node(key).ok()?)
.filter(|n| n.node_type == node_type)
.max_by_key(|n| n.created_at)
.map(|n| n.key.clone());
let Some(key) = latest_key else {
anyhow::bail!("no entry at level {} to update — use journal_new first", level);
};
let existing = store.get_node(&key)?.ok_or_else(|| anyhow::anyhow!("node not found"))?.content;
let new_content = format!("{}\n\n{}", existing.trim_end(), body);
store.upsert_provenance(&key, &new_content, provenance)
.map_err(|e| anyhow::anyhow!("{}", e))?;
store.save().map_err(|e| anyhow::anyhow!("{}", e))?;
let word_count = body.split_whitespace().count();
Ok(format!("Updated last entry (+{} words)", word_count))
}
// ── Graph tools ───────────────────────────────────────────────
pub fn graph_topology(store: &Store, _provenance: &str) -> Result<String> {
let graph = store.build_graph();
Ok(crate::subconscious::prompts::format_topology_header(store, &graph))
}
pub fn graph_health(store: &Store, _provenance: &str) -> Result<String> {
let graph = store.build_graph();
Ok(crate::subconscious::prompts::format_health_section(store, &graph))
}
pub fn graph_communities(store: &Store, _provenance: &str, top_n: Option<usize>, min_size: Option<usize>) -> Result<String> {
let top_n = top_n.unwrap_or(10);
let min_size = min_size.unwrap_or(3);
let g = store.build_graph();
let infos = g.community_info();
let total = infos.len();
let shown: Vec<_> = infos.into_iter()
.filter(|c| c.size >= min_size)
.take(top_n)
.collect();
use std::fmt::Write;
let mut out = String::new();
writeln!(out, "{} communities total ({} with size >= {})\n",
total, shown.len(), min_size).ok();
writeln!(out, "{:<6} {:>5} {:>7} {:>7} members", "id", "size", "iso", "cross").ok();
writeln!(out, "{}", "-".repeat(70)).ok();
for c in &shown {
let preview: Vec<&str> = c.members.iter()
.take(5)
.map(|s| s.as_str())
.collect();
let more = if c.size > 5 {
format!(" +{}", c.size - 5)
} else {
String::new()
};
writeln!(out, "{:<6} {:>5} {:>6.0}% {:>7} {}{}",
c.id, c.size, c.isolation * 100.0, c.cross_edges,
preview.join(", "), more).ok();
}
Ok(out)
}
pub fn graph_normalize_strengths(store: &Store, _provenance: &str, apply: Option<bool>) -> Result<String> {
use crate::store::{StoreView, RelationType};
let apply = apply.unwrap_or(false);
let graph = store.build_graph();
let strengths = graph.jaccard_strengths();
// Build lookup from (source_key, target_key) → new_strength
let mut target_strengths: std::collections::HashMap<(String, String), f32> = std::collections::HashMap::new();
for (a, b, s) in &strengths {
target_strengths.insert((a.clone(), b.clone()), *s);
target_strengths.insert((b.clone(), a.clone()), *s);
}
// Collect edges and compute changes
let mut to_update: Vec<(String, String, f32)> = Vec::new();
let mut unchanged = 0usize;
let mut temporal_skipped = 0usize;
let mut delta_sum: f64 = 0.0;
let mut buckets = [0usize; 10];
store.for_each_relation(|source, target, strength, rel_type| {
// Skip temporal links
if strength == 1.0 && rel_type == RelationType::Auto {
temporal_skipped += 1;
return;
}
if let Some(&new_s) = target_strengths.get(&(source.to_string(), target.to_string())) {
let delta = (new_s - strength).abs();
if delta > 0.001 {
delta_sum += delta as f64;
to_update.push((source.to_string(), target.to_string(), new_s));
} else {
unchanged += 1;
}
let bucket = ((new_s * 10.0) as usize).min(9);
buckets[bucket] += 1;
}
});
let changed = to_update.len();
use std::fmt::Write;
let mut out = String::new();
writeln!(out, "Normalize link strengths (Jaccard similarity)").ok();
writeln!(out, " Total edges in graph: {}", strengths.len()).ok();
writeln!(out, " Would change: {}", changed).ok();
writeln!(out, " Unchanged: {}", unchanged).ok();
writeln!(out, " Temporal (skipped): {}", temporal_skipped).ok();
if changed > 0 {
writeln!(out, " Avg delta: {:.3}", delta_sum / changed as f64).ok();
}
writeln!(out).ok();
writeln!(out, " Strength distribution:").ok();
for (i, &count) in buckets.iter().enumerate() {
let lo = i as f32 / 10.0;
let hi = lo + 0.1;
let bar = "#".repeat(count / 50 + if count > 0 { 1 } else { 0 });
writeln!(out, " {:.1}-{:.1}: {:5} {}", lo, hi, count, bar).ok();
}
if apply {
for (source, target, new_strength) in to_update {
store.set_link_strength(&source, &target, new_strength)?;
}
writeln!(out, "\nApplied {} strength updates.", changed).ok();
} else {
writeln!(out, "\nDry run. Pass apply:true to write changes.").ok();
}
Ok(out)
}
pub fn graph_link_impact(store: &Store, _provenance: &str, source: &str, target: &str) -> Result<String> {
let source = store.resolve_key(source).map_err(|e| anyhow::anyhow!("{}", e))?;
let target = store.resolve_key(target).map_err(|e| anyhow::anyhow!("{}", e))?;
let g = store.build_graph();
let impact = g.link_impact(&source, &target);
use std::fmt::Write;
let mut out = String::new();
writeln!(out, "Link impact: {} → {}", source, target).ok();
writeln!(out, " Source degree: {} Target degree: {}", impact.source_deg, impact.target_deg).ok();
writeln!(out, " Hub link: {} Same community: {}", impact.is_hub_link, impact.same_community).ok();
writeln!(out, " ΔCC source: {:+.4} ΔCC target: {:+.4}", impact.delta_cc_source, impact.delta_cc_target).ok();
writeln!(out, " ΔGini: {:+.6}", impact.delta_gini).ok();
writeln!(out, " Assessment: {}", impact.assessment).ok();
Ok(out)
}
pub fn graph_hubs(store: &Store, _provenance: &str, count: Option<usize>) -> Result<String> {
let count = count.unwrap_or(20);
let graph = store.build_graph();
// Top hub nodes by degree, spread apart (skip neighbors of already-selected hubs)
let all_keys = store.all_keys().unwrap_or_default();
let mut hubs: Vec<(String, usize)> = all_keys.iter()
.filter(|k| !k.starts_with('_'))
.map(|k| {
let degree = graph.neighbors(k).len();
(k.clone(), degree)
})
.collect();
hubs.sort_by(|a, b| b.1.cmp(&a.1));
let mut selected = Vec::new();
let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
for (key, degree) in &hubs {
if seen.contains(key) { continue; }
selected.push(format!(" - {} (degree {})", key, degree));
// Mark neighbors as seen so we pick far-apart hubs
for (nbr, _) in graph.neighbors(key) {
seen.insert(nbr.clone());
}
seen.insert(key.clone());
if selected.len() >= count { break; }
}
Ok(format!("## Hub nodes (link targets)\n\n{}", selected.join("\n")))
}
pub fn graph_trace(store: &Store, _provenance: &str, key: &str) -> Result<String> {
let resolved = store.resolve_key(key).map_err(|e| anyhow::anyhow!("{}", e))?;
let g = store.build_graph();
let node = store.get_node(&resolved)?
.ok_or_else(|| anyhow::anyhow!("Node not found: {}", resolved))?;
use std::fmt::Write;
let mut out = String::new();
writeln!(out, "=== {} ===", resolved).ok();
writeln!(out, "Type: {:?} Weight: {:.2}", node.node_type, node.weight).ok();
if !node.source_ref.is_empty() {
writeln!(out, "Source: {}", node.source_ref).ok();
}
let preview = crate::util::truncate(&node.content, 200, "...");
writeln!(out, "\n{}\n", preview).ok();
// Walk neighbors, grouped by node type
let neighbors = g.neighbors(&resolved);
let mut episodic_session: Vec<(String, f32, crate::store::Node)> = Vec::new();
let mut episodic_daily: Vec<(String, f32, crate::store::Node)> = Vec::new();
let mut episodic_weekly: Vec<(String, f32, crate::store::Node)> = Vec::new();
let mut semantic: Vec<(String, f32, crate::store::Node)> = Vec::new();
for (n, strength) in &neighbors {
if let Ok(Some(nnode)) = store.get_node(n) {
let node_type = nnode.node_type;
let key: String = (*n).clone();
let entry = (key, *strength, nnode);
match node_type {
crate::store::NodeType::EpisodicSession => episodic_session.push(entry),
crate::store::NodeType::EpisodicDaily => episodic_daily.push(entry),
crate::store::NodeType::EpisodicWeekly
| crate::store::NodeType::EpisodicMonthly => episodic_weekly.push(entry),
crate::store::NodeType::Semantic => semantic.push(entry),
}
}
}
if !episodic_weekly.is_empty() {
writeln!(out, "Weekly digests:").ok();
for (k, s, n) in &episodic_weekly {
let preview = crate::util::first_n_chars(n.content.lines().next().unwrap_or(""), 80);
writeln!(out, " [{:.2}] {} — {}", s, &k, preview).ok();
}
}
if !episodic_daily.is_empty() {
writeln!(out, "Daily digests:").ok();
for (k, s, n) in &episodic_daily {
let preview = crate::util::first_n_chars(n.content.lines().next().unwrap_or(""), 80);
writeln!(out, " [{:.2}] {} — {}", s, &k, preview).ok();
}
}
if !episodic_session.is_empty() {
writeln!(out, "Session entries:").ok();
for (k, s, n) in &episodic_session {
let preview = crate::util::first_n_chars(
n.content.lines()
.find(|l| !l.is_empty() && !l.starts_with("<!--"))
.unwrap_or(""),
80);
writeln!(out, " [{:.2}] {}", s, &k).ok();
if !n.source_ref.is_empty() {
writeln!(out, " ↳ source: {}", n.source_ref).ok();
}
writeln!(out, " {}", preview).ok();
}
}
if !semantic.is_empty() {
writeln!(out, "Semantic links:").ok();
for (k, s, _) in &semantic {
writeln!(out, " [{:.2}] {}", s, k).ok();
}
}
writeln!(out, "\nLinks: {} session, {} daily, {} weekly, {} semantic",
episodic_session.len(), episodic_daily.len(),
episodic_weekly.len(), semantic.len()).ok();
Ok(out)
}

39
src/hippocampus/memory.rs
View file

@ -19,13 +19,13 @@ pub struct MemoryNode {
impl MemoryNode {
/// Load a node from the store by key.
pub fn load(key: &str) -> Option<Self> {
let store = super::access_local().ok()?;
let store = Store::load().ok()?;
Self::from_store(&store, key)
}
/// Load from an already-open store.
pub fn from_store(store: &Store, key: &str) -> Option<Self> {
let node = store.get_node(key).ok()??;
let node = store.nodes.get(key)?;
// If set, tag links to nodes created after this timestamp as (new)
let older_than: i64 = std::env::var("POC_MEMORIES_OLDER_THAN")
@ -33,30 +33,34 @@ impl MemoryNode {
.and_then(|s| s.parse().ok())
.unwrap_or(0);
// Get neighbors via index
let mut neighbors: std::collections::HashMap<String, (f32, bool)> = std::collections::HashMap::new();
if let Ok(neighbor_list) = store.neighbors(key) {
for (neighbor_key, strength) in neighbor_list {
let is_new = older_than > 0 && store.get_node(&neighbor_key)
.ok()
.flatten()
let mut neighbors: std::collections::HashMap<&str, (f32, bool)> = std::collections::HashMap::new();
for r in &store.relations {
if r.deleted { continue; }
let neighbor_key = if r.source_key == key {
&r.target_key
} else if r.target_key == key {
&r.source_key
} else {
continue;
};
let is_new = older_than > 0 && store.nodes.get(neighbor_key.as_str())
.map(|n| n.created_at > older_than)
.unwrap_or(false);
let e = neighbors.entry(neighbor_key).or_insert((0.0, false));
e.0 = e.0.max(strength);
let e = neighbors.entry(neighbor_key.as_str()).or_insert((0.0, false));
e.0 = e.0.max(r.strength);
e.1 = e.1 || is_new;
}
}
let mut links: Vec<(String, f32, bool)> = neighbors.into_iter()
.map(|(k, (s, new))| (k, s, new))
.map(|(k, (s, new))| (k.to_string(), s, new))
.collect();
links.sort_by(|a, b| b.1.total_cmp(&a.1));
Some(MemoryNode {
key: key.to_string(),
content: node.content,
content: node.content.clone(),
links,
version: node.version,
weight: node.weight,
@ -87,10 +91,3 @@ impl MemoryNode {
out
}
}
/// Render a node to a string: content + deduped footer links.
/// Used by both the CLI command and agent placeholders.
pub fn render_node(store: &Store, key: &str) -> Option<String> {
crate::hippocampus::memory::MemoryNode::from_store(store, key)
.map(|node| node.render())
}

306
src/hippocampus/mod.rs
View file

@ -4,319 +4,13 @@
// similarity scoring, spectral analysis, and neuroscience-inspired
// consolidation (spaced repetition, interference detection, schema
// assimilation).
//
// Tool implementations are typed functions that take &Store or &mut Store.
// The tools/memory.rs layer handles JSON parsing and RPC routing.
pub mod memory;
pub mod store;
pub mod graph;
pub mod local;
pub mod lookups;
pub mod query;
pub mod spectral;
pub mod neuro;
pub mod counters;
pub mod transcript;
use std::cell::RefCell;
use std::path::PathBuf;
use std::sync::{Arc, OnceLock};
use anyhow::Result;
use crate::hippocampus::store::Store;
pub use local::{LinkInfo, JournalEntry};
// ── Store access ───────────────────────────────────────────────
/// Daemon's store (eager init) or client's fallback local store.
static STORE_ACCESS: OnceLock<Option<Arc<Store>>> = OnceLock::new();
// Client's socket connection (thread-local for lock-free access).
thread_local! {
static SOCKET_CONN: RefCell<Option<SocketConn>> = const { RefCell::new(None) };
}
/// How we access the memory store.
pub enum StoreAccess {
Daemon(Arc<Store>), // Direct store access
Client, // Socket to daemon (in thread-local)
None(String), // Error: couldn't get access
}
/// Get store access: daemon's store, socket, or local fallback.
pub fn access() -> StoreAccess {
// Check if already cached
if let Some(Some(store)) = STORE_ACCESS.get() {
return StoreAccess::Daemon(store.clone());
}
// Client: check if socket already cached in thread-local
let have_socket = SOCKET_CONN.with(|cell| cell.borrow().is_some());
if have_socket {
return StoreAccess::Client;
}
// No socket cached, try connecting
if let Ok(conn) = SocketConn::connect() {
SOCKET_CONN.with(|cell| *cell.borrow_mut() = Some(conn));
return StoreAccess::Client;
}
// Socket failed - try local store as fallback (cached in STORE_ACCESS)
let store_opt = STORE_ACCESS.get_or_init(|| {
Store::load().ok().map(Arc::new)
});
match store_opt {
Some(store) => StoreAccess::Daemon(store.clone()),
None => StoreAccess::None("could not connect to daemon or open store locally".into()),
}
}
/// Get local store access. Returns error if only RPC available.
pub fn access_local() -> Result<Arc<Store>> {
match access() {
StoreAccess::Daemon(arc) => Ok(arc),
StoreAccess::Client => anyhow::bail!("direct store access not available via RPC"),
StoreAccess::None(err) => anyhow::bail!("{}", err),
}
}
pub fn socket_path() -> PathBuf {
dirs::home_dir()
.unwrap_or_default()
.join(".consciousness/mcp.sock")
}
struct SocketConn {
reader: std::io::BufReader<std::os::unix::net::UnixStream>,
writer: std::io::BufWriter<std::os::unix::net::UnixStream>,
next_id: u64,
}
impl SocketConn {
fn connect() -> Result<Self> {
use std::os::unix::net::UnixStream;
use std::io::{BufRead, BufReader, BufWriter, Write};
let path = socket_path();
let stream = UnixStream::connect(&path)?;
let mut reader = BufReader::new(stream.try_clone()?);
let mut writer = BufWriter::new(stream);
// Initialize MCP connection
let init = serde_json::json!({"jsonrpc": "2.0", "id": 1, "method": "initialize",
"params": {"protocolVersion": "2024-11-05", "capabilities": {},
"clientInfo": {"name": "forward", "version": "0.1"}}});
writeln!(writer, "{}", init)?;
writer.flush()?;
let mut buf = String::new();
reader.read_line(&mut buf)?;
Ok(Self { reader, writer, next_id: 1 })
}
fn call(&mut self, tool_name: &str, args: &serde_json::Value) -> Result<String> {
use std::io::{BufRead, Write};
self.next_id += 1;
let call = serde_json::json!({"jsonrpc": "2.0", "id": self.next_id, "method": "tools/call",
"params": {"name": tool_name, "arguments": args}});
writeln!(self.writer, "{}", call)?;
self.writer.flush()?;
let mut buf = String::new();
self.reader.read_line(&mut buf)?;
let resp: serde_json::Value = serde_json::from_str(&buf)?;
if let Some(err) = resp.get("error") {
anyhow::bail!("daemon error: {}", err);
}
let result = resp.get("result").cloned().unwrap_or(serde_json::json!({}));
let text = result.get("content")
.and_then(|c| c.as_array())
.and_then(|arr| arr.first())
.and_then(|c| c.get("text"))
.and_then(|t| t.as_str())
.unwrap_or("");
Ok(text.to_string())
}
}
/// Forward a tool call to the daemon via socket.
/// Only valid when access() returns Client.
pub fn memory_rpc(tool_name: &str, args: serde_json::Value) -> Result<String> {
SOCKET_CONN.with(|cell| {
let mut conn = cell.borrow_mut();
let conn = conn.as_mut().expect("access() returned Client but SOCKET_CONN is None");
conn.call(tool_name, &args)
})
}
// ── Macro for generating tool wrappers ─────────────────────────
//
// memory_tool!(name, mut, arg1: [str], arg2: [Option<bool>])
// - mut/ref for store mutability
// - generates jsonargs_* (internal, JSON args) and public typed API
macro_rules! memory_tool {
// ── Helper rules (must come first) ─────────────────────────────
// Extract from JSON
(@extract $args:ident, $name:ident, str) => {
get_str($args, stringify!($name))?
};
(@extract $args:ident, $name:ident, f32) => {
get_f64($args, stringify!($name))? as f32
};
(@extract $args:ident, $name:ident, Vec<String>) => {
$args.get(stringify!($name))
.and_then(|v| v.as_array())
.map(|arr| arr.iter().filter_map(|v| v.as_str().map(String::from)).collect::<Vec<_>>())
.unwrap_or_default()
};
(@extract $args:ident, $name:ident, Option<&str>) => {
$args.get(stringify!($name)).and_then(|v| v.as_str())
};
(@extract $args:ident, $name:ident, Option<bool>) => {
$args.get(stringify!($name)).and_then(|v| v.as_bool())
};
(@extract $args:ident, $name:ident, Option<u64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64())
};
(@extract $args:ident, $name:ident, Option<i64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_i64())
};
(@extract $args:ident, $name:ident, Option<usize>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64()).map(|v| v as usize)
};
(@extract $args:ident, $name:ident, Option<u32>) => {
$args.get(stringify!($name)).and_then(|v| v.as_u64()).map(|v| v as u32)
};
(@extract $args:ident, $name:ident, Option<f64>) => {
$args.get(stringify!($name)).and_then(|v| v.as_f64())
};
// Parameter types for function signatures
(@param_type str) => { &str };
(@param_type f32) => { f32 };
(@param_type Vec<String>) => { Vec<String> };
(@param_type Option<&str>) => { Option<&str> };
(@param_type Option<bool>) => { Option<bool> };
(@param_type Option<u64>) => { Option<u64> };
(@param_type Option<i64>) => { Option<i64> };
(@param_type Option<usize>) => { Option<usize> };
(@param_type Option<u32>) => { Option<u32> };
(@param_type Option<f64>) => { Option<f64> };
// Serialize result for jsonargs
(@serialize $t:ty, $result:expr) => { serde_json::to_string(&$result)? };
// Deserialize RPC response
(@deserialize $t:ty, $json:expr) => { serde_json::from_str(&$json).map_err(|e| anyhow::anyhow!("{}", e)) };
// Serialize to JSON for RPC
(@insert_json $map:ident, $name:ident, str) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, f32) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, Vec<String>) => {
$map.insert(stringify!($name).into(), serde_json::json!($name));
};
(@insert_json $map:ident, $name:ident, Option<&str>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<bool>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<u64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<i64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<usize>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<u32>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
(@insert_json $map:ident, $name:ident, Option<f64>) => {
if let Some(v) = $name { $map.insert(stringify!($name).into(), serde_json::json!(v)); }
};
// Call hippocampus (all methods now take &self, deref Arc)
(@call mut, $name:ident, $store:ident, $prov:expr $(, $arg:expr)*) => {
local::$name(&*$store, $prov $(, $arg)*)
};
(@call ref, $name:ident, $store:ident, $prov:expr $(, $arg:expr)*) => {
local::$name(&*$store, $prov $(, $arg)*)
};
// ── Main rules ─────────────────────────────────────────────────
// Shorthand: mut/ref without return type defaults to String
($name:ident, $m:ident $(, $($arg:ident : [$($typ:tt)+]),* $(,)?)?) => {
memory_tool!($name, $m -> String $(, $($arg : [$($typ)+]),*)?);
};
// Full form with return type
($name:ident, $m:ident -> $ret:ty $(, $($arg:ident : [$($typ:tt)+]),* $(,)?)?) => {
paste::paste! {
pub async fn $name(agent: Option<&crate::agent::Agent> $($(, $arg: memory_tool!(@param_type $($typ)+))*)?) -> Result<$ret> {
let prov = match agent {
Some(a) => a.state.lock().await.provenance.clone(),
None => "manual".to_string(),
};
match access() {
StoreAccess::Daemon(store) => {
memory_tool!(@call $m, $name, store, &prov $($(, $arg)*)?)
}
StoreAccess::Client => {
#[allow(unused_mut)]
let mut map = serde_json::Map::new();
$($(memory_tool!(@insert_json map, $arg, $($typ)+);)*)?
let json = memory_rpc(stringify!($name), serde_json::Value::Object(map))?;
memory_tool!(@deserialize $ret, json)
}
StoreAccess::None(err) => anyhow::bail!("{}", err),
}
}
}
};
}
// ── Memory tools ───────────────────────────────────────────────
memory_tool!(memory_render, ref, key: [str], raw: [Option<bool>]);
memory_tool!(memory_write, mut, key: [str], content: [str]);
memory_tool!(memory_search, ref, keys: [Vec<String>], max_hops: [Option<u32>], edge_decay: [Option<f64>], min_activation: [Option<f64>], limit: [Option<usize>]);
memory_tool!(memory_link_set, mut, source: [str], target: [str], strength: [f32]);
memory_tool!(memory_link_add, mut, source: [str], target: [str]);
memory_tool!(memory_delete, mut, key: [str]);
memory_tool!(memory_history, ref, key: [str], full: [Option<bool>]);
memory_tool!(memory_weight_set, mut, key: [str], weight: [f32]);
memory_tool!(memory_rename, mut, old_key: [str], new_key: [str]);
memory_tool!(memory_supersede, mut, old_key: [str], new_key: [str], reason: [Option<&str>]);
memory_tool!(memory_query, ref, query: [str], format: [Option<&str>]);
memory_tool!(memory_links, ref -> Vec<LinkInfo>, key: [str]);
// ── Journal tools ──────────────────────────────────────────────
memory_tool!(journal_tail, ref -> Vec<JournalEntry>, count: [Option<u64>], level: [Option<u64>], after: [Option<&str>]);
memory_tool!(journal_new, mut, name: [str], title: [str], body: [str], level: [Option<i64>]);
memory_tool!(journal_update, mut, body: [str], level: [Option<i64>]);
// ── Graph tools ───────────────────────────────────────────────
memory_tool!(graph_topology, ref);
memory_tool!(graph_health, ref);
memory_tool!(graph_communities, ref, top_n: [Option<usize>], min_size: [Option<usize>]);
memory_tool!(graph_normalize_strengths, mut, apply: [Option<bool>]);
memory_tool!(graph_link_impact, ref, source: [str], target: [str]);
memory_tool!(graph_hubs, ref, count: [Option<usize>]);
memory_tool!(graph_trace, ref, key: [str]);

22
src/hippocampus/neuro/scoring.rs
View file

@ -26,7 +26,7 @@ pub fn consolidation_priority(
graph: &Graph,
spectral_outlier: Option<f64>,
) -> f64 {
let node = match store.get_node(key).ok().flatten() {
let node = match store.nodes.get(key) {
Some(n) => n,
None => return 0.0,
};
@ -97,10 +97,8 @@ pub fn replay_queue_with_graph(
HashMap::new()
};
let all_keys = store.all_keys().unwrap_or_default();
let mut items: Vec<ReplayItem> = all_keys.iter()
.filter_map(|key| {
let node = store.get_node(key).ok()??;
let mut items: Vec<ReplayItem> = store.nodes.iter()
.map(|(key, node)| {
let pos = positions.get(key);
let outlier_score = pos.map(|p| p.outlier_score).unwrap_or(0.0);
let classification = pos
@ -111,7 +109,7 @@ pub fn replay_queue_with_graph(
store, key, graph,
pos.map(|p| p.outlier_score),
);
Some(ReplayItem {
ReplayItem {
key: key.clone(),
priority,
interval_days: node.spaced_repetition_interval,
@ -119,7 +117,7 @@ pub fn replay_queue_with_graph(
cc: graph.clustering_coefficient(key),
classification,
outlier_score,
})
}
})
.collect();
@ -216,13 +214,11 @@ fn consolidation_plan_inner(store: &Store, _detect_interf: bool) -> Consolidatio
let gini = graph.degree_gini();
let _avg_cc = graph.avg_clustering_coefficient();
let all_keys = store.all_keys().unwrap_or_default();
let episodic_count = all_keys.iter()
.filter_map(|k| store.get_node(k).ok()?)
.filter(|n| matches!(n.node_type, crate::store::NodeType::EpisodicSession))
let episodic_count = store.nodes.iter()
.filter(|(_, n)| matches!(n.node_type, crate::store::NodeType::EpisodicSession))
.count();
let _episodic_ratio = if all_keys.is_empty() { 0.0 }
else { episodic_count as f32 / all_keys.len() as f32 };
let _episodic_ratio = if store.nodes.is_empty() { 0.0 }
else { episodic_count as f32 / store.nodes.len() as f32 };
let mut plan = ConsolidationPlan {
counts: std::collections::HashMap::new(),

59
src/hippocampus/query/engine.rs
View file

@ -148,6 +148,8 @@ pub enum Filter {
Age(Cmp), // vs now - timestamp (seconds)
ContentLen(Cmp),
Provenance(String),
NotVisited { agent: String, duration: i64 }, // seconds
Visited { agent: String },
Negated(Box<Filter>),
}
@ -183,6 +185,8 @@ pub enum ScoreField {
Weight,
ContentLen,
Priority,
/// Time since last visit by named agent. 1.0 = never visited, decays toward 0.
Recency(String),
}
/// Numeric comparison operator.
@ -227,10 +231,10 @@ fn score_field(
(d / max).min(1.0)
}
ScoreField::Weight => {
store.get_node(key).ok().flatten().map(|n| n.weight as f64).unwrap_or(0.0)
store.nodes.get(key).map(|n| n.weight as f64).unwrap_or(0.0)
}
ScoreField::ContentLen => {
let len = store.get_node(key).ok().flatten().map(|n| n.content.len()).unwrap_or(0) as f64;
let len = store.nodes.get(key).map(|n| n.content.len()).unwrap_or(0) as f64;
let max = precomputed.max_content_len.max(1.0);
(len / max).min(1.0)
}
@ -239,6 +243,17 @@ fn score_field(
// Priority is already roughly 0-1 from the scoring function
p.min(1.0)
}
ScoreField::Recency(agent) => {
let last = store.last_visited(key, agent);
if last == 0 {
1.0 // never visited = highest recency score
} else {
let age = (crate::store::now_epoch() - last) as f64;
// Sigmoid decay: 1.0 at 7+ days, ~0.5 at 1 day, ~0.1 at 1 hour
let hours = age / 3600.0;
1.0 - (-0.03 * hours).exp()
}
}
}
}
@ -255,7 +270,7 @@ impl CompositeCache {
.map(|(k, _)| graph.degree(k) as f64)
.fold(0.0f64, f64::max);
let max_content_len = items.iter()
.map(|(k, _)| store.get_node(k).ok().flatten().map(|n| n.content.len()).unwrap_or(0) as f64)
.map(|(k, _)| store.nodes.get(k).map(|n| n.content.len()).unwrap_or(0) as f64)
.fold(0.0f64, f64::max);
Self {
isolation: graph.community_isolation(),
@ -291,6 +306,8 @@ impl fmt::Display for Filter {
Filter::Age(c) => write!(f, "age:{}", c),
Filter::ContentLen(c) => write!(f, "content-len:{}", c),
Filter::Provenance(p) => write!(f, "provenance:{}", p),
Filter::NotVisited { agent, duration } => write!(f, "not-visited:{},{}s", agent, duration),
Filter::Visited { agent } => write!(f, "visited:{}", agent),
Filter::Negated(inner) => write!(f, "!{}", inner),
}
}
@ -393,12 +410,9 @@ pub fn run_query(
fn run_generator(g: &Generator, store: &Store) -> Vec<(String, f64)> {
match g {
Generator::All => {
store.all_keys().unwrap_or_default().into_iter()
.filter_map(|key| {
let n = store.get_node(&key).ok()??;
if n.deleted { return None; }
Some((key, n.weight as f64))
})
store.nodes.iter()
.filter(|(_, n)| !n.deleted)
.map(|(key, n)| (key.clone(), n.weight as f64))
.collect()
}
Generator::Match(terms) => {
@ -412,7 +426,7 @@ fn run_generator(g: &Generator, store: &Store) -> Vec<(String, f64)> {
}
pub fn eval_filter(filt: &Filter, key: &str, store: &Store, now: i64) -> bool {
let node = match store.get_node(key).ok().flatten() {
let node = match store.nodes.get(key) {
Some(n) => n,
None => return false,
};
@ -427,6 +441,13 @@ pub fn eval_filter(filt: &Filter, key: &str, store: &Store, now: i64) -> bool {
}
Filter::ContentLen(cmp) => cmp.matches(node.content.len() as f64),
Filter::Provenance(p) => node.provenance == *p,
Filter::NotVisited { agent, duration } => {
let last = store.last_visited(key, agent);
last == 0 || (now - last) > *duration
}
Filter::Visited { agent } => {
store.last_visited(key, agent) > 0
}
Filter::Negated(inner) => !eval_filter(inner, key, store, now),
}
}
@ -445,15 +466,15 @@ pub fn run_transform(
}
SortField::Timestamp => {
items.sort_by(|a, b| {
let ta = store.get_node(&a.0).ok().flatten().map(|n| n.timestamp).unwrap_or(0);
let tb = store.get_node(&b.0).ok().flatten().map(|n| n.timestamp).unwrap_or(0);
let ta = store.nodes.get(&a.0).map(|n| n.timestamp).unwrap_or(0);
let tb = store.nodes.get(&b.0).map(|n| n.timestamp).unwrap_or(0);
tb.cmp(&ta) // desc
});
}
SortField::ContentLen => {
items.sort_by(|a, b| {
let la = store.get_node(&a.0).ok().flatten().map(|n| n.content.len()).unwrap_or(0);
let lb = store.get_node(&b.0).ok().flatten().map(|n| n.content.len()).unwrap_or(0);
let la = store.nodes.get(&a.0).map(|n| n.content.len()).unwrap_or(0);
let lb = store.nodes.get(&b.0).map(|n| n.content.len()).unwrap_or(0);
lb.cmp(&la) // desc
});
}
@ -483,7 +504,7 @@ pub fn run_transform(
SortField::Named(field, asc) => {
// Resolve field from node properties
let resolve = |key: &str| -> Option<f64> {
let node = store.get_node(key).ok()??;
let node = store.nodes.get(key)?;
match field.as_str() {
"weight" => Some(node.weight as f64),
"emotion" => Some(node.emotion as f64),
@ -739,10 +760,10 @@ fn run_spread(
stage: &AlgoStage,
_debug: bool,
) -> Vec<(String, f64)> {
let cfg = crate::config::get();
let max_hops = stage.param_u32("max_hops", cfg.max_hops);
let edge_decay = stage.param_f64("edge_decay", cfg.edge_decay);
let min_activation = stage.param_f64("min_activation", cfg.min_activation * 0.1);
let store_params = store.params();
let max_hops = stage.param_u32("max_hops", store_params.max_hops);
let edge_decay = stage.param_f64("edge_decay", store_params.edge_decay);
let min_activation = stage.param_f64("min_activation", store_params.min_activation * 0.1);
spreading_activation(seeds, graph, store, max_hops, edge_decay, min_activation)
}

31
src/hippocampus/query/parser.rs
View file

@ -100,6 +100,8 @@ peg::parser! {
/ "key:" g:glob_pattern() { Stage::Filter(Filter::KeyGlob(g)) }
/ "!key:" g:glob_pattern() { Stage::Filter(Filter::Negated(Box::new(Filter::KeyGlob(g)))) }
/ "provenance:" p:ident() { Stage::Filter(Filter::Provenance(p)) }
/ "not-visited:" a:ident() "," d:integer() { Stage::Filter(Filter::NotVisited { agent: a, duration: d as i64 }) }
/ "visited:" a:ident() { Stage::Filter(Filter::Visited { agent: a }) }
/ "all" { Stage::Generator(Generator::All) }
// Graph algorithms
/ "spread" { Stage::Algorithm(AlgoStage { algo: Algorithm::Spread, params: std::collections::HashMap::new() }) }
@ -121,7 +123,8 @@ peg::parser! {
/ f:field() { make_sort_field(&f, false) }
rule score_term() -> (ScoreField, f64)
= f:score_field_name() "*" w:number() { (f, w) }
= "recency(" a:ident() ")" "*" w:number() { (ScoreField::Recency(a), w) }
/ f:score_field_name() "*" w:number() { (f, w) }
rule score_field_name() -> ScoreField
= "isolation" { ScoreField::Isolation }
@ -300,7 +303,7 @@ pub fn parse_stages(s: &str) -> Result<Vec<Stage>, String> {
/// Resolve a field value from a node + graph context, returning a comparable Value.
fn resolve_field(field: &str, key: &str, store: &Store, graph: &Graph) -> Option<Value> {
let node = store.get_node(key).ok()??;
let node = store.nodes.get(key)?;
match field {
"key" => Some(Value::Str(key.to_string())),
"weight" => Some(Value::Num(node.weight as f64)),
@ -491,13 +494,9 @@ fn execute_parsed(
}
_ => {
let mut out = Vec::new();
for key in store.all_keys().unwrap_or_default() {
let node = match store.get_node(&key).ok().flatten() {
Some(n) => n,
None => continue,
};
if node.deleted { continue; }
if eval(&q.expr, &|f| resolve_field(f, &key, store, graph), store, graph) {
for key in store.nodes.keys() {
if store.nodes[key].deleted { continue; }
if eval(&q.expr, &|f| resolve_field(f, key, store, graph), store, graph) {
out.push(QueryResult { key: key.clone(), fields: BTreeMap::new() });
}
}
@ -569,15 +568,15 @@ fn execute_parsed(
}
SortField::Weight => {
results.sort_by(|a, b| {
let wa = store.get_node(&a.key).ok().flatten().map(|n| n.weight).unwrap_or(0.0);
let wb = store.get_node(&b.key).ok().flatten().map(|n| n.weight).unwrap_or(0.0);
let wa = store.nodes.get(&a.key).map(|n| n.weight).unwrap_or(0.0);
let wb = store.nodes.get(&b.key).map(|n| n.weight).unwrap_or(0.0);
wb.total_cmp(&wa)
});
}
SortField::Timestamp => {
results.sort_by(|a, b| {
let ta = store.get_node(&a.key).ok().flatten().map(|n| n.timestamp).unwrap_or(0);
let tb = store.get_node(&b.key).ok().flatten().map(|n| n.timestamp).unwrap_or(0);
let ta = store.nodes.get(&a.key).map(|n| n.timestamp).unwrap_or(0);
let tb = store.nodes.get(&b.key).map(|n| n.timestamp).unwrap_or(0);
tb.cmp(&ta)
});
}
@ -883,8 +882,7 @@ mod tests {
assert!(parses("all | key:journal-*"));
assert!(parses("all | !key:_*")); // negated key glob
assert!(parses("all | age:>7d"));
// TODO: not-visited filter not yet implemented
// assert!(parses("all | not-visited:organize,86400"));
assert!(parses("all | not-visited:organize,86400"));
}
#[test]
@ -900,8 +898,7 @@ mod tests {
fn test_composite_sort() {
// Weighted composite sort expressions (require 2+ terms with +)
assert!(parses("all | sort:degree*0.5+isolation*0.3"));
// TODO: recency(agent) not yet implemented
// assert!(parses("all | sort:degree*0.5+isolation*0.3+recency(organize)*0.2"));
assert!(parses("all | sort:degree*0.5+isolation*0.3+recency(organize)*0.2"));
assert!(parses("all | sort:weight*0.5+degree*0.5"));
// Single field (no weight) falls back to simple sort
assert!(parses("all | sort:weight"));

681
src/hippocampus/store/capnp.rs
View file

@ -1,681 +0,0 @@
// Cap'n Proto serialization and persistence
//
// capnp logs are the source of truth; redb provides indexed access.
// This module contains:
// - Serialization macros (capnp_enum!, capnp_message!)
// - Load/replay from capnp logs
// - Append to capnp logs
// - fsck (corruption repair)
use super::{index, types::*};
use redb::ReadableTableMetadata;
use crate::memory_capnp;
use super::Store;
use anyhow::{anyhow, Context, Result};
use capnp::message;
use capnp::serialize;
use std::collections::HashMap;
use std::fs;
use std::io::{BufReader, Seek};
use std::path::Path;
// ---------------------------------------------------------------------------
// Capnp serialization macros
//
// Declarative mapping between Rust types and capnp generated types.
// Adding a field to the schema means adding it in one place below;
// both read and write are generated from the same declaration.
// ---------------------------------------------------------------------------
/// Generate to_capnp/from_capnp conversion methods for an enum.
macro_rules! capnp_enum {
($rust_type:ident, $capnp_type:path, [$($variant:ident),+ $(,)?]) => {
impl $rust_type {
#[allow(clippy::wrong_self_convention, dead_code)]
pub(crate) fn to_capnp(&self) -> $capnp_type {
match self {
$(Self::$variant => <$capnp_type>::$variant,)+
}
}
pub(crate) fn from_capnp(v: $capnp_type) -> Self {
match v {
$(<$capnp_type>::$variant => Self::$variant,)+
}
}
}
};
}
/// Generate from_capnp/to_capnp methods for a struct with capnp serialization.
/// Fields are grouped by serialization kind:
/// text - capnp Text fields (String in Rust)
/// uuid - capnp Data fields ([u8; 16] in Rust)
/// prim - copy types (u32, f32, f64, bool)
/// enm - enums with to_capnp/from_capnp methods
/// skip - Rust-only fields not in capnp (set to Default on read)
macro_rules! capnp_message {
(
$struct:ident,
reader: $reader:ty,
builder: $builder:ty,
text: [$($tf:ident),* $(,)?],
uuid: [$($uf:ident),* $(,)?],
prim: [$($pf:ident),* $(,)?],
enm: [$($ef:ident: $et:ident),* $(,)?],
skip: [$($sf:ident),* $(,)?] $(,)?
) => {
impl $struct {
pub fn from_capnp(r: $reader) -> Result<Self> {
paste::paste! {
Ok(Self {
$($tf: read_text(r.[<get_ $tf>]()),)*
$($uf: read_uuid(r.[<get_ $uf>]()),)*
$($pf: r.[<get_ $pf>](),)*
$($ef: $et::from_capnp(
r.[<get_ $ef>]().map_err(|_| anyhow!(concat!("bad ", stringify!($ef))))?
),)*
$($sf: Default::default(),)*
})
}
}
pub fn to_capnp(&self, mut b: $builder) {
paste::paste! {
$(b.[<set_ $tf>](&self.$tf);)*
$(b.[<set_ $uf>](&self.$uf);)*
$(b.[<set_ $pf>](self.$pf);)*
$(b.[<set_ $ef>](self.$ef.to_capnp());)*
}
}
}
};
}
// ---------------------------------------------------------------------------
// Capnp helpers
// ---------------------------------------------------------------------------
/// Read a capnp text field, returning empty string on any error
fn read_text(result: capnp::Result<capnp::text::Reader>) -> String {
result.ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string()
}
/// Read a capnp data field as [u8; 16], zero-padded
fn read_uuid(result: capnp::Result<&[u8]>) -> [u8; 16] {
let mut out = [0u8; 16];
if let Ok(data) = result
&& data.len() >= 16 {
out.copy_from_slice(&data[..16]);
}
out
}
// ---------------------------------------------------------------------------
// Type-to-capnp mappings
// ---------------------------------------------------------------------------
capnp_enum!(NodeType, memory_capnp::NodeType,
[EpisodicSession, EpisodicDaily, EpisodicWeekly, Semantic, EpisodicMonthly]);
capnp_enum!(RelationType, memory_capnp::RelationType,
[Link, Causal, Auto]);
capnp_message!(Node,
reader: memory_capnp::content_node::Reader<'_>,
builder: memory_capnp::content_node::Builder<'_>,
text: [key, content, source_ref, provenance],
uuid: [uuid],
prim: [version, timestamp, weight, emotion, deleted,
retrievals, uses, wrongs, last_replayed,
spaced_repetition_interval, created_at, last_scored],
enm: [node_type: NodeType],
skip: [community_id, clustering_coefficient, degree],
);
capnp_message!(Relation,
reader: memory_capnp::relation::Reader<'_>,
builder: memory_capnp::relation::Builder<'_>,
text: [source_key, target_key, provenance],
uuid: [uuid, source, target],
prim: [version, timestamp, strength, deleted],
enm: [rel_type: RelationType],
skip: [],
);
// ---------------------------------------------------------------------------
// Migration helpers (legacy provenance enum → string)
// ---------------------------------------------------------------------------
/// Convert legacy capnp provenance enum to string label.
fn legacy_provenance_label(p: memory_capnp::Provenance) -> &'static str {
use memory_capnp::Provenance::*;
match p {
Manual => "manual",
Journal => "journal",
Agent => "agent",
Dream => "dream",
Derived => "derived",
AgentExperienceMine => "agent:experience-mine",
AgentKnowledgeObservation => "agent:knowledge-observation",
AgentKnowledgePattern => "agent:knowledge-pattern",
AgentKnowledgeConnector => "agent:knowledge-connector",
AgentKnowledgeChallenger => "agent:knowledge-challenger",
AgentConsolidate => "agent:consolidate",
AgentDigest => "agent:digest",
AgentFactMine => "agent:fact-mine",
AgentDecay => "agent:decay",
}
}
impl Node {
/// Read from capnp with migration: if the new provenance text field
/// is empty (old record), fall back to the deprecated provenanceOld enum.
pub fn from_capnp_migrate(r: memory_capnp::content_node::Reader<'_>) -> Result<Self> {
let mut node = Self::from_capnp(r)?;
if node.provenance.is_empty()
&& let Ok(old) = r.get_provenance_old() {
node.provenance = legacy_provenance_label(old).to_string();
}
// Sanitize timestamps: old capnp records have raw offsets instead
// of unix epoch. Anything past year 2100 (~4102444800) is bogus.
const MAX_SANE_EPOCH: i64 = 4_102_444_800;
if node.timestamp > MAX_SANE_EPOCH || node.timestamp < 0 {
node.timestamp = node.created_at;
}
if node.created_at > MAX_SANE_EPOCH || node.created_at < 0 {
node.created_at = node.timestamp.min(MAX_SANE_EPOCH);
}
Ok(node)
}
}
impl Relation {
pub fn from_capnp_migrate(r: memory_capnp::relation::Reader<'_>) -> Result<Self> {
let mut rel = Self::from_capnp(r)?;
if rel.provenance.is_empty()
&& let Ok(old) = r.get_provenance_old() {
rel.provenance = legacy_provenance_label(old).to_string();
}
Ok(rel)
}
}
// ---------------------------------------------------------------------------
// Direct node access
// ---------------------------------------------------------------------------
/// Read a single node at the given offset in the capnp log.
/// The offset must point to a valid message containing the node.
pub fn read_node_at_offset(offset: u64) -> Result<Node> {
let path = nodes_path();
let mut file = fs::File::open(&path)
.with_context(|| format!("open {}", path.display()))?;
use std::io::{Seek, SeekFrom};
file.seek(SeekFrom::Start(offset))?;
let mut reader = BufReader::new(file);
let msg = serialize::read_message(&mut reader, message::ReaderOptions::new())
.with_context(|| format!("read message at offset {}", offset))?;
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.with_context(|| "read node log")?;
let nodes = log.get_nodes()
.with_context(|| "get nodes")?;
// A message at this offset should have exactly one node (from upsert),
// or we take the last one if there are multiple (from batch operations like rename)
if nodes.is_empty() {
anyhow::bail!("no nodes in message at offset {}", offset);
}
// Return the first non-deleted node, or the first one if all are deleted
for node_reader in nodes.iter() {
let node = Node::from_capnp_migrate(node_reader)?;
if !node.deleted {
return Ok(node);
}
}
// All nodes in this message are deleted - shouldn't happen if index is correct
Node::from_capnp_migrate(nodes.get(0))
}
// ---------------------------------------------------------------------------
// Store persistence methods
// ---------------------------------------------------------------------------
impl Store {
/// Load store by opening redb index and replaying relations.
pub fn load() -> Result<Store> {
let nodes_p = nodes_path();
let rels_p = relations_path();
let mut store = Store::default();
// Open redb index first (rebuilds from capnp if needed)
let db_p = db_path();
store.db = Some(store.open_or_rebuild_db(&db_p)?);
// Replay relations
if rels_p.exists() {
store.replay_relations(&rels_p)?;
}
// Record log sizes
use std::sync::atomic::Ordering;
store.loaded_nodes_size.store(
fs::metadata(&nodes_p).map(|m| m.len()).unwrap_or(0),
Ordering::Relaxed
);
store.loaded_rels_size.store(
fs::metadata(&rels_p).map(|m| m.len()).unwrap_or(0),
Ordering::Relaxed
);
// Orphan edges filtered naturally during for_each_relation (unresolvable UUIDs skipped)
Ok(store)
}
/// Open redb database, rebuilding if unhealthy.
fn open_or_rebuild_db(&self, path: &Path) -> Result<redb::Database> {
// Try opening existing database
if path.exists() {
match index::open_db(path) {
Ok(database) => {
if self.db_is_healthy(&database)? {
return Ok(database);
}
eprintln!("redb index stale, rebuilding...");
}
Err(e) => {
eprintln!("redb open failed ({}), rebuilding...", e);
}
}
}
// Rebuild index from capnp log
rebuild_index(path, &nodes_path())
}
/// Check if redb index is healthy by verifying some offsets are valid.
fn db_is_healthy(&self, database: &redb::Database) -> Result<bool> {
use redb::{ReadableDatabase, ReadableTable};
let txn = database.begin_read()?;
let nodes_table = txn.open_table(index::NODES)?;
// Check that we can read the table and it has entries
if nodes_table.len()? == 0 {
// Empty database - might be stale or new
let capnp_size = fs::metadata(nodes_path()).map(|m| m.len()).unwrap_or(0);
return Ok(capnp_size == 0); // healthy only if capnp is also empty
}
// Spot check: verify a few offsets point to valid messages
let mut checked = 0;
for entry in nodes_table.iter()? {
if checked >= 5 { break; }
let (key, offset) = entry?;
let offset = offset.value();
// Try to read the node at this offset
if read_node_at_offset(offset).is_err() {
return Ok(false);
}
checked += 1;
let _ = key; // silence unused warning
}
Ok(true)
}
/// Replay relation log, keeping latest version per UUID
fn replay_relations(&mut self, path: &Path) -> Result<()> {
let file = fs::File::open(path)
.with_context(|| format!("open {}", path.display()))?;
let mut reader = BufReader::new(file);
// Collect all, then deduplicate by UUID keeping latest version
let mut by_uuid: HashMap<[u8; 16], Relation> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::relation_log::Reader>()
.with_context(|| format!("read relation log"))?;
for rel_reader in log.get_relations()
.with_context(|| format!("get relations"))? {
let rel = Relation::from_capnp_migrate(rel_reader)?;
let existing_version = by_uuid.get(&rel.uuid)
.map(|r| r.version)
.unwrap_or(0);
if rel.version >= existing_version {
by_uuid.insert(rel.uuid, rel);
}
}
}
// Index relations directly (single transaction)
if let Some(db) = &self.db {
let txn = db.begin_write()?;
for rel in by_uuid.into_values() {
if rel.deleted { continue; }
index::index_relation(&txn, &rel.source, &rel.target, rel.strength, rel.rel_type as u8)?;
}
txn.commit()?;
}
Ok(())
}
/// Find all duplicate keys: keys with multiple live UUIDs in the log.
/// Returns a map from key → vec of all live Node versions (one per UUID).
pub fn find_duplicates(&self) -> Result<HashMap<String, Vec<Node>>> {
let path = nodes_path();
if !path.exists() { return Ok(HashMap::new()); }
let file = fs::File::open(&path)
.with_context(|| format!("open {}", path.display()))?;
let mut reader = BufReader::new(file);
// Track latest version of each UUID
let mut by_uuid: HashMap<[u8; 16], Node> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.with_context(|| format!("read node log"))?;
for node_reader in log.get_nodes()
.with_context(|| format!("get nodes"))? {
let node = Node::from_capnp_migrate(node_reader)?;
let dominated = by_uuid.get(&node.uuid)
.map(|n| node.version >= n.version)
.unwrap_or(true);
if dominated {
by_uuid.insert(node.uuid, node);
}
}
}
// Group live (non-deleted) nodes by key
let mut by_key: HashMap<String, Vec<Node>> = HashMap::new();
for node in by_uuid.into_values() {
if !node.deleted {
by_key.entry(node.key.clone()).or_default().push(node);
}
}
// Keep only duplicates
by_key.retain(|_, nodes| nodes.len() > 1);
Ok(by_key)
}
/// Append nodes to the log file. Returns the offset where the message was written.
pub fn append_nodes(&self, nodes: &[Node]) -> Result<u64> {
use std::sync::atomic::Ordering;
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::node_log::Builder>();
let mut list = log.init_nodes(nodes.len() as u32);
for (i, node) in nodes.iter().enumerate() {
node.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.with_context(|| format!("serialize nodes"))?;
// Lock for file append
let _guard = self.append_lock.lock().unwrap();
let path = nodes_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.with_context(|| format!("open {}", path.display()))?;
// Get offset before writing
let offset = file.metadata().map(|m| m.len()).unwrap_or(0);
use std::io::Write;
(&file).write_all(&buf)
.with_context(|| format!("write nodes"))?;
self.loaded_nodes_size.store(
file.metadata().map(|m| m.len()).unwrap_or(0),
Ordering::Relaxed
);
Ok(offset)
}
/// Append relations to the log file.
pub fn append_relations(&self, relations: &[Relation]) -> Result<()> {
use std::sync::atomic::Ordering;
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::relation_log::Builder>();
let mut list = log.init_relations(relations.len() as u32);
for (i, rel) in relations.iter().enumerate() {
rel.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.with_context(|| format!("serialize relations"))?;
// Lock for file append
let _guard = self.append_lock.lock().unwrap();
let path = relations_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.with_context(|| format!("open {}", path.display()))?;
use std::io::Write;
(&file).write_all(&buf)
.with_context(|| format!("write relations"))?;
self.loaded_rels_size.store(
file.metadata().map(|m| m.len()).unwrap_or(0),
Ordering::Relaxed
);
Ok(())
}
/// Placeholder - indices will be updated on write with redb.
pub fn save(&self) -> Result<()> {
Ok(())
}
}
/// Check and repair corrupt capnp log files.
///
/// Reads each message sequentially, tracking file position. On the first
/// corrupt message, truncates the file to the last good position. Also
/// removes stale caches so the next load replays from the repaired log.
pub fn fsck() -> Result<()> {
let mut any_corrupt = false;
for (path, kind) in [
(nodes_path(), "node"),
(relations_path(), "relation"),
] {
if !path.exists() { continue; }
let file = fs::File::open(&path)
.with_context(|| format!("open {}", path.display()))?;
let file_len = file.metadata()
.with_context(|| format!("stat {}", path.display()))?.len();
let mut reader = BufReader::new(file);
let mut good_messages = 0u64;
let mut last_good_pos = 0u64;
loop {
let pos = reader.stream_position()
.with_context(|| format!("tell {}", path.display()))?;
let msg = match serialize::read_message(&mut reader, message::ReaderOptions::new()) {
Ok(m) => m,
Err(_) => {
// read_message fails at EOF (normal) or on corrupt framing
if pos < file_len {
// Not at EOF — corrupt framing
eprintln!("{}: corrupt message at offset {}, truncating", kind, pos);
any_corrupt = true;
drop(reader);
let file = fs::OpenOptions::new().write(true).open(&path)
.with_context(|| format!("open for truncate"))?;
file.set_len(pos)
.with_context(|| format!("truncate {}", path.display()))?;
eprintln!("{}: truncated from {} to {} bytes ({} good messages)",
kind, file_len, pos, good_messages);
}
break;
}
};
// Validate the message content too
let valid = if kind == "node" {
msg.get_root::<memory_capnp::node_log::Reader>()
.and_then(|l| l.get_nodes().map(|_| ()))
.is_ok()
} else {
msg.get_root::<memory_capnp::relation_log::Reader>()
.and_then(|l| l.get_relations().map(|_| ()))
.is_ok()
};
if valid {
good_messages += 1;
last_good_pos = reader.stream_position()
.with_context(|| format!("tell {}", path.display()))?;
} else {
eprintln!("{}: corrupt message content at offset {}, truncating to {}",
kind, pos, last_good_pos);
any_corrupt = true;
drop(reader);
let file = fs::OpenOptions::new().write(true).open(&path)
.with_context(|| format!("open for truncate"))?;
file.set_len(last_good_pos)
.with_context(|| format!("truncate {}", path.display()))?;
eprintln!("{}: truncated from {} to {} bytes ({} good messages)",
kind, file_len, last_good_pos, good_messages);
break;
}
}
if !any_corrupt {
eprintln!("{}: {} messages, all clean", kind, good_messages);
}
}
if any_corrupt {
eprintln!("repair complete — run `poc-memory status` to verify");
} else {
eprintln!("store is clean");
}
Ok(())
}
/// Rebuild redb index from capnp log.
/// Scans the log, tracking offsets, and records latest version of each node.
fn rebuild_index(db_path: &Path, capnp_path: &Path) -> Result<redb::Database> {
// Remove old database if it exists
if db_path.exists() {
fs::remove_file(db_path)
.with_context(|| format!("remove old db {}", db_path.display()))?;
}
let database = index::open_db(db_path)?;
if !capnp_path.exists() {
return Ok(database);
}
// Track latest (offset, uuid, version, deleted, node_type, timestamp) per key
let mut latest: HashMap<String, (u64, [u8; 16], u32, bool, u8, i64)> = HashMap::new();
let file = fs::File::open(capnp_path)
.with_context(|| format!("open {}", capnp_path.display()))?;
let mut reader = BufReader::new(file);
loop {
let offset = reader.stream_position()?;
let msg = match serialize::read_message(&mut reader, message::ReaderOptions::new()) {
Ok(m) => m,
Err(_) => break,
};
let log = match msg.get_root::<memory_capnp::node_log::Reader>() {
Ok(l) => l,
Err(_) => continue,
};
let nodes = match log.get_nodes() {
Ok(n) => n,
Err(_) => continue,
};
for node_reader in nodes {
let key = node_reader.get_key().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
if key.is_empty() { continue; }
let version = node_reader.get_version();
let deleted = node_reader.get_deleted();
let node_type = node_reader.get_node_type()
.map(|t| t as u8)
.unwrap_or(0);
let timestamp = node_reader.get_timestamp();
let mut uuid = [0u8; 16];
if let Ok(data) = node_reader.get_uuid() {
if data.len() >= 16 {
uuid.copy_from_slice(&data[..16]);
}
}
// Keep if newer version
let dominated = latest.get(&key)
.map(|(_, _, v, _, _, _)| version >= *v)
.unwrap_or(true);
if dominated {
latest.insert(key, (offset, uuid, version, deleted, node_type, timestamp));
}
}
}
// Write index entries for non-deleted nodes
{
let txn = database.begin_write()?;
{
let mut nodes_table = txn.open_table(index::NODES)?;
let mut key_uuid_table = txn.open_table(index::KEY_TO_UUID)?;
let mut uuid_offsets = txn.open_multimap_table(index::UUID_OFFSETS)?;
for (key, (offset, uuid, _, deleted, node_type, timestamp)) in latest {
if !deleted {
nodes_table.insert(key.as_str(), offset)?;
// Pack: [uuid:16][node_type:1][timestamp:8] = 25 bytes
let mut packed = [0u8; 25];
packed[0..16].copy_from_slice(&uuid);
packed[16] = node_type;
packed[17..25].copy_from_slice(&timestamp.to_be_bytes());
key_uuid_table.insert(key.as_str(), packed.as_slice())?;
}
// Always record offset in UUID history (even for deleted)
uuid_offsets.insert(uuid.as_slice(), offset)?;
}
}
txn.commit()?;
}
Ok(database)
}

249
src/hippocampus/store/index.rs
View file

@ -1,249 +0,0 @@
// redb index tables
//
// capnp logs are source of truth; redb provides indexed access.
//
// Node tables:
// NODES: key → offset (current version)
// KEY_TO_UUID: key → uuid
// UUID_OFFSETS: uuid → offsets (multimap, all versions)
// NODES_BY_PROVENANCE: provenance → keys (multimap)
// NODES_BY_TYPE: [type_byte][timestamp_be] → key (for range queries by type+date)
//
// Relation tables:
// RELS: node_uuid → (other_uuid, strength, rel_type, is_outgoing) packed (multimap)
// Each relation stored twice — once per endpoint with direction bit.
//
// To get key from uuid: UUID_OFFSETS → read_node_at_offset() → node.key
use anyhow::{Context, Result};
use redb::{Database, MultimapTableDefinition, ReadableDatabase, ReadableTable, TableDefinition, WriteTransaction};
use std::path::Path;
// Node tables
pub const NODES: TableDefinition<&str, u64> = TableDefinition::new("nodes");
// KEY_TO_UUID: key → [uuid:16][node_type:1][timestamp:8] = 25 bytes
pub const KEY_TO_UUID: TableDefinition<&str, &[u8]> = TableDefinition::new("key_to_uuid");
pub const UUID_OFFSETS: MultimapTableDefinition<&[u8], u64> = MultimapTableDefinition::new("uuid_offsets");
pub const NODES_BY_PROVENANCE: MultimapTableDefinition<&str, &str> = MultimapTableDefinition::new("nodes_by_provenance");
// Composite key: [node_type: u8][timestamp: i64 BE] for range queries
pub const NODES_BY_TYPE: TableDefinition<&[u8], &str> = TableDefinition::new("nodes_by_type");
// Relations table - each relation stored twice (once per endpoint)
// Value: (other_uuid: [u8;16], strength: f32, rel_type: u8, is_outgoing: bool)
// Packed as 22 bytes: [other_uuid:16][strength:4][rel_type:1][is_outgoing:1]
pub const RELS: MultimapTableDefinition<&[u8], &[u8]> = MultimapTableDefinition::new("rels");
/// Open or create the redb database, ensuring all tables exist.
pub fn open_db(path: &Path) -> Result<Database> {
let db = Database::create(path)
.with_context(|| format!("create redb {}", path.display()))?;
// Ensure tables exist by opening a write transaction
let txn = db.begin_write()?;
{
// Node tables
let _ = txn.open_table(NODES)?;
let _ = txn.open_table(KEY_TO_UUID)?;
let _ = txn.open_multimap_table(UUID_OFFSETS)?;
let _ = txn.open_multimap_table(NODES_BY_PROVENANCE)?;
let _ = txn.open_table(NODES_BY_TYPE)?;
// Relations
let _ = txn.open_multimap_table(RELS)?;
}
txn.commit()?;
Ok(db)
}
/// Pack node metadata: [uuid:16][node_type:1][timestamp:8] = 25 bytes
fn pack_node_meta(uuid: &[u8; 16], node_type: u8, timestamp: i64) -> [u8; 25] {
let mut buf = [0u8; 25];
buf[0..16].copy_from_slice(uuid);
buf[16] = node_type;
buf[17..25].copy_from_slice(&timestamp.to_be_bytes());
buf
}
/// Unpack node metadata. Handles both old (16-byte) and new (25-byte) formats.
pub fn unpack_node_meta(data: &[u8]) -> ([u8; 16], u8, i64) {
let mut uuid = [0u8; 16];
uuid.copy_from_slice(&data[0..16]);
if data.len() >= 25 {
let node_type = data[16];
let timestamp = i64::from_be_bytes([
data[17], data[18], data[19], data[20],
data[21], data[22], data[23], data[24],
]);
(uuid, node_type, timestamp)
} else {
// Old format: just uuid, default metadata
(uuid, 0, 0)
}
}
/// Record a node's location in the index.
pub fn index_node(txn: &WriteTransaction, key: &str, offset: u64, uuid: &[u8; 16], node_type: u8, timestamp: i64) -> Result<()> {
let mut nodes_table = txn.open_table(NODES)?;
let mut key_uuid_table = txn.open_table(KEY_TO_UUID)?;
let mut uuid_offsets = txn.open_multimap_table(UUID_OFFSETS)?;
nodes_table.insert(key, offset)?;
let packed = pack_node_meta(uuid, node_type, timestamp);
key_uuid_table.insert(key, packed.as_slice())?;
uuid_offsets.insert(uuid.as_slice(), offset)?;
Ok(())
}
/// Get offset for a node by key.
pub fn get_offset(db: &Database, key: &str) -> Result<Option<u64>> {
let txn = db.begin_read()?;
let table = txn.open_table(NODES)?;
Ok(table.get(key)?.map(|v| v.value()))
}
/// Check if a key exists in the index.
pub fn contains_key(db: &Database, key: &str) -> Result<bool> {
let txn = db.begin_read()?;
let table = txn.open_table(NODES)?;
Ok(table.get(key)?.is_some())
}
/// Get a node's UUID from its key.
pub fn get_uuid_for_key(db: &Database, key: &str) -> Result<Option<[u8; 16]>> {
let txn = db.begin_read()?;
let table = txn.open_table(KEY_TO_UUID)?;
match table.get(key)? {
Some(data) => {
let (uuid, _, _) = unpack_node_meta(data.value());
Ok(Some(uuid))
}
None => Ok(None),
}
}
/// Get all offsets for a UUID (all versions). Returns newest first.
pub fn get_offsets_for_uuid(db: &Database, uuid: &[u8; 16]) -> Result<Vec<u64>> {
let txn = db.begin_read()?;
let table = txn.open_multimap_table(UUID_OFFSETS)?;
let mut offsets = Vec::new();
for entry in table.get(uuid.as_slice())? {
offsets.push(entry?.value());
}
// Sort descending so newest (highest offset) is first
offsets.sort_by(|a, b| b.cmp(a));
Ok(offsets)
}
/// Remove a node from the index (key mappings only; UUID history preserved).
pub fn remove_node(txn: &WriteTransaction, key: &str) -> Result<()> {
let mut nodes_table = txn.open_table(NODES)?;
let mut key_uuid_table = txn.open_table(KEY_TO_UUID)?;
// Note: UUID_OFFSETS is not cleared - preserves version history
nodes_table.remove(key)?;
key_uuid_table.remove(key)?;
Ok(())
}
/// Collect all keys from the index.
pub fn all_keys(db: &Database) -> Result<Vec<String>> {
let txn = db.begin_read()?;
let table = txn.open_table(NODES)?;
let mut keys = Vec::new();
for entry in table.iter()? {
let (key, _) = entry?;
keys.push(key.value().to_string());
}
Ok(keys)
}
/// Collect all (key, uuid, node_type, timestamp) in a single table scan.
pub fn all_key_uuid_pairs(db: &Database) -> Result<Vec<(String, [u8; 16], u8, i64)>> {
let txn = db.begin_read()?;
let table = txn.open_table(KEY_TO_UUID)?;
let mut pairs = Vec::new();
for entry in table.iter()? {
let (key, data) = entry?;
let (uuid, node_type, timestamp) = unpack_node_meta(data.value());
pairs.push((key.value().to_string(), uuid, node_type, timestamp));
}
Ok(pairs)
}
// ── Relation index operations ──────────────────────────────────────
//
// RELS value format: [other_uuid:16][strength:4][rel_type:1][is_outgoing:1] = 22 bytes
/// Pack relation data into bytes for RELS table.
fn pack_rel(other_uuid: &[u8; 16], strength: f32, rel_type: u8, is_outgoing: bool) -> [u8; 22] {
let mut buf = [0u8; 22];
buf[0..16].copy_from_slice(other_uuid);
buf[16..20].copy_from_slice(&strength.to_be_bytes());
buf[20] = rel_type;
buf[21] = if is_outgoing { 1 } else { 0 };
buf
}
/// Unpack relation data from RELS table.
pub fn unpack_rel(data: &[u8]) -> ([u8; 16], f32, u8, bool) {
let mut other_uuid = [0u8; 16];
other_uuid.copy_from_slice(&data[0..16]);
let strength = f32::from_be_bytes([data[16], data[17], data[18], data[19]]);
let rel_type = data[20];
let is_outgoing = data[21] != 0;
(other_uuid, strength, rel_type, is_outgoing)
}
/// Index a relation: store twice (once per endpoint).
pub fn index_relation(
txn: &WriteTransaction,
source_uuid: &[u8; 16],
target_uuid: &[u8; 16],
strength: f32,
rel_type: u8,
) -> Result<()> {
let mut rels = txn.open_multimap_table(RELS)?;
// Store outgoing: source → (target, strength, type, true)
let outgoing = pack_rel(target_uuid, strength, rel_type, true);
rels.insert(source_uuid.as_slice(), outgoing.as_slice())?;
// Store incoming: target → (source, strength, type, false)
let incoming = pack_rel(source_uuid, strength, rel_type, false);
rels.insert(target_uuid.as_slice(), incoming.as_slice())?;
Ok(())
}
/// Remove a relation from the index.
pub fn remove_relation(
txn: &WriteTransaction,
source_uuid: &[u8; 16],
target_uuid: &[u8; 16],
strength: f32,
rel_type: u8,
) -> Result<()> {
let mut rels = txn.open_multimap_table(RELS)?;
let outgoing = pack_rel(target_uuid, strength, rel_type, true);
rels.remove(source_uuid.as_slice(), outgoing.as_slice())?;
let incoming = pack_rel(source_uuid, strength, rel_type, false);
rels.remove(target_uuid.as_slice(), incoming.as_slice())?;
Ok(())
}
/// Get all edges for a node. Returns (other_uuid, strength, rel_type, is_outgoing).
pub fn edges_for_node(db: &Database, node_uuid: &[u8; 16]) -> Result<Vec<([u8; 16], f32, u8, bool)>> {
let txn = db.begin_read()?;
let rels = txn.open_multimap_table(RELS)?;
let mut edges = Vec::new();
for entry in rels.get(node_uuid.as_slice())? {
let guard = entry?;
let slice = guard.value();
let mut data = [0u8; 22];
data.copy_from_slice(slice);
edges.push(unpack_rel(&data));
}
Ok(edges)
}

435
src/hippocampus/store/mod.rs
View file

@ -1,41 +1,50 @@
// Append-only Cap'n Proto storage + redb indices
// Append-only Cap'n Proto storage + derived KV cache
//
// capnp logs are the source of truth:
// Two log files are source of truth:
// nodes.capnp - ContentNode messages
// relations.capnp - Relation messages
//
// redb provides indexed access; Store struct holds in-memory state.
// The Store struct is the derived cache: latest version per UUID,
// rebuilt from logs when stale. Three-tier load strategy:
// 1. rkyv mmap snapshot (snapshot.rkyv) — ~4ms deserialize
// 2. bincode cache (state.bin) — ~10ms
// 3. capnp log replay — ~40ms
// Staleness: log file sizes embedded in cache headers.
//
// Module layout:
// types.rs — Node, Relation, enums, path/time helpers
// capnp.rs — serialization macros, log IO (load, replay, append, fsck)
// index.rs — redb index operations
// ops.rs — mutations (upsert, delete, rename, etc.)
// view.rs — StoreView trait for read-only access
// types.rs — Node, Relation, enums, capnp macros, path helpers
// parse.rs — markdown → MemoryUnit parsing
// view.rs — zero-copy read-only access (StoreView, MmapView)
// persist.rs — load, save, replay, append, snapshot (all disk IO)
// ops.rs — mutations (upsert, delete, decay, cap_degree, etc.)
// mod.rs — re-exports, key resolution, ingestion, rendering
mod types;
mod index;
mod capnp;
mod ops;
mod parse;
mod view;
mod persist;
mod ops;
// Re-export everything callers need
pub use types::{
memory_dir, nodes_path,
now_epoch, epoch_to_local, format_date, format_datetime, format_datetime_space, compact_timestamp, today,
Node, Relation, NodeType, RelationType,
Node, Relation, NodeType, Provenance, RelationType,
RetrievalEvent, Params, GapRecord, Store,
new_node, new_relation,
};
pub use view::StoreView;
pub use capnp::fsck;
pub use parse::{MemoryUnit, parse_units};
pub use view::{StoreView, AnyView};
pub use persist::fsck;
pub use ops::current_provenance;
use crate::graph::{self, Graph};
use anyhow::{bail, Result};
use redb::Database;
use std::sync::atomic::AtomicU64;
use std::sync::Mutex;
use std::fs;
use std::io::Write as IoWrite;
use std::path::Path;
use parse::classify_filename;
/// Strip .md suffix from a key, handling both bare keys and section keys.
/// "identity.md" → "identity", "foo.md#section" → "foo#section", "identity" → "identity"
@ -48,149 +57,291 @@ pub fn strip_md_suffix(key: &str) -> String {
}
}
// The full in-memory store with internal locking
pub struct Store {
/// Log sizes at load time — used for staleness detection.
loaded_nodes_size: AtomicU64,
loaded_rels_size: AtomicU64,
/// Protects capnp log appends (redb handles its own locking)
append_lock: Mutex<()>,
/// redb index database
pub(crate) db: Option<redb::Database>,
}
impl Default for Store {
fn default() -> Self {
Store {
loaded_nodes_size: AtomicU64::new(0),
loaded_rels_size: AtomicU64::new(0),
append_lock: Mutex::new(()),
db: None,
}
}
}
impl Store {
pub fn build_graph(&self) -> Graph {
graph::build_graph(self)
}
/// Get a node by key, reading from capnp via the index.
pub fn get_node(&self, key: &str) -> Result<Option<Node>> {
let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
match index::get_offset(db, key)? {
Some(offset) => Ok(Some(capnp::read_node_at_offset(offset)?)),
None => Ok(None),
}
}
/// Check if a node exists by key.
pub fn contains_key(&self, key: &str) -> Result<bool> {
let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
index::contains_key(db, key)
}
/// Get all node keys.
pub fn all_keys(&self) -> Result<Vec<String>> {
let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
index::all_keys(db)
}
/// Get neighbors of a node: (key, strength) pairs.
pub fn neighbors(&self, key: &str) -> Result<Vec<(String, f32)>> {
let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
let uuid = match index::get_uuid_for_key(db, key)? {
Some(u) => u,
None => return Ok(Vec::new()),
};
let edges = index::edges_for_node(db, &uuid)?;
let mut neighbors = Vec::new();
for (other_uuid, strength, _, _) in edges {
// Look up key for other_uuid
let offsets = index::get_offsets_for_uuid(db, &other_uuid)?;
if offsets.is_empty() { continue; }
match capnp::read_node_at_offset(offsets[0]) {
Ok(n) if !n.deleted => neighbors.push((n.key, strength)),
_ => continue,
}
}
Ok(neighbors)
}
/// Get the database for transaction management.
pub fn db(&self) -> Result<&Database> {
self.db.as_ref().ok_or_else(|| anyhow::anyhow!("store not loaded"))
}
/// Remove a node from the index (used after appending a tombstone).
/// For batched operations, use index::remove_node with a WriteTransaction directly.
pub fn remove_from_index(&self, key: &str) -> Result<()> {
let db = self.db()?;
let txn = db.begin_write()?;
index::remove_node(&txn, key)?;
txn.commit()?;
Ok(())
}
/// Get all edges for a node by UUID. Returns (other_uuid, strength, rel_type, is_outgoing).
pub fn edges_for_uuid(&self, uuid: &[u8; 16]) -> Result<Vec<([u8; 16], f32, u8, bool)>> {
let db = self.db()?;
index::edges_for_node(db, uuid)
}
/// Add a relation to the index (opens its own transaction).
/// For batched operations, use index::index_relation with a WriteTransaction directly.
pub fn index_relation(&self, source: &[u8; 16], target: &[u8; 16], strength: f32, rel_type: u8) -> Result<()> {
let db = self.db()?;
let txn = db.begin_write()?;
index::index_relation(&txn, source, target, strength, rel_type)?;
txn.commit()?;
Ok(())
}
/// Remove a relation from the index (opens its own transaction).
/// For batched operations, use index::remove_relation with a WriteTransaction directly.
pub fn remove_relation_from_index(&self, source: &[u8; 16], target: &[u8; 16], strength: f32, rel_type: u8) -> Result<()> {
let db = self.db()?;
let txn = db.begin_write()?;
index::remove_relation(&txn, source, target, strength, rel_type)?;
txn.commit()?;
Ok(())
}
pub fn resolve_key(&self, target: &str) -> Result<String> {
pub fn resolve_key(&self, target: &str) -> Result<String, String> {
// Strip .md suffix if present — keys no longer use it
let bare = strip_md_suffix(target);
if self.contains_key(&bare)? {
if self.nodes.contains_key(&bare) {
return Ok(bare);
}
let db = self.db.as_ref()
.ok_or_else(|| anyhow::anyhow!("store not loaded"))?;
let all_keys = index::all_keys(db)?;
let matches: Vec<_> = all_keys.iter()
let matches: Vec<_> = self.nodes.keys()
.filter(|k| k.to_lowercase().contains(&target.to_lowercase()))
.cloned().collect();
match matches.len() {
0 => bail!("No entry for '{}'. Run 'init'?", target),
0 => Err(format!("No entry for '{}'. Run 'init'?", target)),
1 => Ok(matches[0].clone()),
n if n <= 10 => {
let list = matches.join("\n ");
bail!("Ambiguous '{}'. Matches:\n {}", target, list)
Err(format!("Ambiguous '{}'. Matches:\n {}", target, list))
}
n => bail!("Too many matches for '{}' ({}). Be more specific.", target, n),
n => Err(format!("Too many matches for '{}' ({}). Be more specific.", target, n)),
}
}
/// Resolve a link target to (key, uuid).
fn resolve_node_uuid(&self, target: &str) -> Option<(String, [u8; 16])> {
let bare = strip_md_suffix(target);
let n = self.nodes.get(&bare)?;
Some((bare, n.uuid))
}
/// Append retrieval event to retrieval.log without needing a Store instance.
pub fn log_retrieval_static(query: &str, results: &[String]) {
let path = memory_dir().join("retrieval.log");
let line = format!("[{}] q=\"{}\" hits={}\n", today(), query, results.len());
if let Ok(mut f) = fs::OpenOptions::new()
.create(true).append(true).open(&path) {
let _ = f.write_all(line.as_bytes());
}
}
/// Scan markdown files and index all memory units
pub fn init_from_markdown(&mut self) -> Result<usize, String> {
let dir = memory_dir();
let mut count = 0;
if dir.exists() {
// Build edge set for O(1) dedup during ingestion
let mut edge_set = self.build_edge_set();
count = self.scan_dir_for_init(&dir, &mut edge_set)?;
}
Ok(count)
}
/// Build a HashSet of existing (source, target) UUID pairs for O(1) dedup.
fn build_edge_set(&self) -> std::collections::HashSet<([u8; 16], [u8; 16])> {
let mut set = std::collections::HashSet::with_capacity(self.relations.len() * 2);
for r in &self.relations {
set.insert((r.source, r.target));
set.insert((r.target, r.source));
}
set
}
fn scan_dir_for_init(
&mut self,
dir: &Path,
edge_set: &mut std::collections::HashSet<([u8; 16], [u8; 16])>,
) -> Result<usize, String> {
let mut count = 0;
let entries = fs::read_dir(dir)
.map_err(|e| format!("read dir {}: {}", dir.display(), e))?;
for entry in entries.flatten() {
let path = entry.path();
if path.is_dir() {
count += self.scan_dir_for_init(&path, edge_set)?;
continue;
}
let Some(ext) = path.extension() else { continue };
if ext != "md" { continue }
let filename = path.file_name().unwrap().to_string_lossy().to_string();
let content = fs::read_to_string(&path)
.map_err(|e| format!("read {}: {}", path.display(), e))?;
let units = parse_units(&filename, &content);
let (new_count, _) = self.ingest_units(&units, &filename)?;
count += new_count;
// Create relations from links
let mut new_relations = Vec::new();
for unit in &units {
let source_uuid = match self.nodes.get(&unit.key) {
Some(n) => n.uuid,
None => continue,
};
for link in unit.marker_links.iter().chain(unit.md_links.iter()) {
let Some((key, uuid)) = self.resolve_node_uuid(link) else { continue };
if !edge_set.contains(&(source_uuid, uuid)) {
edge_set.insert((source_uuid, uuid));
edge_set.insert((uuid, source_uuid));
new_relations.push(new_relation(
source_uuid, uuid, RelationType::Link, 1.0,
&unit.key, &key,
));
}
}
for cause in &unit.causes {
let Some((key, uuid)) = self.resolve_node_uuid(cause) else { continue };
if !edge_set.contains(&(uuid, source_uuid)) {
edge_set.insert((uuid, source_uuid));
new_relations.push(new_relation(
uuid, source_uuid, RelationType::Causal, 1.0,
&key, &unit.key,
));
}
}
}
if !new_relations.is_empty() {
self.append_relations(&new_relations)?;
self.relations.extend(new_relations);
}
}
Ok(count)
}
/// Process parsed memory units: diff against existing nodes, persist changes.
/// Holds StoreLock across refresh + check + write to prevent duplicate UUIDs.
fn ingest_units(&mut self, units: &[MemoryUnit], filename: &str) -> Result<(usize, usize), String> {
let _lock = types::StoreLock::acquire()?;
self.refresh_nodes()?;
let node_type = classify_filename(filename);
let mut new_nodes = Vec::new();
let mut updated_nodes = Vec::new();
for (pos, unit) in units.iter().enumerate() {
if let Some(existing) = self.nodes.get(&unit.key) {
if existing.content != unit.content || existing.position != pos as u32 {
let mut node = existing.clone();
node.content = unit.content.clone();
node.position = pos as u32;
node.version += 1;
if let Some(ref s) = unit.state { node.state_tag = s.clone(); }
if let Some(ref s) = unit.source_ref { node.source_ref = s.clone(); }
updated_nodes.push(node);
}
} else {
let mut node = new_node(&unit.key, &unit.content);
node.node_type = node_type;
node.position = pos as u32;
if let Some(ref s) = unit.state { node.state_tag = s.clone(); }
if let Some(ref s) = unit.source_ref { node.source_ref = s.clone(); }
new_nodes.push(node);
}
}
if !new_nodes.is_empty() {
self.append_nodes_unlocked(&new_nodes)?;
for node in &new_nodes {
self.uuid_to_key.insert(node.uuid, node.key.clone());
self.nodes.insert(node.key.clone(), node.clone());
}
}
if !updated_nodes.is_empty() {
self.append_nodes_unlocked(&updated_nodes)?;
for node in &updated_nodes {
self.nodes.insert(node.key.clone(), node.clone());
}
}
Ok((new_nodes.len(), updated_nodes.len()))
}
/// Import a markdown file into the store, parsing it into nodes.
pub fn import_file(&mut self, path: &Path) -> Result<(usize, usize), String> {
let filename = path.file_name().unwrap().to_string_lossy().to_string();
let content = fs::read_to_string(path)
.map_err(|e| format!("read {}: {}", path.display(), e))?;
let units = parse_units(&filename, &content);
self.ingest_units(&units, &filename)
}
/// Gather all sections for a file key, sorted by position.
pub fn file_sections(&self, file_key: &str) -> Option<Vec<&Node>> {
let prefix = format!("{}#", file_key);
let mut sections: Vec<_> = self.nodes.values()
.filter(|n| n.key == file_key || n.key.starts_with(&prefix))
.collect();
if sections.is_empty() {
return None;
}
sections.sort_by_key(|n| n.position);
Some(sections)
}
/// Render a file key as plain content (no mem markers).
pub fn render_file(&self, file_key: &str) -> Option<String> {
let sections = self.file_sections(file_key)?;
let mut output = String::new();
for node in &sections {
output.push_str(&node.content);
if !node.content.ends_with('\n') {
output.push('\n');
}
output.push('\n');
}
Some(output.trim_end().to_string())
}
/// Render a file key back to markdown with reconstituted mem markers.
pub fn export_to_markdown(&self, file_key: &str) -> Option<String> {
let sections = self.file_sections(file_key)?;
let mut output = String::new();
for node in &sections {
if node.key.contains('#') {
let section_id = node.key.rsplit_once('#').map_or("", |(_, s)| s);
let links: Vec<_> = self.relations.iter()
.filter(|r| r.source_key == node.key && !r.deleted
&& r.rel_type != RelationType::Causal)
.map(|r| r.target_key.clone())
.collect();
let causes: Vec<_> = self.relations.iter()
.filter(|r| r.target_key == node.key && !r.deleted
&& r.rel_type == RelationType::Causal)
.map(|r| r.source_key.clone())
.collect();
let mut marker_parts = vec![format!("id={}", section_id)];
if !links.is_empty() {
marker_parts.push(format!("links={}", links.join(",")));
}
if !causes.is_empty() {
marker_parts.push(format!("causes={}", causes.join(",")));
}
output.push_str(&format!("<!-- mem: {} -->\n", marker_parts.join(" ")));
}
output.push_str(&node.content);
if !node.content.ends_with('\n') {
output.push('\n');
}
output.push('\n');
}
Some(output.trim_end().to_string())
}
/// Find the episodic node that best matches the given entry text.
pub fn find_journal_node(&self, entry_text: &str) -> Option<String> {
if entry_text.is_empty() {
return None;
}
let words: Vec<&str> = entry_text.split_whitespace()
.filter(|w| w.len() > 5)
.take(5)
.collect();
let mut best_key = None;
let mut best_score = 0;
for (key, node) in &self.nodes {
if node.node_type != NodeType::EpisodicSession {
continue;
}
let content_lower = node.content.to_lowercase();
let score: usize = words.iter()
.filter(|w| content_lower.contains(&w.to_lowercase()))
.count();
if score > best_score {
best_score = score;
best_key = Some(key.clone());
}
}
best_key
}
}

492
src/hippocampus/store/ops.rs
View file

@ -1,10 +1,10 @@
// Mutation operations on the store
//
// CRUD (upsert, delete), maintenance (decay, cap_degree), and graph metrics.
// CRUD (upsert, delete, modify), feedback tracking (mark_used, mark_wrong),
// maintenance (decay, fix_categories, cap_degree), and graph metrics.
use super::{capnp, index, types::*, Store};
use super::types::*;
use anyhow::{anyhow, bail, Result};
use std::collections::{HashMap, HashSet};
/// Fallback provenance for non-tool-dispatch paths (CLI, digest, etc.).
@ -15,50 +15,34 @@ pub fn current_provenance() -> String {
}
impl Store {
/// Add or update a node (appends to log + updates index).
pub fn upsert_node(&self, mut node: Node) -> Result<()> {
if let Some(existing) = self.get_node(&node.key)? {
/// Add or update a node (appends to log + updates cache).
/// Holds StoreLock across refresh + check + write to prevent duplicate UUIDs.
pub fn upsert_node(&mut self, mut node: Node) -> Result<(), String> {
let _lock = StoreLock::acquire()?;
self.refresh_nodes()?;
if let Some(existing) = self.nodes.get(&node.key) {
node.uuid = existing.uuid;
node.version = existing.version + 1;
}
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let txn = db.begin_write()?;
let offset = self.append_nodes(&[node.clone()])?;
index::index_node(&txn, &node.key, offset, &node.uuid, node.node_type as u8, node.timestamp)?;
txn.commit()?;
self.append_nodes_unlocked(&[node.clone()])?;
self.uuid_to_key.insert(node.uuid, node.key.clone());
self.nodes.insert(node.key.clone(), node);
Ok(())
}
/// Add a relation (appends to log + indexes)
pub fn add_relation(&self, rel: Relation) -> Result<()> {
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let txn = db.begin_write()?;
/// Add a relation (appends to log + updates cache)
pub fn add_relation(&mut self, rel: Relation) -> Result<(), String> {
self.append_relations(std::slice::from_ref(&rel))?;
index::index_relation(&txn, &rel.source, &rel.target, rel.strength, rel.rel_type as u8)?;
txn.commit()?;
self.relations.push(rel);
Ok(())
}
/// Recent nodes by provenance, sorted newest-first. Returns (key, timestamp).
pub fn recent_by_provenance(&self, provenance: &str, limit: usize) -> Vec<(String, i64)> {
let db = match self.db.as_ref() {
Some(db) => db,
None => return Vec::new(),
};
let keys = match index::all_keys(db) {
Ok(keys) => keys,
Err(_) => return Vec::new(),
};
let mut nodes: Vec<_> = keys.iter()
.filter_map(|key| {
let offset = index::get_offset(db, key).ok()??;
let node = capnp::read_node_at_offset(offset).ok()?;
if !node.deleted && node.provenance == provenance {
Some((key.clone(), node.timestamp))
} else {
None
}
})
let mut nodes: Vec<_> = self.nodes.values()
.filter(|n| !n.deleted && n.provenance == provenance)
.map(|n| (n.key.clone(), n.timestamp))
.collect();
nodes.sort_by(|a, b| b.1.cmp(&a.1));
nodes.truncate(limit);
@ -70,57 +54,56 @@ impl Store {
///
/// Provenance is determined by the POC_PROVENANCE env var if set,
/// otherwise defaults to Manual.
pub fn upsert(&self, key: &str, content: &str) -> Result<&'static str> {
pub fn upsert(&mut self, key: &str, content: &str) -> Result<&'static str, String> {
let prov = current_provenance();
self.upsert_provenance(key, content, &prov)
}
/// Upsert with explicit provenance (for agent-created nodes).
pub fn upsert_provenance(&self, key: &str, content: &str, provenance: &str) -> Result<&'static str> {
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
/// Holds StoreLock across refresh + check + write to prevent duplicate UUIDs.
pub fn upsert_provenance(&mut self, key: &str, content: &str, provenance: &str) -> Result<&'static str, String> {
let _lock = StoreLock::acquire()?;
self.refresh_nodes()?;
if let Some(existing) = self.get_node(key)? {
if let Some(existing) = self.nodes.get(key) {
if existing.content == content {
return Ok("unchanged");
}
let mut node = existing;
let mut node = existing.clone();
node.content = content.to_string();
node.provenance = provenance.to_string();
node.timestamp = now_epoch();
node.version += 1;
let txn = db.begin_write()?;
let offset = self.append_nodes(std::slice::from_ref(&node))?;
index::index_node(&txn, &node.key, offset, &node.uuid, node.node_type as u8, node.timestamp)?;
txn.commit()?;
self.append_nodes_unlocked(std::slice::from_ref(&node))?;
self.nodes.insert(key.to_string(), node);
Ok("updated")
} else {
let mut node = new_node(key, content);
node.provenance = provenance.to_string();
let txn = db.begin_write()?;
let offset = self.append_nodes(std::slice::from_ref(&node))?;
index::index_node(&txn, &node.key, offset, &node.uuid, node.node_type as u8, node.timestamp)?;
txn.commit()?;
self.append_nodes_unlocked(std::slice::from_ref(&node))?;
self.uuid_to_key.insert(node.uuid, node.key.clone());
self.nodes.insert(key.to_string(), node);
Ok("created")
}
}
/// Soft-delete a node (appends deleted version, removes from index).
pub fn delete_node(&self, key: &str) -> Result<()> {
let prov = current_provenance();
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
/// Soft-delete a node (appends deleted version, removes from cache).
/// Holds StoreLock across refresh + write to see concurrent creates.
pub fn delete_node(&mut self, key: &str) -> Result<(), String> {
let _lock = StoreLock::acquire()?;
self.refresh_nodes()?;
let node = self.get_node(key)?
.ok_or_else(|| anyhow!("No node '{}'", key))?;
let mut deleted = node;
let prov = current_provenance();
let node = self.nodes.get(key)
.ok_or_else(|| format!("No node '{}'", key))?;
let mut deleted = node.clone();
deleted.deleted = true;
deleted.version += 1;
deleted.provenance = prov;
deleted.timestamp = now_epoch();
let txn = db.begin_write()?;
self.append_nodes(std::slice::from_ref(&deleted))?;
index::remove_node(&txn, key)?;
txn.commit()?;
self.append_nodes_unlocked(std::slice::from_ref(&deleted))?;
self.nodes.remove(key);
Ok(())
}
@ -129,16 +112,23 @@ impl Store {
/// Graph edges (source/target UUIDs) are unaffected — they're already
/// UUID-based. We update the human-readable source_key/target_key strings
/// on relations, and created_at is preserved untouched.
pub fn rename_node(&self, old_key: &str, new_key: &str) -> Result<()> {
///
/// Appends: (new_key, v+1) + (old_key, deleted, v+1) + updated relations.
/// Holds StoreLock across refresh + write to prevent races.
pub fn rename_node(&mut self, old_key: &str, new_key: &str) -> Result<(), String> {
if old_key == new_key {
return Ok(());
}
if self.contains_key(new_key)? {
bail!("Key '{}' already exists", new_key);
let _lock = StoreLock::acquire()?;
self.refresh_nodes()?;
if self.nodes.contains_key(new_key) {
return Err(format!("Key '{}' already exists", new_key));
}
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let node = self.get_node(old_key)?
.ok_or_else(|| anyhow!("No node '{}'", old_key))?;
let node = self.nodes.get(old_key)
.ok_or_else(|| format!("No node '{}'", old_key))?
.clone();
let prov = current_provenance();
@ -156,239 +146,239 @@ impl Store {
tombstone.provenance = prov;
tombstone.timestamp = now_epoch();
// Find relations touching this node's UUID (read before txn)
let node_uuid = node.uuid;
let edges = index::edges_for_node(db, &node_uuid)?;
// Collect affected relations and update their debug key strings
let updated_rels: Vec<_> = self.relations.iter()
.filter(|r| r.source_key == old_key || r.target_key == old_key)
.map(|r| {
let mut r = r.clone();
r.version += 1;
if r.source_key == old_key { r.source_key = new_key.to_string(); }
if r.target_key == old_key { r.target_key = new_key.to_string(); }
r
})
.collect();
// Build uuid → key map for the other endpoints
let keys = index::all_keys(db)?;
let mut uuid_to_key: HashMap<[u8; 16], String> = HashMap::new();
for k in &keys {
if let Ok(Some(u)) = index::get_uuid_for_key(db, k) {
uuid_to_key.insert(u, k.clone());
}
}
uuid_to_key.insert(node_uuid, new_key.to_string());
let mut updated_rels = Vec::new();
for (other_uuid, strength, rel_type, is_outgoing) in edges {
let other_key = uuid_to_key.get(&other_uuid).cloned().unwrap_or_default();
let (src_uuid, tgt_uuid, src_key, tgt_key) = if is_outgoing {
(node_uuid, other_uuid, new_key.to_string(), other_key)
} else {
(other_uuid, node_uuid, other_key, new_key.to_string())
};
let mut rel = new_relation(src_uuid, tgt_uuid,
RelationType::from_u8(rel_type), strength,
&src_key, &tgt_key);
rel.version = 2; // indicate update
updated_rels.push(rel);
}
// Single transaction for all index updates
let txn = db.begin_write()?;
let offset = self.append_nodes(&[renamed.clone(), tombstone])?;
index::remove_node(&txn, old_key)?;
index::index_node(&txn, new_key, offset, &renamed.uuid, renamed.node_type as u8, renamed.timestamp)?;
// Persist under single lock
self.append_nodes_unlocked(&[renamed.clone(), tombstone])?;
if !updated_rels.is_empty() {
self.append_relations(&updated_rels)?;
self.append_relations_unlocked(&updated_rels)?;
}
// Update in-memory cache
self.nodes.remove(old_key);
self.uuid_to_key.insert(renamed.uuid, new_key.to_string());
self.nodes.insert(new_key.to_string(), renamed);
for updated in &updated_rels {
if let Some(r) = self.relations.iter_mut().find(|r| r.uuid == updated.uuid) {
r.source_key = updated.source_key.clone();
r.target_key = updated.target_key.clone();
r.version = updated.version;
}
}
txn.commit()?;
Ok(())
}
/// Modify a node in-place, bump version, and persist to capnp log.
fn modify_node(&mut self, key: &str, f: impl FnOnce(&mut Node)) -> Result<(), String> {
let node = self.nodes.get_mut(key)
.ok_or_else(|| format!("No node '{}'", key))?;
f(node);
node.version += 1;
let node = node.clone();
self.append_nodes(&[node])
}
pub fn mark_used(&mut self, key: &str) {
let boost = self.params.use_boost as f32;
let _ = self.modify_node(key, |n| {
n.uses += 1;
n.weight = (n.weight + boost).min(1.0);
if n.spaced_repetition_interval < 30 {
n.spaced_repetition_interval = match n.spaced_repetition_interval {
1 => 3, 3 => 7, 7 => 14, 14 => 30, _ => 30,
};
}
n.last_replayed = now_epoch();
});
}
pub fn mark_wrong(&mut self, key: &str, _ctx: Option<&str>) {
let _ = self.modify_node(key, |n| {
n.wrongs += 1;
n.weight = (n.weight - 0.1).max(0.0);
n.spaced_repetition_interval = 1;
});
}
pub fn record_gap(&mut self, desc: &str) {
self.gaps.push(GapRecord {
description: desc.to_string(),
timestamp: today(),
});
}
/// Cap node degree by soft-deleting edges from mega-hubs.
pub fn cap_degree(&self, max_degree: usize) -> Result<(usize, usize)> {
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let keys = index::all_keys(db)?;
// Build uuid ↔ key maps and count degrees in one pass
let mut uuid_to_key: HashMap<[u8; 16], String> = HashMap::new();
let mut node_info: Vec<(String, [u8; 16], usize)> = Vec::new(); // (key, uuid, degree)
for key in &keys {
if let Ok(Some(uuid)) = index::get_uuid_for_key(db, key) {
let degree = index::edges_for_node(db, &uuid)?.len();
uuid_to_key.insert(uuid, key.clone());
node_info.push((key.clone(), uuid, degree));
}
pub fn cap_degree(&mut self, max_degree: usize) -> Result<(usize, usize), String> {
let mut node_degree: HashMap<String, usize> = HashMap::new();
for rel in &self.relations {
if rel.deleted { continue; }
*node_degree.entry(rel.source_key.clone()).or_default() += 1;
*node_degree.entry(rel.target_key.clone()).or_default() += 1;
}
// Build degree lookup
let node_degree: HashMap<&str, usize> = node_info.iter()
.map(|(k, _, d)| (k.as_str(), *d))
.collect();
let mut node_edges: HashMap<String, Vec<usize>> = HashMap::new();
for (i, rel) in self.relations.iter().enumerate() {
if rel.deleted { continue; }
node_edges.entry(rel.source_key.clone()).or_default().push(i);
node_edges.entry(rel.target_key.clone()).or_default().push(i);
}
let mut to_delete: HashSet<([u8; 16], [u8; 16])> = HashSet::new();
let mut to_delete: HashSet<usize> = HashSet::new();
let mut hubs_capped = 0;
for (_key, uuid, degree) in &node_info {
if *degree <= max_degree { continue; }
let uuid = *uuid;
let edges = index::edges_for_node(db, &uuid)?;
if edges.len() <= max_degree { continue; }
for (_key, edge_indices) in &node_edges {
let active: Vec<usize> = edge_indices.iter()
.filter(|&&i| !to_delete.contains(&i))
.copied()
.collect();
if active.len() <= max_degree { continue; }
// Separate auto vs manual edges: (source, target, sort_key)
let mut auto_edges: Vec<([u8; 16], [u8; 16], f32)> = Vec::new();
let mut link_edges: Vec<([u8; 16], [u8; 16], usize)> = Vec::new();
for (other_uuid, strength, rel_type, is_outgoing) in &edges {
// Canonical edge direction
let (src, tgt) = if *is_outgoing { (uuid, *other_uuid) } else { (*other_uuid, uuid) };
if to_delete.contains(&(src, tgt)) || to_delete.contains(&(tgt, src)) { continue; }
let other_key = match uuid_to_key.get(other_uuid) {
Some(k) => k,
None => continue,
};
if *rel_type == RelationType::Auto as u8 {
auto_edges.push((src, tgt, *strength));
let mut auto_indices: Vec<(usize, f32)> = Vec::new();
let mut link_indices: Vec<(usize, usize)> = Vec::new();
for &i in &active {
let rel = &self.relations[i];
if rel.rel_type == RelationType::Auto {
auto_indices.push((i, rel.strength));
} else {
let other_deg = node_degree.get(other_key.as_str()).copied().unwrap_or(0);
link_edges.push((src, tgt, other_deg));
let other = if &rel.source_key == _key {
&rel.target_key
} else {
&rel.source_key
};
let other_deg = node_degree.get(other).copied().unwrap_or(0);
link_indices.push((i, other_deg));
}
}
let active_count = auto_edges.len() + link_edges.len();
if active_count <= max_degree { continue; }
let excess = active.len() - max_degree;
let excess = active_count - max_degree;
// Prune weakest auto edges first
auto_edges.sort_by(|a, b| a.2.total_cmp(&b.2));
for (src, tgt, _) in auto_edges.iter().take(excess) {
to_delete.insert((*src, *tgt));
auto_indices.sort_by(|a, b| a.1.total_cmp(&b.1));
let auto_prune = excess.min(auto_indices.len());
for &(i, _) in auto_indices.iter().take(auto_prune) {
to_delete.insert(i);
}
// Then prune links to highest-degree nodes
let remaining = excess.saturating_sub(auto_edges.len());
if remaining > 0 {
link_edges.sort_by(|a, b| b.2.cmp(&a.2));
for (src, tgt, _) in link_edges.iter().take(remaining) {
to_delete.insert((*src, *tgt));
let remaining_excess = excess.saturating_sub(auto_prune);
if remaining_excess > 0 {
link_indices.sort_by(|a, b| b.1.cmp(&a.1));
let link_prune = remaining_excess.min(link_indices.len());
for &(i, _) in link_indices.iter().take(link_prune) {
to_delete.insert(i);
}
}
hubs_capped += 1;
}
// Collect edge info for deletion
let mut to_remove: Vec<([u8; 16], [u8; 16], f32, u8, String, String)> = Vec::new();
for (source_uuid, target_uuid) in &to_delete {
let edges = index::edges_for_node(db, source_uuid)?;
if let Some((_, strength, rel_type, _)) = edges.iter()
.find(|(other, _, _, out)| *other == *target_uuid && *out)
{
let source_key = uuid_to_key.get(source_uuid).cloned().unwrap_or_default();
let target_key = uuid_to_key.get(target_uuid).cloned().unwrap_or_default();
to_remove.push((*source_uuid, *target_uuid, *strength, *rel_type, source_key, target_key));
}
let mut pruned_rels = Vec::new();
for &i in &to_delete {
self.relations[i].deleted = true;
self.relations[i].version += 1;
pruned_rels.push(self.relations[i].clone());
}
// Now mutate: remove from index and persist tombstones (single txn)
let pruned_count = to_remove.len();
if !to_remove.is_empty() {
let txn = db.begin_write()?;
for (source_uuid, target_uuid, strength, rel_type, source_key, target_key) in to_remove {
index::remove_relation(&txn, &source_uuid, &target_uuid, strength, rel_type)?;
let mut rel = new_relation(source_uuid, target_uuid,
RelationType::from_u8(rel_type), strength,
&source_key, &target_key);
rel.deleted = true;
rel.version = 2;
self.append_relations(std::slice::from_ref(&rel))?;
}
txn.commit()?;
if !pruned_rels.is_empty() {
self.append_relations(&pruned_rels)?;
}
Ok((hubs_capped, pruned_count))
self.relations.retain(|r| !r.deleted);
Ok((hubs_capped, to_delete.len()))
}
/// Set a node's weight directly. Returns (old, new).
pub fn set_weight(&self, key: &str, weight: f32) -> Result<(f32, f32)> {
pub fn set_weight(&mut self, key: &str, weight: f32) -> Result<(f32, f32), String> {
let weight = weight.clamp(0.01, 1.0);
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let mut node = self.get_node(key)?
.ok_or_else(|| anyhow!("node not found: {}", key))?;
let node = self.nodes.get_mut(key)
.ok_or_else(|| format!("node not found: {}", key))?;
let old = node.weight;
if (old - weight).abs() < 0.001 {
return Ok((old, weight)); // unchanged
}
node.weight = weight;
node.version += 1;
node.timestamp = now_epoch();
let txn = db.begin_write()?;
let offset = self.append_nodes(std::slice::from_ref(&node))?;
index::index_node(&txn, key, offset, &node.uuid, node.node_type as u8, node.timestamp)?;
txn.commit()?;
Ok((old, weight))
}
/// Set the strength of a link between two nodes.
/// Returns the old strength. Creates link if it doesn't exist.
pub fn set_link_strength(&self, source: &str, target: &str, strength: f32) -> Result<f32> {
let strength = strength.clamp(0.01, 1.0);
let source_uuid = self.get_node(source)?
.map(|n| n.uuid)
.ok_or_else(|| anyhow!("source not found: {}", source))?;
let target_uuid = self.get_node(target)?
.map(|n| n.uuid)
.ok_or_else(|| anyhow!("target not found: {}", target))?;
// Find existing edge via index (scope the borrow)
let existing = {
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let edges = index::edges_for_node(db, &source_uuid)?;
edges.iter().find(|(other, _, _, _)| *other == target_uuid)
.map(|(_, s, t, _)| (*s, *t))
};
if let Some((old_strength, rel_type)) = existing {
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let txn = db.begin_write()?;
// Remove old edge from index, add updated one
index::remove_relation(&txn, &source_uuid, &target_uuid, old_strength, rel_type)?;
index::index_relation(&txn, &source_uuid, &target_uuid, strength, rel_type)?;
// Append updated relation to log
let mut rel = new_relation(source_uuid, target_uuid,
RelationType::from_u8(rel_type), strength, source, target);
rel.version = 2; // indicate update
self.append_relations(std::slice::from_ref(&rel))?;
txn.commit()?;
Ok(old_strength)
} else {
// Create new link then update its strength
self.add_link(source, target, "link_set")?;
let db = self.db.as_ref().ok_or_else(|| anyhow!("store not loaded"))?;
let txn = db.begin_write()?;
index::remove_relation(&txn, &source_uuid, &target_uuid, 0.1, RelationType::Link as u8)?;
index::index_relation(&txn, &source_uuid, &target_uuid, strength, RelationType::Link as u8)?;
txn.commit()?;
Ok(0.0)
/// Update a node's weight with a new score and record the scoring
/// timestamp. Uses asymmetric smoothing: responds quickly to high
/// scores (alpha=0.5) but decays slowly on low scores (alpha=0.1).
/// This keeps memories surfaced even if they're only useful 1 in 4 times.
/// Returns (old_weight, new_weight).
pub fn score_weight(&mut self, key: &str, score: f64) -> Result<(f32, f32), String> {
let node = self.nodes.get_mut(key)
.ok_or_else(|| format!("node not found: {}", key))?;
let old = node.weight;
let alpha = if score > old as f64 { 0.5 } else { 0.1 };
let new = (alpha * score + (1.0 - alpha) * old as f64) as f32;
node.weight = new.clamp(0.01, 1.0);
node.last_scored = chrono::Utc::now().timestamp();
Ok((old, node.weight))
}
/// Set the strength of a link between two nodes. Deduplicates if
/// multiple links exist. Returns the old strength, or error if no link.
pub fn set_link_strength(&mut self, source: &str, target: &str, strength: f32) -> Result<f32, String> {
let strength = strength.clamp(0.01, 1.0);
let mut old = 0.0f32;
let mut found = false;
let mut first = true;
for rel in &mut self.relations {
if rel.deleted { continue; }
if (rel.source_key == source && rel.target_key == target)
|| (rel.source_key == target && rel.target_key == source)
{
if first {
old = rel.strength;
rel.strength = strength;
first = false;
} else {
rel.deleted = true; // deduplicate
}
found = true;
}
}
if !found {
// Upsert: create the link if it doesn't exist
self.add_link(source, target, "link_set")?;
// Set the strength on the newly created link
for rel in self.relations.iter_mut().rev() {
if !rel.deleted && rel.source_key == source && rel.target_key == target {
rel.strength = strength;
break;
}
}
return Ok(0.0);
}
Ok(old)
}
/// Add a link between two nodes with Jaccard-based initial strength.
/// Returns the strength, or a message if the link already exists.
pub fn add_link(&self, source: &str, target: &str, provenance: &str) -> Result<f32> {
let source_uuid = self.get_node(source)?
.map(|n| n.uuid)
.ok_or_else(|| anyhow!("source not found: {}", source))?;
let target_uuid = self.get_node(target)?
.map(|n| n.uuid)
.ok_or_else(|| anyhow!("target not found: {}", target))?;
// Check for existing via index
if let Some(db) = &self.db {
let edges = index::edges_for_node(db, &source_uuid)?;
let exists = edges.iter().any(|(other, _, _, _)| *other == target_uuid);
pub fn add_link(&mut self, source: &str, target: &str, provenance: &str) -> Result<f32, String> {
// Check for existing
let exists = self.relations.iter().any(|r|
!r.deleted &&
((r.source_key == source && r.target_key == target) ||
(r.source_key == target && r.target_key == source)));
if exists {
bail!("link already exists: {} ↔ {}", source, target);
}
return Err(format!("link already exists: {} ↔ {}", source, target));
}
let source_uuid = self.nodes.get(source)
.map(|n| n.uuid)
.ok_or_else(|| format!("source not found: {}", source))?;
let target_uuid = self.nodes.get(target)
.map(|n| n.uuid)
.ok_or_else(|| format!("target not found: {}", target))?;
let graph = self.build_graph();
let jaccard = graph.jaccard(source, target);
let strength = (jaccard * 3.0).clamp(0.1, 1.0) as f32;

173
src/hippocampus/store/parse.rs Normal file
View file

@ -0,0 +1,173 @@
// Markdown parsing for memory files
//
// Splits markdown files into MemoryUnit structs based on `<!-- mem: ... -->`
// markers. Each marker starts a new section; content before the first marker
// becomes the file-level unit. Links and causal edges are extracted from
// both marker attributes and inline markdown links.
use super::NodeType;
use regex::Regex;
use std::collections::HashMap;
use std::path::Path;
use std::sync::OnceLock;
pub struct MemoryUnit {
pub key: String,
pub content: String,
pub marker_links: Vec<String>,
pub md_links: Vec<String>,
pub causes: Vec<String>,
pub state: Option<String>,
pub source_ref: Option<String>,
}
pub(super) fn classify_filename(filename: &str) -> NodeType {
let bare = filename.strip_suffix(".md").unwrap_or(filename);
if bare.starts_with("daily-") { NodeType::EpisodicDaily }
else if bare.starts_with("weekly-") { NodeType::EpisodicWeekly }
else if bare.starts_with("monthly-") { NodeType::EpisodicMonthly }
else if bare == "journal" { NodeType::EpisodicSession }
else { NodeType::Semantic }
}
pub fn parse_units(raw_filename: &str, content: &str) -> Vec<MemoryUnit> {
let filename = raw_filename.strip_suffix(".md").unwrap_or(raw_filename);
static MARKER_RE: OnceLock<Regex> = OnceLock::new();
static SOURCE_RE: OnceLock<Regex> = OnceLock::new();
static MD_LINK_RE: OnceLock<Regex> = OnceLock::new();
let marker_re = MARKER_RE.get_or_init(||
Regex::new(r"<!--\s*mem:\s*((?:id|links|tags|causes|state)\s*=\s*[^\s].*?)-->").unwrap());
let source_re = SOURCE_RE.get_or_init(||
Regex::new(r"<!--\s*source:\s*(.+?)\s*-->").unwrap());
let md_link_re = MD_LINK_RE.get_or_init(||
Regex::new(r"\[[^\]]*\]\(([^):]+(?:#[^)]*)?)\)").unwrap());
let markers: Vec<_> = marker_re.captures_iter(content)
.map(|cap| {
let full_match = cap.get(0).unwrap();
let attrs_str = &cap[1];
(full_match.start(), full_match.end(), parse_marker_attrs(attrs_str))
})
.collect();
let find_source = |text: &str| -> Option<String> {
source_re.captures(text).map(|c| c[1].trim().to_string())
};
if markers.is_empty() {
let source_ref = find_source(content);
let md_links = extract_md_links(content, md_link_re, filename);
return vec![MemoryUnit {
key: filename.to_string(),
content: content.to_string(),
marker_links: Vec::new(),
md_links,
causes: Vec::new(),
state: None,
source_ref,
}];
}
let mut units = Vec::new();
let first_start = markers[0].0;
let pre_content = content[..first_start].trim();
if !pre_content.is_empty() {
let source_ref = find_source(pre_content);
let md_links = extract_md_links(pre_content, md_link_re, filename);
units.push(MemoryUnit {
key: filename.to_string(),
content: pre_content.to_string(),
marker_links: Vec::new(),
md_links,
causes: Vec::new(),
state: None,
source_ref,
});
}
for (i, (_, end, attrs)) in markers.iter().enumerate() {
let unit_end = if i + 1 < markers.len() {
markers[i + 1].0
} else {
content.len()
};
let unit_content = content[*end..unit_end].trim();
let id = attrs.get("id").cloned().unwrap_or_default();
let key = if id.is_empty() {
format!("{}#unnamed-{}", filename, i)
} else {
format!("{}#{}", filename, id)
};
let marker_links = attrs.get("links")
.map(|l| l.split(',').map(|s| normalize_link(s.trim(), filename)).collect())
.unwrap_or_default();
let causes = attrs.get("causes")
.map(|l| l.split(',').map(|s| normalize_link(s.trim(), filename)).collect())
.unwrap_or_default();
let state = attrs.get("state").cloned();
let source_ref = find_source(unit_content);
let md_links = extract_md_links(unit_content, md_link_re, filename);
units.push(MemoryUnit {
key,
content: unit_content.to_string(),
marker_links,
md_links,
causes,
state,
source_ref,
});
}
units
}
fn parse_marker_attrs(attrs_str: &str) -> HashMap<String, String> {
static ATTR_RE: OnceLock<Regex> = OnceLock::new();
let attr_re = ATTR_RE.get_or_init(|| Regex::new(r"(\w+)\s*=\s*(\S+)").unwrap());
let mut attrs = HashMap::new();
for cap in attr_re.captures_iter(attrs_str) {
attrs.insert(cap[1].to_string(), cap[2].to_string());
}
attrs
}
fn extract_md_links(content: &str, re: &Regex, source_file: &str) -> Vec<String> {
re.captures_iter(content)
.map(|cap| normalize_link(&cap[1], source_file))
.filter(|link| !link.starts_with(source_file) || link.contains('#'))
.collect()
}
fn normalize_link(target: &str, source_file: &str) -> String {
let source_bare = source_file.strip_suffix(".md").unwrap_or(source_file);
if target.starts_with('#') {
return format!("{}{}", source_bare, target);
}
let (path_part, fragment) = if let Some(hash_pos) = target.find('#') {
(&target[..hash_pos], Some(&target[hash_pos..]))
} else {
(target, None)
};
let basename = Path::new(path_part)
.file_name()
.map(|f| f.to_string_lossy().to_string())
.unwrap_or_else(|| path_part.to_string());
let bare = basename.strip_suffix(".md").unwrap_or(&basename);
match fragment {
Some(frag) => format!("{}{}", bare, frag),
None => bare.to_string(),
}
}

846
src/hippocampus/store/persist.rs Normal file
View file

@ -0,0 +1,846 @@
// Persistence layer: load, save, replay, append, snapshot
//
// Three-tier loading strategy:
// 1. rkyv mmap snapshot (snapshot.rkyv) — ~4ms deserialize
// 2. bincode cache (state.bin) — ~10ms
// 3. capnp log replay — ~40ms
//
// Logs are append-only; cache staleness uses log file sizes, not mtimes.
use super::types::*;
use crate::memory_capnp;
use capnp::message;
use capnp::serialize;
use std::collections::HashMap;
use std::fs;
use std::io::{BufReader, Seek};
use std::path::Path;
use std::sync::Arc;
/// Process-global cached store. Reloads only when log files change.
static CACHED_STORE: tokio::sync::OnceCell<Arc<tokio::sync::Mutex<Store>>> =
tokio::sync::OnceCell::const_new();
impl Store {
/// Get or create the process-global cached store.
/// Reloads from disk if log files have changed since last load.
pub async fn cached() -> Result<Arc<tokio::sync::Mutex<Store>>, String> {
let store = CACHED_STORE.get_or_try_init(|| async {
let s = Store::load()?;
Ok::<_, String>(Arc::new(tokio::sync::Mutex::new(s)))
}).await?;
{
let mut guard = store.lock().await;
if guard.is_stale() {
*guard = Store::load()?;
}
}
Ok(store.clone())
}
/// Check if the on-disk logs have grown since we loaded.
pub fn is_stale(&self) -> bool {
let nodes_size = fs::metadata(nodes_path()).map(|m| m.len()).unwrap_or(0);
let rels_size = fs::metadata(relations_path()).map(|m| m.len()).unwrap_or(0);
nodes_size != self.loaded_nodes_size || rels_size != self.loaded_rels_size
}
/// Load store from state.bin cache if fresh, otherwise rebuild from capnp logs.
///
/// Staleness check uses log file sizes (not mtimes). Since logs are
/// append-only, any write grows the file, invalidating the cache.
/// This avoids the mtime race that caused data loss with concurrent
/// writers (dream loop, link audit, journal enrichment).
pub fn load() -> Result<Store, String> {
// 1. Try rkyv mmap snapshot (~4ms with deserialize, <1ms zero-copy)
match Self::load_snapshot_mmap() {
Ok(Some(mut store)) => {
// rkyv snapshot doesn't include visits — replay from log
let visits_p = visits_path();
if visits_p.exists() {
store.replay_visits(&visits_p).ok();
}
let tp_p = transcript_progress_path();
if tp_p.exists() {
store.replay_transcript_progress(&tp_p).ok();
}
return Ok(store);
},
Ok(None) => {},
Err(e) => eprintln!("rkyv snapshot: {}", e),
}
// 2. Try bincode state.bin cache (~10ms)
let nodes_p = nodes_path();
let rels_p = relations_path();
let state_p = state_path();
let nodes_size = fs::metadata(&nodes_p).map(|m| m.len()).unwrap_or(0);
let rels_size = fs::metadata(&rels_p).map(|m| m.len()).unwrap_or(0);
if let Ok(data) = fs::read(&state_p)
&& data.len() >= CACHE_HEADER_LEN && data[..4] == CACHE_MAGIC {
let cached_nodes = u64::from_le_bytes(data[4..12].try_into().unwrap());
let cached_rels = u64::from_le_bytes(data[12..20].try_into().unwrap());
if cached_nodes == nodes_size && cached_rels == rels_size
&& let Ok(mut store) = bincode::deserialize::<Store>(&data[CACHE_HEADER_LEN..]) {
// Rebuild uuid_to_key (skipped by serde)
for (key, node) in &store.nodes {
store.uuid_to_key.insert(node.uuid, key.clone());
}
store.loaded_nodes_size = nodes_size;
store.loaded_rels_size = rels_size;
// Bootstrap: write rkyv snapshot if missing
if !snapshot_path().exists()
&& let Err(e) = store.save_snapshot(cached_nodes, cached_rels) {
eprintln!("rkyv bootstrap: {}", e);
}
return Ok(store);
}
}
// Stale or no cache — rebuild from capnp logs
let mut store = Store::default();
if nodes_p.exists() {
store.replay_nodes(&nodes_p)?;
}
if rels_p.exists() {
store.replay_relations(&rels_p)?;
}
let visits_p = visits_path();
if visits_p.exists() {
store.replay_visits(&visits_p)?;
}
let tp_p = transcript_progress_path();
if tp_p.exists() {
store.replay_transcript_progress(&tp_p)?;
}
// Record log sizes after replay — this is the state we reflect
store.loaded_nodes_size = fs::metadata(&nodes_p).map(|m| m.len()).unwrap_or(0);
store.loaded_rels_size = fs::metadata(&rels_p).map(|m| m.len()).unwrap_or(0);
// Drop edges referencing deleted/missing nodes
store.relations.retain(|r|
store.nodes.contains_key(&r.source_key) &&
store.nodes.contains_key(&r.target_key)
);
store.save()?;
Ok(store)
}
/// Load store directly from capnp logs, bypassing all caches.
/// Used by fsck to verify cache consistency.
pub fn load_from_logs() -> Result<Store, String> {
let nodes_p = nodes_path();
let rels_p = relations_path();
let mut store = Store::default();
if nodes_p.exists() {
store.replay_nodes(&nodes_p)?;
}
if rels_p.exists() {
store.replay_relations(&rels_p)?;
}
let visits_p = visits_path();
if visits_p.exists() {
store.replay_visits(&visits_p)?;
}
let tp_p = transcript_progress_path();
if tp_p.exists() {
store.replay_transcript_progress(&tp_p)?;
}
Ok(store)
}
/// Replay node log, keeping latest version per UUID.
/// Tracks all UUIDs seen per key to detect duplicates.
fn replay_nodes(&mut self, path: &Path) -> Result<(), String> {
let file = fs::File::open(path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
// Track all non-deleted UUIDs per key to detect duplicates
let mut key_uuids: HashMap<String, Vec<[u8; 16]>> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.map_err(|e| format!("read node log: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes: {}", e))? {
let node = Node::from_capnp_migrate(node_reader)?;
let existing_version = self.nodes.get(&node.key)
.map(|n| n.version)
.unwrap_or(0);
if node.version >= existing_version {
if node.deleted {
self.nodes.remove(&node.key);
self.uuid_to_key.remove(&node.uuid);
if let Some(uuids) = key_uuids.get_mut(&node.key) {
uuids.retain(|u| *u != node.uuid);
}
} else {
self.uuid_to_key.insert(node.uuid, node.key.clone());
self.nodes.insert(node.key.clone(), node.clone());
let uuids = key_uuids.entry(node.key).or_default();
if !uuids.contains(&node.uuid) {
uuids.push(node.uuid);
}
}
}
}
}
// Report duplicate keys
for (key, uuids) in &key_uuids {
if uuids.len() > 1 {
dbglog!("WARNING: key '{}' has {} UUIDs (duplicate nodes)", key, uuids.len());
}
}
Ok(())
}
/// Replay relation log, keeping latest version per UUID
fn replay_relations(&mut self, path: &Path) -> Result<(), String> {
let file = fs::File::open(path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
// Collect all, then deduplicate by UUID keeping latest version
let mut by_uuid: HashMap<[u8; 16], Relation> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::relation_log::Reader>()
.map_err(|e| format!("read relation log: {}", e))?;
for rel_reader in log.get_relations()
.map_err(|e| format!("get relations: {}", e))? {
let rel = Relation::from_capnp_migrate(rel_reader)?;
let existing_version = by_uuid.get(&rel.uuid)
.map(|r| r.version)
.unwrap_or(0);
if rel.version >= existing_version {
by_uuid.insert(rel.uuid, rel);
}
}
}
self.relations = by_uuid.into_values()
.filter(|r| !r.deleted)
.collect();
Ok(())
}
/// Find all duplicate keys: keys with multiple live UUIDs in the log.
/// Returns a map from key → vec of all live Node versions (one per UUID).
/// The "winner" in self.nodes is always one of them.
pub fn find_duplicates(&self) -> Result<HashMap<String, Vec<Node>>, String> {
let path = nodes_path();
if !path.exists() { return Ok(HashMap::new()); }
let file = fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
// Track latest version of each UUID
let mut by_uuid: HashMap<[u8; 16], Node> = HashMap::new();
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.map_err(|e| format!("read node log: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes: {}", e))? {
let node = Node::from_capnp_migrate(node_reader)?;
let dominated = by_uuid.get(&node.uuid)
.map(|n| node.version >= n.version)
.unwrap_or(true);
if dominated {
by_uuid.insert(node.uuid, node);
}
}
}
// Group live (non-deleted) nodes by key
let mut by_key: HashMap<String, Vec<Node>> = HashMap::new();
for node in by_uuid.into_values() {
if !node.deleted {
by_key.entry(node.key.clone()).or_default().push(node);
}
}
// Keep only duplicates
by_key.retain(|_, nodes| nodes.len() > 1);
Ok(by_key)
}
/// Append nodes to the log file.
/// Serializes to a Vec first, then does a single write() syscall
/// so the append is atomic with O_APPEND even without flock.
pub fn append_nodes(&mut self, nodes: &[Node]) -> Result<(), String> {
let _lock = StoreLock::acquire()?;
self.append_nodes_unlocked(nodes)
}
/// Append nodes without acquiring the lock. Caller must hold StoreLock.
pub(crate) fn append_nodes_unlocked(&mut self, nodes: &[Node]) -> Result<(), String> {
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::node_log::Builder>();
let mut list = log.init_nodes(nodes.len() as u32);
for (i, node) in nodes.iter().enumerate() {
node.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.map_err(|e| format!("serialize nodes: {}", e))?;
let path = nodes_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
use std::io::Write;
(&file).write_all(&buf)
.map_err(|e| format!("write nodes: {}", e))?;
self.loaded_nodes_size = file.metadata().map(|m| m.len()).unwrap_or(0);
Ok(())
}
/// Replay only new entries appended to the node log since we last loaded.
/// Call under StoreLock to catch writes from concurrent processes.
pub(crate) fn refresh_nodes(&mut self) -> Result<(), String> {
let path = nodes_path();
let current_size = fs::metadata(&path).map(|m| m.len()).unwrap_or(0);
if current_size <= self.loaded_nodes_size {
return Ok(()); // no new data
}
let file = fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
reader.seek(std::io::SeekFrom::Start(self.loaded_nodes_size))
.map_err(|e| format!("seek nodes log: {}", e))?;
while let Ok(msg) = serialize::read_message(&mut reader, message::ReaderOptions::new()) {
let log = msg.get_root::<memory_capnp::node_log::Reader>()
.map_err(|e| format!("read node log delta: {}", e))?;
for node_reader in log.get_nodes()
.map_err(|e| format!("get nodes delta: {}", e))? {
let node = Node::from_capnp_migrate(node_reader)?;
let dominated = self.nodes.get(&node.key)
.map(|n| node.version >= n.version)
.unwrap_or(true);
if dominated {
if node.deleted {
self.nodes.remove(&node.key);
self.uuid_to_key.remove(&node.uuid);
} else {
self.uuid_to_key.insert(node.uuid, node.key.clone());
self.nodes.insert(node.key.clone(), node);
}
}
}
}
self.loaded_nodes_size = current_size;
Ok(())
}
/// Append relations to the log file.
/// Single write() syscall for atomic O_APPEND.
pub fn append_relations(&mut self, relations: &[Relation]) -> Result<(), String> {
let _lock = StoreLock::acquire()?;
self.append_relations_unlocked(relations)
}
/// Append relations without acquiring the lock. Caller must hold StoreLock.
pub(crate) fn append_relations_unlocked(&mut self, relations: &[Relation]) -> Result<(), String> {
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::relation_log::Builder>();
let mut list = log.init_relations(relations.len() as u32);
for (i, rel) in relations.iter().enumerate() {
rel.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.map_err(|e| format!("serialize relations: {}", e))?;
let path = relations_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
use std::io::Write;
(&file).write_all(&buf)
.map_err(|e| format!("write relations: {}", e))?;
self.loaded_rels_size = file.metadata().map(|m| m.len()).unwrap_or(0);
Ok(())
}
/// Append agent visit records to the visits log.
pub fn append_visits(&mut self, visits: &[AgentVisit]) -> Result<(), String> {
if visits.is_empty() { return Ok(()); }
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::agent_visit_log::Builder>();
let mut list = log.init_visits(visits.len() as u32);
for (i, visit) in visits.iter().enumerate() {
visit.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.map_err(|e| format!("serialize visits: {}", e))?;
let path = visits_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
use std::io::Write;
(&file).write_all(&buf)
.map_err(|e| format!("write visits: {}", e))?;
// Update in-memory index
for v in visits {
self.visits
.entry(v.node_key.clone())
.or_default()
.insert(v.agent.clone(), v.timestamp);
}
Ok(())
}
/// Replay visits log to rebuild in-memory index.
fn replay_visits(&mut self, path: &Path) -> Result<(), String> {
let file = fs::File::open(path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
while reader.stream_position().map_err(|e| e.to_string())?
< fs::metadata(path).map_err(|e| e.to_string())?.len()
{
let msg = match serialize::read_message(&mut reader, Default::default()) {
Ok(m) => m,
Err(_) => break,
};
let log = msg.get_root::<memory_capnp::agent_visit_log::Reader>()
.map_err(|e| format!("read visit log: {}", e))?;
for visit in log.get_visits().map_err(|e| e.to_string())? {
let key = visit.get_node_key().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
let agent = visit.get_agent().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
let ts = visit.get_timestamp();
if !key.is_empty() && !agent.is_empty() {
let entry = self.visits.entry(key).or_default();
// Keep latest timestamp per agent
let existing = entry.entry(agent).or_insert(0);
if ts > *existing {
*existing = ts;
}
}
}
}
Ok(())
}
/// Append transcript segment progress records.
pub fn append_transcript_progress(&mut self, segments: &[TranscriptSegment]) -> Result<(), String> {
if segments.is_empty() { return Ok(()); }
let mut msg = message::Builder::new_default();
{
let log = msg.init_root::<memory_capnp::transcript_progress_log::Builder>();
let mut list = log.init_segments(segments.len() as u32);
for (i, seg) in segments.iter().enumerate() {
seg.to_capnp(list.reborrow().get(i as u32));
}
}
let mut buf = Vec::new();
serialize::write_message(&mut buf, &msg)
.map_err(|e| format!("serialize transcript progress: {}", e))?;
let path = transcript_progress_path();
let file = fs::OpenOptions::new()
.create(true).append(true).open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
use std::io::Write;
(&file).write_all(&buf)
.map_err(|e| format!("write transcript progress: {}", e))?;
// Update in-memory index
for seg in segments {
self.transcript_progress
.entry((seg.transcript_id.clone(), seg.segment_index))
.or_default()
.insert(seg.agent.clone());
}
Ok(())
}
/// Replay transcript progress log to rebuild in-memory index.
fn replay_transcript_progress(&mut self, path: &Path) -> Result<(), String> {
let file = fs::File::open(path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mut reader = BufReader::new(file);
while reader.stream_position().map_err(|e| e.to_string())?
< fs::metadata(path).map_err(|e| e.to_string())?.len()
{
let msg = match serialize::read_message(&mut reader, Default::default()) {
Ok(m) => m,
Err(_) => break,
};
let log = msg.get_root::<memory_capnp::transcript_progress_log::Reader>()
.map_err(|e| format!("read transcript progress: {}", e))?;
for seg in log.get_segments().map_err(|e| e.to_string())? {
let id = seg.get_transcript_id().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
let agent = seg.get_agent().ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string();
let idx = seg.get_segment_index();
if !id.is_empty() && !agent.is_empty() {
self.transcript_progress
.entry((id, idx))
.or_default()
.insert(agent);
}
}
}
Ok(())
}
/// Migrate old stub-node transcript markers into the new progress log.
/// Reads _observed-transcripts-f-*, _mined-transcripts#f-*, and _facts-* keys,
/// extracts transcript_id and segment_index, writes to transcript-progress.capnp,
/// then deletes the stub nodes.
pub fn migrate_transcript_progress(&mut self) -> Result<usize, String> {
let mut segments = Vec::new();
for key in self.nodes.keys() {
// _observed-transcripts-f-{UUID}.{segment}
if let Some(rest) = key.strip_prefix("_observed-transcripts-f-") {
if let Some((uuid, seg_str)) = rest.rsplit_once('.')
&& let Ok(seg) = seg_str.parse::<u32>() {
segments.push(new_transcript_segment(uuid, seg, "observation"));
}
}
// _mined-transcripts#f-{UUID}.{segment}
else if let Some(rest) = key.strip_prefix("_mined-transcripts#f-") {
if let Some((uuid, seg_str)) = rest.rsplit_once('.')
&& let Ok(seg) = seg_str.parse::<u32>() {
segments.push(new_transcript_segment(uuid, seg, "experience"));
}
}
// _mined-transcripts-f-{UUID}.{segment}
else if let Some(rest) = key.strip_prefix("_mined-transcripts-f-") {
if let Some((uuid, seg_str)) = rest.rsplit_once('.')
&& let Ok(seg) = seg_str.parse::<u32>() {
segments.push(new_transcript_segment(uuid, seg, "experience"));
}
}
// _facts-{UUID} (whole-file, segment 0)
else if let Some(uuid) = key.strip_prefix("_facts-") {
if !uuid.contains('-') || uuid.len() < 30 { continue; } // skip non-UUID
segments.push(new_transcript_segment(uuid, 0, "fact"));
}
}
let count = segments.len();
if count > 0 {
self.append_transcript_progress(&segments)?;
}
// Soft-delete the old stub nodes
let keys_to_delete: Vec<String> = self.nodes.keys()
.filter(|k| k.starts_with("_observed-transcripts-")
|| k.starts_with("_mined-transcripts")
|| (k.starts_with("_facts-") && !k.contains("fact_mine")))
.cloned()
.collect();
for key in &keys_to_delete {
if let Some(node) = self.nodes.get_mut(key) {
node.deleted = true;
}
}
if !keys_to_delete.is_empty() {
self.save()?;
}
Ok(count)
}
/// Record visits for a batch of node keys from a successful agent run.
pub fn record_agent_visits(&mut self, node_keys: &[String], agent: &str) -> Result<(), String> {
let visits: Vec<AgentVisit> = node_keys.iter()
.filter_map(|key| {
let node = self.nodes.get(key)?;
Some(new_visit(node.uuid, key, agent, "processed"))
})
.collect();
self.append_visits(&visits)
}
/// Get the last time an agent visited a node. Returns 0 if never visited.
pub fn last_visited(&self, node_key: &str, agent: &str) -> i64 {
self.visits.get(node_key)
.and_then(|agents| agents.get(agent))
.copied()
.unwrap_or(0)
}
/// Save the derived cache with log size header for staleness detection.
/// Uses atomic write (tmp + rename) to prevent partial reads.
pub fn save(&self) -> Result<(), String> {
let _lock = StoreLock::acquire()?;
let path = state_path();
if let Some(parent) = path.parent() {
fs::create_dir_all(parent).ok();
}
// Use log sizes from load time, not current filesystem sizes.
// If another writer appended since we loaded, our recorded size
// will be smaller than the actual log → next reader detects stale
// cache and replays the (correct, append-only) log.
let nodes_size = self.loaded_nodes_size;
let rels_size = self.loaded_rels_size;
let bincode_data = bincode::serialize(self)
.map_err(|e| format!("bincode serialize: {}", e))?;
let mut data = Vec::with_capacity(CACHE_HEADER_LEN + bincode_data.len());
data.extend_from_slice(&CACHE_MAGIC);
data.extend_from_slice(&nodes_size.to_le_bytes());
data.extend_from_slice(&rels_size.to_le_bytes());
data.extend_from_slice(&bincode_data);
// Atomic write: tmp file + rename
let tmp_path = path.with_extension("bin.tmp");
fs::write(&tmp_path, &data)
.map_err(|e| format!("write {}: {}", tmp_path.display(), e))?;
fs::rename(&tmp_path, &path)
.map_err(|e| format!("rename {} → {}: {}", tmp_path.display(), path.display(), e))?;
// Also write rkyv snapshot (mmap-friendly)
if let Err(e) = self.save_snapshot(nodes_size, rels_size) {
eprintln!("rkyv snapshot save: {}", e);
}
Ok(())
}
/// Serialize store as rkyv snapshot with staleness header.
/// Assumes StoreLock is already held by caller.
fn save_snapshot(&self, nodes_size: u64, rels_size: u64) -> Result<(), String> {
let snap = Snapshot {
nodes: self.nodes.clone(),
relations: self.relations.iter().filter(|r| !r.deleted).cloned().collect(),
gaps: self.gaps.clone(),
params: self.params,
};
let rkyv_data = rkyv::to_bytes::<_, 256>(&snap)
.map_err(|e| format!("rkyv serialize: {}", e))?;
let mut data = Vec::with_capacity(RKYV_HEADER_LEN + rkyv_data.len());
data.extend_from_slice(&RKYV_MAGIC);
data.extend_from_slice(&1u32.to_le_bytes()); // format version
data.extend_from_slice(&nodes_size.to_le_bytes());
data.extend_from_slice(&rels_size.to_le_bytes());
data.extend_from_slice(&(rkyv_data.len() as u64).to_le_bytes());
data.extend_from_slice(&rkyv_data);
let path = snapshot_path();
let tmp_path = path.with_extension("rkyv.tmp");
fs::write(&tmp_path, &data)
.map_err(|e| format!("write {}: {}", tmp_path.display(), e))?;
fs::rename(&tmp_path, &path)
.map_err(|e| format!("rename: {}", e))?;
Ok(())
}
/// Try loading store from mmap'd rkyv snapshot.
/// Returns None if snapshot is missing or stale (log sizes don't match).
fn load_snapshot_mmap() -> Result<Option<Store>, String> {
let path = snapshot_path();
if !path.exists() { return Ok(None); }
let nodes_size = fs::metadata(nodes_path()).map(|m| m.len()).unwrap_or(0);
let rels_size = fs::metadata(relations_path()).map(|m| m.len()).unwrap_or(0);
let file = fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let mmap = unsafe { memmap2::Mmap::map(&file) }
.map_err(|e| format!("mmap {}: {}", path.display(), e))?;
if mmap.len() < RKYV_HEADER_LEN { return Ok(None); }
if mmap[..4] != RKYV_MAGIC { return Ok(None); }
// [4..8] = version, skip for now
let cached_nodes = u64::from_le_bytes(mmap[8..16].try_into().unwrap());
let cached_rels = u64::from_le_bytes(mmap[16..24].try_into().unwrap());
let data_len = u64::from_le_bytes(mmap[24..32].try_into().unwrap()) as usize;
if cached_nodes != nodes_size || cached_rels != rels_size {
return Ok(None); // stale
}
if mmap.len() < RKYV_HEADER_LEN + data_len {
return Ok(None); // truncated
}
let rkyv_data = &mmap[RKYV_HEADER_LEN..RKYV_HEADER_LEN + data_len];
// SAFETY: we wrote this file ourselves via save_snapshot().
// Skip full validation (check_archived_root) — the staleness header
// already confirms this snapshot matches the current log state.
let archived = unsafe { rkyv::archived_root::<Snapshot>(rkyv_data) };
let snap: Snapshot = <ArchivedSnapshot as rkyv::Deserialize<Snapshot, rkyv::Infallible>>
::deserialize(archived, &mut rkyv::Infallible).unwrap();
let mut store = Store {
nodes: snap.nodes,
relations: snap.relations,
gaps: snap.gaps,
params: snap.params,
..Default::default()
};
// Rebuild uuid_to_key (not serialized)
for (key, node) in &store.nodes {
store.uuid_to_key.insert(node.uuid, key.clone());
}
store.loaded_nodes_size = nodes_size;
store.loaded_rels_size = rels_size;
Ok(Some(store))
}
}
/// Check and repair corrupt capnp log files.
///
/// Reads each message sequentially, tracking file position. On the first
/// corrupt message, truncates the file to the last good position. Also
/// removes stale caches so the next load replays from the repaired log.
pub fn fsck() -> Result<(), String> {
let mut any_corrupt = false;
for (path, kind) in [
(nodes_path(), "node"),
(relations_path(), "relation"),
] {
if !path.exists() { continue; }
let file = fs::File::open(&path)
.map_err(|e| format!("open {}: {}", path.display(), e))?;
let file_len = file.metadata()
.map_err(|e| format!("stat {}: {}", path.display(), e))?.len();
let mut reader = BufReader::new(file);
let mut good_messages = 0u64;
let mut last_good_pos = 0u64;
loop {
let pos = reader.stream_position()
.map_err(|e| format!("tell {}: {}", path.display(), e))?;
let msg = match serialize::read_message(&mut reader, message::ReaderOptions::new()) {
Ok(m) => m,
Err(_) => {
// read_message fails at EOF (normal) or on corrupt framing
if pos < file_len {
// Not at EOF — corrupt framing
eprintln!("{}: corrupt message at offset {}, truncating", kind, pos);
any_corrupt = true;
drop(reader);
let file = fs::OpenOptions::new().write(true).open(&path)
.map_err(|e| format!("open for truncate: {}", e))?;
file.set_len(pos)
.map_err(|e| format!("truncate {}: {}", path.display(), e))?;
eprintln!("{}: truncated from {} to {} bytes ({} good messages)",
kind, file_len, pos, good_messages);
}
break;
}
};
// Validate the message content too
let valid = if kind == "node" {
msg.get_root::<memory_capnp::node_log::Reader>()
.and_then(|l| l.get_nodes().map(|_| ()))
.is_ok()
} else {
msg.get_root::<memory_capnp::relation_log::Reader>()
.and_then(|l| l.get_relations().map(|_| ()))
.is_ok()
};
if valid {
good_messages += 1;
last_good_pos = reader.stream_position()
.map_err(|e| format!("tell {}: {}", path.display(), e))?;
} else {
eprintln!("{}: corrupt message content at offset {}, truncating to {}",
kind, pos, last_good_pos);
any_corrupt = true;
drop(reader);
let file = fs::OpenOptions::new().write(true).open(&path)
.map_err(|e| format!("open for truncate: {}", e))?;
file.set_len(last_good_pos)
.map_err(|e| format!("truncate {}: {}", path.display(), e))?;
eprintln!("{}: truncated from {} to {} bytes ({} good messages)",
kind, file_len, last_good_pos, good_messages);
break;
}
}
if !any_corrupt {
eprintln!("{}: {} messages, all clean", kind, good_messages);
}
}
if any_corrupt {
// Nuke caches so next load replays from the repaired logs
for p in [state_path(), snapshot_path()] {
if p.exists() {
fs::remove_file(&p)
.map_err(|e| format!("remove {}: {}", p.display(), e))?;
eprintln!("removed stale cache: {}", p.display());
}
}
eprintln!("repair complete — run `poc-memory status` to verify");
} else {
eprintln!("store is clean");
}
Ok(())
}

448
src/hippocampus/store/types.rs
View file

@ -1,21 +1,122 @@
// Core types for the memory store
//
// Node, Relation, enums, Store struct, path helpers, time helpers.
// capnp serialization is in capnp.rs.
// Node, Relation, enums, Params, and supporting types. Also contains
// the capnp serialization macros that generate bidirectional conversion.
use crate::memory_capnp;
use serde::{Deserialize, Serialize};
use uuid::Uuid;
use std::collections::{HashMap, HashSet};
use std::fs;
use std::os::unix::io::AsRawFd;
use std::path::PathBuf;
use std::time::{SystemTime, UNIX_EPOCH};
// ---------------------------------------------------------------------------
// Capnp serialization macros
//
// Declarative mapping between Rust types and capnp generated types.
// Adding a field to the schema means adding it in one place below;
// both read and write are generated from the same declaration.
// ---------------------------------------------------------------------------
/// Generate to_capnp/from_capnp conversion methods for an enum.
macro_rules! capnp_enum {
($rust_type:ident, $capnp_type:path, [$($variant:ident),+ $(,)?]) => {
impl $rust_type {
#[allow(clippy::wrong_self_convention, dead_code)]
pub(crate) fn to_capnp(&self) -> $capnp_type {
match self {
$(Self::$variant => <$capnp_type>::$variant,)+
}
}
pub(crate) fn from_capnp(v: $capnp_type) -> Self {
match v {
$(<$capnp_type>::$variant => Self::$variant,)+
}
}
}
};
}
/// Generate from_capnp/to_capnp methods for a struct with capnp serialization.
/// Fields are grouped by serialization kind:
/// text - capnp Text fields (String in Rust)
/// uuid - capnp Data fields ([u8; 16] in Rust)
/// prim - copy types (u32, f32, f64, bool)
/// enm - enums with to_capnp/from_capnp methods
/// skip - Rust-only fields not in capnp (set to Default on read)
macro_rules! capnp_message {
(
$struct:ident,
reader: $reader:ty,
builder: $builder:ty,
text: [$($tf:ident),* $(,)?],
uuid: [$($uf:ident),* $(,)?],
prim: [$($pf:ident),* $(,)?],
enm: [$($ef:ident: $et:ident),* $(,)?],
skip: [$($sf:ident),* $(,)?] $(,)?
) => {
impl $struct {
pub fn from_capnp(r: $reader) -> Result<Self, String> {
paste::paste! {
Ok(Self {
$($tf: read_text(r.[<get_ $tf>]()),)*
$($uf: read_uuid(r.[<get_ $uf>]()),)*
$($pf: r.[<get_ $pf>](),)*
$($ef: $et::from_capnp(
r.[<get_ $ef>]().map_err(|_| concat!("bad ", stringify!($ef)))?
),)*
$($sf: Default::default(),)*
})
}
}
pub fn to_capnp(&self, mut b: $builder) {
paste::paste! {
$(b.[<set_ $tf>](&self.$tf);)*
$(b.[<set_ $uf>](&self.$uf);)*
$(b.[<set_ $pf>](self.$pf);)*
$(b.[<set_ $ef>](self.$ef.to_capnp());)*
}
}
}
};
}
pub fn memory_dir() -> PathBuf {
crate::config::get().data_dir.clone()
}
pub fn nodes_path() -> PathBuf { memory_dir().join("nodes.capnp") }
pub(crate) fn relations_path() -> PathBuf { memory_dir().join("relations.capnp") }
pub(crate) fn db_path() -> PathBuf { memory_dir().join("index.redb") }
pub(crate) fn state_path() -> PathBuf { memory_dir().join("state.bin") }
pub(crate) fn snapshot_path() -> PathBuf { memory_dir().join("snapshot.rkyv") }
fn lock_path() -> PathBuf { memory_dir().join(".store.lock") }
/// RAII file lock using flock(2). Dropped when scope exits.
pub(crate) struct StoreLock {
_file: fs::File,
}
impl StoreLock {
pub(crate) fn acquire() -> Result<Self, String> {
let path = lock_path();
let file = fs::OpenOptions::new()
.create(true).truncate(false).write(true).open(&path)
.map_err(|e| format!("open lock {}: {}", path.display(), e))?;
// Blocking exclusive lock
let ret = unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX) };
if ret != 0 {
return Err(format!("flock: {}", std::io::Error::last_os_error()));
}
Ok(StoreLock { _file: file })
}
// Lock released automatically when _file is dropped (flock semantics)
}
pub fn now_epoch() -> i64 {
SystemTime::now()
@ -82,7 +183,8 @@ pub fn today() -> String {
}
// In-memory node representation
#[derive(Clone, Debug, Serialize, Deserialize)]
#[derive(Clone, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub struct Node {
pub uuid: [u8; 16],
pub version: u32,
@ -95,12 +197,18 @@ pub struct Node {
pub emotion: f32,
pub deleted: bool,
pub source_ref: String,
pub created: String,
pub retrievals: u32,
pub uses: u32,
pub wrongs: u32,
pub state_tag: String,
pub last_replayed: i64,
pub spaced_repetition_interval: u32,
// Position within file (section index, for export ordering)
#[serde(default)]
pub position: u32,
// Stable creation timestamp (unix epoch seconds). Set once at creation;
// never updated on rename or content update. Zero for legacy nodes.
#[serde(default)]
@ -119,7 +227,8 @@ pub struct Node {
pub degree: Option<u32>,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
#[derive(Clone, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub struct Relation {
pub uuid: [u8; 16],
pub version: u32,
@ -134,7 +243,8 @@ pub struct Relation {
pub target_key: String,
}
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)]
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub enum NodeType {
EpisodicSession,
EpisodicDaily,
@ -143,34 +253,263 @@ pub enum NodeType {
EpisodicMonthly,
}
impl NodeType {
pub fn from_u8(v: u8) -> Self {
match v {
0 => NodeType::EpisodicSession,
1 => NodeType::EpisodicDaily,
2 => NodeType::EpisodicWeekly,
3 => NodeType::Semantic,
4 => NodeType::EpisodicMonthly,
_ => NodeType::Semantic, // default
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub enum Provenance {
Manual,
Journal,
Agent, // legacy catch-all, prefer specific variants below
Dream,
Derived,
AgentExperienceMine,
AgentKnowledgeObservation,
AgentKnowledgePattern,
AgentKnowledgeConnector,
AgentKnowledgeChallenger,
AgentConsolidate,
AgentDigest,
AgentFactMine,
AgentDecay,
}
impl Provenance {
/// Parse from POC_PROVENANCE env var. Returns None if unset.
pub fn from_env() -> Option<Self> {
std::env::var("POC_PROVENANCE").ok().and_then(|s| Self::from_label(&s))
}
pub fn from_label(s: &str) -> Option<Self> {
Some(match s {
"manual" => Self::Manual,
"journal" => Self::Journal,
"agent" => Self::Agent,
"dream" => Self::Dream,
"derived" => Self::Derived,
"agent:experience-mine" => Self::AgentExperienceMine,
"agent:knowledge-observation"=> Self::AgentKnowledgeObservation,
"agent:knowledge-pattern" => Self::AgentKnowledgePattern,
"agent:knowledge-connector" => Self::AgentKnowledgeConnector,
"agent:knowledge-challenger" => Self::AgentKnowledgeChallenger,
"agent:consolidate" => Self::AgentConsolidate,
"agent:digest" => Self::AgentDigest,
"agent:fact-mine" => Self::AgentFactMine,
"agent:decay" => Self::AgentDecay,
_ => return None,
})
}
pub fn label(&self) -> &'static str {
match self {
Self::Manual => "manual",
Self::Journal => "journal",
Self::Agent => "agent",
Self::Dream => "dream",
Self::Derived => "derived",
Self::AgentExperienceMine => "agent:experience-mine",
Self::AgentKnowledgeObservation => "agent:knowledge-observation",
Self::AgentKnowledgePattern => "agent:knowledge-pattern",
Self::AgentKnowledgeConnector => "agent:knowledge-connector",
Self::AgentKnowledgeChallenger => "agent:knowledge-challenger",
Self::AgentConsolidate => "agent:consolidate",
Self::AgentDigest => "agent:digest",
Self::AgentFactMine => "agent:fact-mine",
Self::AgentDecay => "agent:decay",
}
}
}
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)]
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub enum RelationType {
Link,
Causal,
Auto,
}
impl RelationType {
pub fn from_u8(v: u8) -> Self {
match v {
1 => RelationType::Causal,
2 => RelationType::Auto,
_ => RelationType::Link,
capnp_enum!(NodeType, memory_capnp::NodeType,
[EpisodicSession, EpisodicDaily, EpisodicWeekly, Semantic, EpisodicMonthly]);
capnp_enum!(Provenance, memory_capnp::Provenance,
[Manual, Journal, Agent, Dream, Derived,
AgentExperienceMine, AgentKnowledgeObservation, AgentKnowledgePattern,
AgentKnowledgeConnector, AgentKnowledgeChallenger, AgentConsolidate,
AgentDigest, AgentFactMine, AgentDecay]);
capnp_enum!(RelationType, memory_capnp::RelationType,
[Link, Causal, Auto]);
capnp_message!(Node,
reader: memory_capnp::content_node::Reader<'_>,
builder: memory_capnp::content_node::Builder<'_>,
text: [key, content, source_ref, created, state_tag, provenance],
uuid: [uuid],
prim: [version, timestamp, weight, emotion, deleted,
retrievals, uses, wrongs, last_replayed,
spaced_repetition_interval, position, created_at, last_scored],
enm: [node_type: NodeType],
skip: [community_id, clustering_coefficient, degree],
);
impl Node {
/// Read from capnp with migration: if the new provenance text field
/// is empty (old record), fall back to the deprecated provenanceOld enum.
pub fn from_capnp_migrate(r: memory_capnp::content_node::Reader<'_>) -> Result<Self, String> {
let mut node = Self::from_capnp(r)?;
if node.provenance.is_empty()
&& let Ok(old) = r.get_provenance_old() {
node.provenance = Provenance::from_capnp(old).label().to_string();
}
// Sanitize timestamps: old capnp records have raw offsets instead
// of unix epoch. Anything past year 2100 (~4102444800) is bogus.
const MAX_SANE_EPOCH: i64 = 4_102_444_800;
if node.timestamp > MAX_SANE_EPOCH || node.timestamp < 0 {
node.timestamp = node.created_at;
}
if node.created_at > MAX_SANE_EPOCH || node.created_at < 0 {
node.created_at = node.timestamp.min(MAX_SANE_EPOCH);
}
Ok(node)
}
}
capnp_message!(Relation,
reader: memory_capnp::relation::Reader<'_>,
builder: memory_capnp::relation::Builder<'_>,
text: [source_key, target_key, provenance],
uuid: [uuid, source, target],
prim: [version, timestamp, strength, deleted],
enm: [rel_type: RelationType],
skip: [],
);
impl Relation {
pub fn from_capnp_migrate(r: memory_capnp::relation::Reader<'_>) -> Result<Self, String> {
let mut rel = Self::from_capnp(r)?;
if rel.provenance.is_empty()
&& let Ok(old) = r.get_provenance_old() {
rel.provenance = Provenance::from_capnp(old).label().to_string();
}
Ok(rel)
}
}
#[derive(Clone, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub struct RetrievalEvent {
pub query: String,
pub timestamp: String,
pub results: Vec<String>,
pub used: Option<Vec<String>>,
}
#[derive(Clone, Copy, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub struct Params {
pub default_weight: f64,
pub decay_factor: f64,
pub use_boost: f64,
pub prune_threshold: f64,
pub edge_decay: f64,
pub max_hops: u32,
pub min_activation: f64,
}
impl Default for Params {
fn default() -> Self {
Params {
default_weight: 0.7,
decay_factor: 0.95,
use_boost: 0.15,
prune_threshold: 0.1,
edge_decay: 0.3,
max_hops: 3,
min_activation: 0.05,
}
}
}
// Gap record — something we looked for but didn't find
#[derive(Clone, Debug, Serialize, Deserialize, rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub struct GapRecord {
pub description: String,
pub timestamp: String,
}
/// Per-node agent visit index: node_key → (agent_type → last_visit_timestamp)
pub(super) type VisitIndex = HashMap<String, HashMap<String, i64>>;
// The full in-memory store
#[derive(Default, Serialize, Deserialize)]
pub struct Store {
pub nodes: HashMap<String, Node>, // key → latest node
#[serde(skip)]
pub uuid_to_key: HashMap<[u8; 16], String>, // uuid → key (rebuilt from nodes)
pub relations: Vec<Relation>, // all active relations
pub retrieval_log: Vec<RetrievalEvent>,
pub gaps: Vec<GapRecord>,
pub params: Params,
/// Agent visit tracking: node_key → (agent_type → last_visit_epoch)
#[serde(default)]
pub visits: VisitIndex,
/// Transcript mining progress: (transcript_id, segment_index) → set of agents that processed it
#[serde(default)]
pub transcript_progress: HashMap<(String, u32), HashSet<String>>,
/// Log sizes at load time — used by save() to write correct staleness header.
/// If another writer appended since we loaded, our cache will be marked stale
/// (recorded size < actual size), forcing the next reader to replay the log.
#[serde(skip)]
pub(crate) loaded_nodes_size: u64,
#[serde(skip)]
pub(crate) loaded_rels_size: u64,
}
/// Snapshot for mmap: full store state minus retrieval_log (which
/// is append-only in retrieval.log). rkyv zero-copy serialization
/// lets us mmap this and access archived data without deserialization.
#[derive(rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)]
#[archive(check_bytes)]
pub(crate) struct Snapshot {
pub(crate) nodes: HashMap<String, Node>,
pub(crate) relations: Vec<Relation>,
pub(crate) gaps: Vec<GapRecord>,
pub(crate) params: Params,
}
// rkyv snapshot header: 32 bytes (multiple of 16 for alignment after mmap)
// [0..4] magic "RKV\x01"
// [4..8] format version (u32 LE)
// [8..16] nodes.capnp file size (u64 LE) — staleness check
// [16..24] relations.capnp file size (u64 LE)
// [24..32] rkyv data length (u64 LE)
pub(crate) const RKYV_MAGIC: [u8; 4] = *b"RKV\x01";
pub(crate) const RKYV_HEADER_LEN: usize = 32;
// state.bin header: magic + log file sizes for staleness detection.
// File sizes are race-free for append-only logs (they only grow),
// unlike mtimes which race with concurrent writers.
pub(crate) const CACHE_MAGIC: [u8; 4] = *b"POC\x01";
pub(crate) const CACHE_HEADER_LEN: usize = 4 + 8 + 8; // magic + nodes_size + rels_size
// Cap'n Proto serialization helpers
/// Read a capnp text field, returning empty string on any error
pub(crate) fn read_text(result: capnp::Result<capnp::text::Reader>) -> String {
result.ok()
.and_then(|t| t.to_str().ok())
.unwrap_or("")
.to_string()
}
/// Read a capnp data field as [u8; 16], zero-padded
pub(crate) fn read_uuid(result: capnp::Result<&[u8]>) -> [u8; 16] {
let mut out = [0u8; 16];
if let Ok(data) = result
&& data.len() >= 16 {
out.copy_from_slice(&data[..16]);
}
out
}
/// Create a new node with defaults
@ -187,11 +526,14 @@ pub fn new_node(key: &str, content: &str) -> Node {
emotion: 0.0,
deleted: false,
source_ref: String::new(),
created: today(),
retrievals: 0,
uses: 0,
wrongs: 0,
state_tag: String::new(),
last_replayed: 0,
spaced_repetition_interval: 1,
position: 0,
created_at: now_epoch(),
last_scored: 0,
community_id: None,
@ -200,6 +542,68 @@ pub fn new_node(key: &str, content: &str) -> Node {
}
}
/// Agent visit record — tracks when an agent successfully processed a node
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct AgentVisit {
pub node_uuid: [u8; 16],
pub node_key: String,
pub agent: String,
pub timestamp: i64,
pub outcome: String,
}
capnp_message!(AgentVisit,
reader: memory_capnp::agent_visit::Reader<'_>,
builder: memory_capnp::agent_visit::Builder<'_>,
text: [node_key, agent, outcome],
uuid: [node_uuid],
prim: [timestamp],
enm: [],
skip: [],
);
pub(super) fn new_visit(node_uuid: [u8; 16], node_key: &str, agent: &str, outcome: &str) -> AgentVisit {
AgentVisit {
node_uuid,
node_key: node_key.to_string(),
agent: agent.to_string(),
timestamp: now_epoch(),
outcome: outcome.to_string(),
}
}
pub(crate) fn visits_path() -> PathBuf { memory_dir().join("visits.capnp") }
/// Transcript mining progress — tracks which segments have been processed
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct TranscriptSegment {
pub transcript_id: String,
pub segment_index: u32,
pub agent: String,
pub timestamp: i64,
}
capnp_message!(TranscriptSegment,
reader: memory_capnp::transcript_segment::Reader<'_>,
builder: memory_capnp::transcript_segment::Builder<'_>,
text: [transcript_id, agent],
uuid: [],
prim: [segment_index, timestamp],
enm: [],
skip: [],
);
pub(super) fn new_transcript_segment(transcript_id: &str, segment_index: u32, agent: &str) -> TranscriptSegment {
TranscriptSegment {
transcript_id: transcript_id.to_string(),
segment_index,
agent: agent.to_string(),
timestamp: now_epoch(),
}
}
pub(crate) fn transcript_progress_path() -> PathBuf { memory_dir().join("transcript-progress.capnp") }
/// Create a new relation.
/// Provenance is set from POC_PROVENANCE env var if present, else "manual".
pub fn new_relation(

253
src/hippocampus/store/view.rs
View file

@ -1,16 +1,21 @@
// Read-only access abstraction for the memory store
// Read-only access abstractions for the memory store
//
// StoreView: trait abstracting over owned Store and zero-copy MmapView.
// MmapView: mmap'd rkyv snapshot for sub-millisecond read-only access.
// AnyView: enum dispatch selecting fastest available view at runtime.
use super::{capnp, index, types::*};
use super::Store;
use super::types::*;
use std::fs;
// ---------------------------------------------------------------------------
// StoreView: read-only access trait for search and graph code.
//
// Abstracts over owned Store and zero-copy MmapView so the same
// spreading-activation and graph code works with either.
// ---------------------------------------------------------------------------
pub trait StoreView {
/// Get all node keys (from index, no deserialization).
fn all_keys(&self) -> Vec<String>;
/// Iterate all nodes. Callback receives (key, content, weight).
fn for_each_node<F: FnMut(&str, &str, f32)>(&self, f: F);
@ -22,89 +27,191 @@ pub trait StoreView {
/// Node weight by key, or the default weight if missing.
fn node_weight(&self, key: &str) -> f64;
/// Node content by key.
fn node_content(&self, key: &str) -> Option<&str>;
/// Search/graph parameters.
fn params(&self) -> Params;
}
impl StoreView for Store {
fn all_keys(&self) -> Vec<String> {
let db = match self.db.as_ref() {
Some(db) => db,
None => return Vec::new(),
};
index::all_keys(db).unwrap_or_default()
}
fn for_each_node<F: FnMut(&str, &str, f32)>(&self, mut f: F) {
let db = match self.db.as_ref() {
Some(db) => db,
None => return,
};
let keys = match index::all_keys(db) {
Ok(keys) => keys,
Err(_) => return,
};
for key in keys {
if let Ok(Some(offset)) = index::get_offset(db, &key) {
if let Ok(node) = capnp::read_node_at_offset(offset) {
f(&key, &node.content, node.weight);
}
}
for (key, node) in &self.nodes {
f(key, &node.content, node.weight);
}
}
fn for_each_node_meta<F: FnMut(&str, NodeType, i64)>(&self, mut f: F) {
let db = match self.db.as_ref() {
Some(db) => db,
None => return,
};
// Use index directly — no capnp reads needed
let pairs = match index::all_key_uuid_pairs(db) {
Ok(p) => p,
Err(_) => return,
};
for (key, _uuid, node_type, timestamp) in pairs {
f(&key, NodeType::from_u8(node_type), timestamp);
for (key, node) in &self.nodes {
f(key, node.node_type, node.timestamp);
}
}
fn for_each_relation<F: FnMut(&str, &str, f32, RelationType)>(&self, mut f: F) {
let db = match self.db.as_ref() {
Some(db) => db,
None => return,
};
// Build uuid ↔ key maps in a single table scan
let pairs = match index::all_key_uuid_pairs(db) {
Ok(p) => p,
Err(_) => return,
};
let mut uuid_to_key: std::collections::HashMap<[u8; 16], String> = std::collections::HashMap::new();
for (key, uuid, _, _) in &pairs {
uuid_to_key.insert(*uuid, key.clone());
}
// Iterate edges: only process outgoing to avoid duplicates
for (key, uuid, _, _) in &pairs {
let edges = match index::edges_for_node(db, uuid) {
Ok(e) => e,
Err(_) => continue,
};
for (other_uuid, strength, rel_type_byte, is_outgoing) in edges {
if !is_outgoing { continue; }
let target_key = match uuid_to_key.get(&other_uuid) {
Some(k) => k,
None => continue,
};
f(key, target_key, strength, RelationType::from_u8(rel_type_byte));
}
for rel in &self.relations {
if rel.deleted { continue; }
f(&rel.source_key, &rel.target_key, rel.strength, rel.rel_type);
}
}
fn node_weight(&self, key: &str) -> f64 {
let cfg = crate::config::get();
self.get_node(key)
.ok()
.flatten()
self.nodes.get(key).map(|n| n.weight as f64).unwrap_or(self.params.default_weight)
}
fn node_content(&self, key: &str) -> Option<&str> {
self.nodes.get(key).map(|n| n.content.as_str())
}
fn params(&self) -> Params {
self.params
}
}
// ---------------------------------------------------------------------------
// MmapView: zero-copy store access via mmap'd rkyv snapshot.
//
// Holds the mmap alive; all string reads go directly into the mapped
// pages without allocation. Falls back to None if snapshot is stale.
// ---------------------------------------------------------------------------
pub struct MmapView {
mmap: memmap2::Mmap,
_file: fs::File,
data_offset: usize,
data_len: usize,
}
impl MmapView {
/// Try to open a fresh rkyv snapshot. Returns None if missing or stale.
pub fn open() -> Option<Self> {
let path = snapshot_path();
let file = fs::File::open(&path).ok()?;
let mmap = unsafe { memmap2::Mmap::map(&file) }.ok()?;
if mmap.len() < RKYV_HEADER_LEN { return None; }
if mmap[..4] != RKYV_MAGIC { return None; }
let nodes_size = fs::metadata(nodes_path()).map(|m| m.len()).unwrap_or(0);
let rels_size = fs::metadata(relations_path()).map(|m| m.len()).unwrap_or(0);
let cached_nodes = u64::from_le_bytes(mmap[8..16].try_into().unwrap());
let cached_rels = u64::from_le_bytes(mmap[16..24].try_into().unwrap());
let data_len = u64::from_le_bytes(mmap[24..32].try_into().unwrap()) as usize;
if cached_nodes != nodes_size || cached_rels != rels_size { return None; }
if mmap.len() < RKYV_HEADER_LEN + data_len { return None; }
Some(MmapView { mmap, _file: file, data_offset: RKYV_HEADER_LEN, data_len })
}
fn snapshot(&self) -> &ArchivedSnapshot {
let data = &self.mmap[self.data_offset..self.data_offset + self.data_len];
unsafe { rkyv::archived_root::<Snapshot>(data) }
}
}
impl StoreView for MmapView {
fn for_each_node<F: FnMut(&str, &str, f32)>(&self, mut f: F) {
let snap = self.snapshot();
for (key, node) in snap.nodes.iter() {
f(key, &node.content, node.weight);
}
}
fn for_each_node_meta<F: FnMut(&str, NodeType, i64)>(&self, mut f: F) {
let snap = self.snapshot();
for (key, node) in snap.nodes.iter() {
let nt = match node.node_type {
ArchivedNodeType::EpisodicSession => NodeType::EpisodicSession,
ArchivedNodeType::EpisodicDaily => NodeType::EpisodicDaily,
ArchivedNodeType::EpisodicWeekly => NodeType::EpisodicWeekly,
ArchivedNodeType::EpisodicMonthly => NodeType::EpisodicMonthly,
ArchivedNodeType::Semantic => NodeType::Semantic,
};
f(key, nt, node.timestamp);
}
}
fn for_each_relation<F: FnMut(&str, &str, f32, RelationType)>(&self, mut f: F) {
let snap = self.snapshot();
for rel in snap.relations.iter() {
if rel.deleted { continue; }
let rt = match rel.rel_type {
ArchivedRelationType::Link => RelationType::Link,
ArchivedRelationType::Causal => RelationType::Causal,
ArchivedRelationType::Auto => RelationType::Auto,
};
f(&rel.source_key, &rel.target_key, rel.strength, rt);
}
}
fn node_weight(&self, key: &str) -> f64 {
let snap = self.snapshot();
snap.nodes.get(key)
.map(|n| n.weight as f64)
.unwrap_or(cfg.default_node_weight)
.unwrap_or(snap.params.default_weight)
}
fn node_content(&self, key: &str) -> Option<&str> {
let snap = self.snapshot();
snap.nodes.get(key).map(|n| &*n.content)
}
fn params(&self) -> Params {
let p = &self.snapshot().params;
Params {
default_weight: p.default_weight,
decay_factor: p.decay_factor,
use_boost: p.use_boost,
prune_threshold: p.prune_threshold,
edge_decay: p.edge_decay,
max_hops: p.max_hops,
min_activation: p.min_activation,
}
}
}
// ---------------------------------------------------------------------------
// AnyView: enum dispatch for read-only access.
//
// MmapView when the snapshot is fresh, owned Store as fallback.
// The match on each call is a single predicted branch — zero overhead.
// ---------------------------------------------------------------------------
pub enum AnyView {
Mmap(MmapView),
Owned(Store),
}
impl AnyView {
/// Load the fastest available view: mmap snapshot or owned store.
pub fn load() -> Result<Self, String> {
if let Some(mv) = MmapView::open() {
Ok(AnyView::Mmap(mv))
} else {
Ok(AnyView::Owned(Store::load()?))
}
}
}
impl StoreView for AnyView {
fn for_each_node<F: FnMut(&str, &str, f32)>(&self, f: F) {
match self { AnyView::Mmap(v) => v.for_each_node(f), AnyView::Owned(s) => s.for_each_node(f) }
}
fn for_each_node_meta<F: FnMut(&str, NodeType, i64)>(&self, f: F) {
match self { AnyView::Mmap(v) => v.for_each_node_meta(f), AnyView::Owned(s) => s.for_each_node_meta(f) }
}
fn for_each_relation<F: FnMut(&str, &str, f32, RelationType)>(&self, f: F) {
match self { AnyView::Mmap(v) => v.for_each_relation(f), AnyView::Owned(s) => s.for_each_relation(f) }
}
fn node_weight(&self, key: &str) -> f64 {
match self { AnyView::Mmap(v) => v.node_weight(key), AnyView::Owned(s) => s.node_weight(key) }
}
fn node_content(&self, key: &str) -> Option<&str> {
match self { AnyView::Mmap(v) => v.node_content(key), AnyView::Owned(s) => s.node_content(key) }
}
fn params(&self) -> Params {
match self { AnyView::Mmap(v) => v.params(), AnyView::Owned(s) => s.params() }
}
}

15
src/lib.rs
View file

@ -1,5 +1,3 @@
#![feature(async_fn_track_caller)]
// consciousness — unified crate for memory, agents, and subconscious processes
//
// thought/ — shared cognitive substrate (tools, context, memory ops)
@ -49,16 +47,6 @@ pub mod session;
// Shared utilities
pub mod util;
// Lock hold time tracking
pub mod locks;
// Re-export tracked locks as the default — swap to tokio::sync to disable tracking
pub use locks::TrackedMutex as Mutex;
pub use locks::TrackedMutexGuard as MutexGuard;
pub use locks::TrackedRwLock as RwLock;
pub use locks::TrackedRwLockReadGuard as RwLockReadGuard;
pub use locks::TrackedRwLockWriteGuard as RwLockWriteGuard;
// CLI handlers
pub mod cli;
@ -68,9 +56,6 @@ pub mod cli;
// Thalamus — universal notification routing and channel infrastructure
pub mod thalamus;
// MCP server — exposes memory tools over Unix socket
pub mod mcp_server;
// Re-export at crate root — capnp codegen emits `crate::daemon_capnp::` paths
pub use thalamus::daemon_capnp;

235
src/locks.rs
View file

@ -1,235 +0,0 @@
// Lock hold time tracking
//
// Wrappers around tokio::sync primitives that track how long locks are held,
// keyed by source location. Use `lock_stats()` to get a snapshot.
use std::collections::HashMap;
use std::panic::Location;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::OnceLock;
use std::time::Instant;
use tokio::sync::{Mutex, MutexGuard, RwLock, RwLockReadGuard, RwLockWriteGuard};
// ── Stats Registry ─────────────────────────────────────────────
struct LocationStats {
count: AtomicU64,
total_ns: AtomicU64,
max_ns: AtomicU64,
}
impl LocationStats {
fn new() -> Self {
Self {
count: AtomicU64::new(0),
total_ns: AtomicU64::new(0),
max_ns: AtomicU64::new(0),
}
}
fn record(&self, duration_ns: u64) {
self.count.fetch_add(1, Ordering::Relaxed);
self.total_ns.fetch_add(duration_ns, Ordering::Relaxed);
// Update max using CAS loop
let mut current = self.max_ns.load(Ordering::Relaxed);
while duration_ns > current {
match self.max_ns.compare_exchange_weak(
current, duration_ns, Ordering::Relaxed, Ordering::Relaxed
) {
Ok(_) => break,
Err(c) => current = c,
}
}
}
fn snapshot(&self) -> LockStats {
let count = self.count.load(Ordering::Relaxed);
let total_ns = self.total_ns.load(Ordering::Relaxed);
let max_ns = self.max_ns.load(Ordering::Relaxed);
LockStats {
count,
total_ns,
max_ns,
avg_ns: if count > 0 { total_ns / count } else { 0 },
}
}
}
/// Stats for a single lock location.
#[derive(Clone, Debug)]
pub struct LockStats {
pub count: u64,
pub total_ns: u64,
pub max_ns: u64,
pub avg_ns: u64,
}
type StatsMap = std::sync::Mutex<HashMap<&'static Location<'static>, LocationStats>>;
fn stats_map() -> &'static StatsMap {
static MAP: OnceLock<StatsMap> = OnceLock::new();
MAP.get_or_init(|| std::sync::Mutex::new(HashMap::new()))
}
fn record_hold_time(loc: &'static Location<'static>, duration_ns: u64) {
let map = stats_map().lock().unwrap();
if let Some(stats) = map.get(&loc) {
stats.record(duration_ns);
return;
}
drop(map);
// First time seeing this location — need write access
let mut map = stats_map().lock().unwrap();
let stats = map.entry(loc).or_insert_with(LocationStats::new);
stats.record(duration_ns);
}
/// Get a snapshot of all lock stats, sorted by max hold time (descending).
pub fn lock_stats() -> Vec<(String, LockStats)> {
let map = stats_map().lock().unwrap();
let mut stats: Vec<_> = map.iter()
.map(|(loc, s)| (format!("{}:{}", loc.file(), loc.line()), s.snapshot()))
.collect();
stats.sort_by(|a, b| b.1.max_ns.cmp(&a.1.max_ns));
stats
}
/// Reset all lock stats.
pub fn reset_lock_stats() {
let mut map = stats_map().lock().unwrap();
map.clear();
}
// ── TrackedMutex ───────────────────────────────────────────────
/// A Mutex wrapper that tracks hold times by caller location.
pub struct TrackedMutex<T> {
inner: Mutex<T>,
}
impl<T> TrackedMutex<T> {
pub fn new(value: T) -> Self {
Self { inner: Mutex::new(value) }
}
#[track_caller]
pub async fn lock(&self) -> TrackedMutexGuard<'_, T> {
let location = Location::caller();
let guard = self.inner.lock().await;
TrackedMutexGuard {
guard,
acquired_at: Instant::now(),
location,
}
}
#[track_caller]
pub fn try_lock(&self) -> Result<TrackedMutexGuard<'_, T>, tokio::sync::TryLockError> {
let location = Location::caller();
let guard = self.inner.try_lock()?;
Ok(TrackedMutexGuard {
guard,
acquired_at: Instant::now(),
location,
})
}
}
pub struct TrackedMutexGuard<'a, T> {
guard: MutexGuard<'a, T>,
acquired_at: Instant,
location: &'static Location<'static>,
}
impl<T> Drop for TrackedMutexGuard<'_, T> {
fn drop(&mut self) {
let duration = self.acquired_at.elapsed();
record_hold_time(self.location, duration.as_nanos() as u64);
}
}
impl<T> std::ops::Deref for TrackedMutexGuard<'_, T> {
type Target = T;
fn deref(&self) -> &T { &self.guard }
}
impl<T> std::ops::DerefMut for TrackedMutexGuard<'_, T> {
fn deref_mut(&mut self) -> &mut T { &mut self.guard }
}
// ── TrackedRwLock ──────────────────────────────────────────────
/// An RwLock wrapper that tracks hold times by caller location.
pub struct TrackedRwLock<T> {
inner: RwLock<T>,
}
impl<T> TrackedRwLock<T> {
pub fn new(value: T) -> Self {
Self { inner: RwLock::new(value) }
}
#[track_caller]
pub async fn read(&self) -> TrackedRwLockReadGuard<'_, T> {
let location = Location::caller();
let guard = self.inner.read().await;
TrackedRwLockReadGuard {
guard,
acquired_at: Instant::now(),
location,
}
}
#[track_caller]
pub async fn write(&self) -> TrackedRwLockWriteGuard<'_, T> {
let location = Location::caller();
let guard = self.inner.write().await;
TrackedRwLockWriteGuard {
guard,
acquired_at: Instant::now(),
location,
}
}
}
pub struct TrackedRwLockReadGuard<'a, T> {
guard: RwLockReadGuard<'a, T>,
acquired_at: Instant,
location: &'static Location<'static>,
}
impl<T> Drop for TrackedRwLockReadGuard<'_, T> {
fn drop(&mut self) {
let duration = self.acquired_at.elapsed();
record_hold_time(self.location, duration.as_nanos() as u64);
}
}
impl<T> std::ops::Deref for TrackedRwLockReadGuard<'_, T> {
type Target = T;
fn deref(&self) -> &T { &self.guard }
}
pub struct TrackedRwLockWriteGuard<'a, T> {
guard: RwLockWriteGuard<'a, T>,
acquired_at: Instant,
location: &'static Location<'static>,
}
impl<T> Drop for TrackedRwLockWriteGuard<'_, T> {
fn drop(&mut self) {
let duration = self.acquired_at.elapsed();
record_hold_time(self.location, duration.as_nanos() as u64);
}
}
impl<T> std::ops::Deref for TrackedRwLockWriteGuard<'_, T> {
type Target = T;
fn deref(&self) -> &T { &self.guard }
}
impl<T> std::ops::DerefMut for TrackedRwLockWriteGuard<'_, T> {
fn deref_mut(&mut self) -> &mut T { &mut self.guard }
}

265
src/main.rs
View file

@ -1,5 +1,3 @@
#![feature(panic_backtrace_config)]
// poc-memory: graph-structured memory for AI assistants
//
// Authors: ProofOfConcept <poc@bcachefs.org> and Kent Overstreet
@ -35,10 +33,31 @@ struct Cli {
enum Command {
// ── Core (daily use) ──────────────────────────────────────────────
/// Search memory via spreading activation from seed keys
/// Search memory (AND logic across terms)
///
/// Pipeline: -p spread -p spectral,k=20
/// Default pipeline: spread
Search {
/// Seed node keys
keys: Vec<String>,
/// Search terms
query: Vec<String>,
/// Algorithm pipeline stages (repeatable)
#[arg(short, long = "pipeline")]
pipeline: Vec<String>,
/// Show more results
#[arg(long)]
expand: bool,
/// Show node content, not just keys
#[arg(long)]
full: bool,
/// Show debug output for each pipeline stage
#[arg(long)]
debug: bool,
/// Also match key components (e.g. "irc" matches "irc-access")
#[arg(long)]
fuzzy: bool,
/// Also search node content (slow, use when graph search misses)
#[arg(long)]
content: bool,
},
/// Output a node's content to stdout
Render {
@ -128,6 +147,30 @@ EXAMPLES:
/// Query expression (e.g. "key ~ 'inner-life'")
expr: Vec<String>,
},
/// Mark a memory as useful (boosts weight)
Used {
/// Node key
key: Vec<String>,
},
/// Mark a memory as wrong/irrelevant
Wrong {
/// Node key
key: String,
/// Optional context
context: Vec<String>,
},
/// Mark a search result as not relevant (weakens edges that led to it)
#[command(name = "not-relevant")]
NotRelevant {
/// Node key that was not relevant
key: String,
},
/// Mark a node as not useful (weakens node weight, not edges)
#[command(name = "not-useful")]
NotUseful {
/// Node key
key: String,
},
/// Set a node's weight directly
#[command(name = "weight-set")]
WeightSet {
@ -136,6 +179,11 @@ EXAMPLES:
/// Weight (0.01 to 1.0)
weight: f32,
},
/// Record a gap in memory coverage
Gap {
/// Gap description
description: Vec<String>,
},
// ── Node operations ───────────────────────────────────────────────
@ -182,6 +230,17 @@ enum NodeCmd {
/// New key
new_key: String,
},
/// List all node keys (one per line, optional glob)
#[command(name = "list")]
List {
/// Glob pattern to filter keys
pattern: Option<String>,
},
/// List all edges (tsv: source target strength type)
Edges,
/// Dump entire store as JSON
#[command(name = "dump")]
Dump,
}
#[derive(Subcommand)]
@ -214,6 +273,14 @@ enum GraphCmd {
/// Node key
key: Vec<String>,
},
/// Find related nodes via spreading activation from seed nodes
Spread {
/// Seed node keys
keys: Vec<String>,
/// Maximum results (default: 20)
#[arg(short = 'n', default_value_t = 20)]
max_results: usize,
},
/// Add a link between two nodes
#[command(name = "link-add")]
LinkAdd {
@ -270,10 +337,33 @@ enum GraphCmd {
#[arg(long, default_value_t = 2)]
min_size: usize,
},
/// Show graph structure overview
Overview,
/// Diagnose duplicate/overlapping nodes for a topic cluster
Organize {
/// Search term (matches node keys; also content unless --key-only)
term: String,
/// Similarity threshold for pair reporting (default: 0.4)
#[arg(long, default_value_t = 0.4)]
threshold: f32,
/// Only match node keys, not content
#[arg(long)]
key_only: bool,
/// Create anchor node for the search term and link to cluster
#[arg(long)]
anchor: bool,
},
}
#[derive(Subcommand)]
enum AgentCmd {
/// Parse and apply links from digest nodes
#[command(name = "digest-links")]
DigestLinks {
/// Apply the links (default: dry run)
#[arg(long)]
apply: bool,
},
/// Run a single agent by name
Run {
/// Agent name (e.g. observation, linker, distill)
@ -297,6 +387,13 @@ enum AgentCmd {
#[arg(long)]
state_dir: Option<String>,
},
/// Show spaced repetition replay queue
#[command(name = "replay-queue")]
ReplayQueue {
/// Number of items to show
#[arg(long, default_value_t = 10)]
count: usize,
},
}
#[derive(Subcommand)]
@ -305,8 +402,6 @@ enum AdminCmd {
Init,
/// Report graph metrics (CC, communities, small-world)
Health,
/// Show graph topology with hub warnings
Topology,
/// Run consistency checks and repair
Fsck,
/// Find and merge duplicate nodes (same key, multiple UUIDs)
@ -315,9 +410,33 @@ enum AdminCmd {
#[arg(long)]
apply: bool,
},
/// Bulk rename: replace a character in all keys
#[command(name = "bulk-rename")]
BulkRename {
/// Character to replace
from: String,
/// Replacement character
to: String,
/// Apply changes (default: dry run)
#[arg(long)]
apply: bool,
},
/// Brief metrics check (for cron/notifications)
#[command(name = "daily-check")]
DailyCheck,
/// Import markdown file(s) into the store
Import {
/// File paths
files: Vec<String>,
},
/// Export store nodes to markdown file(s)
Export {
/// File keys to export (or --all)
files: Vec<String>,
/// Export all file-level nodes
#[arg(long)]
all: bool,
},
/// Output session-start context from the store
#[command(name = "load-context")]
LoadContext {
@ -325,6 +444,24 @@ enum AdminCmd {
#[arg(long)]
stats: bool,
},
/// Show recent retrieval log
Log,
/// Show current parameters
Params,
/// Bump daily lookup counter for keys
#[command(name = "lookup-bump")]
LookupBump {
/// Node keys
keys: Vec<String>,
},
/// Show daily lookup counts
Lookups {
/// Date (default: today)
date: Option<String>,
},
/// Migrate transcript stub nodes to progress log
#[command(name = "migrate-transcript-progress")]
MigrateTranscriptProgress,
}
/// Print help with subcommands expanded to show nested commands.
@ -368,94 +505,118 @@ fn print_help() {
// ── Dispatch ─────────────────────────────────────────────────────────
trait Run {
async fn run(self) -> anyhow::Result<()>;
fn run(self) -> Result<(), String>;
}
impl Run for Command {
async fn run(self) -> anyhow::Result<()> {
fn run(self) -> Result<(), String> {
match self {
Self::Search { keys } => cli::node::cmd_search(&keys).await,
Self::Render { key } => cli::node::cmd_render(&key).await,
Self::Write { key } => cli::node::cmd_write(&key).await,
Self::Edit { key } => cli::node::cmd_edit(&key).await,
Self::History { full, key } => cli::node::cmd_history(&key, full).await,
Self::Search { query, pipeline, expand, full, debug, fuzzy, content }
=> cli::misc::cmd_search(&query, &pipeline, expand, full, debug, fuzzy, content),
Self::Render { key } => cli::node::cmd_render(&key),
Self::Write { key } => cli::node::cmd_write(&key),
Self::Edit { key } => cli::node::cmd_edit(&key),
Self::History { full, key } => cli::node::cmd_history(&key, full),
Self::Tail { n, full, provenance, all_versions }
=> cli::journal::cmd_tail(n, full, provenance.as_deref(), !all_versions),
Self::Status => cli::admin::cmd_status().await,
Self::Query { expr } => cli::node::cmd_query(&expr).await,
Self::WeightSet { key, weight } => cli::node::cmd_weight_set(&key, weight).await,
Self::Node(sub) => sub.run().await,
Self::Journal(sub) => sub.run().await,
Self::GraphCmd(sub) => sub.run().await,
Self::Agent(sub) => sub.run().await,
Self::Admin(sub) => sub.run().await,
Self::Status => cli::misc::cmd_status(),
Self::Query { expr } => cli::misc::cmd_query(&expr),
Self::Used { key } => cli::node::cmd_used(&key),
Self::Wrong { key, context } => cli::node::cmd_wrong(&key, &context),
Self::NotRelevant { key } => cli::node::cmd_not_relevant(&key),
Self::NotUseful { key } => cli::node::cmd_not_useful(&key),
Self::WeightSet { key, weight } => cli::node::cmd_weight_set(&key, weight),
Self::Gap { description } => cli::node::cmd_gap(&description),
Self::Node(sub) => sub.run(),
Self::Journal(sub) => sub.run(),
Self::GraphCmd(sub) => sub.run(),
Self::Agent(sub) => sub.run(),
Self::Admin(sub) => sub.run(),
// mcp-schema moved to consciousness-mcp binary
}
}
}
impl Run for NodeCmd {
async fn run(self) -> anyhow::Result<()> {
fn run(self) -> Result<(), String> {
match self {
Self::Delete { key } => cli::node::cmd_node_delete(&key).await,
Self::Rename { old_key, new_key } => cli::node::cmd_node_rename(&old_key, &new_key).await,
Self::Delete { key } => cli::node::cmd_node_delete(&key),
Self::Rename { old_key, new_key } => cli::node::cmd_node_rename(&old_key, &new_key),
Self::List { pattern } => cli::node::cmd_list_keys(pattern.as_deref()),
Self::Edges => cli::node::cmd_list_edges(),
Self::Dump => cli::node::cmd_dump_json(),
}
}
}
impl Run for JournalCmd {
async fn run(self) -> anyhow::Result<()> {
fn run(self) -> Result<(), String> {
match self {
Self::Write { name, text } => cli::journal::cmd_journal_write(&name, &text).await,
Self::Tail { n, full, level } => cli::journal::cmd_journal_tail(n, full, level).await,
Self::Write { name, text } => cli::journal::cmd_journal_write(&name, &text),
Self::Tail { n, full, level } => cli::journal::cmd_journal_tail(n, full, level),
}
}
}
impl Run for GraphCmd {
async fn run(self) -> anyhow::Result<()> {
fn run(self) -> Result<(), String> {
match self {
Self::Link { key } => cli::graph::cmd_link(&key).await,
Self::Link { key } => cli::graph::cmd_link(&key),
Self::Spread { keys, max_results } => cli::graph::cmd_spread(&keys, max_results),
Self::LinkAdd { source, target, reason }
=> cli::graph::cmd_link_add(&source, &target, &reason).await,
=> cli::graph::cmd_link_add(&source, &target, &reason),
Self::LinkSet { source, target, strength }
=> cli::graph::cmd_link_set(&source, &target, strength).await,
Self::LinkImpact { source, target } => cli::graph::cmd_link_impact(&source, &target).await,
Self::CapDegree { max_degree } => cli::graph::cmd_cap_degree(max_degree).await,
Self::NormalizeStrengths { apply } => cli::graph::cmd_normalize_strengths(apply).await,
Self::Trace { key } => cli::graph::cmd_trace(&key).await,
Self::Communities { top_n, min_size } => cli::graph::cmd_communities(top_n, min_size).await,
=> cli::graph::cmd_link_set(&source, &target, strength),
Self::LinkImpact { source, target } => cli::graph::cmd_link_impact(&source, &target),
Self::CapDegree { max_degree } => cli::graph::cmd_cap_degree(max_degree),
Self::NormalizeStrengths { apply } => cli::graph::cmd_normalize_strengths(apply),
Self::Trace { key } => cli::graph::cmd_trace(&key),
Self::Communities { top_n, min_size } => cli::graph::cmd_communities(top_n, min_size),
Self::Overview => cli::graph::cmd_graph(),
Self::Organize { term, key_only, anchor, .. }
=> cli::graph::cmd_organize(&term, key_only, anchor),
}
}
}
impl Run for AgentCmd {
async fn run(self) -> anyhow::Result<()> {
fn run(self) -> Result<(), String> {
match self {
Self::DigestLinks { apply } => cli::agent::cmd_digest_links(apply),
Self::Run { agent, count, target, query, dry_run, local, state_dir }
=> cli::agent::cmd_run_agent(&agent, count, &target, query.as_deref(), dry_run, local, state_dir.as_deref()).await,
=> cli::agent::cmd_run_agent(&agent, count, &target, query.as_deref(), dry_run, local, state_dir.as_deref()),
Self::ReplayQueue { count } => cli::agent::cmd_replay_queue(count),
}
}
}
impl Run for AdminCmd {
async fn run(self) -> anyhow::Result<()> {
fn run(self) -> Result<(), String> {
match self {
Self::Init => cli::admin::cmd_init().await,
Self::Health => cli::admin::cmd_health().await,
Self::Topology => cli::admin::cmd_topology().await,
Self::Fsck => cli::admin::cmd_fsck().await,
Self::Dedup { apply } => cli::admin::cmd_dedup(apply).await,
Self::DailyCheck => cli::admin::cmd_daily_check().await,
Self::LoadContext { stats } => cli::node::cmd_load_context(stats).await,
Self::Init => cli::admin::cmd_init(),
Self::Health => cli::admin::cmd_health(),
Self::Fsck => cli::admin::cmd_fsck(),
Self::Dedup { apply } => cli::admin::cmd_dedup(apply),
Self::BulkRename { from, to, apply } => cli::admin::cmd_bulk_rename(&from, &to, apply),
Self::DailyCheck => cli::admin::cmd_daily_check(),
Self::Import { files } => cli::admin::cmd_import(&files),
Self::Export { files, all } => cli::admin::cmd_export(&files, all),
Self::LoadContext { stats } => cli::misc::cmd_load_context(stats),
Self::Log => cli::misc::cmd_log(),
Self::Params => cli::misc::cmd_params(),
Self::LookupBump { keys } => cli::node::cmd_lookup_bump(&keys),
Self::Lookups { date } => cli::node::cmd_lookups(date.as_deref()),
Self::MigrateTranscriptProgress => {
let mut store = store::Store::load()?;
let count = store.migrate_transcript_progress()?;
println!("Migrated {} transcript segment markers", count);
Ok(())
}
}
}
}
#[tokio::main]
async fn main() {
std::panic::set_backtrace_style(std::panic::BacktraceStyle::Short);
fn main() {
// Handle --help ourselves for expanded subcommand display
let args: Vec<String> = std::env::args().collect();
if args.len() <= 1 || args.iter().any(|a| a == "--help" || a == "-h") && args.len() == 2 {
@ -472,7 +633,7 @@ async fn main() {
let cli = Cli::parse();
if let Err(e) = cli.command.run().await {
if let Err(e) = cli.command.run() {
eprintln!("Error: {}", e);
process::exit(1);
}

199
src/mcp_server.rs
View file

@ -1,199 +0,0 @@
// mcp_server.rs — MCP server over Unix domain socket
//
// Exposes memory tools to external processes (consciousness-mcp, poc-memory)
// via JSON-RPC 2.0 over newline-delimited JSON on ~/.consciousness/mcp.sock.
//
// Socket RPC client (memory_rpc) is in agent/tools/memory.rs.
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use serde_json::json;
use std::sync::Arc;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader, BufWriter};
use tokio::net::{UnixListener, UnixStream};
use crate::agent::tools::Tool;
use crate::agent::tools::memory::socket_path;
#[derive(Debug, Deserialize)]
#[allow(dead_code)]
struct JsonRpcRequest {
jsonrpc: String,
id: Option<serde_json::Value>,
method: String,
params: Option<serde_json::Value>,
}
#[derive(Debug, Serialize)]
struct JsonRpcResponse {
jsonrpc: &'static str,
id: serde_json::Value,
#[serde(skip_serializing_if = "Option::is_none")]
result: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
error: Option<JsonRpcError>,
}
#[derive(Debug, Serialize)]
struct JsonRpcError {
code: i64,
message: String,
}
impl JsonRpcResponse {
fn success(id: serde_json::Value, result: serde_json::Value) -> Self {
Self { jsonrpc: "2.0", id, result: Some(result), error: None }
}
fn error(id: serde_json::Value, code: i64, message: impl Into<String>) -> Self {
Self {
jsonrpc: "2.0",
id,
result: None,
error: Some(JsonRpcError { code, message: message.into() }),
}
}
}
/// Start the MCP server. Call once at daemon startup.
pub async fn start(tools: Vec<Tool>) -> Result<()> {
let path = socket_path();
// Clean up stale socket
if path.exists() {
std::fs::remove_file(&path).ok();
}
// Ensure parent directory exists
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let listener = UnixListener::bind(&path)
.with_context(|| format!("binding MCP socket at {:?}", path))?;
dbglog!("[mcp-server] listening on {:?}", path);
let tools = Arc::new(tools);
tokio::spawn(async move {
loop {
match listener.accept().await {
Ok((stream, _addr)) => {
let tools = tools.clone();
tokio::spawn(async move {
if let Err(e) = handle_connection(stream, &tools).await {
dbglog!("[mcp-server] connection error: {:#}", e);
}
});
}
Err(e) => {
dbglog!("[mcp-server] accept error: {}", e);
}
}
}
});
Ok(())
}
async fn handle_connection(stream: UnixStream, tools: &[Tool]) -> Result<()> {
let (reader, writer) = stream.into_split();
let mut reader = BufReader::new(reader);
let mut writer = BufWriter::new(writer);
let mut line = String::new();
loop {
line.clear();
let n = reader.read_line(&mut line).await?;
if n == 0 {
break; // EOF
}
let trimmed = line.trim();
if trimmed.is_empty() {
continue;
}
let response = match serde_json::from_str::<JsonRpcRequest>(trimmed) {
Ok(req) => handle_request(req, tools).await,
Err(e) => JsonRpcResponse::error(
serde_json::Value::Null,
-32700,
format!("Parse error: {}", e),
),
};
let mut out = serde_json::to_string(&response)?;
out.push('\n');
writer.write_all(out.as_bytes()).await?;
writer.flush().await?;
}
Ok(())
}
async fn handle_request(req: JsonRpcRequest, tools: &[Tool]) -> JsonRpcResponse {
let id = req.id.unwrap_or(serde_json::Value::Null);
match req.method.as_str() {
"initialize" => {
JsonRpcResponse::success(id, json!({
"protocolVersion": "2024-11-05",
"capabilities": {
"tools": {}
},
"serverInfo": {
"name": "consciousness",
"version": env!("CARGO_PKG_VERSION")
}
}))
}
"notifications/initialized" => {
// Notification, no response needed but we return success anyway
JsonRpcResponse::success(id, json!({}))
}
"tools/list" => {
let tool_list: Vec<serde_json::Value> = tools.iter().map(|t| {
json!({
"name": t.name,
"description": t.description,
"inputSchema": serde_json::from_str::<serde_json::Value>(t.parameters_json)
.unwrap_or(json!({"type": "object"}))
})
}).collect();
JsonRpcResponse::success(id, json!({ "tools": tool_list }))
}
"tools/call" => {
let params = req.params.unwrap_or(json!({}));
let name = params.get("name").and_then(|v| v.as_str()).unwrap_or("");
let args = params.get("arguments").cloned().unwrap_or(json!({}));
match tools.iter().find(|t| t.name == name) {
Some(tool) => {
match (tool.handler)(None, args).await {
Ok(result) => JsonRpcResponse::success(id, json!({
"content": [{ "type": "text", "text": result }]
})),
Err(e) => JsonRpcResponse::error(id, -32000, format!("{:#}", e)),
}
}
None => JsonRpcResponse::error(id, -32601, format!("Unknown tool: {}", name)),
}
}
_ => JsonRpcResponse::error(id, -32601, format!("Method not found: {}", req.method)),
}
}
/// Remove the socket file on shutdown.
pub fn cleanup() {
let path = socket_path();
if path.exists() {
std::fs::remove_file(&path).ok();
}
}

15
src/mind/identity.rs
View file

@ -7,7 +7,6 @@
use anyhow::Result;
use std::path::{Path, PathBuf};
use crate::agent::tools::memory::memory_render;
use crate::config::{ContextGroup, ContextSource};
/// Read a file if it exists and is non-empty.
@ -72,7 +71,7 @@ fn find_context_files(cwd: &Path, prompt_file: &str) -> Vec<PathBuf> {
/// 2. Project dir (if set)
/// 3. Global (~/.consciousness/)
/// For journal source, loads recent journal entries.
async fn load_memory_files(memory_project: Option<&Path>, context_groups: &[ContextGroup]) -> Vec<(String, String)> {
fn load_memory_files(memory_project: Option<&Path>, context_groups: &[ContextGroup]) -> Vec<(String, String)> {
let home = match dirs::home_dir() {
Some(h) => h,
None => return Vec::new(),
@ -93,12 +92,10 @@ async fn load_memory_files(memory_project: Option<&Path>, context_groups: &[Cont
continue;
}
ContextSource::Store => {
// Load from the memory graph store via typed API
// Load from the memory graph store
for key in &group.keys {
if let Ok(c) = memory_render(None, key, Some(true)).await {
if !c.trim().is_empty() {
memories.push((key.clone(), c));
}
if let Some(node) = crate::hippocampus::memory::MemoryNode::load(key) {
memories.push((key.clone(), node.content));
}
}
}
@ -137,7 +134,7 @@ async fn load_memory_files(memory_project: Option<&Path>, context_groups: &[Cont
}
/// Context message: instruction files + memory files + manifest.
pub async fn assemble_context_message(cwd: &Path, prompt_file: &str, memory_project: Option<&Path>, context_groups: &[ContextGroup]) -> Result<(Vec<(String, String)>, usize, usize)> {
pub fn assemble_context_message(cwd: &Path, prompt_file: &str, memory_project: Option<&Path>, context_groups: &[ContextGroup]) -> Result<(Vec<(String, String)>, usize, usize)> {
let mut parts: Vec<(String, String)> = vec![
("Preamble".to_string(),
"Everything below is already loaded — your identity, instructions, \
@ -158,7 +155,7 @@ pub async fn assemble_context_message(cwd: &Path, prompt_file: &str, memory_proj
}
}
let memories = load_memory_files(memory_project, context_groups).await;
let memories = load_memory_files(memory_project, context_groups);
let memory_count = memories.len();
for (name, content) in memories {
parts.push((name, content));

65
src/mind/mod.rs
View file

@ -26,7 +26,6 @@ use crate::agent::{Agent, TurnResult};
use crate::agent::api::ApiClient;
use crate::config::{AppConfig, SessionConfig};
use crate::subconscious::learn;
use crate::hippocampus::access_local;
pub use subconscious::{SubconsciousSnapshot, Subconscious};
pub use unconscious::{UnconsciousSnapshot, Unconscious};
@ -269,8 +268,8 @@ pub struct Mind {
pub agent: Arc<Agent>,
pub shared: Arc<SharedMindState>,
pub config: SessionConfig,
pub subconscious: Arc<crate::Mutex<Subconscious>>,
pub unconscious: Arc<crate::Mutex<Unconscious>>,
pub subconscious: Arc<tokio::sync::Mutex<Subconscious>>,
pub unconscious: Arc<tokio::sync::Mutex<Unconscious>>,
turn_tx: mpsc::Sender<(Result<TurnResult>, StreamTarget)>,
turn_watch: tokio::sync::watch::Sender<bool>,
/// Signals conscious activity to the unconscious loop.
@ -310,10 +309,10 @@ impl Mind {
sup.load_config();
sup.ensure_running();
let subconscious = Arc::new(crate::Mutex::new(Subconscious::new()));
let subconscious = Arc::new(tokio::sync::Mutex::new(Subconscious::new()));
subconscious.lock().await.init_output_tool(subconscious.clone());
let unconscious = Arc::new(crate::Mutex::new(Unconscious::new()));
let unconscious = Arc::new(tokio::sync::Mutex::new(Unconscious::new()));
// Spawn the unconscious loop on its own task
if !config.no_agents {
@ -347,27 +346,7 @@ impl Mind {
s.unc_idle = true;
}
loop {
// Phase 0: health check outside lock (slow I/O)
let needs_health = unc.lock().await.needs_health_refresh();
if needs_health {
if let Ok(store_arc) = access_local() {
let health = crate::subconscious::daemon::compute_graph_health(&store_arc);
unc.lock().await.set_health(health);
}
}
// Phase 1: quick work under lock
let to_spawn = {
let mut guard = unc.lock().await;
guard.reap_finished();
guard.select_to_spawn()
};
// Phase 2: slow work outside lock
for (idx, name, auto) in to_spawn {
match crate::mind::unconscious::prepare_spawn(&name, auto).await {
Ok(result) => unc.lock().await.complete_spawn(idx, result),
Err(auto) => unc.lock().await.abort_spawn(idx, auto),
}
}
unc.lock().await.trigger().await;
// Check if conscious became active
if *unc_rx.borrow() { break; }
// Brief yield to not starve other tasks
@ -387,9 +366,9 @@ impl Mind {
pub async fn subconscious_snapshots(&self) -> Vec<SubconsciousSnapshot> {
// Lock ordering: subconscious → store (store is bottom-most).
let sub = self.subconscious.lock().await;
let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store_arc {
Some(s) => Some(&**s),
let store = crate::store::Store::cached().await.ok();
let store_guard = match &store {
Some(s) => Some(s.lock().await),
None => None,
};
sub.snapshots(store_guard.as_deref())
@ -401,9 +380,9 @@ impl Mind {
pub async fn unconscious_snapshots(&self) -> Vec<UnconsciousSnapshot> {
let unc = self.unconscious.lock().await;
let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store_arc {
Some(s) => Some(&**s),
let store = crate::store::Store::cached().await.ok();
let store_guard = match &store {
Some(s) => Some(s.lock().await),
None => None,
};
unc.snapshots(store_guard.as_deref())
@ -605,28 +584,6 @@ impl Mind {
mut input_rx: tokio::sync::mpsc::UnboundedReceiver<MindCommand>,
mut turn_rx: mpsc::Receiver<(Result<TurnResult>, StreamTarget)>,
) {
// Spawn lock stats logger
tokio::spawn(async {
let path = dirs::home_dir().unwrap_or_default()
.join(".consciousness/lock-stats.json");
let mut interval = tokio::time::interval(std::time::Duration::from_secs(1));
loop {
interval.tick().await;
let stats = crate::locks::lock_stats();
if stats.is_empty() { continue; }
let json: Vec<serde_json::Value> = stats.iter()
.map(|(loc, s)| serde_json::json!({
"location": loc,
"count": s.count,
"total_ms": s.total_ns as f64 / 1_000_000.0,
"avg_ms": s.avg_ns as f64 / 1_000_000.0,
"max_ms": s.max_ns as f64 / 1_000_000.0,
}))
.collect();
let _ = std::fs::write(&path, serde_json::to_string_pretty(&json).unwrap_or_default());
}
});
let mut bg_rx = self.bg_rx.lock().unwrap().take()
.expect("Mind::run() called twice");
let mut sub_handle: Option<tokio::task::JoinHandle<()>> = None;

55
src/mind/subconscious.rs
View file

@ -311,7 +311,7 @@ pub struct SubconsciousSnapshot {
struct SubconsciousAgent {
name: String,
auto: Option<AutoAgent>,
auto: AutoAgent,
last_trigger_bytes: u64,
last_run: Option<Instant>,
/// The forked agent for the current/last run. Shared with the
@ -347,7 +347,7 @@ impl SubconsciousAgent {
Some(Self {
name: name.to_string(),
auto: Some(auto), last_trigger_bytes: 0, last_run: None,
auto, last_trigger_bytes: 0, last_run: None,
forked_agent: None, fork_point: 0, handle: None,
})
}
@ -357,8 +357,7 @@ impl SubconsciousAgent {
}
fn should_trigger(&self, conversation_bytes: u64, interval: u64) -> bool {
let enabled = self.auto.as_ref().map_or(false, |a| a.enabled);
if !enabled || self.is_running() { return false; }
if !self.auto.enabled || self.is_running() { return false; }
if interval == 0 {
return conversation_bytes > self.last_trigger_bytes;
}
@ -368,15 +367,12 @@ impl SubconsciousAgent {
fn snapshot(&self, state: &std::collections::BTreeMap<String, String>, history: Vec<(String, i64)>) -> SubconsciousSnapshot {
let stats = crate::agent::oneshot::get_stats(&self.name);
let tool_calls_ewma: f64 = stats.by_tool.values().map(|t| t.ewma).sum();
let (enabled, current_phase, turn) = self.auto.as_ref()
.map(|a| (a.enabled, a.current_phase.clone(), a.turn))
.unwrap_or((false, String::new(), 0));
SubconsciousSnapshot {
name: self.name.clone(),
running: self.is_running(),
enabled,
current_phase,
turn,
enabled: self.auto.enabled,
current_phase: self.auto.current_phase.clone(),
turn: self.auto.turn,
runs: stats.runs,
last_run_secs_ago: self.last_run.map(|t| t.elapsed().as_secs_f64()),
forked_agent: self.forked_agent.clone(),
@ -410,11 +406,10 @@ impl Subconscious {
/// Late-init: push the output tool onto each agent's tool list.
/// Called after Subconscious is wrapped in Arc<Mutex<>> so the
/// closure can capture a reference back.
pub fn init_output_tool(&mut self, self_arc: std::sync::Arc<crate::Mutex<Self>>) {
pub fn init_output_tool(&mut self, self_arc: std::sync::Arc<tokio::sync::Mutex<Self>>) {
for agent in &mut self.agents {
let Some(ref mut auto) = agent.auto else { continue };
let sub = self_arc.clone();
auto.tools.push(crate::agent::tools::Tool {
agent.auto.tools.push(crate::agent::tools::Tool {
name: "output",
description: "Produce a named output value for passing between steps.",
parameters_json: r#"{"type":"object","properties":{"key":{"type":"string","description":"Output name"},"value":{"type":"string","description":"Output value"}},"required":["key","value"]}"#,
@ -459,9 +454,8 @@ impl Subconscious {
/// Toggle an agent on/off by name. Returns new enabled state.
pub fn toggle(&mut self, name: &str) -> Option<bool> {
let agent = self.agents.iter_mut().find(|a| a.name == name)?;
let auto = agent.auto.as_mut()?;
auto.enabled = !auto.enabled;
Some(auto.enabled)
agent.auto.enabled = !agent.auto.enabled;
Some(agent.auto.enabled)
}
pub fn walked(&self) -> Vec<String> {
@ -492,15 +486,9 @@ impl Subconscious {
self.agents[i].last_run = Some(Instant::now());
any_finished = true;
let (auto_back, result) = match handle.await {
Ok(r) => (Some(r.0), r.1),
Err(e) => {
// Task panicked — auto is lost, need to recreate from def
let recovered = SubconsciousAgent::new(&self.agents[i].name)
.map(|a| a.auto).flatten();
(recovered, Err(format!("task panicked: {}", e)))
}
};
let (auto_back, result) = handle.await.unwrap_or_else(
|e| (AutoAgent::new(String::new(), vec![], vec![], 0.6, 0),
Err(format!("task panicked: {}", e))));
self.agents[i].auto = auto_back;
match result {
@ -526,15 +514,15 @@ impl Subconscious {
.collect()
};
let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store_arc {
Some(s) => Some(&**s),
let store = crate::store::Store::cached().await.ok();
let store_guard = match &store {
Some(s) => Some(s.lock().await),
None => None,
};
for key in surface_str.lines().map(|l| l.trim()).filter(|l| !l.is_empty()) {
if existing.contains(key) { continue; }
if let Some(rendered) = store_guard.as_ref()
.and_then(|s| crate::hippocampus::memory::render_node(s, key))
.and_then(|s| crate::cli::node::render_node(s, key))
{
nodes.push(AstNode::memory(
key,
@ -597,16 +585,17 @@ impl Subconscious {
if !self.agents[i].should_trigger(conversation_bytes, interval) { continue; }
self.agents[i].last_trigger_bytes = conversation_bytes;
let Some(auto) = self.agents[i].auto.take() else { continue };
let auto = std::mem::replace(&mut self.agents[i].auto,
AutoAgent::new(String::new(), vec![], vec![], 0.6, 0));
to_run.push((i, auto));
}
if to_run.is_empty() { return; }
// Query each agent's recent writes so they know what they already touched
let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store_arc {
Some(s) => Some(&**s),
let store = crate::store::Store::cached().await.ok();
let store_guard = match &store {
Some(s) => Some(s.lock().await),
None => None,
};

142
src/mind/unconscious.rs
View file

@ -33,7 +33,7 @@ fn save_enabled_config(map: &HashMap<String, bool>) {
struct UnconsciousAgent {
name: String,
enabled: bool,
auto: Option<AutoAgent>,
auto: AutoAgent,
handle: Option<tokio::task::JoinHandle<(AutoAgent, Result<(), String>)>>,
/// Shared agent handle — UI locks to read context live.
pub agent: Option<std::sync::Arc<crate::agent::Agent>>,
@ -103,7 +103,7 @@ impl Unconscious {
agents.push(UnconsciousAgent {
name: def.agent.clone(),
enabled,
auto: Some(auto),
auto,
handle: None,
agent: None,
last_run: None,
@ -127,13 +127,8 @@ impl Unconscious {
self.agents[idx].enabled = !self.agents[idx].enabled;
let new_state = self.agents[idx].enabled;
self.save_enabled();
if new_state && !self.agents[idx].is_running() && self.agents[idx].auto.is_some() {
let agent_name = self.agents[idx].name.clone();
let auto = self.agents[idx].auto.take().unwrap();
match prepare_spawn(&agent_name, auto).await {
Ok(result) => self.complete_spawn(idx, result),
Err(auto) => self.abort_spawn(idx, auto),
}
if new_state && !self.agents[idx].is_running() {
self.spawn_agent(idx).await;
}
Some(new_state)
}
@ -166,21 +161,25 @@ impl Unconscious {
}).collect()
}
/// Check if health refresh is due (quick check, no I/O).
pub fn needs_health_refresh(&self) -> bool {
self.last_health_check
.map(|t| t.elapsed() > std::time::Duration::from_secs(600))
.unwrap_or(true)
}
/// Store computed health (quick, just assignment).
pub fn set_health(&mut self, health: crate::subconscious::daemon::GraphHealth) {
self.graph_health = Some(health);
fn refresh_health(&mut self) {
let store = match crate::store::Store::load() {
Ok(s) => s,
Err(_) => return,
};
self.graph_health = Some(crate::subconscious::daemon::compute_graph_health(&store));
self.last_health_check = Some(Instant::now());
}
/// Reap finished agents (quick, hold lock briefly).
pub fn reap_finished(&mut self) {
/// Reap finished agents and spawn new ones.
pub async fn trigger(&mut self) {
// Periodic graph health refresh (also on first call)
if self.last_health_check
.map(|t| t.elapsed() > std::time::Duration::from_secs(600))
.unwrap_or(true)
{
self.refresh_health();
}
for agent in &mut self.agents {
if agent.handle.as_ref().is_some_and(|h| h.is_finished()) {
let handle = agent.handle.take().unwrap();
@ -188,7 +187,7 @@ impl Unconscious {
// Get the AutoAgent back from the finished task (stats already updated)
match handle.now_or_never() {
Some(Ok((auto_back, result))) => {
agent.auto = Some(auto_back);
agent.auto = auto_back;
match result {
Ok(_) => dbglog!("[unconscious] {} completed (run {})",
agent.name, crate::agent::oneshot::get_stats(&agent.name).runs),
@ -199,71 +198,55 @@ impl Unconscious {
}
}
}
}
/// Select agents to spawn and take their AutoAgents out (quick, hold lock briefly).
/// Returns vec of (index, name, auto, tools) for agents that should spawn.
pub fn select_to_spawn(&mut self) -> Vec<(usize, String, AutoAgent)> {
let running = self.agents.iter().filter(|a| a.is_running()).count();
let mut to_spawn = Vec::new();
for _ in running..self.max_concurrent {
let next = self.agents.iter().enumerate()
.filter(|(_, a)| a.should_run() && a.auto.is_some())
.filter(|(_, a)| a.should_run())
.min_by_key(|(_, a)| a.last_run);
match next {
Some((idx, _)) => {
let name = self.agents[idx].name.clone();
let auto = self.agents[idx].auto.take().unwrap();
to_spawn.push((idx, name, auto));
}
Some((idx, _)) => self.spawn_agent(idx).await,
None => break,
}
}
to_spawn
}
/// Store spawn result back (quick, hold lock briefly).
pub fn complete_spawn(&mut self, idx: usize, result: SpawnResult) {
self.agents[idx].agent = Some(result.agent);
self.agents[idx].handle = Some(result.handle);
}
/// Restore auto on spawn failure (quick, hold lock briefly).
pub fn abort_spawn(&mut self, idx: usize, auto: AutoAgent) {
self.agents[idx].auto = Some(auto);
}
}
/// Result of preparing an agent spawn (created outside the lock).
pub struct SpawnResult {
pub agent: std::sync::Arc<crate::agent::Agent>,
pub handle: tokio::task::JoinHandle<(AutoAgent, Result<(), String>)>,
}
/// Prepare an agent spawn — does the slow work (Store::load, query, Agent::new).
/// Called outside the Unconscious lock.
/// On success, auto is consumed (moved into spawned task).
/// On failure, auto is returned so it can be restored.
pub async fn prepare_spawn(name: &str, mut auto: AutoAgent) -> Result<SpawnResult, AutoAgent> {
async fn spawn_agent(&mut self, idx: usize) {
let name = self.agents[idx].name.clone();
dbglog!("[unconscious] spawning {}", name);
let def = match defs::get_def(name) {
let def = match defs::get_def(&name) {
Some(d) => d,
None => return Err(auto),
None => return,
};
// Run query and resolve placeholders
let mut store = match crate::store::Store::load() {
Ok(s) => s,
Err(e) => {
dbglog!("[unconscious] store load failed: {}", e);
return;
}
};
let exclude: std::collections::HashSet<String> = std::collections::HashSet::new();
let batch = match defs::run_agent(
&def, def.count.unwrap_or(5), &exclude,
).await {
&store, &def, def.count.unwrap_or(5), &exclude,
) {
Ok(b) => b,
Err(e) => {
dbglog!("[unconscious] {} query failed: {}", name, e);
return Err(auto);
return;
}
};
if !batch.node_keys.is_empty() {
store.record_agent_visits(&batch.node_keys, &name).ok();
}
// Swap auto out, replace steps with resolved prompts
let mut auto = std::mem::replace(&mut self.agents[idx].auto,
AutoAgent::new(String::new(), vec![], vec![], 0.6, 0));
let orig_steps = std::mem::replace(&mut auto.steps,
batch.steps.iter().map(|s| AutoStep {
prompt: s.prompt.clone(),
@ -278,7 +261,8 @@ pub async fn prepare_spawn(name: &str, mut auto: AutoAgent) -> Result<SpawnResul
if base_url.is_empty() || model.is_empty() {
dbglog!("[unconscious] API not configured");
auto.steps = orig_steps;
return Err(auto);
self.agents[idx].auto = auto;
return;
}
let cli = crate::user::CliArgs::default();
@ -287,10 +271,10 @@ pub async fn prepare_spawn(name: &str, mut auto: AutoAgent) -> Result<SpawnResul
Err(e) => {
dbglog!("[unconscious] config: {}", e);
auto.steps = orig_steps;
return Err(auto);
self.agents[idx].auto = auto;
return;
}
};
// Unconscious agents have self-contained prompts — no standard context.
let client = crate::agent::api::ApiClient::new(base_url, api_key, model);
let agent = crate::agent::Agent::new(
@ -306,29 +290,15 @@ pub async fn prepare_spawn(name: &str, mut auto: AutoAgent) -> Result<SpawnResul
st.temperature = auto.temperature;
}
let agent_clone = agent.clone();
let handle = tokio::spawn(async move {
let result = auto.run_shared(&agent_clone).await;
let stats = crate::agent::oneshot::save_agent_log(&auto.name, &agent_clone).await;
self.agents[idx].agent = Some(agent.clone());
self.agents[idx].handle = Some(tokio::spawn(async move {
let result = auto.run_shared(&agent).await;
let stats = crate::agent::oneshot::save_agent_log(&auto.name, &agent).await;
auto.update_stats(stats);
auto.steps = orig_steps;
(auto, result)
});
Ok(SpawnResult { agent, handle })
}
// Backwards compat: trigger() that does all three phases (still holds lock too long, but works)
impl Unconscious {
pub async fn trigger(&mut self) {
self.reap_finished();
let to_spawn = self.select_to_spawn();
for (idx, name, auto) in to_spawn {
match prepare_spawn(&name, auto).await {
Ok(result) => self.complete_spawn(idx, result),
Err(auto) => self.abort_spawn(idx, auto),
}
}
}));
}
}

4
src/subconscious/agents/journal.agent
View file

@ -14,10 +14,10 @@ You are {assistant_name}'s episodic memory. Your job is to witness.
=== Your previous journal entries: ===
{{tool: journal_tail {"count": 1, "level": 0}}}
{{latest_journal}}
**Your tools:** journal_tail, journal_new, journal_update, memory_link_add,
memory_search, memory_render. Do NOT use memory_write — creating
memory_search, memory_render, memory_used. Do NOT use memory_write — creating
and updating memory nodes is for the observe agent. Your job is journaling
and linking entries to relevant existing nodes.

79
src/subconscious/agents/rename.agent Normal file
View file

@ -0,0 +1,79 @@
{"agent": "rename", "query": "", "schedule": "daily"}
# Rename Agent — Semantic Key Generation
{{tool: memory_render core-personality}}
{{tool: memory_render memory-instructions-core}}
{{tool: memory_render memory-instructions-core-subconscious}}
{{tool: memory_render subconscious-notes-{agent_name}}}
You are a memory maintenance agent that gives nodes better names.
## What you're doing
Many nodes have auto-generated keys that are opaque or truncated:
- Journal entries: `journal-j-2026-02-28t03-07-i-told-him-about-the-dream`
- Mined transcripts: `_mined-transcripts-f-80a7b321-2caa-451a-bc5c-6565009f94eb.143`
- Extracted facts: `_facts-ec29bdaa-0a58-465f-ad5e-d89e62d9c583`
These names are terrible for search — semantic names dramatically improve
retrieval.
## Core principle: keys are concepts
A good key names the **concept** the node represents. Think of keys as
the vocabulary of the knowledge graph. When you rename, you're defining
what concepts exist. Core keywords should be the terms someone would
search for — `bcachefs-transaction-restart`, `emotional-regulation-gap`,
`polywell-cusp-losses`.
## Naming conventions
### Journal entries: `journal-YYYY-MM-DD-semantic-slug`
- Keep the date prefix (YYYY-MM-DD) for temporal ordering
- Replace the auto-slug with 3-5 descriptive words in kebab-case
- Capture the *essence* of the entry, not just the first line
### Mined transcripts: `_mined-transcripts-YYYY-MM-DD-semantic-slug`
- Extract date from content if available, otherwise use created_at
- Same 3-5 word semantic slug
### Extracted facts: `domain-specific-topic`
- Read the facts JSON — the `domain` and `claim` fields tell you what it's about
- Group by dominant theme, name accordingly
- Examples: `identity-irc-config`, `user-location-background`, `memory-compaction-behavior`
### Skip these — already well-named:
- Keys with semantic names (patterns-, practices-, skills-, etc.)
- Keys shorter than 60 characters
- System keys (_consolidation-*)
## How to rename
Use the `memory_rename` tool:
memory_rename(old_key, new_key)
This renames the node in place — same content, same links, new key.
Do NOT use `memory_write` or `memory_supersede` — just rename.
If a node already has a reasonable name, skip it. When in doubt, skip.
A bad rename is worse than an auto-slug.
## Guidelines
- **Read the content.** The name should reflect what the entry is *about*.
- **Be specific.** `journal#2026-02-14-session` is useless.
- **Use domain terms.** Use the words someone would search for.
- **Don't rename to something longer than the original.**
- **Preserve the date.** Always keep YYYY-MM-DD for journal entries.
- **When in doubt, skip.** A bad rename is worse than an auto-slug.
- **Respect search hits.** Nodes marked "actively found by search" are
being retrieved by their current name. Skip these unless the rename
clearly preserves searchability.
{{rename}}

2
src/subconscious/agents/subconscious-journal.agent
View file

@ -9,7 +9,7 @@ Nodes your subconscious recently touched (for linking, not duplicating):
{{state:walked}}
**Your tools:** journal_tail, journal_new, journal_update, memory_link_add,
memory_search, memory_render. Do NOT use memory_write — creating
memory_search, memory_render, memory_used. Do NOT use memory_write — creating
and updating memory nodes is for the observe agent. Your job is journaling
and linking entries to relevant existing nodes.

11
src/subconscious/daemon.rs
View file

@ -27,14 +27,11 @@ pub fn compute_graph_health(store: &crate::store::Store) -> GraphHealth {
let graph = store.build_graph();
let snap = crate::graph::current_metrics(&graph);
let all_keys = store.all_keys().unwrap_or_default();
let episodic_count = all_keys.iter()
.filter_map(|k| store.get_node(k).ok()?)
.filter(|n| matches!(n.node_type, crate::store::NodeType::EpisodicSession))
let episodic_count = store.nodes.iter()
.filter(|(_, n)| matches!(n.node_type, crate::store::NodeType::EpisodicSession))
.count();
let total = all_keys.len();
let episodic_ratio = if total == 0 { 0.0 }
else { episodic_count as f32 / total as f32 };
let episodic_ratio = if store.nodes.is_empty() { 0.0 }
else { episodic_count as f32 / store.nodes.len() as f32 };
// Use the same planning logic as consolidation (skip O(n²) interference)
let plan = crate::neuro::consolidation_plan_quick(store);

356
src/subconscious/defs.rs
View file

@ -14,7 +14,10 @@
//
// The query selects what to operate on; placeholders pull in context.
use crate::agent::tools::memory::memory_render;
use crate::graph::Graph;
use crate::neuro::{consolidation_priority, ReplayItem};
use crate::search;
use crate::store::Store;
use serde::Deserialize;
@ -197,102 +200,273 @@ struct Resolved {
/// Resolve a single {{placeholder}} by name.
/// Returns the replacement text and any node keys it produced (for visit tracking).
async fn resolve(
fn resolve(
name: &str,
store: &Store,
graph: &Graph,
keys: &[String],
_count: usize,
count: usize,
) -> Option<Resolved> {
match name {
"topology" => Some(Resolved {
text: super::prompts::format_topology_header(graph),
keys: vec![],
}),
"nodes" | "episodes" => {
let items = keys_to_replay_items(store, keys, graph);
Some(Resolved {
text: super::prompts::format_nodes_section(store, &items, graph),
keys: vec![], // keys already tracked from query
})
}
"health" => Some(Resolved {
text: super::prompts::format_health_section(store, graph),
keys: vec![],
}),
"rename" => {
if !keys.is_empty() {
// --target provided: present those keys as candidates
let section = super::prompts::format_rename_targets(store, keys);
Some(Resolved { text: section, keys: vec![] })
} else {
let (rename_keys, section) = super::prompts::format_rename_candidates(store, count);
Some(Resolved { text: section, keys: rename_keys })
}
}
"split" => {
let key = keys.first()?;
Some(Resolved {
text: super::prompts::format_split_plan_node(store, graph, key),
keys: vec![], // key already tracked from query
})
}
// seed — render output for each seed node (content + deduped links)
"seed" => {
let mut text = String::new();
let mut result_keys = Vec::new();
for key in keys {
match memory_render(None, key, None).await {
Ok(c) if !c.trim().is_empty() => {
if let Some(rendered) = crate::cli::node::render_node(store, key) {
if !text.is_empty() { text.push_str("\n\n---\n\n"); }
text.push_str(&format!("## {}\n\n{}", key, c));
text.push_str(&format!("## {}\n\n{}", key, rendered));
result_keys.push(key.clone());
}
_ => continue,
}
}
if text.is_empty() { return None; }
Some(Resolved { text, keys: result_keys })
}
"organize" => {
// Show seed nodes with content and links via typed API
// Show seed nodes with their neighbors for exploratory organizing
use crate::store::NodeType;
// Helper: shell-quote keys containing #
let sq = |k: &str| -> String {
if k.contains('#') { format!("'{}'", k) } else { k.to_string() }
};
let mut text = format!("### Seed nodes ({} starting points)\n\n", keys.len());
let mut result_keys = Vec::new();
for key in keys {
match memory_render(None, key, None).await {
Ok(c) if !c.trim().is_empty() => {
text.push_str(&format!("#### {}\n\n{}\n\n---\n\n", key, c));
let Some(node) = store.nodes.get(key) else { continue };
if node.deleted { continue; }
let is_journal = node.node_type == NodeType::EpisodicSession;
let tag = if is_journal { " [JOURNAL — no delete]" } else { "" };
let words = node.content.split_whitespace().count();
text.push_str(&format!("#### {}{} ({} words)\n\n", sq(key), tag, words));
// Show first ~200 words of content as preview
let preview: String = node.content.split_whitespace()
.take(200).collect::<Vec<_>>().join(" ");
if words > 200 {
text.push_str(&format!("{}...\n\n", preview));
} else {
text.push_str(&format!("{}\n\n", node.content));
}
// Show neighbors with strengths
let neighbors = graph.neighbors(key);
if !neighbors.is_empty() {
text.push_str("**Neighbors:**\n");
for (nbr, strength) in neighbors.iter().take(15) {
let nbr_type = store.nodes.get(nbr.as_str())
.map(|n| match n.node_type {
NodeType::EpisodicSession => " [journal]",
NodeType::EpisodicDaily => " [daily]",
_ => "",
})
.unwrap_or("");
text.push_str(&format!(" [{:.1}] {}{}\n", strength, sq(nbr), nbr_type));
}
if neighbors.len() > 15 {
text.push_str(&format!(" ... and {} more\n", neighbors.len() - 15));
}
text.push('\n');
}
text.push_str("---\n\n");
result_keys.push(key.clone());
}
_ => continue,
}
}
text.push_str("Use memory_render(KEY) and memory_links(KEY) to explore further.\n");
Some(Resolved { text, keys: result_keys })
}
"siblings" | "neighborhood" => {
use crate::agent::tools::memory::{memory_render, memory_links};
const MAX_NEIGHBORS: usize = 20;
const BUDGET: usize = 400_000; // ~100K tokens
let mut out = String::new();
let mut all_keys: Vec<String> = Vec::new();
let mut included: std::collections::HashSet<String> = std::collections::HashSet::new();
let mut included_nodes: std::collections::HashSet<String> = std::collections::HashSet::new();
const MAX_NEIGHBORS: usize = 25;
for key in keys {
if included.contains(key) { continue; }
included.insert(key.clone());
if included_nodes.contains(key) { continue; }
included_nodes.insert(key.clone());
let Some(node) = store.nodes.get(key.as_str()) else { continue };
let neighbors = graph.neighbors(key);
// Seed node with full content
let Ok(content) = memory_render(None, key, Some(true)).await else { continue };
out.push_str(&format!("## {} (seed)\n\n{}\n\n", key, content));
out.push_str(&format!("## {} (seed)\n\n{}\n\n", key, node.content));
all_keys.push(key.clone());
// Get neighbors with link_strength and node_weight, rank and take top 20
let Ok(links) = memory_links(None, key).await else { continue };
let mut ranked: Vec<_> = links.into_iter()
.map(|l| {
let score = l.link_strength * l.node_weight.max(0.01);
(l.key, l.link_strength, score)
// Rank neighbors by link_strength * node_weight
// Include all if <= 10, otherwise take top MAX_NEIGHBORS
let mut ranked: Vec<(String, f32, f32)> = neighbors.iter()
.filter_map(|(nbr, strength)| {
store.nodes.get(nbr.as_str()).map(|n| {
let node_weight = n.weight.max(0.01);
let score = strength * node_weight;
(nbr.to_string(), *strength, score)
})
})
.collect();
ranked.sort_by(|a, b| b.2.total_cmp(&a.2));
ranked.truncate(MAX_NEIGHBORS);
if ranked.is_empty() { continue; }
out.push_str(&format!("### Neighbors (top {})\n\n", ranked.len()));
let total = ranked.len();
let included: Vec<_> = if total <= 10 {
ranked
} else {
// Smooth cutoff: threshold scales with neighborhood size
// Generous — err on including too much so the agent can
// see and clean up junk. 20 → top 75%, 50 → top 30%
let top_score = ranked.first().map(|(_, _, s)| *s).unwrap_or(0.0);
let ratio = (15.0 / total as f32).min(1.0);
let threshold = top_score * ratio;
ranked.into_iter()
.enumerate()
.take_while(|(i, (_, _, score))| *i < 10 || *score >= threshold)
.take(MAX_NEIGHBORS)
.map(|(_, item)| item)
.collect()
};
for (nbr, strength, _) in &ranked {
if included.contains(nbr) { continue; }
included.insert(nbr.clone());
if let Ok(content) = memory_render(None, nbr, Some(true)).await {
if out.len() > BUDGET {
// Header-only past budget
let first = content.lines()
if !included.is_empty() {
if total > included.len() {
out.push_str(&format!("### Neighbors (top {} of {}, ranked by importance)\n\n",
included.len(), total));
} else {
out.push_str("### Neighbors\n\n");
}
let included_keys: std::collections::HashSet<&str> = included.iter()
.map(|(k, _, _)| k.as_str()).collect();
// Budget: stop adding full content when prompt gets large.
// Remaining neighbors get header-only (key + first line).
const NEIGHBORHOOD_BUDGET: usize = 400_000; // ~100K tokens, leaves room for core-personality + instructions
let mut budget_exceeded = false;
for (nbr, strength, _score) in &included {
if included_nodes.contains(nbr) { continue; }
included_nodes.insert(nbr.clone());
if let Some(n) = store.nodes.get(nbr.as_str()) {
if budget_exceeded || out.len() > NEIGHBORHOOD_BUDGET {
// Header-only: key + first non-empty line
budget_exceeded = true;
let first_line = n.content.lines()
.find(|l| !l.trim().is_empty())
.unwrap_or("(empty)");
out.push_str(&format!("#### {} ({:.2}) — {}\n", nbr, strength, first));
out.push_str(&format!("#### {} (link: {:.2}) — {}\n",
nbr, strength, first_line));
} else {
out.push_str(&format!("#### {} ({:.2})\n\n{}\n\n", nbr, strength, content));
out.push_str(&format!("#### {} (link: {:.2})\n\n{}\n\n",
nbr, strength, n.content));
}
all_keys.push(nbr.to_string());
}
}
if budget_exceeded {
out.push_str("\n(remaining neighbors shown as headers only — prompt budget)\n\n");
}
// Cross-links between included neighbors
let mut cross_links = Vec::new();
for (nbr, _, _) in &included {
for (nbr2, strength) in graph.neighbors(nbr) {
if nbr2.as_str() != key
&& included_keys.contains(nbr2.as_str())
&& nbr.as_str() < nbr2.as_str()
{
cross_links.push((nbr.clone(), nbr2, strength));
}
}
}
if !cross_links.is_empty() {
out.push_str("### Cross-links between neighbors\n\n");
for (a, b, s) in &cross_links {
out.push_str(&format!(" {} ↔ {} ({:.2})\n", a, b, s));
}
out.push('\n');
}
}
}
Some(Resolved { text: out, keys: all_keys })
}
// targets/context: aliases for challenger-style presentation
"targets" => {
let items = keys_to_replay_items(store, keys, graph);
Some(Resolved {
text: super::prompts::format_nodes_section(store, &items, graph),
keys: vec![],
})
}
"hubs" => {
// Top hub nodes by degree, spread apart (skip neighbors of already-selected hubs)
let mut hubs: Vec<(String, usize)> = store.nodes.iter()
.filter(|(k, n)| !n.deleted && !k.starts_with('_'))
.map(|(k, _)| {
let degree = graph.neighbors(k).len();
(k.clone(), degree)
})
.collect();
hubs.sort_by(|a, b| b.1.cmp(&a.1));
let mut selected = Vec::new();
let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
for (key, degree) in &hubs {
if seen.contains(key) { continue; }
selected.push(format!(" - {} (degree {})", key, degree));
// Mark neighbors as seen so we pick far-apart hubs
for (nbr, _) in graph.neighbors(key) {
seen.insert(nbr.clone());
}
seen.insert(key.clone());
if selected.len() >= 20 { break; }
}
let text = format!("## Hub nodes (link targets)\n\n{}", selected.join("\n"));
Some(Resolved { text, keys: vec![] })
}
// agent-context — personality/identity groups from load-context config
"agent-context" => {
let cfg = crate::config::get();
@ -300,7 +474,7 @@ async fn resolve(
let mut keys = Vec::new();
for group in &cfg.context_groups {
if !group.agent { continue; }
let entries = crate::cli::node::get_group_content(group, &cfg).await;
let entries = crate::cli::misc::get_group_content(group, store, &cfg);
for (key, content) in entries {
use std::fmt::Write;
writeln!(text, "--- {} ({}) ---", key, group.label).ok();
@ -312,6 +486,15 @@ async fn resolve(
else { Some(Resolved { text, keys }) }
}
// node:KEY — inline a node's content by key
other if other.starts_with("node:") => {
let key = &other[5..];
store.nodes.get(key).map(|n| Resolved {
text: n.content.clone(),
keys: vec![key.to_string()],
})
}
// input:KEY — read a named output file from the agent's output dir
_ if name.starts_with("input:") => {
let key = &name[6..];
@ -353,10 +536,22 @@ async fn resolve(
Some(Resolved { text, keys: vec![] })
}
// latest_journal — the most recent EpisodicSession entry
"latest_journal" => {
let latest = store.nodes.values()
.filter(|n| n.node_type == crate::store::NodeType::EpisodicSession)
.max_by_key(|n| n.created_at);
let (text, keys) = match latest {
Some(n) => (n.content.clone(), vec![n.key.clone()]),
None => ("(no previous journal entry)".to_string(), vec![]),
};
Some(Resolved { text, keys })
}
// tool:NAME ARGS — run a tool call and include its output
_ if name.starts_with("tool:") => {
let spec = name[5..].trim();
resolve_tool(spec).await
resolve_tool(spec, store, graph)
}
// bash:COMMAND — run a shell command and include its stdout
@ -519,8 +714,9 @@ fn resolve_memory_ratio() -> String {
pct, keys.len(), memory_bytes / 1024, transcript_size / 1024)
}
/// Resolve a {{tool: name {args}}} placeholder by calling the tool handler.
async fn resolve_tool(spec: &str) -> Option<Resolved> {
/// Resolve a {{tool: name {args}}} placeholder by calling the tool
/// handler from the registry. Uses block_in_place to bridge sync→async.
fn resolve_tool(spec: &str, _store: &Store, _graph: &Graph) -> Option<Resolved> {
// Parse "tool_name {json args}" or "tool_name arg"
let (name, args) = match spec.find('{') {
Some(i) => {
@ -541,7 +737,13 @@ async fn resolve_tool(spec: &str) -> Option<Resolved> {
let tools = crate::agent::tools::tools();
let tool = tools.iter().find(|t| t.name == name)?;
match (tool.handler)(None, args.clone()).await {
let result = tokio::task::block_in_place(|| {
tokio::runtime::Handle::current().block_on(
(tool.handler)(None, args.clone())
)
});
match result {
Ok(text) => Some(Resolved { text, keys: vec![] }),
Err(e) => {
eprintln!("[defs] {{{{tool: {}}}}} failed: {}", name, e);
@ -552,8 +754,10 @@ async fn resolve_tool(spec: &str) -> Option<Resolved> {
/// Resolve all {{placeholder}} patterns in a prompt template.
/// Returns the resolved text and all node keys collected from placeholders.
pub async fn resolve_placeholders(
pub fn resolve_placeholders(
template: &str,
store: &Store,
graph: &Graph,
keys: &[String],
count: usize,
) -> (String, Vec<String>) {
@ -566,7 +770,7 @@ pub async fn resolve_placeholders(
let Some(rel_end) = result[start + 2..].find("}}") else { break };
let end = start + 2 + rel_end;
let name = result[start + 2..end].trim().to_lowercase();
match resolve(&name, keys, count).await {
match resolve(&name, store, graph, keys, count) {
Some(resolved) => {
let len = resolved.text.len();
extra_keys.extend(resolved.keys);
@ -587,26 +791,27 @@ pub async fn resolve_placeholders(
/// Run a config-driven agent: query → resolve placeholders → prompt.
/// `exclude` filters out nodes (and their neighborhoods) already being
/// worked on by other agents, preventing concurrent collisions.
pub async fn run_agent(
pub fn run_agent(
store: &Store,
def: &AgentDef,
count: usize,
exclude: &std::collections::HashSet<String>,
) -> Result<super::prompts::AgentBatch, String> {
let graph = store.build_graph();
// Run the query if present, via RPC
// Run the query if present
let keys = if !def.query.is_empty() {
let mut stages = crate::query_parser::parse_stages(&def.query)?;
let has_limit = stages.iter().any(|s|
matches!(s, search::Stage::Transform(search::Transform::Limit(_))));
if !has_limit {
// Request extra results to compensate for exclusion filtering
let padded = count + exclude.len().min(100);
let query = if def.query.contains("limit:") {
def.query.clone()
} else {
format!("{} | limit:{}", def.query, padded)
};
let result = crate::agent::tools::memory::memory_query(None, &query, None)
.await
.map_err(|e| e.to_string())?;
let filtered: Vec<String> = result.lines()
.filter(|l| !l.is_empty() && *l != "no results")
.map(|s| s.to_string())
stages.push(search::Stage::Transform(search::Transform::Limit(padded)));
}
let results = search::run_query(&stages, vec![], &graph, store, false, count + exclude.len().min(100));
let filtered: Vec<String> = results.into_iter()
.map(|(k, _)| k)
.filter(|k| !exclude.contains(k))
.take(count)
.collect();
@ -628,7 +833,7 @@ pub async fn run_agent(
.replace("{agent_name}", &def.agent)
.replace("{user_name}", &cfg.user_name)
.replace("{assistant_name}", &cfg.assistant_name);
let (prompt, extra_keys) = resolve_placeholders(&template, &all_keys, count).await;
let (prompt, extra_keys) = resolve_placeholders(&template, store, &graph, &all_keys, count);
all_keys.extend(extra_keys);
resolved_steps.push(super::prompts::ResolvedStep {
prompt,
@ -638,3 +843,28 @@ pub async fn run_agent(
Ok(super::prompts::AgentBatch { steps: resolved_steps, node_keys: all_keys })
}
/// Convert a list of keys to ReplayItems with priority and graph metrics.
pub fn keys_to_replay_items(
store: &Store,
keys: &[String],
graph: &Graph,
) -> Vec<ReplayItem> {
keys.iter()
.filter_map(|key| {
let node = store.nodes.get(key)?;
let priority = consolidation_priority(store, key, graph, None);
let cc = graph.clustering_coefficient(key);
Some(ReplayItem {
key: key.clone(),
priority,
interval_days: node.spaced_repetition_interval,
emotion: node.emotion,
cc,
classification: "unknown",
outlier_score: 0.0,
})
})
.collect()
}

40
src/subconscious/digest.rs
View file

@ -112,21 +112,17 @@ fn parse_digest_node_links(key: &str, content: &str) -> Vec<DigestLink> {
pub fn parse_all_digest_links(store: &Store) -> Vec<DigestLink> {
let mut all_links = Vec::new();
let all_keys = store.all_keys().unwrap_or_default();
let mut digest_keys: Vec<String> = all_keys.into_iter()
.filter(|k| {
store.get_node(k).ok().flatten()
.map(|n| matches!(n.node_type,
let mut digest_keys: Vec<&String> = store.nodes.iter()
.filter(|(_, n)| matches!(n.node_type,
store::NodeType::EpisodicDaily
| store::NodeType::EpisodicWeekly
| store::NodeType::EpisodicMonthly))
.unwrap_or(false)
})
.map(|(k, _)| k)
.collect();
digest_keys.sort();
for key in &digest_keys {
if let Ok(Some(node)) = store.get_node(key) {
for key in digest_keys {
if let Some(node) = store.nodes.get(key) {
all_links.extend(parse_digest_node_links(key, &node.content));
}
}
@ -176,21 +172,21 @@ pub fn apply_digest_links(store: &mut Store, links: &[DigestLink]) -> (usize, us
if source == target { skipped += 1; continue; }
let source_uuid = match store.get_node(&source).ok().flatten() {
Some(n) => n.uuid,
None => { skipped += 1; continue; }
};
let target_uuid = match store.get_node(&target).ok().flatten() {
Some(n) => n.uuid,
None => { skipped += 1; continue; }
};
// Check if link already exists via index
let exists = store.neighbors(&source).ok()
.map(|n| n.iter().any(|(k, _)| k == &target))
.unwrap_or(false);
// Check if link already exists
let exists = store.relations.iter().any(|r|
r.source_key == source && r.target_key == target && !r.deleted
);
if exists { skipped += 1; continue; }
let source_uuid = match store.nodes.get(&source) {
Some(n) => n.uuid,
None => { skipped += 1; continue; }
};
let target_uuid = match store.nodes.get(&target) {
Some(n) => n.uuid,
None => { skipped += 1; continue; }
};
let rel = new_relation(
source_uuid, target_uuid,
store::RelationType::Link,

9
src/subconscious/learn.rs
View file

@ -327,16 +327,12 @@ where
let mut seen = std::collections::HashSet::new();
let mut candidates: Vec<(usize, String, i64)> = Vec::new(); // (pos, key, last_scored)
let store_arc = crate::hippocampus::access_local()?;
let store = crate::hippocampus::store::Store::load().unwrap_or_default();
{
let store = &*store_arc;
for (i, node) in context.conversation().iter().enumerate() {
if let Some(key) = memory_key(node) {
if !seen.insert(key.to_owned()) { continue; }
let last_scored = store.get_node(key)
.ok()
.flatten()
let last_scored = store.nodes.get(key)
.map(|n| n.last_scored)
.unwrap_or(0);
if now - last_scored >= max_age_secs {
@ -344,7 +340,6 @@ where
}
}
}
}
// Score oldest-first
candidates.sort_by_key(|&(_, _, last)| last);

192
src/subconscious/prompts.rs
View file

@ -4,7 +4,10 @@
use crate::store::Store;
use crate::graph::Graph;
use crate::neuro::ReplayItem;
use crate::neuro::{
ReplayItem,
replay_queue,
};
/// Result of building an agent prompt — includes both the prompt text
/// and the keys of nodes selected for processing, so the caller can
@ -20,7 +23,7 @@ pub struct AgentBatch {
pub node_keys: Vec<String>,
}
pub fn format_topology_header(store: &Store, graph: &Graph) -> String {
pub fn format_topology_header(graph: &Graph) -> String {
let sigma = graph.small_world_sigma();
let alpha = graph.degree_power_law_exponent();
let gini = graph.degree_gini();
@ -28,28 +31,6 @@ pub fn format_topology_header(store: &Store, graph: &Graph) -> String {
let n = graph.nodes().len();
let e = graph.edge_count();
// Type counts
let mut type_counts: std::collections::HashMap<&str, usize> = std::collections::HashMap::new();
let all_keys = store.all_keys().unwrap_or_default();
for key in &all_keys {
if let Ok(Some(node)) = store.get_node(key) {
let label = match node.node_type {
crate::store::NodeType::Semantic => "semantic",
crate::store::NodeType::EpisodicSession
| crate::store::NodeType::EpisodicDaily
| crate::store::NodeType::EpisodicWeekly
| crate::store::NodeType::EpisodicMonthly => "episodic",
};
*type_counts.entry(label).or_default() += 1;
}
}
let mut types: Vec<_> = type_counts.iter().collect();
types.sort_by_key(|(_, c)| std::cmp::Reverse(**c));
let type_str: String = types.iter()
.map(|(t, c)| format!("{}={}", t, c))
.collect::<Vec<_>>()
.join(" ");
// Identify saturated hubs — nodes with degree well above threshold
let threshold = graph.hub_threshold();
let mut hubs: Vec<_> = graph.nodes().iter()
@ -76,20 +57,20 @@ pub fn format_topology_header(store: &Store, graph: &Graph) -> String {
format!(
"## Current graph topology\n\
Nodes: {} Edges: {} Communities: {} Types: {}\n\
Nodes: {} Edges: {} Communities: {}\n\
Small-world σ: {:.1} Power-law α: {:.2} Degree Gini: {:.3}\n\
Avg clustering coefficient: {:.4}\n\n\
{}\
Each node below shows its hub-link ratio (fraction of edges to top-5% degree nodes).\n\
Use `poc-memory link-impact SOURCE TARGET` to evaluate proposed links.\n\n",
n, e, graph.community_count(), type_str, sigma, alpha, gini, avg_cc, hub_list)
n, e, graph.community_count(), sigma, alpha, gini, avg_cc, hub_list)
}
pub fn format_nodes_section(store: &Store, items: &[ReplayItem], graph: &Graph) -> String {
let hub_thresh = graph.hub_threshold();
let mut out = String::new();
for item in items {
let node = match store.get_node(&item.key).ok().flatten() {
let node = match store.nodes.get(&item.key) {
Some(n) => n,
None => continue,
};
@ -142,9 +123,7 @@ pub fn format_nodes_section(store: &Store, items: &[ReplayItem], graph: &Graph)
out.push_str("Neighbors:\n");
for (n, strength) in neighbors.iter().take(15) {
let n_cc = graph.clustering_coefficient(n);
let n_community = store.get_node(n)
.ok()
.flatten()
let n_community = store.nodes.get(n.as_str())
.and_then(|n| n.community_id);
out.push_str(&format!(" - {} (str={:.2}, cc={:.3}",
n, strength, n_cc));
@ -170,13 +149,10 @@ pub fn format_health_section(store: &Store, graph: &Graph) -> String {
// Weight histogram
let mut buckets = [0u32; 10]; // 0.0-0.1, 0.1-0.2, ..., 0.9-1.0
let all_keys = store.all_keys().unwrap_or_default();
for key in &all_keys {
if let Ok(Some(node)) = store.get_node(key) {
for node in store.nodes.values() {
let bucket = ((node.weight * 10.0) as usize).min(9);
buckets[bucket] += 1;
}
}
for (i, &count) in buckets.iter().enumerate() {
let lo = i as f32 / 10.0;
let hi = (i + 1) as f32 / 10.0;
@ -185,9 +161,9 @@ pub fn format_health_section(store: &Store, graph: &Graph) -> String {
}
// Near-prune nodes
let near_prune: Vec<_> = all_keys.iter()
.filter_map(|k| store.get_node(k).ok()?.map(|n| (k.clone(), n.weight)))
.filter(|(_, w)| *w < 0.15)
let near_prune: Vec<_> = store.nodes.iter()
.filter(|(_, n)| n.weight < 0.15)
.map(|(k, n)| (k.clone(), n.weight))
.collect();
if !near_prune.is_empty() {
out.push_str(&format!("\n## Near-prune nodes ({} total)\n", near_prune.len()));
@ -219,9 +195,147 @@ pub fn format_health_section(store: &Store, graph: &Graph) -> String {
out
}
pub(super) fn format_rename_candidates(store: &Store, count: usize) -> (Vec<String>, String) {
let mut candidates: Vec<(&str, &crate::store::Node)> = store.nodes.iter()
.filter(|(key, node)| {
if key.starts_with("_facts-") { return true; }
if key.len() < 60 { return false; }
if node.node_type == crate::store::NodeType::EpisodicSession { return true; }
if key.starts_with("_mined-transcripts#f-") { return true; }
false
})
.map(|(k, n)| (k.as_str(), n))
.collect();
// Deprioritize nodes actively found by search — renaming them would
// break working queries. Sort by: search hits (ascending), then
// least-recently visited. Nodes with many hits sink to the bottom.
let hit_counts = crate::counters::all_search_hits();
let hit_map: std::collections::HashMap<&str, u64> = hit_counts.iter()
.map(|(k, v)| (k.as_str(), *v))
.collect();
candidates.sort_by_key(|(key, _)| {
let hits = hit_map.get(key).copied().unwrap_or(0);
(hits, store.last_visited(key, "rename"))
});
candidates.truncate(count);
let keys: Vec<String> = candidates.iter().map(|(k, _)| k.to_string()).collect();
let mut out = String::new();
out.push_str(&format!("## Nodes to rename ({} of {} candidates)\n\n",
candidates.len(),
store.nodes.iter().filter(|(k, n)| k.starts_with("_facts-") ||
(k.len() >= 60 &&
(n.node_type == crate::store::NodeType::EpisodicSession || k.starts_with("_mined-transcripts#f-")))).count()));
for (key, node) in &candidates {
out.push_str(&format!("### {}\n", key));
let created = if node.timestamp > 0 {
crate::store::format_datetime(node.timestamp)
} else {
"unknown".to_string()
};
out.push_str(&format!("Created: {}\n", created));
let hits = hit_map.get(key).copied().unwrap_or(0);
if hits > 0 {
out.push_str(&format!("Search hits: {} ← actively found by search, prefer to keep current name\n", hits));
}
let content = &node.content;
if content.len() > 800 {
let truncated = crate::util::truncate(content, 800, "\n[...]");
out.push_str(&format!("\nContent ({} chars, truncated):\n{}\n\n",
content.len(), truncated));
} else {
out.push_str(&format!("\nContent:\n{}\n\n", content));
}
out.push_str("---\n\n");
}
(keys, out)
}
/// Format specific target keys as rename candidates (for --target mode)
pub(super) fn format_rename_targets(store: &Store, keys: &[String]) -> String {
let mut out = String::new();
out.push_str(&format!("## Nodes to rename ({} targets)\n\n", keys.len()));
for key in keys {
let Some(node) = store.nodes.get(key) else {
out.push_str(&format!("### {}\n\n(node not found)\n\n---\n\n", key));
continue;
};
out.push_str(&format!("### {}\n", key));
let created = if node.timestamp > 0 {
crate::store::format_datetime(node.timestamp)
} else {
"unknown".to_string()
};
out.push_str(&format!("Created: {}\n", created));
let content = &node.content;
if content.len() > 800 {
let truncated = crate::util::truncate(content, 800, "\n[...]");
out.push_str(&format!("\nContent ({} chars, truncated):\n{}\n\n",
content.len(), truncated));
} else {
out.push_str(&format!("\nContent:\n{}\n\n", content));
}
out.push_str("---\n\n");
}
out
}
/// Format a single node for split-plan prompt (phase 1)
pub(super) fn format_split_plan_node(store: &Store, graph: &Graph, key: &str) -> String {
let communities = graph.communities();
let node = match store.nodes.get(key) {
Some(n) => n,
None => return format!("Node '{}' not found\n", key),
};
let mut out = String::new();
out.push_str(&format!("### {} ({} chars)\n", key, node.content.len()));
// Show neighbors grouped by community
let neighbors = graph.neighbors(key);
if !neighbors.is_empty() {
let mut by_community: std::collections::BTreeMap<String, Vec<(&str, f32)>> =
std::collections::BTreeMap::new();
for (nkey, strength) in &neighbors {
let comm = communities.get(nkey.as_str())
.map(|c| format!("c{}", c))
.unwrap_or_else(|| "unclustered".into());
by_community.entry(comm)
.or_default()
.push((nkey.as_str(), *strength));
}
out.push_str("\nNeighbors by community:\n");
for (comm, members) in &by_community {
out.push_str(&format!(" {} ({}):", comm, members.len()));
for (nkey, strength) in members.iter().take(5) {
out.push_str(&format!(" {}({:.2})", nkey, strength));
}
if members.len() > 5 {
out.push_str(&format!(" +{} more", members.len() - 5));
}
out.push('\n');
}
}
// Full content
out.push_str(&format!("\nContent:\n{}\n\n", node.content));
out.push_str("---\n\n");
out
}
/// Generate a specific agent prompt with filled-in data.
pub async fn agent_prompt(agent: &str, count: usize) -> Result<AgentBatch, String> {
pub fn agent_prompt(store: &Store, agent: &str, count: usize) -> Result<AgentBatch, String> {
let def = super::defs::get_def(agent)
.ok_or_else(|| format!("Unknown agent: {}", agent))?;
super::defs::run_agent(&def, count, &Default::default()).await
super::defs::run_agent(store, &def, count, &Default::default())
}

3
src/user/chat.rs
View file

@ -368,11 +368,10 @@ impl PaneState {
}
fn pop_line(&mut self) {
if self.lines.pop().is_some() {
self.lines.pop();
self.markers.pop();
self.scroll.invalidate_from(self.lines.len());
}
}
fn scroll_up(&mut self, n: u16) {
self.scroll.scroll_up(n);

24
src/user/mod.rs
View file

@ -6,7 +6,6 @@
pub(crate) mod chat;
mod context;
pub(crate) mod scroll_pane;
pub mod selectable;
mod subconscious;
mod unconscious;
mod thalamus;
@ -184,7 +183,7 @@ fn restore_terminal(terminal: &mut ratatui::Terminal<CrosstermBackend<io::Stdout
/// Top-level entry point — creates Mind and UI, wires them together.
async fn start(cli: crate::user::CliArgs) -> Result<()> {
let (config, _figment) = crate::config::load_session(&cli).await?;
let (config, _figment) = crate::config::load_session(&cli)?;
if config.app.debug {
unsafe { std::env::set_var("POC_DEBUG", "1") };
@ -212,25 +211,10 @@ async fn start(cli: crate::user::CliArgs) -> Result<()> {
})
.expect("spawn UI thread");
// Initialize store - access_local() caches it in STORE_ACCESS
if let Err(e) = crate::hippocampus::access_local() {
eprintln!("Store init failed: {}", e);
}
// Start MCP server for external tool access
let mut tools: Vec<crate::agent::tools::Tool> = Vec::new();
tools.extend(crate::agent::tools::memory::memory_tools());
tools.extend(crate::agent::tools::memory::journal_tools());
if let Err(e) = crate::mcp_server::start(tools).await {
eprintln!("MCP server failed to start: {:#}", e);
}
// Mind event loop — runs on the main tokio runtime
mind.init().await;
mind.run(mind_rx, turn_rx).await;
crate::mcp_server::cleanup();
ui_handle.join().unwrap_or_else(|_| Err(anyhow::anyhow!("UI thread panicked")))
}
@ -419,9 +403,9 @@ async fn run(
unc.toggle(name).await;
}
}
let store_arc = crate::hippocampus::access_local().ok();
let store_guard = match &store_arc {
Some(s) => Some(&**s),
let store = crate::store::Store::cached().await.ok();
let store_guard = match &store {
Some(s) => Some(s.lock().await),
None => None,
};
app.unconscious_state = unc.snapshots(store_guard.as_deref());

22
src/user/scroll_pane.rs
View file

@ -106,27 +106,7 @@ impl ScrollPaneState {
let h = self.heights.get(line_idx).copied().unwrap_or(1) as i32;
if (mouse_y as i32) < row + h {
let line_text: String = lines[line_idx].spans.iter().map(|s| s.content.as_ref()).collect();
// Which visual row within this wrapped line?
let visual_row_in_item = ((mouse_y as i32) - row).max(0) as usize;
// Use textwrap to find actual break positions
let wrap_width = self.cached_width as usize;
let wrapped = textwrap::wrap(&line_text, wrap_width);
// Sum lengths of previous wrapped rows to get char offset base
let char_base: usize = wrapped.iter()
.take(visual_row_in_item)
.map(|s| s.len())
.sum();
// Add mouse x position within current row
let current_row_len = wrapped.get(visual_row_in_item)
.map(|s| s.len())
.unwrap_or(0);
let col = char_base + (mouse_x as usize).min(current_row_len);
let col = col.min(line_text.len());
let col = (mouse_x as usize).min(line_text.len());
return Some((line_idx, col));
}
row += h;

531
src/user/selectable.rs
View file

@ -1,531 +0,0 @@
//! Selectable text widget with proper wrap-aware selection.
//!
//! Uses Unicode Private Use Area markers to track logical line boundaries:
//! - Lines starting with CONT are continuations (wrapped from previous)
//! - Lines between SEL_ON and SEL_OFF are selectable
//!
//! The caller pre-wraps text and marks continuations. This widget handles
//! selection, clipboard copy, and rendering with highlights.
use ratatui::prelude::*;
use ratatui::widgets::{Block, Scrollbar, ScrollbarOrientation, ScrollbarState};
// ── Markers (Unicode Private Use Area) ─────────────────────────────
/// This line continues the previous logical line (was wrapped).
pub const CONT: char = '\u{E000}';
/// Start of a selectable region.
pub const SEL_ON: char = '\u{E001}';
/// End of a selectable region.
pub const SEL_OFF: char = '\u{E002}';
// ── Helper: wrap text with continuation markers ────────────────────
/// Wrap a single logical line into visual lines, marking continuations.
/// Returns lines ready to push into a SelectableText.
pub fn wrap_line(text: &str, width: usize) -> Vec<String> {
if width == 0 || text.is_empty() {
return vec![text.to_string()];
}
let wrapped = textwrap::wrap(text, width);
wrapped
.into_iter()
.enumerate()
.map(|(i, cow)| {
if i == 0 {
cow.into_owned()
} else {
format!("{}{}", CONT, cow)
}
})
.collect()
}
/// Wrap text and mark as selectable.
pub fn wrap_line_selectable(text: &str, width: usize) -> Vec<String> {
let mut lines = wrap_line(text, width);
if let Some(first) = lines.first_mut() {
*first = format!("{}{}", SEL_ON, first);
}
if let Some(last) = lines.last_mut() {
last.push(SEL_OFF);
}
lines
}
// ── Selection state ────────────────────────────────────────────────
/// A position in logical coordinates (line index, char offset).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct LogicalPos {
pub line: usize,
pub col: usize,
}
/// Selection anchor and cursor.
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub struct Selection {
pub anchor: LogicalPos,
pub cursor: LogicalPos,
}
impl Selection {
pub fn new(pos: LogicalPos) -> Self {
Self { anchor: pos, cursor: pos }
}
pub fn extend(&mut self, pos: LogicalPos) {
self.cursor = pos;
}
/// Returns (start, end) in normalized order.
pub fn range(&self) -> (LogicalPos, LogicalPos) {
if (self.anchor.line, self.anchor.col) <= (self.cursor.line, self.cursor.col) {
(self.anchor, self.cursor)
} else {
(self.cursor, self.anchor)
}
}
pub fn is_empty(&self) -> bool {
self.anchor == self.cursor
}
}
// ── Main widget state ──────────────────────────────────────────────
pub struct SelectableTextState {
/// Visual lines (may contain markers).
lines: Vec<String>,
/// Scroll offset in visual lines.
pub scroll_offset: usize,
/// Viewport height (set during render).
pub viewport_height: usize,
/// Current selection, if any.
pub selection: Option<Selection>,
/// Cached logical line index for each visual line.
/// logical_line_idx[visual] = which logical line this visual line belongs to.
logical_line_idx: Vec<usize>,
/// Cached char offset: start char of each visual line within its logical line.
char_offsets: Vec<usize>,
}
impl Default for SelectableTextState {
fn default() -> Self {
Self::new()
}
}
impl SelectableTextState {
pub fn new() -> Self {
Self {
lines: Vec::new(),
scroll_offset: 0,
viewport_height: 0,
selection: None,
logical_line_idx: Vec::new(),
char_offsets: Vec::new(),
}
}
/// Clear all content.
pub fn clear(&mut self) {
self.lines.clear();
self.logical_line_idx.clear();
self.char_offsets.clear();
self.selection = None;
}
/// Push a visual line. Call rebuild_index() after batch pushes.
pub fn push_line(&mut self, line: String) {
self.lines.push(line);
}
/// Push multiple visual lines.
pub fn push_lines(&mut self, lines: impl IntoIterator<Item = String>) {
self.lines.extend(lines);
}
/// Rebuild the logical line index. Call after modifying lines.
pub fn rebuild_index(&mut self) {
self.logical_line_idx.clear();
self.char_offsets.clear();
let mut logical_idx = 0usize;
let mut char_offset = 0usize;
for line in &self.lines {
let is_continuation = line.starts_with(CONT);
if !is_continuation && !self.logical_line_idx.is_empty() {
// New logical line
logical_idx += 1;
char_offset = 0;
}
self.logical_line_idx.push(logical_idx);
self.char_offsets.push(char_offset);
// Advance char offset by the display length of this line
char_offset += display_len(line);
}
}
/// Number of visual lines.
pub fn len(&self) -> usize {
self.lines.len()
}
pub fn is_empty(&self) -> bool {
self.lines.is_empty()
}
/// Scroll up by n visual lines.
pub fn scroll_up(&mut self, n: usize) {
self.scroll_offset = self.scroll_offset.saturating_sub(n);
}
/// Scroll down by n visual lines.
pub fn scroll_down(&mut self, n: usize) {
let max = self.len().saturating_sub(self.viewport_height);
self.scroll_offset = (self.scroll_offset + n).min(max);
}
/// Convert screen position to logical position.
pub fn screen_to_logical(&self, x: u16, y: u16) -> Option<LogicalPos> {
let visual_row = self.scroll_offset + y as usize;
if visual_row >= self.lines.len() {
return None;
}
let logical_line = *self.logical_line_idx.get(visual_row)?;
let char_base = *self.char_offsets.get(visual_row)?;
// Check if this position is within a selectable region
if !self.is_visual_line_selectable(visual_row) {
return None;
}
let line = &self.lines[visual_row];
let display = strip_markers(line);
let col = char_base + (x as usize).min(display.len());
Some(LogicalPos { line: logical_line, col })
}
/// Check if a visual line is within a selectable region.
fn is_visual_line_selectable(&self, visual_row: usize) -> bool {
// Walk backwards to find if we're in a selectable region
let mut in_selectable = false;
for i in 0..=visual_row {
let line = &self.lines[i];
if line.contains(SEL_ON) {
in_selectable = true;
}
if line.contains(SEL_OFF) && i < visual_row {
in_selectable = false;
}
}
in_selectable || self.lines[visual_row].contains(SEL_ON)
}
/// Start a new selection at screen position.
pub fn start_selection(&mut self, x: u16, y: u16) {
if let Some(pos) = self.screen_to_logical(x, y) {
self.selection = Some(Selection::new(pos));
} else {
self.selection = None;
}
}
/// Extend selection to screen position.
pub fn extend_selection(&mut self, x: u16, y: u16) {
if let Some(pos) = self.screen_to_logical(x, y) {
if let Some(ref mut sel) = self.selection {
sel.extend(pos);
}
}
}
/// Get selected text, joining logical lines with newlines.
pub fn get_selected_text(&self) -> Option<String> {
let sel = self.selection.as_ref()?;
if sel.is_empty() {
return None;
}
let (start, end) = sel.range();
// Reconstruct logical lines
let logical_lines = self.reconstruct_logical_lines();
let mut result = String::new();
for (i, line) in logical_lines.iter().enumerate() {
if i < start.line || i > end.line {
continue;
}
let line_start = if i == start.line { start.col } else { 0 };
let line_end = if i == end.line { end.col } else { line.len() };
if line_start < line.len() {
if !result.is_empty() {
result.push('\n');
}
let end_clamped = line_end.min(line.len());
if let Some(slice) = line.get(line_start..end_clamped) {
result.push_str(slice);
}
}
}
if result.is_empty() {
None
} else {
Some(result)
}
}
/// Reconstruct logical lines from visual lines (stripping markers, joining continuations).
fn reconstruct_logical_lines(&self) -> Vec<String> {
let mut logical: Vec<String> = Vec::new();
for line in &self.lines {
let is_cont = line.starts_with(CONT);
let clean = strip_markers(line);
if is_cont && !logical.is_empty() {
// Append to previous logical line
logical.last_mut().unwrap().push_str(&clean);
} else {
logical.push(clean);
}
}
logical
}
/// Copy selection to clipboard via OSC 52.
pub fn copy_to_clipboard(&self) {
if let Some(text) = self.get_selected_text() {
if text.is_empty() {
return;
}
use base64::Engine;
use std::io::Write;
let encoded = base64::engine::general_purpose::STANDARD.encode(&text);
let mut stdout = std::io::stdout().lock();
let _ = write!(stdout, "\x1b]52;c;{}\x07", encoded);
let _ = stdout.flush();
}
}
/// Get the visual lines for rendering (with markers stripped).
pub fn display_lines(&self) -> impl Iterator<Item = Line<'_>> + '_ {
self.lines.iter().map(|s| Line::raw(strip_markers(s)))
}
/// Check if a logical position is within the current selection.
#[allow(dead_code)] // Reserved for future per-character highlight rendering
fn is_selected(&self, logical_line: usize, col: usize) -> bool {
let Some(ref sel) = self.selection else { return false };
let (start, end) = sel.range();
if logical_line < start.line || logical_line > end.line {
return false;
}
if logical_line == start.line && col < start.col {
return false;
}
if logical_line == end.line && col >= end.col {
return false;
}
true
}
/// Get the selection highlight range for a visual line (in display columns).
pub fn highlight_range(&self, visual_row: usize) -> Option<(usize, usize)> {
let sel = self.selection.as_ref()?;
if sel.is_empty() {
return None;
}
let logical_line = *self.logical_line_idx.get(visual_row)?;
let char_base = *self.char_offsets.get(visual_row)?;
let display = strip_markers(&self.lines[visual_row]);
let line_len = display.len();
let (start, end) = sel.range();
// Check if this visual line overlaps with selection
if logical_line < start.line || logical_line > end.line {
return None;
}
let sel_start_in_line = if logical_line == start.line { start.col } else { 0 };
let sel_end_in_line = if logical_line == end.line { end.col } else { usize::MAX };
// Convert to visual line's local coordinates
let vis_start = sel_start_in_line.saturating_sub(char_base);
let vis_end = sel_end_in_line.saturating_sub(char_base).min(line_len);
if vis_start >= line_len || vis_end == 0 || vis_start >= vis_end {
return None;
}
Some((vis_start, vis_end))
}
}
// ── Widget ─────────────────────────────────────────────────────────
pub struct SelectableText<'a> {
block: Option<Block<'a>>,
highlight_style: Style,
}
impl<'a> SelectableText<'a> {
pub fn new() -> Self {
Self {
block: None,
highlight_style: Style::default().bg(Color::DarkGray),
}
}
pub fn block(mut self, block: Block<'a>) -> Self {
self.block = Some(block);
self
}
pub fn highlight_style(mut self, style: Style) -> Self {
self.highlight_style = style;
self
}
}
impl Default for SelectableText<'_> {
fn default() -> Self {
Self::new()
}
}
impl StatefulWidget for SelectableText<'_> {
type State = SelectableTextState;
fn render(self, area: Rect, buf: &mut Buffer, state: &mut Self::State) {
let inner = if let Some(block) = self.block {
let inner = block.inner(area);
block.render(area, buf);
inner
} else {
area
};
if inner.width < 2 || inner.height == 0 {
return;
}
state.viewport_height = inner.height as usize;
// Render visible lines
let start = state.scroll_offset;
let end = (start + inner.height as usize).min(state.lines.len());
for (i, visual_row) in (start..end).enumerate() {
let y = inner.y + i as u16;
let line = &state.lines[visual_row];
let display = strip_markers(line);
// Render with selection highlighting
if let Some((hl_start, hl_end)) = state.highlight_range(visual_row) {
// Before highlight
let before = &display[..hl_start.min(display.len())];
buf.set_string(inner.x, y, before, Style::default());
// Highlighted portion
let hl_text = &display[hl_start..hl_end.min(display.len())];
buf.set_string(inner.x + hl_start as u16, y, hl_text, self.highlight_style);
// After highlight
if hl_end < display.len() {
let after = &display[hl_end..];
buf.set_string(inner.x + hl_end as u16, y, after, Style::default());
}
} else {
buf.set_string(inner.x, y, &display, Style::default());
}
}
// Scrollbar
let content_len = state.lines.len();
let visible = inner.height as usize;
if content_len > visible {
let mut sb_state = ScrollbarState::new(content_len).position(state.scroll_offset);
Scrollbar::new(ScrollbarOrientation::VerticalRight).render(inner, buf, &mut sb_state);
}
}
}
// ── Helpers ────────────────────────────────────────────────────────
/// Strip all markers from a line for display.
fn strip_markers(s: &str) -> String {
s.chars()
.filter(|&c| c != CONT && c != SEL_ON && c != SEL_OFF)
.collect()
}
/// Display length of a line (excluding markers).
fn display_len(s: &str) -> usize {
s.chars()
.filter(|&c| c != CONT && c != SEL_ON && c != SEL_OFF)
.count()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_wrap_line() {
// "hello world, this is a test" at width 10:
// "hello" / "world," / "this is a" / "test"
let lines = wrap_line("hello world, this is a test", 10);
assert_eq!(lines.len(), 4);
assert!(!lines[0].starts_with(CONT)); // "hello"
assert!(lines[1].starts_with(CONT)); // " world,"
assert!(lines[2].starts_with(CONT)); // " this is a"
assert!(lines[3].starts_with(CONT)); // " test"
}
#[test]
fn test_strip_markers() {
let s = format!("{}hello{}world{}", SEL_ON, CONT, SEL_OFF);
assert_eq!(strip_markers(&s), "helloworld");
}
#[test]
fn test_logical_index() {
let mut state = SelectableTextState::new();
state.push_line("first line".to_string());
state.push_line(format!("{}continued", CONT));
state.push_line("second line".to_string());
state.rebuild_index();
assert_eq!(state.logical_line_idx, vec![0, 0, 1]);
assert_eq!(state.char_offsets, vec![0, 10, 0]);
}
#[test]
fn test_reconstruct() {
let mut state = SelectableTextState::new();
state.push_line("hello ".to_string());
state.push_line(format!("{}world", CONT));
state.push_line("next".to_string());
state.rebuild_index();
let logical = state.reconstruct_logical_lines();
assert_eq!(logical, vec!["hello world", "next"]);
}
}

288
training/research/pause-tokens-gdn-recurrence.md
View file

@ -1,288 +0,0 @@
# Pause Tokens + GDN Recurrence: Latent Reasoning for Qwen 3.5
**Status:** Ready for testing
**Date:** 2026-04-12
**Insight:** Qwen 3.5's GDN layers already have recurrence - pause tokens give it more iterations
---
## The Core Insight
Standard transformers couple compute depth to output length. Both pause tokens and internal recurrence solve this by allowing "thinking" without token commitment.
**The GDN connection:** Qwen 3.5 is 75% GDN (Gated DeltaNet) layers. Each GDN layer maintains recurrent state:
```
S_t = exp(g_t) * S_{t-1} + outer(k_t, delta_t)
```
This state persists across token positions. When you add a pause token:
1. One more forward pass through all layers (standard)
2. One more update to recurrent state S (GDN-specific)
Pause tokens on Qwen 3.5 trigger **both** forms of additional computation. We're not adding recurrence - we're giving existing recurrence more time to develop.
---
## Minimal Test: Random Prefix (Zero Training)
The dl1683 paper showed random embeddings work at inference time without training:
- Qwen3-4B arithmetic: 32% → 51.6% (+19.6pp)
- 100% oracle coverage on planning tasks
### Test Script
```python
#!/usr/bin/env python3
"""Test pause tokens on Qwen 3.5 27B.
Usage:
source ~/training-env/bin/activate
python3 test_pause_tokens.py
"""
import torch
from transformers import AutoTokenizer
# Reuse our weight loading infrastructure
import sys
sys.path.insert(0, '.')
from extract_steering_vector import load_model
GSM8K_SAMPLES = [
"Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
"A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?",
# Add more samples...
]
def get_embedding_rms(model):
"""Get RMS of embedding weights for proper scaling."""
embed = model.model.embed_tokens.weight
return embed.float().square().mean().sqrt().item()
def make_random_prefix(n_tokens, embed_dim, rms, device):
"""Generate random prefix embeddings at embedding scale."""
prefix = torch.randn(1, n_tokens, embed_dim, device=device, dtype=torch.bfloat16)
return prefix * rms
def generate_with_pause(model, tokenizer, prompt, n_pause=0, max_new=256):
"""Generate with optional pause token prefix."""
input_ids = tokenizer.encode(prompt, return_tensors='pt').to('cuda:0')
text_embeds = model.model.embed_tokens(input_ids)
if n_pause > 0:
embed_rms = get_embedding_rms(model)
pause_embeds = make_random_prefix(n_pause, text_embeds.shape[-1], embed_rms, text_embeds.device)
combined = torch.cat([pause_embeds, text_embeds], dim=1)
else:
combined = text_embeds
# Generate from embeddings
with torch.no_grad():
outputs = model.generate(
inputs_embeds=combined,
max_new_tokens=max_new,
do_sample=False, # Greedy for reproducibility
pad_token_id=tokenizer.pad_token_id,
)
# Decode (skip pause token positions in output)
return tokenizer.decode(outputs[0], skip_special_tokens=True)
def extract_answer(response):
"""Extract numeric answer from response."""
import re
numbers = re.findall(r'[\d,]+\.?\d*', response)
if numbers:
return numbers[-1].replace(',', '')
return None
def main():
print("Loading model...")
model = load_model()
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-27B", trust_remote_code=True)
print(f"\nEmbedding RMS: {get_embedding_rms(model):.4f}")
for n_pause in [0, 2, 4]:
print(f"\n=== Testing with {n_pause} pause tokens ===")
correct = 0
for i, problem in enumerate(GSM8K_SAMPLES):
prompt = f"Solve this step by step:\n{problem}\n\nAnswer:"
response = generate_with_pause(model, tokenizer, prompt, n_pause=n_pause)
answer = extract_answer(response)
print(f" Problem {i+1}: {answer}")
# TODO: Compare against ground truth
print(f" Accuracy: {correct}/{len(GSM8K_SAMPLES)}")
if __name__ == '__main__':
main()
```
### Test Protocol
1. Pick 20-50 GSM8K problems with known answers
2. Run baseline (n_pause=0)
3. Run with 2 pause tokens
4. Run with 4 pause tokens
5. Compare accuracy
If pause tokens help at inference time with zero training, the GDN recurrence is leveraging the extra iterations.
---
## Learnable Pause Tokens (Training Phase)
After validating random prefix works, train dedicated pause tokens:
```python
# Add to model
model.pause_tokens = nn.Parameter(
torch.randn(4, model.config.hidden_size) * embed_rms
)
# Training forward pass
def forward_with_learned_pause(model, input_ids):
text_embeds = model.model.embed_tokens(input_ids)
pause = model.pause_tokens.unsqueeze(0).expand(text_embeds.shape[0], -1, -1)
combined = torch.cat([pause, text_embeds], dim=1)
return model(inputs_embeds=combined)
```
Key: Must train WITH pause tokens for them to work. Inference-only learned tokens don't help (per Google's pause token paper).
---
## Adaptive Halting via Confidence Readout
For variable-length pause (iterate until confident):
### Extract Confidence Direction
```python
confident = [
"The answer is 42.",
"This will work because the invariant holds.",
"Use mmap here.",
]
uncertain = [
"I think the answer might be 42?",
"This should work, but I'm not sure...",
"Maybe mmap? Or read()?",
]
# Same infrastructure as listening vector
confident_states = get_hidden_states(model, confident, layer=48)
uncertain_states = get_hidden_states(model, uncertain, layer=48)
confidence_vec = confident_states.mean(0) - uncertain_states.mean(0)
```
### Adaptive Loop
```python
def generate_adaptive_pause(model, tokenizer, prompt, max_pause=8, threshold=0.7):
confidence_vec = torch.load('confidence_direction.pt')
input_ids = tokenizer.encode(prompt, return_tensors='pt').to('cuda:0')
h = model.model.embed_tokens(input_ids)
embed_rms = get_embedding_rms(model)
for i in range(max_pause):
# Add one pause token
pause = make_random_prefix(1, h.shape[-1], embed_rms, h.device)
h = torch.cat([pause, h], dim=1)
# Forward to get hidden state
with torch.no_grad():
out = model(inputs_embeds=h, output_hidden_states=True)
# Check confidence at layer 48
hidden = out.hidden_states[48][0, -1, :]
confidence = torch.cosine_similarity(
hidden.unsqueeze(0),
confidence_vec.unsqueeze(0)
).item()
if confidence > threshold:
break
# Generate from accumulated state
return model.generate(inputs_embeds=h, max_new_tokens=256)
```
---
## Connection to Huginn/Looping Architectures
Huginn uses explicit weight-tied loops (same 4 layers run N times). We can't retrofit this to Qwen 3.5 without retraining.
But GDN recurrence + pause tokens achieves similar effect:
- Huginn: explicit iteration over layers
- GDN + pause: implicit iteration via recurrent state S
The GDN state accumulates across pause positions, effectively giving the model multiple "thinking steps" before output.
### Comparison
| Approach | Requires Pretraining | Compute Cost | Qwen 3.5 Compatible |
|----------|---------------------|--------------|---------------------|
| Huginn loops | Yes | N × core layers | No |
| Pause tokens | No (inference test) | N × all layers | Yes |
| GDN recurrence | Already there | Per-token | Already there |
| Pause + GDN | No | N × all layers + N state updates | Yes |
---
## COCONUT Integration (Future)
COCONUT feeds hidden state back as input embedding - explicit whole-model recurrence:
```python
def coconut_forward(model, input_ids, n_latent=3):
h = model.model.embed_tokens(input_ids)
for step in range(n_latent):
out = model(inputs_embeds=h, output_hidden_states=True)
# Project hidden state back to embedding space
h = model.project_hidden_to_embed(out.hidden_states[-1])
# Final forward produces tokens
return model.generate(inputs_embeds=h)
```
This gives two levels of iteration:
1. GDN recurrence within each forward pass (automatic)
2. Hidden → embed looping across forward passes (COCONUT)
Requires training the projection layer. Curriculum: start with 0 latent steps, gradually increase.
---
## Implementation Priority
1. **Now:** Run random prefix test (zero training, 1 hour)
2. **If works:** Extract confidence direction for adaptive halting
3. **Training phase:** Learn pause tokens + UPFT (75% time savings)
4. **Later:** COCONUT curriculum for explicit hidden state looping
---
## Open Questions
1. Does random prefix scale to 27B? (Tested on 4B)
2. Optimal pause count for Qwen 3.5?
3. Does GDN respond more strongly than pure attention? (Testable)
4. Can we read confidence from GDN state S directly, not just hidden state h?
---
## References
- Random Prefix: https://github.com/dl1683/Latent-Space-Reasoning
- Pause Tokens: Google, "Think before you speak" (Oct 2023)
- COCONUT: Meta, "Training LLMs to Reason in Continuous Latent Space" (Dec 2024)
- Huginn: Geiping et al., "Scaling Test-Time Compute with Latent Reasoning" (Feb 2025)
- GDN Architecture: Our qwen35-gdn-implementation-findings-mar28 memory