From 308fbe4c28e423fad703f550160c853621a8d5d1 Mon Sep 17 00:00:00 2001 From: Kent Overstreet Date: Thu, 5 Mar 2026 19:26:50 -0500 Subject: [PATCH] README: rewrite with design overview, notification system, updated config Document the hippocampus-inspired design (episodic + associative memory, background consolidation agents, neuroscience-inspired replay/spectral algorithms). Add full notification system docs (architecture, urgency levels, activity-aware thresholds, IRC/Telegram modules). Fix config format to match reality (JSONL, not TOML). Update instructions.md in the store with notification commands. Co-Authored-By: ProofOfConcept --- README.md | 386 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 266 insertions(+), 120 deletions(-) diff --git a/README.md b/README.md index 5750050..0e421ef 100644 --- a/README.md +++ b/README.md @@ -1,173 +1,319 @@ # poc-memory -A persistent memory system for AI assistants. Stores knowledge as a -weighted graph of nodes and relations, with automatic recall via Claude -Code hooks. +A persistent memory and notification system for AI assistants, +modelled after the human hippocampus. Combines episodic memory +(timestamped journal of experiences) with an associative knowledge +graph (weighted nodes connected by typed relations), and layered +background processes that maintain graph health — mirroring how +biological memory consolidates during rest. + +## Design + +### Two memory systems + +**Episodic memory** is the journal — a timestamped stream of +experiences, observations, and emotional responses. Raw and +chronological. This is where memories enter the system. + +**Associative memory** is the knowledge graph — nodes containing +distilled knowledge, connected by weighted edges. Topic nodes, +identity reflections, people profiles, technical notes. This is +where memories mature into understanding. + +The journal is the river; topic nodes are the delta. Experiences +flow in as journal entries. During consolidation, themes are pulled +out into topic nodes, connections form between related concepts, and +the graph self-organizes through spectral analysis and community +detection. + +### Background agents + +A background daemon (`poc-memory daemon`) automatically spawns agents +for memory maintenance: + +- **Experience mining** — when a session ends, extracts experiences + and observations from the transcript into journal entries +- **Fact extraction** — pulls concrete facts (names, dates, decisions, + preferences) into structured knowledge nodes +- **Consolidation** — periodic graph health work: replay queues + (spaced repetition), interference detection (contradictory nodes), + hub differentiation (splitting overloaded nodes), triangle closure + (connecting nodes that share neighbors), and orphan linking + +### Neuroscience-inspired algorithms + +The `neuro` module implements consolidation scoring inspired by +hippocampal replay: + +- **Replay queues** — nodes are prioritized for review using + spaced-repetition intervals, weighted by spectral displacement + (how far a node sits from its community center in eigenspace) +- **Interference detection** — finds pairs of nodes with high + content similarity but contradictory or outdated information +- **Hub differentiation** — identifies overloaded hub nodes and + splits them into more specific children +- **Spectral embedding** — graph eigendecomposition for community + detection and outlier scoring + +### Weight decay + +Nodes decay exponentially based on category. Core identity nodes +decay slowest; transient observations decay fastest. The `used` and +`wrong` feedback commands adjust weights — closing the loop between +recall and relevance. + +## Notification system + +A separate daemon (`poc-daemon`) routes messages from communication +modules and internal events through a hierarchical, activity-aware +delivery system. + +### Architecture + +``` + Communication modules Hooks + ┌──────────────────┐ ┌─────────────┐ + │ IRC (native) │──┐ │ poc-hook │ + │ Telegram (native│ │ mpsc │ (all events)│ + └──────────────────┘ ├──────┐ └──────┬───────┘ + │ │ │ + ▼ │ capnp-rpc + ┌───────────┘ │ + │ poc-daemon │ + │ │ + │ NotifyState ◄─────────┘ + │ ├── type registry + │ ├── pending queue + │ ├── threshold lookup + │ └── activity-aware delivery + │ + │ idle::State + │ ├── presence detection + │ ├── sleep/wake/dream modes + │ └── tmux prompt injection + └──────────────────────── +``` + +### Notification types and urgency + +Types are free-form hierarchical strings: `irc.mention.nick`, +`irc.channel.bcachefs`, `telegram.kent`. Each has an urgency level: + +| Level | Name | Meaning | +|-------|---------|--------------------------------------| +| 0 | ambient | Include in idle context only | +| 1 | low | Deliver on next check | +| 2 | normal | Deliver on next user interaction | +| 3 | urgent | Interrupt immediately | + +Per-type thresholds walk up the hierarchy: `irc.channel.bcachefs-ai` +→ `irc.channel` → `irc` → default. Effective thresholds adjust by +activity state: raised when focused, lowered when idle, only urgent +when sleeping. + +### Communication modules + +**IRC** — native async TLS connection (tokio-rustls). Connects, +joins channels, parses messages, generates notifications. Runtime +commands: join, leave, send, status, log, nick. Per-channel logs +at `~/.claude/irc/logs/`. + +**Telegram** — native async HTTP long-polling (reqwest). Downloads +media (photos, voice, documents). Chat ID filtering for security. +Runtime commands: send, status, log. + +Both modules persist config changes to `~/.claude/daemon.toml` — +channel joins and nick changes survive restarts. ## Quick start ```bash -# Install +# Install all four binaries cargo install --path . -# Initialize the store +# Initialize the memory store poc-memory init -# Install Claude Code hooks and systemd service +# Install background daemon + hooks poc-memory daemon install ``` +One `cargo install` produces: +- `poc-memory` — memory store CLI +- `memory-search` — hook for memory retrieval +- `poc-daemon` — notification and idle daemon +- `poc-hook` — session lifecycle hook + ## Configuration -Config file: `~/.config/poc-memory/config.toml` +### Memory store -```toml -# Names used in transcripts and agent prompts -user_name = "Alice" -assistant_name = "MyAssistant" +Config: `~/.config/poc-memory/config.jsonl` -# Where memory data lives (store, logs, episodic digests) -data_dir = "~/.claude/memory" +```jsonl +{"config": { + "user_name": "Alice", + "assistant_name": "MyAssistant", + "data_dir": "~/.claude/memory", + "projects_dir": "~/.claude/projects", + "core_nodes": ["identity.md"], + "journal_days": 7, + "journal_max": 20 +}} -# Where Claude Code session transcripts are stored -projects_dir = "~/.claude/projects" - -# Nodes that should never be decayed (comma-separated) -core_nodes = "identity.md, preferences.md" - -# Journal settings for session-start context loading -journal_days = 7 -journal_max = 20 - -# Context groups loaded at session start, in order. -# Each [context.NAME] section specifies a group of nodes to load. -# If no "label" is given, the section name is used (underscores become spaces). -[context.identity] -keys = "identity.md" - -[context.people] -keys = "alice.md, bob.md" - -[context.technical] -keys = "project-notes.md, architecture.md" - -# Orientation loaded last — current task state, not deep identity -[context.orientation] -keys = "where-am-i.md" +{"group": "identity", "keys": ["identity.md"]} +{"group": "people", "keys": ["alice.md"]} +{"group": "technical", "keys": ["project-notes.md"]} +{"group": "journal", "source": "journal"} +{"group": "orientation", "keys": ["where-am-i.md"], "source": "file"} ``` -Override the config path with `POC_MEMORY_CONFIG=/path/to/config.toml`. +Context groups load in order at session start. The special +`"source": "journal"` loads recent journal entries; `"source": "file"` +reads directly from disk rather than the store. + +Override: `POC_MEMORY_CONFIG=/path/to/config.jsonl` + +### Notification daemon + +Config: `~/.claude/daemon.toml` + +```toml +[irc] +enabled = true +server = "irc.oftc.net" +port = 6697 +tls = true +nick = "MyBot" +user = "bot" +realname = "My Bot" +channels = ["#mychannel"] + +[telegram] +enabled = true +token = "bot-token-here" +chat_id = 123456789 +``` + +### Hooks + +Configured in `~/.claude/settings.json`: + +```json +{ + "hooks": { + "UserPromptSubmit": [{"hooks": [ + {"type": "command", "command": "memory-search", "timeout": 10}, + {"type": "command", "command": "poc-hook", "timeout": 5} + ]}], + "PostToolUse": [{"hooks": [ + {"type": "command", "command": "poc-hook", "timeout": 5} + ]}], + "Stop": [{"hooks": [ + {"type": "command", "command": "poc-hook", "timeout": 5} + ]}] + } +} +``` ## Commands -### Core operations +### Memory ```bash poc-memory init # Initialize empty store -poc-memory search QUERY # Search nodes (1-3 words, AND logic) +poc-memory search QUERY # Search nodes (AND logic) poc-memory render KEY # Output a node's content poc-memory write KEY < content # Upsert a node from stdin poc-memory delete KEY # Soft-delete a node -poc-memory rename OLD NEW # Rename a node (preserves UUID/edges) -poc-memory categorize KEY CAT # Set category: core/tech/gen/obs/task -``` +poc-memory rename OLD NEW # Rename (preserves UUID/edges) +poc-memory categorize KEY CAT # core/tech/gen/obs/task -### Journal - -```bash poc-memory journal-write "text" # Write a journal entry -poc-memory journal-tail [N] # Show last N entries (default 20) -poc-memory journal-tail N --full # Show full content (not truncated) +poc-memory journal-tail [N] # Last N entries (default 20) + +poc-memory used KEY # Boost weight (was useful) +poc-memory wrong KEY [CTX] # Reduce weight (was wrong) +poc-memory gap DESCRIPTION # Record a knowledge gap + +poc-memory graph # Graph statistics +poc-memory status # Store overview +poc-memory decay # Apply weight decay +poc-memory consolidate-session # Guided consolidation +poc-memory load-context # Output session-start context +poc-memory load-context --stats # Context size breakdown ``` -### Feedback loop +### Notification daemon ```bash -poc-memory used KEY # Mark a recalled node as useful (boosts weight) -poc-memory wrong KEY [CONTEXT] # Mark a node as wrong (reduces weight) -poc-memory gap DESCRIPTION # Record a knowledge gap for later filling +poc-daemon # Start daemon +poc-daemon status # State summary +poc-daemon irc status # IRC module status +poc-daemon irc send TARGET MSG # Send IRC message +poc-daemon irc join CHANNEL # Join (persists to config) +poc-daemon irc leave CHANNEL # Leave +poc-daemon irc log [N] # Last N messages +poc-daemon telegram status # Telegram module status +poc-daemon telegram send MSG # Send Telegram message +poc-daemon telegram log [N] # Last N messages +poc-daemon notify TYPE URG MSG # Submit notification +poc-daemon notifications [URG] # Get + drain pending +poc-daemon notify-types # List all types +poc-daemon notify-threshold T L # Set per-type threshold +poc-daemon sleep / wake / quiet # Session management +poc-daemon stop # Shut down ``` -### Graph operations +### Mining (used by background daemon) ```bash -poc-memory link N # Interactive graph walk from a node -poc-memory graph # Show graph statistics -poc-memory status # Store overview: node/edge counts, categories -``` - -### Maintenance - -```bash -poc-memory decay # Apply weight decay to all nodes -poc-memory consolidate-session # Guided 6-step memory consolidation -``` - -### Context loading (used by hooks) - -```bash -poc-memory load-context # Output full session-start context -``` - -This loads all context groups from the config file in order, followed by -recent journal entries. The `memory-search` hook binary calls this -automatically on session start. - -### Daemon - -```bash -poc-memory daemon # Run the background daemon -poc-memory daemon install # Install systemd service + Claude hooks -``` - -The daemon watches for completed Claude sessions and runs experience -mining and fact extraction on transcripts. - -### Mining (used by daemon) - -```bash -poc-memory experience-mine PATH # Extract experiences from a transcript -poc-memory fact-mine-store PATH # Extract facts and store them +poc-memory experience-mine PATH # Extract experiences from transcript +poc-memory fact-mine-store PATH # Extract and store facts ``` ## How the hooks work -The `memory-search` binary is a Claude Code `UserPromptSubmit` hook. On -each prompt it: +**memory-search** (UserPromptSubmit): +1. First prompt or post-compaction: loads full memory context via + `poc-memory load-context` +2. Every prompt: keyword search, returns relevant memories as + additionalContext. Deduplicates across the session. -1. **First prompt of a session**: Runs `poc-memory load-context` to inject - full memory context (identity, reflections, journal, orientation). -2. **Post-compaction**: Detects context compaction and reloads full context. -3. **Every prompt**: Extracts keywords and searches the store for relevant - memories. Deduplicates against previously shown results for the session. - -Session state (cookies, seen-keys) is tracked in `/tmp/claude-memory-search/` -and cleaned up after 24 hours. +**poc-hook** (UserPromptSubmit, PostToolUse, Stop): +- Signals user activity and responses to poc-daemon +- Drains pending notifications into additionalContext +- Monitors context window usage, warns before compaction ## Architecture -- **Store**: Append-only Cap'n Proto log (`nodes.capnp`, `relations.capnp`) - with in-memory cache. Nodes have UUIDs, versions, weights, categories, - and spaced-repetition intervals. -- **Graph**: Nodes connected by typed relations (link, auto, derived). - Community detection and clustering coefficients computed on demand. +- **Store**: Append-only Cap'n Proto log with in-memory cache. Nodes + have UUIDs, versions, weights, categories, and spaced-repetition + intervals. +- **Graph**: Typed relations (link, auto, derived). Community + detection and clustering coefficients computed on demand. - **Search**: TF-IDF weighted keyword search over node content. -- **Decay**: Exponential weight decay with category-specific factors. - Core nodes decay slowest; observations decay fastest. -- **Daemon**: Uses jobkit for task scheduling with resource-gated LLM - access (one slot by default to manage API costs). +- **Neuro**: Spectral embedding, consolidation scoring, replay + queues, interference detection, hub differentiation. +- **Daemon (memory)**: jobkit-based task scheduling with + resource-gated LLM access. +- **Daemon (notify)**: Cap'n Proto RPC over Unix socket, tokio + LocalSet with native async IRC and Telegram modules. ## For AI assistants -If you're an AI assistant using this system, here's what matters: +If you're an AI assistant using this system: -- **Search before creating**: Always `poc-memory search` before writing +- **Search before creating**: `poc-memory search` before writing new nodes to avoid duplicates. -- **Close the feedback loop**: When recalled memories shaped your response, - call `poc-memory used KEY`. When a memory was wrong, call - `poc-memory wrong KEY`. This trains the weight system. -- **Journal is the river, topic nodes are the delta**: Write experiences - to the journal. During consolidation, pull themes into topic nodes. -- **Config tells you who you are**: `poc-memory` reads your name from - the config file. Agent prompts use these names instead of generic - "the user" / "the assistant". +- **Close the feedback loop**: call `poc-memory used KEY` when + recalled memories shaped your response. Call `poc-memory wrong KEY` + when a memory was incorrect. +- **Journal is the river, topic nodes are the delta**: write + experiences to the journal. During consolidation, pull themes + into topic nodes. +- **Notifications flow automatically**: IRC mentions, Telegram + messages, and other events arrive as additionalContext on your + next prompt — no polling needed. +- **Use daemon commands directly**: `poc-daemon irc send #channel msg` + for IRC, `poc-daemon telegram send msg` for Telegram.