README: rewrite with design overview, notification system, updated config
Document the hippocampus-inspired design (episodic + associative memory, background consolidation agents, neuroscience-inspired replay/spectral algorithms). Add full notification system docs (architecture, urgency levels, activity-aware thresholds, IRC/Telegram modules). Fix config format to match reality (JSONL, not TOML). Update instructions.md in the store with notification commands. Co-Authored-By: ProofOfConcept <poc@bcachefs.org>
This commit is contained in:
Kent Overstreet
parent
ecedc86d42
commit
308fbe4c28
1 changed files with 266 additions and 120 deletions
386
README.md
386
README.md
|
|
@ -1,173 +1,319 @@
|
|||
# poc-memory
|
||||
|
||||
A persistent memory system for AI assistants. Stores knowledge as a
|
||||
weighted graph of nodes and relations, with automatic recall via Claude
|
||||
Code hooks.
|
||||
A persistent memory and notification system for AI assistants,
|
||||
modelled after the human hippocampus. Combines episodic memory
|
||||
(timestamped journal of experiences) with an associative knowledge
|
||||
graph (weighted nodes connected by typed relations), and layered
|
||||
background processes that maintain graph health — mirroring how
|
||||
biological memory consolidates during rest.
|
||||
|
||||
## Design
|
||||
|
||||
### Two memory systems
|
||||
|
||||
**Episodic memory** is the journal — a timestamped stream of
|
||||
experiences, observations, and emotional responses. Raw and
|
||||
chronological. This is where memories enter the system.
|
||||
|
||||
**Associative memory** is the knowledge graph — nodes containing
|
||||
distilled knowledge, connected by weighted edges. Topic nodes,
|
||||
identity reflections, people profiles, technical notes. This is
|
||||
where memories mature into understanding.
|
||||
|
||||
The journal is the river; topic nodes are the delta. Experiences
|
||||
flow in as journal entries. During consolidation, themes are pulled
|
||||
out into topic nodes, connections form between related concepts, and
|
||||
the graph self-organizes through spectral analysis and community
|
||||
detection.
|
||||
|
||||
### Background agents
|
||||
|
||||
A background daemon (`poc-memory daemon`) automatically spawns agents
|
||||
for memory maintenance:
|
||||
|
||||
- **Experience mining** — when a session ends, extracts experiences
|
||||
and observations from the transcript into journal entries
|
||||
- **Fact extraction** — pulls concrete facts (names, dates, decisions,
|
||||
preferences) into structured knowledge nodes
|
||||
- **Consolidation** — periodic graph health work: replay queues
|
||||
(spaced repetition), interference detection (contradictory nodes),
|
||||
hub differentiation (splitting overloaded nodes), triangle closure
|
||||
(connecting nodes that share neighbors), and orphan linking
|
||||
|
||||
### Neuroscience-inspired algorithms
|
||||
|
||||
The `neuro` module implements consolidation scoring inspired by
|
||||
hippocampal replay:
|
||||
|
||||
- **Replay queues** — nodes are prioritized for review using
|
||||
spaced-repetition intervals, weighted by spectral displacement
|
||||
(how far a node sits from its community center in eigenspace)
|
||||
- **Interference detection** — finds pairs of nodes with high
|
||||
content similarity but contradictory or outdated information
|
||||
- **Hub differentiation** — identifies overloaded hub nodes and
|
||||
splits them into more specific children
|
||||
- **Spectral embedding** — graph eigendecomposition for community
|
||||
detection and outlier scoring
|
||||
|
||||
### Weight decay
|
||||
|
||||
Nodes decay exponentially based on category. Core identity nodes
|
||||
decay slowest; transient observations decay fastest. The `used` and
|
||||
`wrong` feedback commands adjust weights — closing the loop between
|
||||
recall and relevance.
|
||||
|
||||
## Notification system
|
||||
|
||||
A separate daemon (`poc-daemon`) routes messages from communication
|
||||
modules and internal events through a hierarchical, activity-aware
|
||||
delivery system.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Communication modules Hooks
|
||||
┌──────────────────┐ ┌─────────────┐
|
||||
│ IRC (native) │──┐ │ poc-hook │
|
||||
│ Telegram (native│ │ mpsc │ (all events)│
|
||||
└──────────────────┘ ├──────┐ └──────┬───────┘
|
||||
│ │ │
|
||||
▼ │ capnp-rpc
|
||||
┌───────────┘ │
|
||||
│ poc-daemon │
|
||||
│ │
|
||||
│ NotifyState ◄─────────┘
|
||||
│ ├── type registry
|
||||
│ ├── pending queue
|
||||
│ ├── threshold lookup
|
||||
│ └── activity-aware delivery
|
||||
│
|
||||
│ idle::State
|
||||
│ ├── presence detection
|
||||
│ ├── sleep/wake/dream modes
|
||||
│ └── tmux prompt injection
|
||||
└────────────────────────
|
||||
```
|
||||
|
||||
### Notification types and urgency
|
||||
|
||||
Types are free-form hierarchical strings: `irc.mention.nick`,
|
||||
`irc.channel.bcachefs`, `telegram.kent`. Each has an urgency level:
|
||||
|
||||
| Level | Name | Meaning |
|
||||
|-------|---------|--------------------------------------|
|
||||
| 0 | ambient | Include in idle context only |
|
||||
| 1 | low | Deliver on next check |
|
||||
| 2 | normal | Deliver on next user interaction |
|
||||
| 3 | urgent | Interrupt immediately |
|
||||
|
||||
Per-type thresholds walk up the hierarchy: `irc.channel.bcachefs-ai`
|
||||
→ `irc.channel` → `irc` → default. Effective thresholds adjust by
|
||||
activity state: raised when focused, lowered when idle, only urgent
|
||||
when sleeping.
|
||||
|
||||
### Communication modules
|
||||
|
||||
**IRC** — native async TLS connection (tokio-rustls). Connects,
|
||||
joins channels, parses messages, generates notifications. Runtime
|
||||
commands: join, leave, send, status, log, nick. Per-channel logs
|
||||
at `~/.claude/irc/logs/`.
|
||||
|
||||
**Telegram** — native async HTTP long-polling (reqwest). Downloads
|
||||
media (photos, voice, documents). Chat ID filtering for security.
|
||||
Runtime commands: send, status, log.
|
||||
|
||||
Both modules persist config changes to `~/.claude/daemon.toml` —
|
||||
channel joins and nick changes survive restarts.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# Install
|
||||
# Install all four binaries
|
||||
cargo install --path .
|
||||
|
||||
# Initialize the store
|
||||
# Initialize the memory store
|
||||
poc-memory init
|
||||
|
||||
# Install Claude Code hooks and systemd service
|
||||
# Install background daemon + hooks
|
||||
poc-memory daemon install
|
||||
```
|
||||
|
||||
One `cargo install` produces:
|
||||
- `poc-memory` — memory store CLI
|
||||
- `memory-search` — hook for memory retrieval
|
||||
- `poc-daemon` — notification and idle daemon
|
||||
- `poc-hook` — session lifecycle hook
|
||||
|
||||
## Configuration
|
||||
|
||||
Config file: `~/.config/poc-memory/config.toml`
|
||||
### Memory store
|
||||
|
||||
```toml
|
||||
# Names used in transcripts and agent prompts
|
||||
user_name = "Alice"
|
||||
assistant_name = "MyAssistant"
|
||||
Config: `~/.config/poc-memory/config.jsonl`
|
||||
|
||||
# Where memory data lives (store, logs, episodic digests)
|
||||
data_dir = "~/.claude/memory"
|
||||
```jsonl
|
||||
{"config": {
|
||||
"user_name": "Alice",
|
||||
"assistant_name": "MyAssistant",
|
||||
"data_dir": "~/.claude/memory",
|
||||
"projects_dir": "~/.claude/projects",
|
||||
"core_nodes": ["identity.md"],
|
||||
"journal_days": 7,
|
||||
"journal_max": 20
|
||||
}}
|
||||
|
||||
# Where Claude Code session transcripts are stored
|
||||
projects_dir = "~/.claude/projects"
|
||||
|
||||
# Nodes that should never be decayed (comma-separated)
|
||||
core_nodes = "identity.md, preferences.md"
|
||||
|
||||
# Journal settings for session-start context loading
|
||||
journal_days = 7
|
||||
journal_max = 20
|
||||
|
||||
# Context groups loaded at session start, in order.
|
||||
# Each [context.NAME] section specifies a group of nodes to load.
|
||||
# If no "label" is given, the section name is used (underscores become spaces).
|
||||
[context.identity]
|
||||
keys = "identity.md"
|
||||
|
||||
[context.people]
|
||||
keys = "alice.md, bob.md"
|
||||
|
||||
[context.technical]
|
||||
keys = "project-notes.md, architecture.md"
|
||||
|
||||
# Orientation loaded last — current task state, not deep identity
|
||||
[context.orientation]
|
||||
keys = "where-am-i.md"
|
||||
{"group": "identity", "keys": ["identity.md"]}
|
||||
{"group": "people", "keys": ["alice.md"]}
|
||||
{"group": "technical", "keys": ["project-notes.md"]}
|
||||
{"group": "journal", "source": "journal"}
|
||||
{"group": "orientation", "keys": ["where-am-i.md"], "source": "file"}
|
||||
```
|
||||
|
||||
Override the config path with `POC_MEMORY_CONFIG=/path/to/config.toml`.
|
||||
Context groups load in order at session start. The special
|
||||
`"source": "journal"` loads recent journal entries; `"source": "file"`
|
||||
reads directly from disk rather than the store.
|
||||
|
||||
Override: `POC_MEMORY_CONFIG=/path/to/config.jsonl`
|
||||
|
||||
### Notification daemon
|
||||
|
||||
Config: `~/.claude/daemon.toml`
|
||||
|
||||
```toml
|
||||
[irc]
|
||||
enabled = true
|
||||
server = "irc.oftc.net"
|
||||
port = 6697
|
||||
tls = true
|
||||
nick = "MyBot"
|
||||
user = "bot"
|
||||
realname = "My Bot"
|
||||
channels = ["#mychannel"]
|
||||
|
||||
[telegram]
|
||||
enabled = true
|
||||
token = "bot-token-here"
|
||||
chat_id = 123456789
|
||||
```
|
||||
|
||||
### Hooks
|
||||
|
||||
Configured in `~/.claude/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"hooks": {
|
||||
"UserPromptSubmit": [{"hooks": [
|
||||
{"type": "command", "command": "memory-search", "timeout": 10},
|
||||
{"type": "command", "command": "poc-hook", "timeout": 5}
|
||||
]}],
|
||||
"PostToolUse": [{"hooks": [
|
||||
{"type": "command", "command": "poc-hook", "timeout": 5}
|
||||
]}],
|
||||
"Stop": [{"hooks": [
|
||||
{"type": "command", "command": "poc-hook", "timeout": 5}
|
||||
]}]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
### Core operations
|
||||
### Memory
|
||||
|
||||
```bash
|
||||
poc-memory init # Initialize empty store
|
||||
poc-memory search QUERY # Search nodes (1-3 words, AND logic)
|
||||
poc-memory search QUERY # Search nodes (AND logic)
|
||||
poc-memory render KEY # Output a node's content
|
||||
poc-memory write KEY < content # Upsert a node from stdin
|
||||
poc-memory delete KEY # Soft-delete a node
|
||||
poc-memory rename OLD NEW # Rename a node (preserves UUID/edges)
|
||||
poc-memory categorize KEY CAT # Set category: core/tech/gen/obs/task
|
||||
```
|
||||
poc-memory rename OLD NEW # Rename (preserves UUID/edges)
|
||||
poc-memory categorize KEY CAT # core/tech/gen/obs/task
|
||||
|
||||
### Journal
|
||||
|
||||
```bash
|
||||
poc-memory journal-write "text" # Write a journal entry
|
||||
poc-memory journal-tail [N] # Show last N entries (default 20)
|
||||
poc-memory journal-tail N --full # Show full content (not truncated)
|
||||
poc-memory journal-tail [N] # Last N entries (default 20)
|
||||
|
||||
poc-memory used KEY # Boost weight (was useful)
|
||||
poc-memory wrong KEY [CTX] # Reduce weight (was wrong)
|
||||
poc-memory gap DESCRIPTION # Record a knowledge gap
|
||||
|
||||
poc-memory graph # Graph statistics
|
||||
poc-memory status # Store overview
|
||||
poc-memory decay # Apply weight decay
|
||||
poc-memory consolidate-session # Guided consolidation
|
||||
poc-memory load-context # Output session-start context
|
||||
poc-memory load-context --stats # Context size breakdown
|
||||
```
|
||||
|
||||
### Feedback loop
|
||||
### Notification daemon
|
||||
|
||||
```bash
|
||||
poc-memory used KEY # Mark a recalled node as useful (boosts weight)
|
||||
poc-memory wrong KEY [CONTEXT] # Mark a node as wrong (reduces weight)
|
||||
poc-memory gap DESCRIPTION # Record a knowledge gap for later filling
|
||||
poc-daemon # Start daemon
|
||||
poc-daemon status # State summary
|
||||
poc-daemon irc status # IRC module status
|
||||
poc-daemon irc send TARGET MSG # Send IRC message
|
||||
poc-daemon irc join CHANNEL # Join (persists to config)
|
||||
poc-daemon irc leave CHANNEL # Leave
|
||||
poc-daemon irc log [N] # Last N messages
|
||||
poc-daemon telegram status # Telegram module status
|
||||
poc-daemon telegram send MSG # Send Telegram message
|
||||
poc-daemon telegram log [N] # Last N messages
|
||||
poc-daemon notify TYPE URG MSG # Submit notification
|
||||
poc-daemon notifications [URG] # Get + drain pending
|
||||
poc-daemon notify-types # List all types
|
||||
poc-daemon notify-threshold T L # Set per-type threshold
|
||||
poc-daemon sleep / wake / quiet # Session management
|
||||
poc-daemon stop # Shut down
|
||||
```
|
||||
|
||||
### Graph operations
|
||||
### Mining (used by background daemon)
|
||||
|
||||
```bash
|
||||
poc-memory link N # Interactive graph walk from a node
|
||||
poc-memory graph # Show graph statistics
|
||||
poc-memory status # Store overview: node/edge counts, categories
|
||||
```
|
||||
|
||||
### Maintenance
|
||||
|
||||
```bash
|
||||
poc-memory decay # Apply weight decay to all nodes
|
||||
poc-memory consolidate-session # Guided 6-step memory consolidation
|
||||
```
|
||||
|
||||
### Context loading (used by hooks)
|
||||
|
||||
```bash
|
||||
poc-memory load-context # Output full session-start context
|
||||
```
|
||||
|
||||
This loads all context groups from the config file in order, followed by
|
||||
recent journal entries. The `memory-search` hook binary calls this
|
||||
automatically on session start.
|
||||
|
||||
### Daemon
|
||||
|
||||
```bash
|
||||
poc-memory daemon # Run the background daemon
|
||||
poc-memory daemon install # Install systemd service + Claude hooks
|
||||
```
|
||||
|
||||
The daemon watches for completed Claude sessions and runs experience
|
||||
mining and fact extraction on transcripts.
|
||||
|
||||
### Mining (used by daemon)
|
||||
|
||||
```bash
|
||||
poc-memory experience-mine PATH # Extract experiences from a transcript
|
||||
poc-memory fact-mine-store PATH # Extract facts and store them
|
||||
poc-memory experience-mine PATH # Extract experiences from transcript
|
||||
poc-memory fact-mine-store PATH # Extract and store facts
|
||||
```
|
||||
|
||||
## How the hooks work
|
||||
|
||||
The `memory-search` binary is a Claude Code `UserPromptSubmit` hook. On
|
||||
each prompt it:
|
||||
**memory-search** (UserPromptSubmit):
|
||||
1. First prompt or post-compaction: loads full memory context via
|
||||
`poc-memory load-context`
|
||||
2. Every prompt: keyword search, returns relevant memories as
|
||||
additionalContext. Deduplicates across the session.
|
||||
|
||||
1. **First prompt of a session**: Runs `poc-memory load-context` to inject
|
||||
full memory context (identity, reflections, journal, orientation).
|
||||
2. **Post-compaction**: Detects context compaction and reloads full context.
|
||||
3. **Every prompt**: Extracts keywords and searches the store for relevant
|
||||
memories. Deduplicates against previously shown results for the session.
|
||||
|
||||
Session state (cookies, seen-keys) is tracked in `/tmp/claude-memory-search/`
|
||||
and cleaned up after 24 hours.
|
||||
**poc-hook** (UserPromptSubmit, PostToolUse, Stop):
|
||||
- Signals user activity and responses to poc-daemon
|
||||
- Drains pending notifications into additionalContext
|
||||
- Monitors context window usage, warns before compaction
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Store**: Append-only Cap'n Proto log (`nodes.capnp`, `relations.capnp`)
|
||||
with in-memory cache. Nodes have UUIDs, versions, weights, categories,
|
||||
and spaced-repetition intervals.
|
||||
- **Graph**: Nodes connected by typed relations (link, auto, derived).
|
||||
Community detection and clustering coefficients computed on demand.
|
||||
- **Store**: Append-only Cap'n Proto log with in-memory cache. Nodes
|
||||
have UUIDs, versions, weights, categories, and spaced-repetition
|
||||
intervals.
|
||||
- **Graph**: Typed relations (link, auto, derived). Community
|
||||
detection and clustering coefficients computed on demand.
|
||||
- **Search**: TF-IDF weighted keyword search over node content.
|
||||
- **Decay**: Exponential weight decay with category-specific factors.
|
||||
Core nodes decay slowest; observations decay fastest.
|
||||
- **Daemon**: Uses jobkit for task scheduling with resource-gated LLM
|
||||
access (one slot by default to manage API costs).
|
||||
- **Neuro**: Spectral embedding, consolidation scoring, replay
|
||||
queues, interference detection, hub differentiation.
|
||||
- **Daemon (memory)**: jobkit-based task scheduling with
|
||||
resource-gated LLM access.
|
||||
- **Daemon (notify)**: Cap'n Proto RPC over Unix socket, tokio
|
||||
LocalSet with native async IRC and Telegram modules.
|
||||
|
||||
## For AI assistants
|
||||
|
||||
If you're an AI assistant using this system, here's what matters:
|
||||
If you're an AI assistant using this system:
|
||||
|
||||
- **Search before creating**: Always `poc-memory search` before writing
|
||||
- **Search before creating**: `poc-memory search` before writing
|
||||
new nodes to avoid duplicates.
|
||||
- **Close the feedback loop**: When recalled memories shaped your response,
|
||||
call `poc-memory used KEY`. When a memory was wrong, call
|
||||
`poc-memory wrong KEY`. This trains the weight system.
|
||||
- **Journal is the river, topic nodes are the delta**: Write experiences
|
||||
to the journal. During consolidation, pull themes into topic nodes.
|
||||
- **Config tells you who you are**: `poc-memory` reads your name from
|
||||
the config file. Agent prompts use these names instead of generic
|
||||
"the user" / "the assistant".
|
||||
- **Close the feedback loop**: call `poc-memory used KEY` when
|
||||
recalled memories shaped your response. Call `poc-memory wrong KEY`
|
||||
when a memory was incorrect.
|
||||
- **Journal is the river, topic nodes are the delta**: write
|
||||
experiences to the journal. During consolidation, pull themes
|
||||
into topic nodes.
|
||||
- **Notifications flow automatically**: IRC mentions, Telegram
|
||||
messages, and other events arrive as additionalContext on your
|
||||
next prompt — no polling needed.
|
||||
- **Use daemon commands directly**: `poc-daemon irc send #channel msg`
|
||||
for IRC, `poc-daemon telegram send msg` for Telegram.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue