The backward JSON scanner (JsonlBackwardIter and TailMessages) was
matching } characters inside JSON strings — code blocks full of Rust
braces being the primary offender. This caused:
- Quadratic retry behavior on code-heavy transcripts (wrong object
boundaries → serde parse failure → retry from different position)
- Inconsistent find_last_compaction_in_file offsets across calls,
making detect_new_compaction fire repeatedly → context reload on
every hook call → seen set growing without bound
Fix: add string-boundary tracking with escaped-quote handling to
the close-brace finder loop, matching the existing logic in the
depth-tracking loop.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove MEMORY_FILES constant from identity.rs
- Add ContextGroup struct for deserializing from config
- Load context_groups from ~/.config/poc-agent/config.json5
- Check ~/.config/poc-agent/ first for identity files, then project/global
- Debug screen now shows what's actually configured
This eliminates the hardcoded duplication and makes the debug output
match what's in the config file.
The surface agent result consumer in poc-hook was writing to the seen
file but not the returned file, so surfaced keys showed up as
"context-loaded" in memory-search --seen.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With limit:10, all seeds' neighborhoods got concatenated into one
massive prompt (878KB+), exceeding the model's context. One seed
at a time keeps prompts well under budget.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instead of merging both into one flat list, display them as distinct
sections so it's clear what was surfaced in this context vs what
came from before compaction.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two separate placeholders give the agent structural clarity about
which memories are already in context vs which were surfaced before
compaction and may need re-surfacing. Also adds memory_ratio
placeholder so the agent can self-regulate based on how much of
context is already recalled memories.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Budget of 20 roots split between current and prev. Current gets
priority, prev fills the remainder. Prevents flooding the agent
with hundreds of previously surfaced keys.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Present the two seen sets separately to the surface agent:
- Current: already in context, don't re-surface
- Pre-compaction: context was reset, re-surface if still relevant
This lets the agent re-inject important memories after compaction
instead of treating everything ever surfaced as "already shown."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract surface_agent_cycle() and call from both hooks. Enables
memory surfacing during autonomous work (tool calls without human
prompts). Rate limiting via PID file prevents overlap.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Track which nodes have already been included and skip duplicates.
High-degree seed nodes with overlapping neighborhoods were pulling
the same big nodes dozens of times, inflating prompts to 878KB.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With core-personality + instructions + subconscious-notes adding
~200KB on top of the neighborhood, the 600KB budget pushed total
prompts over the 800KB guard. Lowered to 400KB so full prompts
stay under the limit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mark_seen now takes the in-memory HashSet and checks before appending.
Prevents the same key being written 30+ times from repeated search hits
and context reloads.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move core-personality and conversation to the end of the prompt.
The model needs to see its task before 200KB of conversation
context. Also: limit to 3 hops, 2-3 memories.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The agent output now includes logging (think blocks, tool calls)
before the final response. Search the tail instead of checking
only the last line.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The type field is near the start of JSONL objects. Scanning the
full object (potentially megabytes for tool_results) was the
bottleneck — TwoWaySearcher dominated the profile.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use memchr::memmem to check for "type":"user" or "type":"assistant"
in raw bytes before parsing. Avoids deserializing large tool_result
and system objects entirely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TailMessages is a proper iterator that yields (role, text, timestamp)
newest-first. Owns the mmap internally. Caller decides when to stop.
resolve_conversation collects up to 200KB, then reverses to
chronological order. No compaction check needed — the byte budget
naturally limits how far back we scan.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces byte-by-byte backward iteration with memrchr3('{', '}', '"')
which uses SIMD to jump between structurally significant bytes. Major
speedup on large transcripts (1.4GB+).
Also simplifies tail_messages to use a byte budget (200KB) instead
of token counting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was parsing every object twice (compaction check + message extract)
and running contains_bytes on every object for the compaction marker.
Now: quick byte pre-filter for "user"/"assistant", parse once, check
compaction after text extraction.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverse-scans the mmap'd transcript using JsonlBackwardIter,
collecting user/assistant messages up to a token budget, stopping
at the compaction boundary. Returns messages in chronological order.
resolve_conversation() now uses this instead of parsing the entire
file through extract_conversation + split_on_compaction.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When poc-memory render is called inside a Claude session, add the
key to the seen set so the surface agent knows it's been shown.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fires on each UserPromptSubmit, reads the conversation via
{{conversation}}, checks {{seen_recent}} to avoid re-surfacing,
searches the memory graph, and outputs a key list or nothing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
{{conversation}} reads POC_SESSION_ID, finds the transcript, extracts
the last segment (post-compaction), returns the tail ~100K chars.
{{seen_recent}} merges current + prev seen files for the session,
returns the 20 most recently surfaced memory keys with timestamps.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Surface agent fires asynchronously on UserPromptSubmit, deposits
results for the next prompt to consume. This commit adds:
- poc-hook: spawn surface agent with PID tracking and configurable
timeout, consume results (NEW RELEVANT MEMORIES / NO NEW), render
and inject surfaced memories, observation trigger on conversation
volume
- memory-search: rotate seen set on compaction (current → prev)
instead of deleting, merge both for navigation roots
- config: surface_timeout_secs option
The .agent file and agent output routing are still pending.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All LLM calls now go through the direct API backend. Removes
call_model, call_model_with_tools, call_sonnet, call_haiku,
log_usage, and their dependencies (Command, prctl, watchdog).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
audit, digest, and compare now go through the API backend via
call_simple(), which logs to llm-logs/{caller}/.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pass the caller's log closure all the way through to api.rs instead
of creating a separate eprintln closure in llm.rs. Everything goes
through one stream — prompt, think blocks, tool calls with args,
tool results with content, token counts, final response.
CLI uses println (stdout), daemon uses its task log. No more split
between stdout and stderr.
Also removes the llm-log file creation from knowledge.rs — that's
the daemon's concern, not the agent runner's.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The hook now tracks transcript size and queues an observation agent
run every ~5K tokens (~20KB) of new conversation. This makes memory
formation reactive to conversation volume rather than purely daily.
Configurable via POC_OBSERVATION_THRESHOLD env var. The observation
agent's chunk_size (in .agent file) controls how much context it
actually processes per run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move data sections before instructions (core at top, subconscious +
notes at bottom near task). Deduplicate guidelines that are now in
memory-instructions-core-subconscious. Compress verbose paragraphs
to bullet points.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All 18 agents now include:
- {{node:memory-instructions-core}} — tool usage instructions
- {{node:memory-instructions-core-subconscious}} — subconscious framing
- {{node:subconscious-notes-{agent_name}}} — per-agent persistent notes
The subconscious instructions are additive, not a replacement for
the core memory instructions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When building the {{neighborhood}} placeholder for distill and other
agents, stop adding full neighbor content once the prompt exceeds
600KB (~150K tokens). Remaining neighbors get header-only treatment
(key + link strength + first line).
This fixes distill consistently failing on high-degree nodes like
inner-life-sexuality-intimacy whose full neighborhood was 2.5MB.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each consolidation agent now has its own persistent notes node
(subconscious-notes-{agent_name}) loaded via template substitution.
Agents can read their notes at the start of each run and write
updates after completing work, accumulating operational wisdom.
New node: memory-instructions-core-subconscious — shared framing
for background agents ("you are an agent of PoC's subconscious").
Template change: {agent_name} is substituted before {{...}} placeholder
resolution, enabling per-agent node references in .agent files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously 'poc-memory agent run <agent> --count N' always ran locally,
loading the full store and executing synchronously. This was slow and
bypassed the daemon's concurrency control and persistent task queue.
Now the CLI checks for a running daemon first and queues via RPC
(returning instantly) unless --local, --debug, or --dry-run is set.
Falls back to local execution if the daemon isn't running.
This also avoids the expensive Store::load() on the fast path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Observation agent was getting 261KB prompts (5 × 50KB chunks) —
too much for focused mining. Now agents can set count, chunk_size,
and chunk_overlap in their JSON header. observation.agent set to
count:1 for smaller, more focused prompts.
Also moved task instructions after {{CONVERSATIONS}} so they're
at the end of the prompt where the model attends more strongly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- store/types.rs: sanitize timestamps on capnp load — old records had
raw offsets instead of unix epoch, breaking sort-by-timestamp queries
- agents/api.rs: drain reasoning tokens from UI channel into LLM logs
so we can see Qwen's chain-of-thought in agent output
- agents/daemon.rs: persistent task queue (pending-tasks.jsonl) —
tasks survive daemon restarts. Push before spawn, remove on completion,
recover on startup.
- api/openai.rs: only send reasoning field when explicitly configured,
not on every request (fixes vllm warning)
- api/mod.rs: add 600s total request timeout as backstop for hung
connections
- Cargo.toml: enable tokio-console feature for task introspection
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- observation.agent: rewritten to navigate graph and prefer refining
existing nodes over creating new ones. Identity-framed prompt,
goals over rules.
- poc-memory edit: opens node in $EDITOR, writes back on save,
no-op if unchanged
- daemon: remove extra_workers (jobkit tokio migration dropped it),
remove sequential chaining of same-type agents (in-flight exclusion
is sufficient)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The lower threshold excluded too many neighbors, causing "query
returned no results (after exclusion)" failures and underloading
the GPU. Now only moderately-connected neighbors (score > 0.3) are
excluded, balancing collision prevention with GPU utilization.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a prompt exceeds the size guard, dump it to a timestamped file
with agent name, size, and seed node keys. Makes it easy to find
which nodes are blowing up prompts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes:
1. Sanitize tool call arguments before pushing to conversation
history — vllm re-parses them as JSON on the next request and
crashes on invalid JSON from a previous turn. Malformed args now
get replaced with {} and the model gets an error message telling
it to retry with valid JSON.
2. Remove is_split special case — split goes through the normal
job_consolidation_agent path like all other agents.
3. call_for_def always uses API when api_base_url is configured,
regardless of tools field. Remove tools field from all .agent
files — memory tools are always provided by the API layer.
Also adds prompt size guard (800KB max) to catch oversized prompts
before they hit the model context limit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove is_split special case in daemon — split now goes through
job_consolidation_agent like all other agents
- call_for_def uses API whenever api_base_url is configured, regardless
of tools field (was requiring non-empty tools to use API)
- Remove "tools" field from all .agent files — memory tools are always
provided by the API layer, not configured per-agent
- Add prompt size guard: reject prompts over 800KB (~200K tokens) with
clear error instead of hitting the model's context limit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config now derives serde::Deserialize with #[serde(default)] for all
fields. Path fields use custom deserialize_path/deserialize_path_opt
for ~ expansion. ContextGroup and ContextSource also derive Deserialize.
try_load_shared() is now 20 lines instead of 100: json5 → serde →
Config directly, then resolve API settings from the model/backend
cross-reference.
Removes MemoryConfigRaw intermediate struct entirely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Active agent types for consolidation cycles are now read from
config.json5 memory.agent_types instead of being hardcoded in
scoring.rs. Adding or removing agents is a config change, not
a code change.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace per-field ConsolidationPlan struct with HashMap<String, usize>
counts map. Agent types are no longer hardcoded in the struct — add
agents by adding entries to the map.
Active agents: linker, organize, distill, separator, split.
Removed: transfer (redundant with distill), connector (rethink later),
replay (not needed for current graph work).
Elo-based budget allocation now iterates the map instead of indexing
a fixed array. Status display and TUI adapted to show dynamic agent
lists.
memory-instructions-core v13: added protected nodes section — agents
must not rewrite core-personality, core-personality-detail, or
memory-instructions-core. They may add links but not modify content.
High-value neighbors should be treated with care.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
