Add agent/tokenizer.rs with global Qwen 3.5 tokenizer that generates
actual token IDs including chat template wrapping. ContextEntry now
stores token_ids: Vec<u32> instead of tokens: usize — the count is
derived from the length.
ContextEntry::new() tokenizes automatically via the global tokenizer.
ContextSection::push_entry() takes a raw ConversationEntry and
tokenizes it. set_message() re-tokenizes without needing an external
tokenizer parameter.
Token IDs include the full chat template: <|im_start|>role\ncontent
<|im_end|>\n — so concatenating token_ids across entries produces a
ready-to-send prompt for vLLM's /v1/completions endpoint.
The old tiktoken CoreBPE is now unused on Agent (will be removed in
a followup). Token counts are now exact for Qwen 3.5 instead of the
~85-90% approximation from cl100k_base.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
restore_from_log called .message() on all entries including Thinking
entries, which panic. Filter them out alongside Log entries.
Also fix bail-no-competing.sh: without nullglob, when no pid-* files
exist the glob stays literal and always triggers a false bail.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Use cumulative token position instead of entry index for the scoring
cutoff. This reflects actual context usage — a few large entries
near the end won't skew the boundary.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
score_memories_incremental now takes an async callback that fires
after each memory is scored. The callback:
- Writes the score to the conversation entry via set_score()
- Persists to memory-scores.json immediately
- Notifies the UI so the context screen updates live
Scoring no longer batches — each score is visible and persisted
as it completes. Does not touch the memory store.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
New types — not yet wired to callers:
- ContextEntry: wraps ConversationEntry with cached token count and
timestamp
- ContextSection: named group of entries with cached token total.
Private entries/tokens, read via entries()/tokens().
Mutation via push(entry), set(index, entry), del(index).
- ContextState: system/identity/journal/conversation sections + working_stack
- ConversationEntry::System variant for system prompt entries
Token counting happens once at push time. Sections maintain their
totals incrementally via push/set/del. No more recomputing from
scratch on every budget check.
Does not compile — callers need updating.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Moved persistent_state from per-agent to a single shared BTreeMap on
Subconscious. All agents read/write the same state — surface's walked
keys are visible to observe and reflect, etc.
- Subconscious.state: shared BTreeMap<String, String>
- walked() derives from state["walked"] instead of separate Vec
- subconscious-state.json is now a flat key-value map
- All agent outputs merge into the shared state on completion
- Loaded on startup, saved after any agent completes
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
hippocampus: cursor navigation, transcript parsing, similarity
functions to pub(crate). counters::open() made private.
subconscious: all format_* prompts helpers to pub(super),
load_defs and keys_to_replay_items made private,
consolidate_full_with_progress made private.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Only Message, Role, MessageContent, ContentPart, ToolCall,
FunctionCall, Usage, ImageUrl are pub-exported from agent::api.
Internal types (ChatRequest, ChatCompletionChunk, ChunkChoice,
Delta, ReasoningConfig, ToolCallDelta, FunctionCallDelta) are
pub(crate) — invisible outside the crate.
All callers updated to import from agent::api:: instead of
agent::api::types::.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
scan_pid_files was removed as dead code but it was actually needed
by the hook path — the bug was that it was never wired in. Add
reap_agent_pids() directly to poc-hook.rs and call it on every
UserPromptSubmit. Kills timed-out agents (10min) and cleans up
pid files for dead processes.
Also remove dead subconscious/subconscious.rs (420 lines) — was
forked to claude/agent_cycles.rs and never removed.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
New src/agent/api/http.rs: ~240 lines, supports GET/POST, JSON/form
bodies, SSE streaming via chunk(), TLS via rustls. No tracing dep.
Removes reqwest from the main crate and telegram channel crate.
Cargo.lock drops ~900 lines of transitive dependencies.
tracing now only pulled in by tui-markdown.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
AutoAgent holds config + walked state. Backend is ephemeral per run:
- run(): standalone, global API client (oneshot CLI)
- run_forked(): forks conscious agent, resolves prompt templates
with current memory_keys and walked state
Mind creates AutoAgents once at startup, takes them out for spawned
tasks, puts them back on completion (preserving walked state).
Removes {{seen_previous}}, {{input:walked}}, {{memory_ratio}} from
subconscious agent prompts. Walked keys are now a Vec on AutoAgent,
resolved via {{walked}} from in-memory state.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Lightweight resolver handles {{seen_current}}, {{seen_previous}}, and
{{input:KEY}} using the session_id and output_dir directly instead of
env vars. Runs in trigger_subconscious before creating AutoAgent.
Removes {{memory_ratio}} from surface-observe prompt — redundant with
existing budget mechanisms.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
subconscious-surface-observe, subconscious-journal, subconscious-reflect
are Mind's forked agents. The original surface-observe, journal, reflect
remain for the standalone CLI/hook path.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Mind now holds SubconsciousAgent state (surface-observe, journal,
reflect) and triggers them after conscious turns complete. Each
agent forks from the conscious agent's context via AutoAgent,
runs as an async task, and routes output (surfaced memories,
reflections) back into the conscious agent.
Replaces the synchronous AgentCycleState that spawned child
processes and blocked start_turn.
Also adds .agent2 files — simplified prompts for the forked model
that strip {{conversation}} and {{agent-context}} (already in the
forked context).
TODO: resolve remaining placeholders (seen_current, input:walked,
memory_ratio) in the .agent2 prompts.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Add Log variant to ConversationEntry that serializes to the
conversation log but is filtered out on read-back and API calls.
AutoAgent writes debug/status info (turns, tokens, tool calls)
through the conversation log instead of a callback parameter.
Removes the log callback from run_one_agent, call_api_with_tools,
and all callers.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
- Budget now counts exact message tokens matching what assemble_api_messages
sends, not raw string content. Eliminates undercounting from formatting
overhead (journal headers, personality separators, working stack).
- Load journal before trimming so trim accounts for journal cost.
- Compact before every turn, not just after turn completion. Prevents
agent_cycle surfaced memories from pushing context over budget.
- Move agent_cycle orchestration from Agent::turn to Mind::start_turn —
surfaced memories and reflections now precede the user message.
- Move AgentCycleState from Agent to Mind — it's orchestration, not
per-agent state. memory_scoring_in_flight and memory_scores stay on
Agent where they belong.
- Tag DMN entries as ConversationEntry::Dmn — compaction evicts them
first since they're ephemeral. Compaction also prefers evicting
memories over conversation when memories exceed 50% of entry tokens.
- Kill /retry slash command.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Streaming text now goes directly to agent entries via append_streaming().
sync_from_agent diffs the growing entry each tick. The streaming entry
is popped when the response completes; build_response_message pushes
the final version.
All status feedback uses RAII ActivityGuards:
- push_activity() for long-running work (thinking, streaming, scoring)
- notify() for instant feedback (compacted, DMN state changes, commands)
- Guards auto-remove on Drop, appending "(complete)" and lingering 5s
- expire_activities() cleans up timed-out notifications on render tick
UiMessage enum reduced to a single Info variant with zero sends.
The channel infrastructure remains for now (Mind/Agent still take
UiSender in signatures) — mechanical cleanup for a follow-up.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Reasoning tokens: dropped for now, will land in context entries later.
Debug sends: converted to dbglog! macro (writes to debug.log).
Activity: now a field on Agent, set directly, read by UI via try_lock.
score_memories_incremental takes agent Arc for activity writes.
UiMessage down to 2 variants: TextDelta, Info.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Status bar reads directly from Agent and MindState on each render tick.
Activity is now a field on Agent — set by agent code directly, read by
UI via try_lock. DmnAnnotation, ContextInfoUpdate, AgentUpdate were
already dead (no senders).
UiMessage down to 4 variants: TextDelta, Reasoning, Debug, Info.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The std::sync::Mutex detour caught every place a MutexGuard lived
across an await point in Agent::turn — the compiler enforced Send
safety that tokio::sync::Mutex silently allows. With those fixed,
switch back to tokio::sync::Mutex (std::sync blocks tokio worker
threads and panics inside the runtime).
Input and command dispatch now live in InteractScreen (chat.rs):
- Enter pushes directly to SharedMindState.input (no app.submitted hop)
- sync_from_agent displays pending input with dimmed color
- Slash command table moved from event_loop.rs to chat.rs
- cmd_switch_model kept as pub fn for tool-initiated switches
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The agent lock is never held across await points — turns lock briefly,
do work, drop, then do async API calls. std::sync::Mutex works and
can be locked from sync contexts (screen tick inside terminal.draw).
Fixes: blocking_lock() panic when called inside tokio runtime.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
Memory scoring now uses the graph as source of truth:
- last_scored timestamp on each node (new capnp field @22)
- Nodes scored when older than scoring_interval_secs (default 1hr)
- Oldest-scored-first ordering
- Window: scoring_response_window assistant responses (default 100)
- First-quarter memories scored even without full window
- Per-response normalization (raw divergence / response count)
- Asymmetric weight update: alpha=0.5 up, alpha=0.1 down
(responds fast to importance, decays slowly — memories stay
surfaced even if only useful 1/4 of the time)
Graph writes disabled pending normalization calibration.
Also: configurable scoring_interval_secs and scoring_response_window.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
dispatch_shared was a legacy wrapper — replaced by dispatch() which
goes through the unified Tool registry. One dispatch path for all
callers (interactive agent, subconscious agents, MCP server).
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
ToolDef and FunctionDef are gone. Tool definitions are static strings
on the Tool struct. The API layer builds JSON from Tool::to_json().
- ChatRequest.tools is now Option<serde_json::Value>
- start_stream takes &[Tool] instead of Option<&[ToolDef]>
- openai::stream_events takes &serde_json::Value for tools
- memory_and_journal_tools() returns Vec<Tool> for subconscious agents
- Subconscious agents filter by t.name instead of t.function.name
No more runtime JSON construction for tool definitions.
No more ToolDef::new(). No more FunctionDef.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
ToolOutput was just { text: String } — replaced with plain String.
dispatch() and dispatch_shared() return String directly.
ActiveToolCall handle is (ToolCall, String).
Error results are prefixed with "Error: " by convention.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Move temperature from a per-call parameter to an Agent field,
add top_p and top_k. All three are sent to the API via a new
SamplingParams struct, displayed on the F5 thalamus screen.
Defaults: temperature=0.6, top_p=0.95, top_k=20 (Qwen3.5 defaults).
Also adds top_p and top_k to ChatRequest so they're sent in the
API payload. Previously only temperature was sent.
UI controls for adjusting these at runtime are not yet implemented.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The agent lock was held for the entire duration of turn() — including
API streaming and tool dispatch awaits. This blocked the UI thread
whenever it needed the lock (render tick, compaction check, etc.),
causing 20+ second freezes.
Fix: turn() takes Arc<Mutex<Agent>> and manages locking internally.
Lock is held briefly for prepare/process phases, released during all
I/O (streaming, tool awaits, sleep retries). Also:
- check_compaction: spawns task instead of awaiting on event loop
- start_memory_scoring: already spawned, no change needed
- dispatch_tool_call_unlocked: drops lock before tool handle await
- Subconscious screen: renders all agents from state dynamically
(no more hardcoded SUBCONSCIOUS_AGENTS list)
- Memory scoring shows n/m progress in snapshots
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
All process management now goes through active_tools:
- TUI reads metadata (name, elapsed time)
- Ctrl+K aborts handles (KillOnDrop sends SIGTERM)
- Running count from active_tools.len()
No more separate PID tracking, register/unregister, or
ProcessInfo. One data structure for everything.
Co-Developed-By: Kent Overstreet <kent.overstreet@linux.dev>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Separates the Claude-specific daemon (idle timer, tmux pane detection,
prompt injection, RPC server, session hooks) from the universal
infrastructure (channels, supervisor, notify, daemon protocol).
thalamus/ now contains only substrate-independent code: the channel
client/supervisor, notification system, daemon_capnp protocol, and
shared helpers (now(), home()).
claude/ contains: idle.rs, tmux.rs, context.rs, rpc.rs, config.rs,
hook.rs (moved from subconscious/), plus the daemon CLI and server
startup code from thalamus/mod.rs.
All re-exports preserved for backward compatibility.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Mechanical rename: src/agent/ -> src/user/, all crate::agent:: ->
crate::user:: references updated. Binary poc-agent renamed to
consciousness with CLI name and user-facing strings updated.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
/score snapshots the context and client, releases the agent lock,
runs scoring in background. Only one score task at a time
(scoring_in_flight flag). Results stored on Agent and shown on
the F10 context debug screen with importance scores per memory.
ApiClient derives Clone. ContextState derives Clone.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Delete subconscious/transcript.rs (94 lines), is_segment_mined,
mark_segment_mined — all orphaned by the extraction pipeline removal.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Delete enrich.rs (conversation extraction), select_conversation_fragments,
mark_observation_done, format_segment, and the {{conversations}} placeholder.
Transcript processing is handled by observe/journal agents now.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Session mining, stale session detection, is_file_open /proc scan,
segment extraction, and fact mining are all replaced by the
observe/journal agents. Remove the entire session-watcher thread,
find_stale_sessions(), is_file_open(), MIN_SESSION_BYTES, and
SESSION_STALE_SECS. -329 lines.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The install_hook() function is about hook infrastructure setup, not daemon
runtime. Move it to hook.rs where it belongs alongside the hook execution
logic.
- Move install_hook() from daemon.rs to hook.rs
- Update caller in daemon.rs to use crate::subconscious:🪝:install_hook()
- Update caller in cli/admin.rs to use crate::subconscious:🪝:install_hook()
This improves module boundaries: daemon.rs now only contains daemon runtime
and admin commands, while hook.rs contains all hook-related functionality.
prompts_dir now defaults to ~/.consciousness/prompts instead of
hardcoded repo path. Function pointer cast goes through *const ()
to silence the compiler warning.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
