Add agent/tokenizer.rs with global Qwen 3.5 tokenizer that generates
actual token IDs including chat template wrapping. ContextEntry now
stores token_ids: Vec<u32> instead of tokens: usize — the count is
derived from the length.
ContextEntry::new() tokenizes automatically via the global tokenizer.
ContextSection::push_entry() takes a raw ConversationEntry and
tokenizes it. set_message() re-tokenizes without needing an external
tokenizer parameter.
Token IDs include the full chat template: <|im_start|>role\ncontent
<|im_end|>\n — so concatenating token_ids across entries produces a
ready-to-send prompt for vLLM's /v1/completions endpoint.
The old tiktoken CoreBPE is now unused on Agent (will be removed in
a followup). Token counts are now exact for Qwen 3.5 instead of the
~85-90% approximation from cl100k_base.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Individual memory nodes now show their score in the status column
after the token count, not embedded in the name. Unscored memories
have an empty status column.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
SectionView gains a status field for extra info displayed after the
token count column. Memory nodes section shows "N scored, M unscored"
in the status column instead of burying it in the title.
Renderer uses fixed-width columns (40 name, 16 tokens, status) for
consistent alignment across sections.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Introduced SectionView {name, tokens, content, children} as a
UI-only tree node, separate from the data ContextSection. The widget
SectionTree renders SectionView with the old recursive expand/collapse
behavior — children for sub-sections, content for text expansion.
section_to_view() converts data sections to UI views, using
ConversationEntry::label() for names and content_text() for
expandable content.
read_context_views() builds the same tree the old context_state_summary
did: System, Identity, Journal, Memory nodes (scored/unscored counts,
expandable to show content), Conversation entries.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Build a synthetic "Memory nodes (N scored, M unscored)" section in
the context screen by extracting Memory entries from the conversation
section. Each node shows its key and score. Inserted before the
conversation section so scores are visible at a glance.
This makes it easy to see whether scoring is keeping up — if unscored
count is high relative to scored, scoring needs to run more often.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
New types — not yet wired to callers:
- ContextEntry: wraps ConversationEntry with cached token count and
timestamp
- ContextSection: named group of entries with cached token total.
Private entries/tokens, read via entries()/tokens().
Mutation via push(entry), set(index, entry), del(index).
- ContextState: system/identity/journal/conversation sections + working_stack
- ConversationEntry::System variant for system prompt entries
Token counting happens once at push time. Sections maintain their
totals incrementally via push/set/del. No more recomputing from
scratch on every budget check.
Does not compile — callers need updating.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
SectionTree:
- 'e': expand all nodes
- 'c': collapse all nodes
- Home/End already wired from previous commit
Key legend shown at bottom border of each focused pane:
- Tree panes: nav, expand/collapse, expand/collapse all, paging
- Agent list: select, tab
- History: scroll, paging
Legend only appears on the focused pane to avoid clutter.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
SectionTree.handle_nav() now takes viewport height:
- PgUp/PgDn move both cursor and viewport by one page, keeping the
cursor at the same screen position
- Home/End jump to first/last item
- scroll_to_selected() uses actual viewport height instead of
hardcoded 30
Added render_scrollable() in widgets.rs: renders a Paragraph with a
vertical Scrollbar when content exceeds the viewport. Used by the
conscious and subconscious screens.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
New layout for F3 screen:
- Top-left: agent list using ratatui List widget with ListState
- Middle-left: expandable agent state (persistent across runs)
- Bottom-left: memory store activity by provenance, walked keys
- Right: context tree from fork point, reusing SectionTree
Tab/Shift-Tab cycles focus clockwise between panes; focused pane
gets white border. Each pane handles its own input when focused.
Extracted user/widgets.rs:
- SectionTree (moved from mod.rs): expand/collapse tree for ContextSection
- pane_block_focused(): standard bordered block with focus indicator
- format_age()/format_ts_age(): shared duration formatting
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
ConversationEntry::Memory gains score: Option<f64>. The scorer
writes scores directly onto entries when results arrive. Removes
Agent.memory_scores Vec and the memory_scores parameter from
context_state_summary().
Scores are serialized to/from the conversation log as memory_score.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
F1 and F2 screens now call agent.context_state_summary() directly
via try_lock/lock instead of reading from a shared RwLock cache.
Removes SharedContextState, publish_context_state(), and
publish_context_state_with_scores().
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
InteractScreen in chat.rs owns conversation/autonomous/tools panes,
textarea, input history, scroll state. App is now just shared state
(status, sampling params, agent_state, channel_status, idle_info).
Event loop holds InteractScreen separately for UiMessage routing.
Overlay screens (F2-F5) in screens vec. F-key switching preserves
state across screen changes.
handle_ui_message moved from App to InteractScreen.
handle_key split: global keys on App, screen keys in tick().
draw dispatch eliminated — each screen draws itself.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
Convert F2-F5 screens to ScreenView trait with tick() method.
Each screen owns its view state (scroll, selection, expanded).
State persists across screen switches.
- ThalamusScreen: owns sampling_selected, scroll
- ConsciousScreen: owns scroll, selected, expanded
- SubconsciousScreen: owns selected, log_view, scroll
- UnconsciousScreen: owns scroll
Removed from App: Screen enum, debug_scroll, debug_selected,
debug_expanded, agent_selected, agent_log_view, sampling_selected,
set_screen(), per-screen key handling, draw dispatch.
App now only draws the interact (F1) screen. Overlay screens are
drawn by the event loop via ScreenView::tick. F-key routing and
screen instantiation to be wired in event_loop next.
InteractScreen (state-driven, reading from agent entries) is the
next step — will eliminate the input display race condition.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
