Add agent/tokenizer.rs with global Qwen 3.5 tokenizer that generates
actual token IDs including chat template wrapping. ContextEntry now
stores token_ids: Vec<u32> instead of tokens: usize — the count is
derived from the length.
ContextEntry::new() tokenizes automatically via the global tokenizer.
ContextSection::push_entry() takes a raw ConversationEntry and
tokenizes it. set_message() re-tokenizes without needing an external
tokenizer parameter.
Token IDs include the full chat template: <|im_start|>role\ncontent
<|im_end|>\n — so concatenating token_ids across entries produces a
ready-to-send prompt for vLLM's /v1/completions endpoint.
The old tiktoken CoreBPE is now unused on Agent (will be removed in
a followup). Token counts are now exact for Qwen 3.5 instead of the
~85-90% approximation from cl100k_base.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Thinking/reasoning content is now a first-class entry type:
- Serialized as {"thinking": "..."} in conversation log
- 0 tokens for budgeting (doesn't count against context window)
- Filtered from assemble_api_messages (not sent back to model)
- Displayed in UI with "thinking: ..." label and expandable content
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
SectionView gains a status field for extra info displayed after the
token count column. Memory nodes section shows "N scored, M unscored"
in the status column instead of burying it in the title.
Renderer uses fixed-width columns (40 name, 16 tokens, status) for
consistent alignment across sections.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Introduced SectionView {name, tokens, content, children} as a
UI-only tree node, separate from the data ContextSection. The widget
SectionTree renders SectionView with the old recursive expand/collapse
behavior — children for sub-sections, content for text expansion.
section_to_view() converts data sections to UI views, using
ConversationEntry::label() for names and content_text() for
expandable content.
read_context_views() builds the same tree the old context_state_summary
did: System, Identity, Journal, Memory nodes (scored/unscored counts,
expandable to show content), Conversation entries.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
New types — not yet wired to callers:
- ContextEntry: wraps ConversationEntry with cached token count and
timestamp
- ContextSection: named group of entries with cached token total.
Private entries/tokens, read via entries()/tokens().
Mutation via push(entry), set(index, entry), del(index).
- ContextState: system/identity/journal/conversation sections + working_stack
- ConversationEntry::System variant for system prompt entries
Token counting happens once at push time. Sections maintain their
totals incrementally via push/set/del. No more recomputing from
scratch on every budget check.
Does not compile — callers need updating.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
SectionTree:
- 'e': expand all nodes
- 'c': collapse all nodes
- Home/End already wired from previous commit
Key legend shown at bottom border of each focused pane:
- Tree panes: nav, expand/collapse, expand/collapse all, paging
- Agent list: select, tab
- History: scroll, paging
Legend only appears on the focused pane to avoid clutter.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
SectionTree.handle_nav() now takes viewport height:
- PgUp/PgDn move both cursor and viewport by one page, keeping the
cursor at the same screen position
- Home/End jump to first/last item
- scroll_to_selected() uses actual viewport height instead of
hardcoded 30
Added render_scrollable() in widgets.rs: renders a Paragraph with a
vertical Scrollbar when content exceeds the viewport. Used by the
conscious and subconscious screens.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
New layout for F3 screen:
- Top-left: agent list using ratatui List widget with ListState
- Middle-left: expandable agent state (persistent across runs)
- Bottom-left: memory store activity by provenance, walked keys
- Right: context tree from fork point, reusing SectionTree
Tab/Shift-Tab cycles focus clockwise between panes; focused pane
gets white border. Each pane handles its own input when focused.
Extracted user/widgets.rs:
- SectionTree (moved from mod.rs): expand/collapse tree for ContextSection
- pane_block_focused(): standard bordered block with focus indicator
- format_age()/format_ts_age(): shared duration formatting
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
