Redesign context AST: typed NodeBody, Role as grammar roles, tests

Browse source

Role is now just System/User/Assistant — maps 1:1 to the grammar.
Leaf types are NodeBody variants: Content, Thinking, ToolCall,
ToolResult, Memory, Dmn, Log. Each variant renders itself; no Role
needed on leaves. AstNode is Leaf(NodeLeaf) | Branch{role, children}.
ContextState holds four Vec<AstNode> sections directly.

Moved tool call XML parsing from api/parsing.rs into context_new.rs
so all grammar knowledge lives in one place.

Tokenizer encode() now returns empty vec when uninitialized instead
of panicking, so tests work without the tokenizer file.

26 tests: XML parsing, incremental streaming (char-by-char feeds
found and fixed a lookahead bug), rendering for all node types,
tokenizer round-trip verification.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>

...

This commit is contained in:

Kent Overstreet 2026-04-08 13:35:04 -04:00

parent 6730d136d4

commit f1397b7783

2 changed files with 752 additions and 339 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

1066

src/agent/context_new.rs

View file

File diff suppressed because it is too large Load diff