WIP: Context AST design — AstNode with Leaf{text,token_ids}/Branch
New context_new.rs with the AST-based context window design: - AstNode: role + NodeBody (Leaf with text+token_ids, or Branch with children) - Tokens only on leaves, branches walk children - render() produces UTF-8, tokenize produces token IDs, same path - ResponseParser state machine for streaming assistant responses - Role enum covers all node types including sections Still needs: fix remaining pattern match issues, add ContextState wrapper, wire into mod.rs, replace old context.rs. Does not compile yet — this is a design checkpoint. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
This commit is contained in:
parent
64157d8fd7
commit
29dc339f54
2 changed files with 524 additions and 0 deletions
|
|
@ -15,6 +15,7 @@
|
|||
|
||||
pub mod api;
|
||||
pub mod context;
|
||||
pub mod context_new;
|
||||
pub mod oneshot;
|
||||
pub mod tokenizer;
|
||||
pub mod tools;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue