kent/consciousness
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Kill tiktoken — all token counting now uses Qwen 3.5 tokenizer

Browse source 
Remove tiktoken-rs dependency, CoreBPE field on Agent, and the
msg_token_count() function. All tokenization now goes through the
global HuggingFace tokenizer in agent/tokenizer.rs.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
This commit is contained in:
Kent Overstreet 2026-04-08 11:25:28 -04:00
parent 5e4067c04f
commit 67e3228c32
4 changed files with 1 additions and 78 deletions
Download patch file Download diff file Expand all files Collapse all files

1
Cargo.toml
View file

@ -59,7 +59,6 @@ futures = "0.3"
capnp = "0.25"
capnp-rpc = "0.25"
tiktoken-rs = "0.9.1"
tokenizers = "0.21"
skillratings = "0.28"