JsonlBackwardIter: use memrchr3 for SIMD-accelerated scanning
Replaces byte-by-byte backward iteration with memrchr3('{', '}', '"')
which uses SIMD to jump between structurally significant bytes. Major
speedup on large transcripts (1.4GB+).
Also simplifies tail_messages to use a byte budget (200KB) instead
of token counting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
d7d631d77d
commit
6c41b50e04
3 changed files with 23 additions and 23 deletions
|
|
@ -16,6 +16,7 @@ clap = { version = "4", features = ["derive"] }
|
|||
libc = "0.2"
|
||||
faer = "0.24.0"
|
||||
rkyv = { version = "0.7", features = ["validation", "std"] }
|
||||
memchr = "2"
|
||||
memmap2 = "0.9"
|
||||
rayon = "1"
|
||||
peg = "0.8"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue