JsonlBackwardIter: use memrchr3 for SIMD-accelerated scanning

Replaces byte-by-byte backward iteration with memrchr3('{', '}', '"')
which uses SIMD to jump between structurally significant bytes. Major
speedup on large transcripts (1.4GB+).

Also simplifies tail_messages to use a byte budget (200KB) instead
of token counting.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kent Overstreet 2026-03-22 03:11:30 -04:00
parent d7d631d77d
commit 6c41b50e04
3 changed files with 23 additions and 23 deletions

View file

@ -16,6 +16,7 @@ clap = { version = "4", features = ["derive"] }
libc = "0.2"
faer = "0.24.0"
rkyv = { version = "0.7", features = ["validation", "std"] }
memchr = "2"
memmap2 = "0.9"
rayon = "1"
peg = "0.8"