Previously NodeLeaf.timestamp and AstNode::Branch.timestamp accepted
null or missing via a deserialize_timestamp_or_epoch fallback — legacy
entries in conversation.jsonl from before Branch timestamps existed
(and from before chrono serialization was wired up) would load with
UNIX_EPOCH as a sentinel. Downstream, node_timestamp_ns() returned
Option<i64> and callers had to handle None as "old entry, skip."
That second filter was silently dropping every candidate in
score_finetune_candidates when scoring an older session — the F6
screen showed "0 above threshold" even when max_divergence was
orders of magnitude above the threshold, because every entry was
failing the None check, not the divergence check.
The fix, in three parts:
1. src/bin/fix-timestamps.rs — one-off migration tool that walks a
conversation.jsonl, linearly interpolates timestamps for entries
stuck at UNIX_EPOCH (using surrounding real timestamps as anchors),
propagates to child leaves with per-sibling ns offsets, and bumps
any collisions by 1 ns for uniqueness. Ran against the current
session's log: 11887 entries, 72289 ns bumps, all unique.
2. context.rs — drop default_timestamp and
deserialize_timestamp_or_epoch. NodeLeaf and Branch now require a
present non-null timestamp on deserialize. Tests flip from
"missing/null → UNIX_EPOCH" to "missing/null → Err."
3. subconscious/learn.rs — node_timestamp_ns now returns i64, not
Option<i64>. The matching caller in score_finetune_candidates
collapses from a Some/None match to a single trained-set check.
mind/log.rs's oldest_timestamp no longer filters UNIX_EPOCH.
Every line currently on disk has already been migrated. Going
forward, new AstNodes always carry real timestamps (Utc::now() at
construction time), so the strict schema is the invariant, not an
aspiration.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Two related changes to the learn subsystem:
1. AST node timestamps are now non-optional — both Leaf and Branch
variants carry a DateTime<Utc>. UNIX_EPOCH means "unset" (old entries
deserialized from on-disk conversation logs).
Training uses timestamps as unique keys for dedup, so we promote to
nanosecond precision: node_timestamp_ns(), TrainData.timestamp_ns,
FinetuneCandidate.timestamp_ns, mark_trained(ns).
2. build_token_ids() now also returns token-position ranges of assistant
messages. These are passed to vLLM's /score endpoint via the new
score_ranges field so only scored-position logprobs are returned —
cuts bandwidth/compute when scoring small windows.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Uses JsonlBackwardIter (SIMD memrchr3) to scan the conversation log
newest-first without reading/parsing the whole file. Stops as soon
as the conversation budget is full. Only the kept nodes get
retokenized and pushed into context.
18MB log → only tokenize the ~50 nodes that fit in the budget.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
