Replace rkyv/bincode caching with redb indices

Remove three-tier loading (rkyv snapshot, bincode cache, capnp replay)
in favor of direct capnp log replay + redb for indexed access.

- Remove all rkyv derives from types (Node, Relation, enums, etc.)
- Remove Snapshot struct, RKYV_MAGIC, CACHE_MAGIC constants
- Remove load_snapshot_mmap(), save(), save_snapshot()
- Remove MmapView, AnyView from view.rs (keep StoreView trait)
- Simplify Store::load() to just replay capnp logs
- Add db.rs with redb schema: nodes, uuid_to_key, visits, transcript_progress
- Simplify cmd_fsck to just check capnp integrity + graph health

capnp logs remain source of truth; redb indices will be rebuilt on demand.

Co-Authored-By: Proof of Concept <poc@bcachefs.org>
This commit is contained in:
Kent Overstreet 2026-04-13 18:30:58 -04:00
parent 1d88293ccf
commit 2caccf875d
8 changed files with 201 additions and 636 deletions

View file

@ -1,21 +1,16 @@
// Append-only Cap'n Proto storage + derived KV cache
// Append-only Cap'n Proto storage + redb indices
//
// Two log files are source of truth:
// capnp logs are the source of truth:
// nodes.capnp - ContentNode messages
// relations.capnp - Relation messages
//
// The Store struct is the derived cache: latest version per UUID,
// rebuilt from logs when stale. Three-tier load strategy:
// 1. rkyv mmap snapshot (snapshot.rkyv) — ~4ms deserialize
// 2. bincode cache (state.bin) — ~10ms
// 3. capnp log replay — ~40ms
// Staleness: log file sizes embedded in cache headers.
// redb provides indexed access; Store struct holds in-memory state.
//
// Module layout:
// types.rs — Node, Relation, enums, capnp macros, path helpers
// parse.rs — markdown → MemoryUnit parsing
// view.rs — zero-copy read-only access (StoreView, MmapView)
// persist.rs — load, save, replay, append, snapshot (all disk IO)
// view.rs — StoreView trait for read-only access
// persist.rs — load, replay, append (capnp IO)
// ops.rs — mutations (upsert, delete, decay, cap_degree, etc.)
// mod.rs — re-exports, key resolution, ingestion, rendering
@ -24,6 +19,7 @@ mod parse;
mod view;
mod persist;
mod ops;
pub mod db;
// Re-export everything callers need
pub use types::{
@ -34,7 +30,7 @@ pub use types::{
new_node, new_relation,
};
pub use parse::{MemoryUnit, parse_units};
pub use view::{StoreView, AnyView};
pub use view::StoreView;
pub use persist::fsck;
pub use ops::current_provenance;