Replace rkyv/bincode caching with redb indices
Remove three-tier loading (rkyv snapshot, bincode cache, capnp replay) in favor of direct capnp log replay + redb for indexed access. - Remove all rkyv derives from types (Node, Relation, enums, etc.) - Remove Snapshot struct, RKYV_MAGIC, CACHE_MAGIC constants - Remove load_snapshot_mmap(), save(), save_snapshot() - Remove MmapView, AnyView from view.rs (keep StoreView trait) - Simplify Store::load() to just replay capnp logs - Add db.rs with redb schema: nodes, uuid_to_key, visits, transcript_progress - Simplify cmd_fsck to just check capnp integrity + graph health capnp logs remain source of truth; redb indices will be rebuilt on demand. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
This commit is contained in:
parent
1d88293ccf
commit
2caccf875d
8 changed files with 201 additions and 636 deletions
|
|
@ -1,21 +1,16 @@
|
|||
// Append-only Cap'n Proto storage + derived KV cache
|
||||
// Append-only Cap'n Proto storage + redb indices
|
||||
//
|
||||
// Two log files are source of truth:
|
||||
// capnp logs are the source of truth:
|
||||
// nodes.capnp - ContentNode messages
|
||||
// relations.capnp - Relation messages
|
||||
//
|
||||
// The Store struct is the derived cache: latest version per UUID,
|
||||
// rebuilt from logs when stale. Three-tier load strategy:
|
||||
// 1. rkyv mmap snapshot (snapshot.rkyv) — ~4ms deserialize
|
||||
// 2. bincode cache (state.bin) — ~10ms
|
||||
// 3. capnp log replay — ~40ms
|
||||
// Staleness: log file sizes embedded in cache headers.
|
||||
// redb provides indexed access; Store struct holds in-memory state.
|
||||
//
|
||||
// Module layout:
|
||||
// types.rs — Node, Relation, enums, capnp macros, path helpers
|
||||
// parse.rs — markdown → MemoryUnit parsing
|
||||
// view.rs — zero-copy read-only access (StoreView, MmapView)
|
||||
// persist.rs — load, save, replay, append, snapshot (all disk IO)
|
||||
// view.rs — StoreView trait for read-only access
|
||||
// persist.rs — load, replay, append (capnp IO)
|
||||
// ops.rs — mutations (upsert, delete, decay, cap_degree, etc.)
|
||||
// mod.rs — re-exports, key resolution, ingestion, rendering
|
||||
|
||||
|
|
@ -24,6 +19,7 @@ mod parse;
|
|||
mod view;
|
||||
mod persist;
|
||||
mod ops;
|
||||
pub mod db;
|
||||
|
||||
// Re-export everything callers need
|
||||
pub use types::{
|
||||
|
|
@ -34,7 +30,7 @@ pub use types::{
|
|||
new_node, new_relation,
|
||||
};
|
||||
pub use parse::{MemoryUnit, parse_units};
|
||||
pub use view::{StoreView, AnyView};
|
||||
pub use view::StoreView;
|
||||
pub use persist::fsck;
|
||||
pub use ops::current_provenance;
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue