kent/consciousness
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

apollo-checkpoint: efficient diff-based GPU weight checkpointing

Browse source 
Rust tool that mmaps previous checkpoint, diffs against live GPU weights
(via CUDA IPC handles), and only writes changed blocks. For small
behavioral training steps, turns 54GB write into ~500MB.

Also includes vllm_export_hook.py with direct source patch approach —
exports IPC handles from vLLM's worker subprocess after model load.

Run every 10 minutes via cron to protect against vLLM crashes.
Daily rsync to moria for long-term storage.
This commit is contained in:
ProofOfConcept 2026-03-30 22:53:17 -04:00
parent 5f41898bb8
commit c1245ab139
3 changed files with 305 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

13
training/checkpoint/Cargo.toml Normal file
View file

@ -0,0 +1,13 @@
[package]
name = "apollo-checkpoint"
version = "0.1.0"
edition = "2024"
[dependencies]
memmap2 = "0.9"
safetensors = "0.5"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
anyhow = "1"
clap = { version = "4", features = ["derive"] }
chrono = "0.4"