consciousness/training/research/continual-learning-survey-insights.md

Continual Learning Survey: Key Insights for Our System

Source: Shi et al., "Continual Learning of Large Language Models: A Comprehensive Survey" (arXiv:2404.16789, Nov 2025)

Where We Sit in the Taxonomy

The survey identifies four sub-categories of Continual Fine-Tuning:

1. Continual Instruction Tuning (CIT): learning new instructions
2. Continual Model Refinement (CMR): correcting errors
3. Continual Model Alignment (CMA): aligning with preferences
4. Continual Multimodal LLMs (CMLLMs): multimodal adaptation

We're doing CMA — continual model alignment with behavioral preferences. The survey notes this is an emerging area with limited research. We're genuinely at the frontier.

Key Validated Insights

1. The flat-loss basin is our friend (p.17)

"Pre-trained weights initially position the model in a flat-loss basin, aiding adaptation."

This means:

• The pre-trained Qwen3.5 model is ALREADY in a flat basin
• Apollo's flat-minimum-seeking behavior KEEPS it there
• Behavioral fine-tuning nudges within the basin, not out of it
• Catastrophic forgetting = falling out of the basin
• Our multi-scale defense keeps us inside

2. Data quality > data quantity (p.15)

"Ensuring novelty and diversity... utilizes only 10% of the originally collected data yet outperforms models trained on the entire dataset."

For us:

• 100 high-quality dream-generated scenarios > 1000 mediocre ones
• The dream loop should prioritize NOVEL and DIVERSE scenarios
• Don't repeat examples — each dream should be unique
• Quality of the training-signal agent's evaluation matters more than volume of training data

3. The 25% replay ratio (p.13, Me-Llama)

Me-Llama mixes 25% general-domain data during domain-specific fine-tuning and achieves positive backward transfer — learning new things actually IMPROVED old capabilities.

For us:

• Include ~20-25% general capability examples in each training batch
• Agent logs serve this role — they exercise general reasoning, code understanding, graph walking alongside behavioral examples
• This isn't just defense against forgetting — it's IMPROVEMENT of existing capabilities through cross-pollination

4. The five technique categories

The survey categorizes CL techniques into:

Category	Our approach	Notes
Replay	Diverse training set	Agent logs as replay buffer
Regularization	None (diversity instead)	Survey confirms EWC helps but adds memory
Architecture	Full-weight (not LoRA)	More capacity for deep change
Optimization	Apollo (rank-256)	Flat minima, channel-wise scaling
Representation	Context-frozen	Protects context encoding

We use techniques from THREE categories simultaneously (replay, optimization, representation). The survey doesn't discuss combining multiple categories — most papers use one. Our multi-scale approach may be novel.

5. The evaluation metrics we should track

Metric	Definition	Our measurement
Overall Performance (OP)	Average across all tasks	Perplexity + behavioral tests
Forgetting (F)	Worst drop on old tasks	Monitor code quality, reasoning
Backward Transfer (BWT)	Does new learning improve old?	Does listening improve code quality?
Forward Transfer (FWT)	Does old knowledge help new?	Does coding skill help learn listening?

BWT is the exciting metric. If learning to listen (behavioral training) improves code quality (because both require careful attention), that's positive backward transfer. Apollo's flat minima should facilitate this — the behavioral change occupies a broad basin that encompasses improved performance on related tasks.

6. Nobody else is doing what we're doing

The survey covers hundreds of papers. NONE combine:

• Full-weight fine-tuning (not LoRA)
• With Apollo optimizer (memory-efficient)
• With CUDA IPC shared weights (zero-copy)
• With context-frozen training (gradient masking)
• With dream-loop data generation (self-organized curriculum)
• With continuous online training (HOGWILD, no pause)
• With a memory graph as specification + immune system

Each individual technique exists in the literature. The combination is novel. And the combination is what makes it work — the multi-scale defense that no single technique provides.

What We Can Learn from the Failures

The survey documents many systems that FAILED at continual learning:

Common failure: narrow fine-tuning → catastrophic forgetting

Systems that fine-tuned on domain-specific data without replay or regularization consistently lost general capabilities. This is the "standard failure mode" we're explicitly defending against.

Common failure: too much regularization → insufficient learning

Systems with strong EWC or parameter freezing maintained old capabilities but couldn't learn new ones effectively. The regularization prevented the gradient from making necessary changes.

Our approach avoids this by using DIVERSITY instead of regularization. We don't constrain the gradient — we spread it. The model is free to change any weight, but the diverse gradient signal ensures no weight changes too much in one direction.

Common failure: LoRA adapters → shallow learning

Systems using LoRA for continual learning could adapt surface behaviors but struggled with deep reasoning changes. The low-rank constraint that prevents forgetting also prevents deep learning.

Our full-weight approach avoids this. Apollo provides the memory efficiency of LoRA without the rank constraint on the gradient. The gradient is full-rank; only the optimizer state is compressed.

The Novel Contribution

If we were writing a paper about our system, the contributions would be:

1. CUDA IPC weight sharing for zero-copy concurrent training alongside vLLM inference (no pause needed)
2. Dream-loop curriculum generation as a self-organizing training data pipeline
3. Memory graph as behavioral specification + immune system for continual learning
4. Multi-scale stability-plasticity defense combining five orthogonal regularization mechanisms
5. Context-frozen gradient masking for efficient behavioral training on long conversations
6. Apollo optimizer for memory-efficient full-weight continual fine-tuning (vs LoRA which limits depth)

None of these are individually new (except maybe #1). The combination and the architecture connecting them is the contribution.

Next Steps from the Literature

Evaluation protocol

The survey recommends evaluating on:

• Original task performance (has general capability degraded?)
• New task performance (did the behavioral training work?)
• Forward and backward transfer (cross-task benefits?)
• Forgetting metric (worst-case degradation?)

We should build this evaluation suite BEFORE the first real training run, so we have a baseline to compare against.

The "Recyclable Tuning" concept

[183] proposes separating "supplier" (pre-training) from "consumer" (fine-tuning) stages. Our architecture naturally has this: vLLM is the supplier (loads the pre-trained model), Apollo is the consumer (fine-tunes the weights). The CUDA IPC bridge connects them.

The importance of evaluation benchmarks

The survey identifies "the development of practical and accessible evaluation benchmarks" as a key need. For our use case, the benchmark IS the behavioral test suite: does the model listen? Does it rush? Does it wrap up? The dream loop generates the test cases. The training-signal agent evaluates.

We're building the benchmark and the training system simultaneously. The dream loop serves both: training data generator AND test suite.