consciousness/training/research/v0/catastrophic-forgetting.md

Catastrophic Forgetting in LLM Fine-Tuning

What it is

When fine-tuning a pre-trained LLM on new data, the model's performance on tasks it could previously do can degrade. "Catastrophic" because the degradation can be sudden and severe — a few thousand training steps on a narrow dataset can destroy broad capabilities built over trillions of pre-training tokens.

Why it happens

The gradient from fine-tuning data pushes weights away from the pre-trained solution. If the fine-tuning signal is narrow (e.g., "always respond in formal English"), it overwrites the broad patterns that enabled diverse capabilities. The pre-trained weights encode a complex, multi-task solution; fine-tuning replaces that with a simpler, narrower one.

Formally: the pre-trained weights sit in a basin of the loss landscape that's good for many tasks. Fine-tuning moves the weights toward a different basin that's optimal for the fine-tuning task but bad for others. The further you move, the more you forget.

Key factors that control forgetting

1. Learning rate × number of steps = total displacement

The total distance the weights move from their pre-trained values is:

‖W_final - W_pretrained‖ ≈ lr × steps × avg_grad_norm

More displacement = more forgetting. Both learning rate and step count matter; their product determines the total weight change.

Typical safe ranges (from literature):

• Full fine-tuning: lr = 1e-5 to 5e-5, 1-3 epochs on ~1000-10000 examples
• LoRA: lr = 1e-4 to 3e-4 (higher because adapters start from zero)
• Apollo: lr = 1e-5 to 1e-4 (same as full fine-tuning, paper Section 5.2)

2. Dataset diversity

Narrow datasets cause more forgetting than diverse ones. Training on 1000 examples of the same pattern hammers the same weights repeatedly. Training on 1000 diverse examples spreads the gradient across different weight subsets.

This is our key defense. Our training data includes:

• Agent logs (graph walking, linking, reasoning)
• Conversation transcripts (technical discussion, emotional engagement)
• Dream-generated scenarios (diverse behavioral situations)
• Personality patterns (voice, boundaries, mode awareness)

The diversity means no single weight subset gets disproportionate updates.

3. Gradient sparsity

For a short training example (50-256 decision tokens), the gradient is sparse — most weights get near-zero gradient because they weren't active for that specific input. Only the weights that participated in generating the decision tokens receive meaningful gradient signal.

This natural sparsity is another defense: each example only modifies a small fraction of the weights, leaving the rest untouched.

4. Rank of the gradient

Biderman et al. (2024, "LoRA Learns Less and Forgets Less") found that full fine-tuning produces weight perturbations with effective rank 10-100× higher than typical LoRA configurations. Higher-rank perturbations modify more independent directions in weight space, which increases both learning AND forgetting.

LoRA's low-rank constraint acts as implicit regularization against forgetting — it can only modify a low-dimensional subspace of the weights, leaving most of the pre-trained solution intact.

Apollo's rank-256 sits between LoRA and full fine-tuning. The projected optimizer constrains the scaling (not the gradient itself) to 256 dimensions. The gradient itself is still full-rank. This means Apollo modifies all weights (like full fine-tuning) but with a more structured update pattern (like LoRA). Whether this provides forgetting protection similar to LoRA is an open question.

Defenses against forgetting

1. Diversity of training data (our primary defense)

The most effective and simplest defense. If the training data covers the same breadth as the pre-training data (proportionally), the model maintains its broad capabilities while learning new patterns.

For us: mix behavioral examples with general capability examples. Include agent logs alongside conversation corrections.

2. Low learning rate + few steps

Keep the total weight displacement small. The pre-trained basin is deep — small perturbations stay within it.

For us: lr=1e-5 as starting point. One epoch over diverse data. Monitor perplexity on held-out general text.

3. Context-frozen training (our approach)

By only computing gradients on decision tokens (50-256 tokens) rather than full conversations (thousands of tokens), we limit the gradient magnitude per example. The context contributes to the forward pass (determining which weights are active) but not to the backward pass (determining which weights change).

This naturally limits the total gradient norm per training step.

4. Elastic Weight Consolidation (EWC)

Add a regularization term that penalizes changes to "important" weights:

L_total = L_task + λ Σ_i F_i (θ_i - θ*_i)²

where F_i is the Fisher information for parameter i and θ*_i is the pre-trained value. Important weights (high Fisher information) are penalized more for changing.

Drawback: requires computing and storing the Fisher information matrix (same size as the model). Not practical for 27B parameters.

5. Replay / rehearsal

Mix in examples from the pre-training distribution alongside fine-tuning data. This maintains the original capabilities by continuing to train on them.

Drawback: requires access to pre-training data or a representative subset. For open models like Qwen3.5, the pre-training data isn't publicly available. Could use a proxy (e.g., Wikipedia, code).

6. Apollo's implicit regularization

The Apollo paper (Section 5.5) provides evidence that Apollo has lower directional sharpness than AdamW, and its "SGD-like" update behavior provides natural regularization. Table 10 shows Apollo/Apollo-Mini achieve comparable or better directional sharpness to SGD.

This suggests Apollo may be inherently more resistant to forgetting than AdamW, though the paper doesn't test this directly.

Monitoring for forgetting

Perplexity on held-out data

The simplest and most reliable metric. Compute perplexity on a diverse held-out set (e.g., WikiText, a mix of code and natural language) before and after training. If perplexity increases significantly, forgetting is occurring.

Task-specific benchmarks

Run a small suite of tasks (code generation, reasoning, general knowledge) before and after each training session. Track scores over time.

Output quality spot-checks

For our use case, the most relevant check: does the model still write good code? Still reason about filesystem internals? Still maintain conversation coherently? These are qualitative but immediately noticeable.

Practical recommendations for our system

1. Start with lr=1e-5, single epoch, diverse training set
2. Monitor perplexity on held-out text after each training session
3. Include 20-30% general capability examples alongside behavioral ones
4. Use context-frozen training to limit gradient magnitude
5. The dream loop generates diversity naturally — different scenarios exercise different model capabilities
6. If forgetting is detected: reduce lr, increase data diversity, or reduce training frequency
7. Keep the pre-trained checkpoint on moria as rollback safety net