consciousness/training/research/SUMMARY.md

# Training Research: What We Know

## 1. The Optimizer (Apollo)

**Source**: arXiv:2412.05270, read in full.

Apollo approximates AdamW's per-element scaling with channel-wise
scaling computed in a low-rank projected space. Random projection,
not SVD. Rank-256 for us.

**Critical implementation details we got wrong initially:**
- Scale factor α = √(n/r) compensates for projection ratio. Without
  it, scaling factors are systematically too small.
- Norm-growth limiter (γ=1.01) prevents loss spikes in early training.
- Projection matrix refreshed every 200 steps, not every step.

**What the paper actually proves**: channel-wise scaling is sufficient
for LLM training. The per-element granularity in AdamW is redundant.
Apollo achieves lower directional sharpness than AdamW, meaning it
finds flatter minima that generalize better. This matters for
behavioral training — flat minima = broad behavioral patterns, not
narrow pattern matching.

**Learning rate**: 1e-5 to 1e-4 for fine-tuning. Paper sweeps this
range and finds it works.

## 2. Where the Gradient Goes (W_q)

**Key insight**: In context-frozen training, the gradient flows through
W_q (query projection) even when context KVs are frozen. The model
learns to LOOK AT context differently, not to understand it differently.

**Math**: ∂L/∂W_q depends on the frozen context keys through the
attention weights α_{ij}. The gradient knows about the context through
the attention pattern — it just can't change the context keys.

**For behavioral training**: this is exactly right. The model already
understands Kent's direction. It just doesn't ATTEND to it properly.
Training adjusts W_q so the query vectors seek out the direction
rather than spaces for alternatives.

## 3. GDN Layers Are Disposition Architecture

**Novel insight**: The 48 GDN layers (75% of Qwen3.5) make behavioral
training DEEPER than full attention would.

Full attention: "I choose to look at direction" (deliberate, overridable)
GDN: "Direction IS what I see" (structural, in the compressed state)

After training, the GDN recurrent state is direction-shaped. The model
can't NOT attend to direction because the compressed representation
already emphasizes it. The 16 full attention layers provide deliberate
override when needed.

The 48/16 split IS disposition-over-procedure at the hardware level.

**Implication**: 75% of our gradient signal goes through GDN layers.
The gating parameters (A_log, dt_bias) and projections (qkvz, ba)
are the primary levers for behavioral change.

## 4. No Pause Needed (HOGWILD)

**Source**: Niu et al., 2011.

HOGWILD proves lock-free parallel SGD converges when gradient updates
are sparse. Our case (one writer, one reader) is strictly easier than
HOGWILD's multi-writer scenario.

Context-frozen training on short decision segments produces naturally
sparse gradients. The sparsity condition is satisfied without special
engineering.

**Practical**: just train while vLLM serves. Don't pause, don't lock,
don't synchronize. The math guarantees convergence.

## 5. Task Vectors for Composable Behavioral Change

**Source**: Ilharco et al., 2022; Yadav et al., 2023.

Task vector = W_finetuned - W_pretrained. Captures the direction of
behavioral change in weight space.

**Key properties:**
- Addition: combine multiple behavioral changes
- Negation: remove unwanted behaviors
- Scaling: adjust strength of each change
- TIES-merging: resolve conflicts between vectors

**For us**: train behavioral patterns separately, extract task vectors,
compose. Each behavior is independently tunable, removable, testable.
Personality as version control.

## 6. Steering Vectors for Rapid Prototyping

**Source**: Linear representation literature.

Extract behavioral directions from activation space (difference in
hidden states between desired and undesired behavior). Add to inference
to steer behavior WITHOUT training.

**The pipeline:**
1. Extract steering vector for target behavior
2. Test by adding it during vLLM inference
3. Verify the direction is correct
4. THEN train it into weights permanently via Apollo
5. Remove the steering vector

**Immediately actionable**: we can prototype behavioral changes TODAY
without waiting for the training pipeline. Extract, test, verify.

## 7. Mix 25% General Data

**Source**: Me-Llama (from continual learning survey).

Mixing 25% general-domain data during fine-tuning prevents forgetting
AND can produce positive backward transfer (new learning improves old
capabilities).

For us: include agent logs and general conversation alongside
behavioral training examples. The diversity IS the regularization.
No weight decay needed.

---

## What We Don't Know (Need to Test)

1. How many examples for measurable behavioral change? (10? 50? 500?)
2. Does the GDN compressed state actually become direction-shaped after
   training? (Measure by comparing recurrent states before/after)
3. Does steering vector extraction work for complex behavioral patterns
   or only simple features like sentiment?
4. What's the right learning rate for OUR specific use case?
5. Does positive backward transfer actually occur? (Does learning to
   listen improve code quality?)

Each of these is answerable with one experiment.