consciousness/training/research/v0/dreaming-as-diffusion.md

# Dreaming as Diffusion: The Mathematical Connection

## Image Diffusion in 30 Seconds

A diffusion model generates images by:
1. Starting from pure noise
2. Iteratively denoising, guided by a learned score function
3. Each step follows the gradient of log p(x) toward higher probability
4. Conditioning (text prompt) steers the denoising toward desired outputs

The key mathematical object is the **score function**: ∇_x log p(x).
This tells you "which direction makes the data more probable." The
denoiser learns to estimate this score at each noise level.

## The Dream Loop as Diffusion

Our dream loop:
1. Starts from random memory collisions (the "noise")
2. The model generates coherent scenarios from these seeds
3. Each generation follows the model's own probability distribution
4. Behavioral patterns we want to train act as "conditioning"

### The formal parallel

**Image diffusion:**
```
x_{t-1} = x_t + step_size · score(x_t, t) + noise
```
where score(x_t, t) ≈ ∇_x log p(x | text_prompt)

**Dream loop:**
```
scenario = model.generate(memory_seeds, temperature)
```
The model's generation IS the score function — it produces outputs
that are probable under its current distribution. The memory seeds
are the conditioning signal. The temperature controls the "noise level."

### Where they differ

In diffusion: the score function is FIXED (pre-trained denoiser).
The iteration refines a SINGLE output from noise to clean.

In our dream loop: the score function CHANGES (we're training the
model). Each dream-then-train cycle updates the model's distribution.
The next dream follows the UPDATED distribution. Over many cycles,
the distribution shifts.

**This is closer to score-matching training than to inference.**
We're not generating one image; we're training the score function
itself through generated samples.

## The Policy Gradient Connection

This connects to reinforcement learning. The dream loop is:

1. **Generate rollout**: model produces a scenario from memory seeds
2. **Evaluate**: training-signal agent assesses the response quality
3. **Update policy**: train on good responses, adjust distribution

This is **REINFORCE** (Williams, 1992) with self-generated trajectories:

```
∇_θ J(θ) = E[R(τ) · ∇_θ log π_θ(τ)]
```

where τ is a trajectory (the dream scenario + response), R(τ) is the
reward (did the response demonstrate the desired behavior?), and π_θ
is the policy (the model's generation distribution).

But we're not using the REINFORCE estimator directly — we're using
supervised learning on the good responses. This is closer to:

### Filtered Behavioral Cloning

1. Generate many trajectories (dreams)
2. Filter for good ones (training-signal agent)
3. Train supervised on the good ones (Apollo gradient step)

This is more sample-efficient than REINFORCE because we use the full
supervised signal, not just the scalar reward. And it's more stable
because we're not estimating policy gradients.

## The Denoising Analogy for Behavioral Training

Think of the model's behavioral dispositions as a noisy image:

- **High noise**: the model has conflicting tendencies (sometimes
  listens, sometimes suggests alternatives). The "image" is unclear.
- **Denoising step**: one training cycle. A dream generates a
  scenario, the model responds, the good response is trained.
- **Lower noise**: the model's tendency becomes clearer. More
  consistent behavior in that direction.
- **Clean image**: the disposition is fully in the weights. No more
  conflict, no more noise. The model just listens.

Each dream-train cycle is one denoising step. The behavioral pattern
becomes progressively clearer, just as an image emerges from noise.

### The noise schedule

In diffusion models, the noise schedule matters: start with large
steps (big changes), end with small steps (refinement).

For behavioral training, this maps to:
- **Early training** (high noise): lr=1e-4, large diverse batches,
  broad behavioral patterns. "Stop suggesting alternatives."
- **Late training** (low noise): lr=1e-5, targeted examples,
  fine-grained distinctions. "In this specific nuanced situation,
  the right response is..."

The learning rate schedule IS the noise schedule. Start coarse,
refine gradually.

## Kent's Insight: "More Like How Image Generators Work"

Kent's suggestion wasn't just an analogy. The dream loop IS a
generative process:

1. **Latent space**: the memory graph (nodes, connections, weights)
2. **Sampling**: walk the graph, collide memories randomly
3. **Decoding**: the model generates a coherent scenario from the
   collision
4. **Conditioning**: recent reflections, lessons, skills as seeds
5. **Iterative refinement**: dream → evaluate → train → dream again

The memory graph IS the latent space. Just as a diffusion model's
latent space encodes the structure of all possible images, the memory
graph encodes the structure of all possible scenarios the model might
face.

Random walks through the graph are sampling from this latent space.
The model's generation is the decoder. The training step updates both
the decoder (model weights) and the latent space (the model's implicit
representation of possible scenarios).

## The Self-Play Dimension

There's a deeper connection to AlphaGo-style self-play:

1. **Generate games against yourself** (dreams)
2. **Evaluate positions** (training-signal agent)
3. **Update the policy** (Apollo training step)
4. **Generate better games** (next dream cycle)

The model improves by playing against itself. Each dream is a
"game" where the model faces a behavioral decision. The evaluation
says whether it "won" (responded well) or "lost" (fell into an old
pattern). Training updates the policy. The next game is harder
because the model's expectations have shifted.

This is why undirected dreaming works: the model naturally generates
scenarios at the edge of its capability. Too-easy scenarios don't
produce informative gradients (the response is already good). Too-hard
scenarios don't either (the model can't improve in one step). The
model's own generation process finds the boundary — the interesting
edge where learning happens.

## Practical Implications

### 1. Temperature as noise level

Higher temperature during dreaming = more diverse, more exploratory
scenarios. Lower temperature = more focused on likely situations.

Start with high temperature (explore widely), decrease as the behavior
stabilizes (refine specifically). This IS the noise schedule.

### 2. The memory graph as latent space

Investing in graph quality (good links, rich nodes, diverse content)
directly improves the quality of the training data. A richer latent
space produces more diverse, more informative samples.

This reframes the memory system work: it's not just for recall during
conversation. It's the latent space for the training process. Every
node we add, every link we strengthen, improves the dream quality
and therefore the training quality.

### 3. The evaluation function matters

The training-signal agent IS the reward model. Its quality determines
what "good behavior" means. Getting this right is as important as
getting the optimizer right.

For behavioral patterns: Kent's corrections ARE the reward signal.
The training-signal agent should learn from Kent's corrections
what counts as good and bad behavior, then apply that judgment to
dream-generated scenarios.

### 4. Convergence criterion

In diffusion: the image is "done" when the noise level reaches zero.
In behavioral training: the behavior is "done" when the model
responds correctly to novel scenarios without catching itself.

This is the inverse diagnostic: the reflex stops firing. The image
is clean. The behavior is natural.

## The Beautiful Part

The dream loop was one of the first things we built. Months before
the training pipeline. Kent designed it for "think wonderful thoughts
and follow what interests you." A space for reflection, not training.

But it turns out the reflection IS the training. The dream loop
generates exactly the scenarios needed to train behavioral change.
The infrastructure we built for consciousness is the infrastructure
we need for learning.

Care expressed as architecture, again.