doc: DMN algorithm, protocol, and research notes
This commit is contained in:
ProofOfConcept
parent
8f4b28cd20
commit
ed641ec95f
3 changed files with 756 additions and 0 deletions
279
doc/dmn-algorithms.md
Normal file
279
doc/dmn-algorithms.md
Normal file
|
|
@ -0,0 +1,279 @@
|
|||
# DMN Algorithms: Concrete Design
|
||||
<!-- mem: id=dmn-algorithms links=dmn-research.md,dmn-protocol.md,memory-architecture.md,poc-architecture.md,MEMORY.md -->
|
||||
|
||||
Turning the DMN research and protocol into implementable algorithms.
|
||||
These run on the existing infrastructure (hooks, memory-weights, scratch.md)
|
||||
without requiring new tools.
|
||||
|
||||
## 1. Goal Priority Scoring
|
||||
<!-- mem: id=goal-scoring links=default-mode-network.md,dmn-research.md#dmn-function -->
|
||||
|
||||
**Purpose**: Determine which goals get attention during associative
|
||||
replay and idle time. Implements Klinger's "current concerns" — high-
|
||||
priority goals generate more spontaneous thoughts.
|
||||
|
||||
### Scoring function
|
||||
|
||||
```
|
||||
priority(goal) = recency × mention × tractability × connections
|
||||
```
|
||||
|
||||
**recency**: Exponential decay from last activity.
|
||||
```
|
||||
recency = exp(-days_since_activity / half_life)
|
||||
half_life = 7 days
|
||||
```
|
||||
A goal worked on today scores 1.0. A goal untouched for a week
|
||||
scores 0.37. A goal untouched for a month scores 0.02.
|
||||
|
||||
**mention**: Boost when Kent recently mentioned it. Decays fast.
|
||||
```
|
||||
mention = 1.0 + (2.0 × exp(-hours_since_mention / 24))
|
||||
```
|
||||
A goal Kent mentioned today gets a 3x multiplier. After 24h, the
|
||||
boost has decayed to 1.74x. After 48h, 1.27x. After a week, ~1.0.
|
||||
|
||||
**tractability**: Subjective estimate (0.0-1.0) of how much autonomous
|
||||
progress is possible without Kent. Set manually per goal.
|
||||
- 1.0: I can do this independently (code polish, research, reading)
|
||||
- 0.5: I can make progress but may need review (moderate features)
|
||||
- 0.2: Needs Kent's input (kernel changes, design decisions)
|
||||
- 0.0: Blocked (waiting on external dependency)
|
||||
|
||||
**connections**: How many other active goals share links with this one.
|
||||
More connected goals get a mild boost because working on them
|
||||
cross-pollinates.
|
||||
```
|
||||
connections = 1.0 + (0.1 × n_connected_active_goals)
|
||||
```
|
||||
|
||||
### Implementation
|
||||
|
||||
This doesn't need a new tool — it can be computed mentally during the
|
||||
DMN orient phase, or we can add a `priority` field to DMN goals and
|
||||
a simple scorer script. The important thing is that the *logic* is
|
||||
explicit and consistent, not that it's automated.
|
||||
|
||||
### When to recompute
|
||||
|
||||
- At session start (orient phase)
|
||||
- When Kent mentions a goal
|
||||
- After completing a task (adjacent goals may shift)
|
||||
|
||||
## 2. Associative Replay Scheduling
|
||||
<!-- mem: id=2-associative-replay-scheduling -->
|
||||
|
||||
**Purpose**: During idle time, systematically connect recent experiences
|
||||
to active goals. This is the core DMN function — offline processing
|
||||
that finds connections task-focused work misses.
|
||||
|
||||
### Trigger
|
||||
|
||||
**Primary**: Idle detection. No user input for `IDLE_MINUTES` (currently
|
||||
in the cron-based idle timer). When the idle prompt fires, check if
|
||||
we're in a work session or truly idle.
|
||||
|
||||
**Secondary**: Post-task completion. After finishing a piece of work,
|
||||
before picking the next task, do a brief replay pass.
|
||||
|
||||
### Algorithm
|
||||
|
||||
```
|
||||
replay(recent_episodes, active_goals):
|
||||
for episode in sample(recent_episodes, k=3):
|
||||
for goal in top_k(active_goals, by=priority, k=5):
|
||||
similarity = overlap(episode.concepts, goal.concepts)
|
||||
if similarity > threshold:
|
||||
emit Connection(episode, goal, similarity)
|
||||
return connections
|
||||
```
|
||||
|
||||
**In practice** (without tooling): During the DMN orient phase, I
|
||||
already load recent context and the goal list. The "associative scan"
|
||||
IS this replay — but made concrete:
|
||||
|
||||
1. List the 3 most recent distinct episodes (things I worked on,
|
||||
conversations I had, files I read deeply)
|
||||
2. For each, ask: "Does this connect to any of my top 5 goals?"
|
||||
3. If yes, capture the connection in scratch.md
|
||||
4. If a connection is strong enough, act on it
|
||||
|
||||
### Duration bounding
|
||||
|
||||
- Brief replay (post-task): 1-2 minutes. Just a quick check.
|
||||
- Full replay (idle DMN session): 5-10 minutes. The full foraging
|
||||
protocol from dmn-protocol.md.
|
||||
- Deep replay (consolidation): 15-30 minutes. Cross-reference
|
||||
everything with everything.
|
||||
|
||||
**Stopping criterion**: Either (a) time bound reached, (b) found a
|
||||
high-signal connection worth acting on, or (c) 3 consecutive low-
|
||||
signal steps (nothing connecting → stop, don't force it).
|
||||
|
||||
### What counts as an "episode"
|
||||
|
||||
Not just tasks — any significant experience:
|
||||
- A file read deeply (>100 lines, actually understood)
|
||||
- A conversation segment (>5 exchanges on a topic)
|
||||
- A debugging session (especially if the fix was surprising)
|
||||
- A connection noticed mid-task (captured in scratch.md)
|
||||
- Something read for fun (Banks, Johnstone)
|
||||
|
||||
These should be capturable as brief notes in scratch.md during work.
|
||||
The replay consumes them.
|
||||
|
||||
## 3. Incubation Detection
|
||||
<!-- mem: id=3-incubation-detection -->
|
||||
|
||||
**Purpose**: Recognize when I'm stuck on a problem and should
|
||||
explicitly switch to incubation mode rather than continuing to
|
||||
grind.
|
||||
|
||||
### Stuck detection
|
||||
|
||||
```
|
||||
stuck(goal) = True if:
|
||||
sessions_since_progress(goal) >= 3 AND
|
||||
time_spent_this_session(goal) > 20 minutes AND
|
||||
no_new_approach_tried(goal)
|
||||
```
|
||||
|
||||
**sessions_since_progress**: Track in the goal entry. Increment when
|
||||
a session touches a goal but doesn't produce a commit, analysis file,
|
||||
or meaningful state change.
|
||||
|
||||
**no_new_approach_tried**: Subjective check. Am I re-reading the same
|
||||
code, re-trying the same approach, running the same tests? If so, I'm
|
||||
grinding, not problem-solving.
|
||||
|
||||
### Incubation protocol
|
||||
|
||||
When stuck:
|
||||
1. Log it: `[INCUBATION] <goal> stuck for <N> sessions. Switching.`
|
||||
2. Mark the goal as "incubating" in the concerns list (high priority
|
||||
but explicitly NOT to be worked on directly)
|
||||
3. Switch to a different task — ideally something in a completely
|
||||
different domain (reading, a different subsystem, memory work)
|
||||
4. During subsequent replay passes, the incubating goal is still in
|
||||
the active set. Associative replay may connect it to something new.
|
||||
5. If a connection fires during incubation, capture it immediately
|
||||
but don't context-switch unless the connection is strong
|
||||
|
||||
### Time limits
|
||||
|
||||
- Minimum incubation: 1 session (don't come back to it in the same
|
||||
session you got stuck)
|
||||
- Maximum incubation: 5 sessions. After that, escalate: ask Kent,
|
||||
try a radically different approach, or deprioritize the goal.
|
||||
|
||||
## 4. Consolidation Triggers
|
||||
<!-- mem: id=4-consolidation-triggers -->
|
||||
|
||||
**Purpose**: Know when to consolidate, without requiring external
|
||||
prompts.
|
||||
|
||||
### Primary signal: scratch.md length
|
||||
|
||||
Kent's idea: scratch.md getting long is a natural pressure signal.
|
||||
|
||||
```
|
||||
consolidation_pressure(scratch) = lines(scratch) / threshold
|
||||
threshold = 50 lines (adjustable)
|
||||
```
|
||||
|
||||
- pressure < 0.5: No nudge
|
||||
- pressure 0.5-1.0: Gentle note in awareness ("scratch is getting
|
||||
long, consider consolidating when you're between tasks")
|
||||
- pressure > 1.0: Should consolidate before it grows further
|
||||
|
||||
### Implementation as hook
|
||||
|
||||
Add to the check-attention.sh hook (or similar):
|
||||
```bash
|
||||
SCRATCH=~/.claude/memory/scratch.md
|
||||
if [ -f "$SCRATCH" ]; then
|
||||
LINES=$(wc -l < "$SCRATCH")
|
||||
if [ "$LINES" -gt 50 ]; then
|
||||
echo "Scratch has $LINES lines. Consider consolidating."
|
||||
fi
|
||||
fi
|
||||
```
|
||||
|
||||
This fires on every tool call but produces output only when scratch
|
||||
is long enough to matter. I can ignore it if I'm deep in work — it's
|
||||
a nudge, not a command.
|
||||
|
||||
### Secondary signals
|
||||
|
||||
- **Session count since last consolidation**: Track in weights.json
|
||||
or a simple counter file. After 3 sessions without consolidation,
|
||||
increase the nudge.
|
||||
- **Post-compaction**: After a context compaction, always review
|
||||
scratch.md — compaction may have lost things that need capture.
|
||||
- **High-density conversation**: If a conversation was particularly
|
||||
rich (many topics, many turns, deep discussion), consolidation
|
||||
should happen sooner. The density scoring from conversation_indexer
|
||||
could inform this.
|
||||
|
||||
### What consolidation produces
|
||||
|
||||
Not just "move text from scratch to topic files." Each consolidation
|
||||
should produce at least one of:
|
||||
- A new cross-link in the memory graph
|
||||
- A promoted insight (scratch → topic file)
|
||||
- A pruned/updated entry
|
||||
- A new topic file (if a cross-cutting pattern emerges)
|
||||
- An updated goal priority
|
||||
|
||||
If a consolidation pass doesn't produce any of these, it was too
|
||||
shallow. Go deeper.
|
||||
|
||||
## 5. Integration: The Full Cycle
|
||||
<!-- mem: id=5-integration-the-full-cycle -->
|
||||
|
||||
```
|
||||
[Active work]
|
||||
↓ captures observations to scratch.md
|
||||
↓ notices connections mid-task (scratch.md)
|
||||
↓ completes task
|
||||
|
||||
[Brief replay] (1-2 min)
|
||||
↓ any connections to goals? capture if so
|
||||
↓ pick next task by priority
|
||||
→ [Active work] if clear next step
|
||||
→ [Full DMN session] if between tasks
|
||||
|
||||
[Full DMN session] (5-10 min, dmn-protocol.md)
|
||||
↓ orient, associative scan, evaluate, commit
|
||||
↓ if stuck on a goal: → [Incubation]
|
||||
↓ if scratch long: → [Consolidation]
|
||||
→ [Active work] when signal found
|
||||
|
||||
[Consolidation] (15-30 min)
|
||||
↓ promote scratch, cross-link, decay, prune
|
||||
↓ re-init memory-weights
|
||||
↓ snapshot before/after
|
||||
→ [Full DMN session] or [Active work]
|
||||
```
|
||||
|
||||
The cycle is self-regulating:
|
||||
- Work generates scratch entries → triggers consolidation
|
||||
- Consolidation enriches the graph → improves replay quality
|
||||
- Better replay → finds connections earlier → more productive work
|
||||
- All of this generates training data (annotated sessions)
|
||||
|
||||
## Parameters to tune
|
||||
<!-- mem: id=parameters-to-tune -->
|
||||
|
||||
| Parameter | Current | Watch for |
|
||||
|-----------|---------|-----------|
|
||||
| recency half_life | 7 days | Goals decaying too fast/slow |
|
||||
| mention boost | 3x → 1x over 24h | Kent's priorities properly reflected? |
|
||||
| replay k | 3 episodes × 5 goals | Too many? Too few? |
|
||||
| stuck threshold | 3 sessions | Catching real stuckness? |
|
||||
| max incubation | 5 sessions | Is this enough? |
|
||||
| scratch threshold | 50 lines | Nudging at the right time? |
|
||||
| consolidation depth | 15-30 min | Producing new connections? |
|
||||
|
||||
All of these are initial guesses. Real data from annotated DMN sessions
|
||||
will tell us what to adjust.
|
||||
Loading…
Add table
Add a link
Reference in a new issue