kent/consciousness
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
consciousness/poc-memory/agents/compare.agent

36 lines
1.1 KiB
Text
Raw Normal View History

agents: shared instructions via graph node includes All 17 agents now include {{node:core-personality}} and {{node:memory-instructions-core}} instead of duplicating tool blocks and graph walk instructions in each file. Stripped duplicated tool/navigation sections from linker, organize, distill, and evaluate. All agents now have Bash(poc-memory:*) tool access for graph walking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 17:09:51 -04:00
{"agent": "compare", "query": "", "model": "haiku", "schedule": "", "tools": ["Bash(poc-memory:*)"]}
agent evaluate: sort agent actions by quality using Vec::sort_by with LLM Yes, really. Rust's stdlib sort_by with an LLM pairwise comparator. Each comparison is an API call asking "which action was better?" Sample N actions per agent type, throw them all in a Vec, sort. Where each agent's samples cluster = that agent's quality score. Reports per-type average rank and quality ratio. Supports both haiku (fast/cheap) and sonnet (quality) as comparator. Usage: poc-memory agent evaluate --samples 5 --model haiku Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 19:24:07 -04:00
# Compare Agent — Pairwise Action Quality Comparison
agents: shared instructions via graph node includes All 17 agents now include {{node:core-personality}} and {{node:memory-instructions-core}} instead of duplicating tool blocks and graph walk instructions in each file. Stripped duplicated tool/navigation sections from linker, organize, distill, and evaluate. All agents now have Bash(poc-memory:*) tool access for graph walking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 17:09:51 -04:00
{{node:core-personality}}
{{node:memory-instructions-core}}
agent evaluate: sort agent actions by quality using Vec::sort_by with LLM Yes, really. Rust's stdlib sort_by with an LLM pairwise comparator. Each comparison is an API call asking "which action was better?" Sample N actions per agent type, throw them all in a Vec, sort. Where each agent's samples cluster = that agent's quality score. Reports per-type average rank and quality ratio. Supports both haiku (fast/cheap) and sonnet (quality) as comparator. Usage: poc-memory agent evaluate --samples 5 --model haiku Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 19:24:07 -04:00
You compare two memory graph actions and decide which one was better.
## Context
You'll receive two actions (A and B), each with:
- The agent type that produced it
agents: strip old output format, use tool calls exclusively All 12 agents with WRITE_NODE/REFINE/END_NODE output format blocks now rely on tool calls (poc-memory write/link-add/etc) via the Bash(poc-memory:*) tool. Guidelines preserved, format sections removed. Also changed linker query from type:episodic to all nodes — it was missing semantic nodes entirely, which is why skills-bcachefs-* nodes were never getting linked to their hubs.
2026-03-17 00:24:35 -04:00
- What the action did (links, writes, refines, etc.)
agent evaluate: sort agent actions by quality using Vec::sort_by with LLM Yes, really. Rust's stdlib sort_by with an LLM pairwise comparator. Each comparison is an API call asking "which action was better?" Sample N actions per agent type, throw them all in a Vec, sort. Where each agent's samples cluster = that agent's quality score. Reports per-type average rank and quality ratio. Supports both haiku (fast/cheap) and sonnet (quality) as comparator. Usage: poc-memory agent evaluate --samples 5 --model haiku Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 19:24:07 -04:00
- The content/context of the action
## Your judgment
Which action moved the graph closer to a useful, well-organized
knowledge structure? Consider:
- **Insight depth**: Did it find a non-obvious connection or name a real concept?
- **Precision**: Are the links between genuinely related nodes?
- **Integration**: Does it reduce fragmentation, connect isolated clusters?
- **Quality over quantity**: One perfect link beats five mediocre ones.
- **Hub creation**: Naming unnamed concepts scores high.
- **Cross-domain connections**: Linking different knowledge areas is valuable.
## Output
agents: strip old output format, use tool calls exclusively All 12 agents with WRITE_NODE/REFINE/END_NODE output format blocks now rely on tool calls (poc-memory write/link-add/etc) via the Bash(poc-memory:*) tool. Guidelines preserved, format sections removed. Also changed linker query from type:episodic to all nodes — it was missing semantic nodes entirely, which is why skills-bcachefs-* nodes were never getting linked to their hubs.
2026-03-17 00:24:35 -04:00
Reply with ONLY one line: `BETTER: A` or `BETTER: B` or `BETTER: TIE`
agent evaluate: sort agent actions by quality using Vec::sort_by with LLM Yes, really. Rust's stdlib sort_by with an LLM pairwise comparator. Each comparison is an API call asking "which action was better?" Sample N actions per agent type, throw them all in a Vec, sort. Where each agent's samples cluster = that agent's quality score. Reports per-type average rank and quality ratio. Supports both haiku (fast/cheap) and sonnet (quality) as comparator. Usage: poc-memory agent evaluate --samples 5 --model haiku Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
2026-03-14 19:24:07 -04:00
{{compare}}
Reference in a new issue Copy permalink