evaluate: switch to Elo ratings with skillratings crate
Replace sort-based ranking with proper Elo system: - Each agent TYPE has a persistent Elo rating (agent-elo.json) - Each matchup: pick two random types, grab a recent action from each, LLM compares, update ratings - Ratings persist across daily evaluations — natural recency bias from continuous updates against current opponents - K=32 for fast adaptation to prompt changes Usage: poc-memory agent evaluate --matchups 30 --model haiku Co-Authored-By: Kent Overstreet <kent.overstreet@linux.dev>
This commit is contained in:
parent
e2a6bc4c8b
commit
16777924d0
4 changed files with 129 additions and 71 deletions
|
|
@ -24,6 +24,7 @@ jobkit-daemon = { path = "../jobkit-daemon" }
|
|||
redb = "2"
|
||||
log = "0.4"
|
||||
ratatui = "0.29"
|
||||
skillratings = "0.28"
|
||||
crossterm = { version = "0.28", features = ["event-stream"] }
|
||||
|
||||
[build-dependencies]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue