forked from kent/consciousness
Waffles
6c26cee86e
Cloud API support: - Add chat_api config flag to BackendConfig, threaded through SessionConfig → ResolvedModel → Agent → Mind - New StreamToken::TextDelta variant for chat completions streaming - stream_chat_completion() method on ApiClient: builds messages array, sends to /v1/chat/completions, parses SSE stream - ChatMessage struct and wire_messages() on ContextState: converts the AST (system/identity/journal/conversation nodes) into a messages array for the chat API, handling images as base64 data URIs - ResponseParser handles TextDelta alongside Token variants - TUI rendering fix: tokens() returns byte-length estimate (~4 bytes/token) when tokenizer isn't loaded, so the change detector actually triggers re-renders - Gate all vLLM-specific scoring (memory scoring, finetune scoring, compare scoring) behind !chat_api checks Per-agent model override: - Add model field to agent definition headers (.agent files) - Thread through AutoAgent → prepare_spawn → resolve_model - Agents fall back to default_backend when model is unset - Enables cheaper backends (e.g. Kimi) for graph maintenance agents while keeping Sonnet for conversation Tested: end-to-end with Poe API + Haiku, chat_api: true in config. TUI starts, messages send, responses stream and render. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
47 lines
1.7 KiB
Rust
47 lines
1.7 KiB
Rust
// generate.rs — Continuation generation for scoring / comparison flows.
|
|
//
|
|
// Shared by the finetune pipeline (learn.rs) and the compare screen:
|
|
// given a context prefix and a skip predicate, generate what the model
|
|
// would say as the next assistant turn.
|
|
|
|
use crate::agent::api::{ApiClient, SamplingParams, StreamToken};
|
|
use crate::agent::context::{AstNode, ContextState};
|
|
use crate::agent::tokenizer;
|
|
|
|
/// Generate an assistant continuation from the context up to `entry_idx`,
|
|
/// with `skip` applied to identity + conversation entries during prompt
|
|
/// assembly. The model is whichever `client` points at — the default
|
|
/// runtime client for memory-ablation alternates, a test-model client
|
|
/// for F7 comparison.
|
|
pub async fn gen_continuation<F>(
|
|
context: &ContextState,
|
|
entry_idx: usize,
|
|
skip: F,
|
|
client: &ApiClient,
|
|
) -> anyhow::Result<String>
|
|
where F: FnMut(&AstNode) -> bool,
|
|
{
|
|
let (mut prompt, images, _) = context.wire_prompt(0..entry_idx, skip);
|
|
|
|
prompt.push(tokenizer::IM_START);
|
|
prompt.extend(tokenizer::encode("assistant\n"));
|
|
|
|
let sampling = SamplingParams {
|
|
temperature: 0.6,
|
|
top_p: 0.95,
|
|
top_k: 20,
|
|
};
|
|
let (mut rx, _guard) = client.stream_completion_mm(&prompt, &images, sampling, Some(-5));
|
|
|
|
let mut tokens = Vec::new();
|
|
while let Some(tok) = rx.recv().await {
|
|
match tok {
|
|
StreamToken::Token { id, .. } => tokens.push(id),
|
|
StreamToken::TextDelta(text) => tokens.extend(tokenizer::encode(&text)),
|
|
StreamToken::Done { .. } => break,
|
|
StreamToken::Error(e) => anyhow::bail!("generation error: {}", e),
|
|
}
|
|
}
|
|
|
|
Ok(tokenizer::decode(&tokens))
|
|
}
|