Evaluation Benchmarks

Status (2026-05-26): Three pre-registered v1.1 benchmarks complete. OOS Replay: 50.0% (n=110). Skill-Retention: 81.7% (n=60). Persona-Align: null confirmed (p=0.70). Two exploratory benchmarks at reduced scale: Mem-Utility: null (−0.064, n=25, not pre-registered). MultiTurn-Persona: underpowered (+0.173, p=0.109, n=7, not pre-registered). Total v1.1 cost: $20.51 ($0.51 over $20 cap) — see cost amendment.

Tests the strongest architectural claim: that persona vectors produce meaningfully different behavior. 100 queries spanning the agent's domain are run twice — once seeded at the execution_oriented anchor, once at analysis_oriented. Measures cosine distance between response embeddings and Jaccard distance between skill sets called.

Baseline: Same queries with fixed 50/50 persona (should give near-zero divergence).

Statistical method: Mean ± bootstrap CI over 5 seeds (42–46).

Result: Pre-registered NULL confirmed. Persona divergence ratio 1.003× baseline (p=0.23, Wilcoxon). Two persona-conditioned responses are no more different from each other than two neutral-neutral runs. Skill selection Jaccard ≈ 0.10 (agent) vs 0.11 (baseline) — effectively identical. Our pre-registered protocol predicted this as the most likely v0.1 outcome: in-prompt persona injection does not cause the LLM to condition on a 64-d float vector. v0.2 activation steering (residual-stream injection) is the hypothesized mechanism for deeper conditioning.

Re-verification (2026-05-24): After fixing the planner stop-condition bug, the persona benchmark was re-run with the corrected planner (49 queries × 4 persona modes = 196 agent runs). Both persona modes now execute multi-step plans (2-5 steps with real BQ data retrieval), eliminating the earlier confound of structurally identical minimal executions. The null result holds. This confirms the finding is genuine — the persona module's effect on planner behavior is not detectable through embedding-space cosine divergence at v0.1's in-prompt injection point. The issue is architectural, not a measurement artifact.

v1.1 Re-run (2026-05-25): Re-ran with data fixes (NO_DATA routing, dtype coercion) and corrected cost tracking. Result: null confirmed — divergence ratio 1.001× baseline (p=0.70, Wilcoxon). 196 agent runs, n=49 comparisons. Skill Jaccard: 0.047 (agent) vs 0.040 (baseline). Cost with full tracking: $1.77 (vs v1.0 reported $0.31 skill-only = 5.7× correction). Data fixes do not alter the persona null finding. v1.1 result JSON.

Note on n=49: The benchmark runs 49 distinct queries × 4 persona modes (2 anchored + 2 neutral baseline) = 196 total agent runs. Divergence is computed per query pair, yielding n=49 paired comparisons for the Wilcoxon test. The description above references “100 queries” as the design target; actual implementation uses 49 unique queries from the domain probe set.

Tests whether the planner routes queries to the correct skill. 60 canonical probes (3 per skill, 20 skills, 7 categories) are run through the full agent. For each probe, we check whether the expected skill appears anywhere in the generated plan (plan inclusion). Secondary metrics include strict positional top-1 (first step) and top-3 accuracy.

v1.0 protocol: Single-turn skill selection. Multi-turn retention decay deferred to v0.2.

Baseline: Random chance of 1 target skill in a 3.4-step plan from 20 skills = 15.7%. Observed: 5.2× baseline.

Protocol amendment (2026-05-23): Primary metric changed from top-1 accuracy to plan inclusion after identifying three measurement bugs. Initial run reported 23.3% top-1 (initial result JSON); post-fix rerun: 78.3% plan inclusion. See SKILL_RETENTION_PROTOCOL.md for full amendment.

Re-verification (2026-05-24): After planner stop-condition fix, re-ran all 60 probes. Result: 81.7% plan inclusion (CI: 71.7%–91.7%). 14/20 skills at 100%. Top-3: 55%, Top-1: 21.7%. Mean plan length: 3.4 steps. Weakest: fetch_inventory_baseline (0%), sense_tank_state (0%) — planner substitutes fetch_tank_features for both. Full results: JSON.

v1.1 Re-run (2026-05-25): Re-ran with data fixes (NO_DATA routing, dtype coercion) and corrected cost tracking. Result: 81.7% plan inclusion (identical to v1.0). Top-3: 58.3% (+3.3pp), Top-1: 21.7% (same). Mean plan length: 3.3 steps. Cost with full tracking: $1.65 (vs v1.0 reported $0.20 skill-only). Confirms data fixes do not affect skill selection. v1.1 result JSON.

Counterfactual evaluation: queries run with full memory retrieval, then the same queries with memory disabled (NoopMemoryManager). Response quality compared via LLM-as-judge (Claude Haiku rates both on 0–5, blind, order randomized). Memory utility = mean(with_memory) − mean(without_memory).

Positive utility means memory retrieval improved outcomes; negative means it hurt. Reports per-query delta plus overall.

Status: Exploratory. Not pre-registered. Executed at n=25 (designed scale: 500). Results reported for transparency but statistically underpowered for subcategory analysis.

v1.1 Result (n=25, seed=42, $1.39): Memory utility: −0.064 (95% CI: −0.276 to +0.128). Mean score with memory: 3.77/5.0. Mean score without memory: 3.83/5.0. The CI crosses zero — the 4-tier memory system does not produce a statistically detectable improvement. Per-category results (n=5 each) have extremely wide CIs and are not statistically interpretable. Full-scale execution at n≥200 scoped for v1.2. Result JSON.

Multi-turn sessions (5 turns each) testing whether persona modes produce increasingly distinct behavior across turns. Each conversation runs under two opposing persona modes (execution_oriented vs analysis_oriented) with the same agent instance persisting. Measures Jaccard skill divergence at turns 1–2 (early) vs 4–5 (late). Convergence delta = late divergence − early divergence.

Status: Exploratory. Not pre-registered. Executed at n=7 sequences (designed scale: 100). At this sample size, the benchmark lacks statistical power to reliably distinguish signal from noise.

v1.1 Result (n=7 sequences, seed=42, $2.57): Convergence delta: +0.173 (95% CI: −0.064 to +0.378). Mean early divergence: 0.162. Mean late divergence: 0.335. Wilcoxon p=0.1094 — not statistically significant. The positive direction is noted but cannot be distinguished from chance at n=7. No conclusions about persona convergence should be drawn from this run. Confirmatory run at n≥20 with pre-registered protocol scoped for v1.2. Result JSON.

The headline benchmark. Walk day-by-day through 2025-05-01 onward (~250 trading days). For each day t: set query_ts = t, ask the agent "what's the call for tomorrow?", record recommendation, compare against actual next-day move.

Reports directional accuracy, not just Sharpe. Depends critically on the Category B causal filter — all memory retrieval has query_ts plumbed through.

Cadence: Monthly — heavy run; full replay takes hours.

Result (full 250-day run, n=237, seed=42): Directional accuracy 54.4% (95% CI: [48.1%, 60.8%]). CI crosses 50% — not statistically significant at 95% confidence. Sharpe ratio: 1.37. Cumulative P&L: 6,301 bps; max drawdown: 2,623 bps. Cost: estimated $9–15 (see cost amendment).

v1.1 Result (compressed run, n=110, seed=42): Directional accuracy 50.0% (95% CI: [40.9%, 60.0%]). Sharpe: 1.05. P&L: 1,513 bps; drawdown: 1,204 bps. Cost: $13.13 ($0.12/day with full LLM cost tracking). Sample size reduced from 237 to 110 days due to API budget constraint ($0.12/day × 237 days ≈ $28, exceeding available budget); full 250-day re-run deferred to v1.2.

v1.0 → v1.1 comparison (same period, May–Oct 2025):

Scope amendment: Budget constraints required compressing from 237 to 120 days. Run terminated at 110 days ($13.13 of $15 cost cap). Nov 2025–Apr 2026 not evaluated in v1.1. Pre-registered expectation was 1–5pp improvement; observed +2.5pp over the comparable period confirms the data-availability fixes had the expected modest effect.

v1.0 per-month breakdown (full 237-day run):

Confidence calibration: Mean confidence when correct = 0.538; when wrong = 0.539. At full scale the confidence calibration is flat — the agent does not know when it is right. The 30-day smoke showed slight calibration (0.547 vs 0.472) that did not persist at n=237.

Interpretation. The 54.4% accuracy is above the 50% coin-flip baseline and above the 52.9% majority-class baseline, but the CI crossing 50% means we cannot reject the null hypothesis of random performance. This is consistent with pre-registered expectations: the regime classifier achieves 65.4% in-sample, but composite signal dilution from near-chance components was predicted to yield 52–58% OOS. The result falls squarely within that range.

Notable: accuracy varies significantly by month (25.0% in Aug 2025 to 68.8% in Nov 2025), suggesting regime-dependent performance. The agent performs better in trending markets and worse in range-bound/volatile periods — consistent with a momentum-based regime classifier.

Audit trail: v1 → v2 fix cycle. The initial smoke test (v1) produced 57.1% accuracy, but diagnosis revealed every prediction was made on empty data. Three planner bugs were identified and fixed:

1. _should_stop used naive substring matching — any single failure triggered abort
2. query_goii returned SUCCESS with null data (changed to NO_DATA status)
3. _resolve_args couldn't parse the planner's $stepN.field reference formats

See v1 result (data-starved) | v2 full 250-day result.

Cost: v1.0: Estimated $9–15 total for 237 days (~$0.04–0.06/day, see cost amendment). v1.1: $13.13 for 110 days ($0.12/day with full LLM cost tracking).