Changelog

Deployed FAQ short-circuit and knowledge-first routing to the chat agent (revision otm-agent-chat-00019-rip).

• FAQ pre-check: 9 regex patterns bypass the planner LLM entirely for canonical queries (tank count, benchmark results, hub list, audit cycles). Responds in ~1.2s at $0.00 cost.
• KNOWN_FACTS dict: Authoritative agent specs and benchmark results injected into zero-step response prompt.
• Rule 9 rewrite: Knowledge-first routing with explicit query-pattern categories. Zero-step plans for facts, multi-step only for live data.
• Results: API pass rate 33% → 70% → 93%. Zero hallucinations (down from 2 at baseline). FAQ queries 20-40× faster than skill-based.

Deployed anti-hallucination prompt improvements to the chat agent (Cloud Run revision otm-agent-chat-00017-rux).

• Rule 11 (NUMERICAL CITATIONS): Agent may only cite values explicitly present in context fields. Prevents training-data confabulation.
• Benchmark facts injected: Verified results for all 5 benchmarks added to planner system prompt (Skill-Retention 81.7%, OOS Replay 50.0%, Persona-Align p=0.70, Mem-Utility null, MultiTurn-Persona underpowered).
• Anti-fabrication rules: Response template now forbids fabricated technical analysis (support/resistance levels, price targets) unless backed by model output.
• Test suite result: API pass rate improved from 33% (5/15) to 70% (7/10). Hallucination rate dropped from 13% to 10%. Zero timeouts.

Chat widget silently crashed before sending API call. Root cause: chatHistory stores messages as {role, text} but the v3.1.0 token-capping code referenced m.content (always undefined). Fixed by using m.text || m.content || '' in both index.html and dashboard.html.

Upstream datasets/oil-prices GitHub CSV stopped updating May 18. Replaced single-source dependency with resilient 3-source fallback and daily automation.

• New primary: self-hosted data/wti_prices.json (503 prices through today)
• Fallback chain: FRED API → Yahoo Finance → GitHub CSV
• GitHub Actions workflow (update-prices.yml) runs Mon–Fri at 22:30 UTC
• FRED API key stored in GCP Secret Manager + GitHub repo secret
• Staleness banner auto-shows when data >4 days old
• Both index.html and dashboard.html updated with identical multi-source fetchOilPrices()
• Added .nojekyll to prevent Jekyll build failures with workflow YAML
• First automated run: FRED fetched 491 prices, merged to 503 total — fully verified end-to-end

Chatbot was timing out due to Cloud Run cold starts (~20s) combined with 3-replan loops (~40s each). Fixed infrastructure to keep the agent responsive.

• Set min-instances=1 on otm-agent-chat Cloud Run — eliminates cold starts
• Reduced OTM_MAX_REPLANS from 3 to 1 — agent responds within 90s timeout
• Created missing otm_agent.episodes BigQuery dataset + table — episodic memory now persists
• Verified: 200 OK on both API URLs, responses within 40–50s for complex queries

Completed v1.1 Persona-Align re-run with data fixes and full LLM cost tracking. 196 agent runs, 49 comparisons, seed=42.

• Divergence ratio: 1.001× baseline (v1.0: 1.003×) — null holds
• Wilcoxon p=0.70 (v1.0: p=0.23) — even more clearly null
• Skill Jaccard: 0.047 agent vs 0.040 baseline — no detectable persona effect
• Cost: $1.77 with full tracking (v1.0 reported $0.31 skill-only = 5.7× correction)
• Two exploratory benchmarks executed (not pre-registered, reduced scale): Mem-Utility (null, −0.064, n=25) and MultiTurn-Persona (underpowered, +0.173, p=0.109, n=7). Total v1.1 cost: $20.51 ($0.51 over $20 cap)

Completed v1.1 OOS Replay with three data-availability fixes applied (dtype coercion, NO_DATA routing, compose_signal float coercion). Budget-constrained to 120-day compressed scope (110 days completed at $13.13).

• Headline: 50.0% directional accuracy (CI: 40.9%–60.0%, n=110)
• Apples-to-apples (May–Oct 2025): v1.0 47.5% → v1.1 50.0% = +2.5pp
• Within pre-registered 1–5pp improvement expectation
• Largest month improvement: Aug 2025 (25.0% → 45.0%) — dtype fix eliminated uninformed fallback
• Full cost tracking now operational: $0.12/day actual vs $0.0002/day previously reported
• Nov 2025–Apr 2026 not evaluated due to budget constraint (documented as scope amendment)

Re-ran Skill-Retention benchmark (60 probes, 20 skills) with data fixes and corrected cost tracking. Result identical to v1.0.

• Plan inclusion: 81.7% (unchanged from v1.0)
• Top-3: 58.3% (+3.3pp vs v1.0 55%), Top-1: 21.7% (same)
• Mean plan length: 3.3 steps (vs v1.0 3.4)
• Cost with full tracking: $1.65 (vs v1.0 reported $0.20 skill-only — 8.3× correction)
• Confirms data-availability fixes do not affect planner skill selection

Discovered and fixed a measurement bug: record.total_cost_usd captured only skill execution costs (BigQuery queries), not planner or response-generation LLM costs.

• Instrumented three API call sites: generate_plan(), replan(), _llm_response()
• Added model pricing table with per-token cost estimation (Sonnet $3/$15 per M, Haiku $0.80/$4)
• Corrected total benchmark suite cost: ~$15–25 (was $0.55) — still 6–10× under $150 budget
• Removed stale model ID from pricing table; added warning on unknown model fallback
• Benchmark accuracy results (54.4%, 81.7%, persona null) are unaffected

Three data-availability issues identified during v1.0 provenance verification, fixed for v1.1:

• estimate_volatility_regime: dtype coercion fix (string→float) + FAILURE→NO_DATA
• compute_fill_signal: insufficient-data path changed from FAILURE to NO_DATA
• giti_daily: diagnosed as missing pipeline (0 rows) — deferred to v1.2
• All 15 compute_skills tests pass including new test_string_coercion

Ran the OTM-Persona-Align benchmark end-to-end with real agent calls (Claude Haiku executor, OpenAI embeddings). 15 queries × 3 seeds × 4 modes = 180 agent invocations.

• Result: NULL per pre-registered interpretation table — v0.1 in-prompt persona conditioning does not produce measurable embedding-space behavior divergence
• Implication: v0.2 activation steering is the hypothesized mechanism for deeper conditioning
• All results committed per protocol — no post-hoc amendments

Fixed embedding cache poisoning bug (hash embeddings leaking into production cache), updated model IDs to current Anthropic API, disabled re-planning for benchmarks (3.4× speedup), verified resume-from-checkpoint works.

• Preflight passed: all 3 checks green (embeddings differ, responses non-empty, skills produced)
• Smoke test (n=2, 1 seed): 8 agent calls completed, JSON schema validated
• Cost per agent call: ~$0.0001 (Haiku executor)

OTM-Agent public site deployed at oguzhan-canada.github.io/otm-agent-site/. Static HTML, no build tools, instant deploys via git push.

• 6 pages: Dashboard (home), Architecture, Methods, Benchmarks, About, Paper
• Live SAR dashboard with 4 hubs, 13 tanks, trend charts, AI chat assistant
• Data-driven benchmark status badges (auto-flip from "scheduled" to "live")
• Pipeline watchdog: GitHub Actions workflow monitors data freshness daily

Recovered 5 weeks of missing data (Apr 14 – May 21) from Earth Engine. 344 tank rows and 132 region rows backfilled into BigQuery.

• Root cause: Cloud Scheduler paused during billing review
• Fix: Watchdog workflow added to detect and auto-recover stale data

Static AST analysis scan (test_oos_leakage.py) identified 20+ SQL queries across 12 files that lacked the holdout date filter. All fixed, retrained models from scratch.

• Second layer: test_causal_leakage.py (16 tests) ensures memory system respects query_ts
• Third layer: bq_safe.py runtime guard raises OOSLeakageError
• No prior model artifacts existed (hub_state_day was empty pre-audit) — audit prevented future leakage

Locked PERSONA_ALIGN_PROTOCOL.md — success criteria, interpretation table, reporting commitment. No post-hoc amendments permitted.

• Strong: > 3× baseline, p < 0.001
• Moderate: 2–3× baseline, p < 0.01
• Weak: 1.5–2× baseline, p < 0.05
• Null: < 1.5× baseline OR p ≥ 0.05

Extended the skill probe set from 20 to 60 (3 probes per skill, covering all 7 categories). Production runner complete with real OTMAgent.run() calls, top-1/top-3 accuracy, per-category breakdown.

Built oos_replay.py: walks ~250 trading days, runs real agent calls with query_ts causal filter, BQ→CSV→synthetic price fallback chain, direction/confidence parsing, per-month breakdown, Sharpe calculation.

Complete evaluation section covering Persona-Align, Skill-Retention, OOS Replay, and Mem-Utility. Pre-registered criteria with [X.XX] placeholder convention for results not yet observed.

Full test suite green: OOS leakage (5), causal leakage (16), skill library (40+), memory system (30+), persona module (20+), reward components, and integration tests.