Methods

Evaluation methodology, integrity controls, and reward design. Every claim on this site is either backed by a measured number or clearly marked as design intent.

Out-of-Sample Holdout Policy

All training data is restricted to dates before May 1, 2025. The subsequent 12+ months constitute the evaluation window that no model has seen during training.

OOS Cutoff

2025-05-01

static, enforced in CI

Holdout Window

12+ mo

evaluation data

Date Columns Checked

date, obs_date, ts, …

Static Enforcement (CI-Blocking)

test_oos_leakage.py runs 5 tests in the CI pipeline:

test_sql_files_filter_training_tables — scans all .sql files; any query referencing 19 known training tables must include WHERE <date_col> < '2025-05-01'
test_python_inline_sql_filters_training_tables — AST parses all Python files to extract SQL strings (including f-strings with JoinedStr nodes); same filter requirement
test_oos_pragma_only_in_allowlisted_files — the @oos-eval-only pragma (for evaluation code that intentionally reads OOS data) must only appear in allowlisted files
test_allowlist_entries_exist — validates every allowlisted file actually exists
test_oos_cutoff_date_is_consistent — confirms cutoff date in test matches dashboard constants

Runtime Enforcement

bq_safe.py provides a TrainingBQClient wrapper that raises OOSLeakageError if a query touches a training table without the cutoff filter — defence-in-depth for dynamically constructed queries that static analysis might miss.

OOS Filter Impact on Training Data

Table	Total Rows	Pre-Cutoff	Removed	%
sentinel1_weekly_tank_features	5,124	4,578	546	10.7%
calendar_spreads	20,276	19,804	472	2.3%
market_prices	40,014	39,062	952	2.4%
release_clock	1,148	1,099	49	4.3%
cot_positions	327	75	252	77.1%

Causal Leakage Prevention

During out-of-sample replay, the agent traverses historical decision points chronologically. Without safeguards, memory retrieval could access episodes recorded after the simulated timestamp — letting the agent "remember the future."

All 8 memory retrieval paths accept an optional query_ts parameter. When provided, each path applies a strict less-than filter:

Retrieval Site	Filter	Column
EpisodicMemory._bq_keyword_search()	`AND ts < @query_ts`	ts
EpisodicMemory.get_recent()	`WHERE ts < @query_ts`	ts
EpisodicMemory._local_search()	`ep.timestamp < query_ts`	timestamp
SemanticMemory._bq_search()	`AND first_observed < @query_ts`	first_observed
SemanticMemory.retrieve_relevant()	`fact.first_observed < query_ts`	first_observed
ProceduralMemory._bq_lookup()	`AND last_used < @query_ts`	last_used
ProceduralMemory.get_best_sequences()	`record.last_used < query_ts`	last_used
compare_to_history skill	`AND trade_date < @query_ts`	trade_date

Enforced by 16 dedicated tests in test_causal_leakage.py that verify local cache filtering, backward compatibility, and BQ query source code analysis.

Reward Function Design

Multi-Objective Formulation

R_total = R_directional + α·R_task + β·R_memory + γ·R_consistency + δ·R_retention − λ·C_cost + penalties

Default weights: α = 1.0, β = 0.5, γ = 0.3, δ = 0.2, λ = 0.1

Component	Definition	Signal Source
R_directional	`tanh(pnl_realized / 10.0)`	Environment — subsequent price movement
R_task	Skill completion quality [0,1]	LLM judge or heuristic
R_memory	reward_with − reward_without memory	Counterfactual differential
R_consistency	Directional agreement across turns	Internal coherence
R_retention	probe_pass_count / probe_total	Retention probe tests
C_cost	`clip(cost / $0.50, 0, 1)`	API costs

Penalties

−0.5 per hallucinated skill (skill name not in registry)
−1.0 for position sizes exceeding safety limits (|size| > 100,000 units)

DPO Training Pipeline

Preference pairs are generated from environment feedback (not human annotation): trajectories whose recommended direction aligns with subsequent price movement are preferred. Target base model: Qwen 2.5 7B. Infrastructure is built; execution is deferred pending strategy refinement to ensure the P&L signal carries sufficient edge.

Statistical Methodology

95% confidence intervals via bootstrap resampling (n = 10,000, seed = 42)
Pairwise comparisons: Wilcoxon signed-rank test, p < 0.05
All models use random_state=42 across 22 model constructors
Model version: v0.2.0-oos-clean