Eval: adopt "time to resolution" as a first-class axis and scoring dimension

[smodee]

Summary

Three related changes to the eval stage, all aimed at making time to resolution a first-class quantity:

1. Plot scores against time to resolution rather than forecast index.
2. Record time to resolution as a column on every scored forecast.
3. Aggregate scores with a dwell-time-weighted time-average, so holding an accurate forecast for longer scores better.

All three are blocked on one prerequisite: the question schema has no resolution date.

Motivation

An accurate forecast made one day before resolution is much less impressive than the same forecast made a month out. Nothing in the eval stage currently distinguishes the two — a forecast made one day before resolution counts exactly as much as one made eight weeks out, both in the per-question table and in every pooled summary.

Time to resolution is also the natural x-axis for cross-question comparison. Calendar date is not: it pools questions with different resolution dates, and it bakes in an assumed forecasting cadence. Human forecasts are aggregated from individuals who don't submit synchronously, and our own cadence may change; a horizon axis is invariant to both.

Current state

Axis. _timeline_column() (visualisation.py:60) uses forecast_date when present, else forecast_version. Either way the values are plotted at x = np.arange(len(versions)) (visualisation.py:161, :234) — evenly-spaced ordinal slots, ascending, so real gaps between forecasts are not respected. versions pools distinct dates across all questions, so a forecast 3 days from one question's resolution shares a column with one 200 days from another's.

Horizon metric. question_level_metrics.csv:

question_id,topic,question_type,forecast_source,forecast_version,forecast_date,
resolved_option,brier_score,log_score,accuracy,accuracy_error,rps,
top_probability,normalized_entropy,true_probability

No horizon column. score_all_forecasts() (pipeline.py:211) uses forecast_date only as a grouping key; it never enters a metric.

Aggregation. plot_score_timeline_boxplots() calls _latest_rows_for_source() (visualisation.py:87), which keeps only each source's most recent forecast and discards the rest — the box spread is across questions at that single timepoint. _summarise_group() (compare.py:61) computes unweighted mean/median/std/q1/q3, and no weights parameter exists anywhere in the module.

Blocker: no resolution date in the question schema

All three question CSVs share the same 11 columns, none of which is a resolution date:

question_id | topic | question_text | question_type | resolution_criteria |
created_date | question_status | resolved_option | comparison_to_outcome |
takeaways | relevant_links

• bioscancast_questions.csv (;-delimited)
• bioscancast_questions_resolved.csv (;-delimited)
• bfg_summer_2026_questions.csv (,-delimited, has a BOM)

created_date gives the open date — the other end of the interval — but there is no close date.

The only current route to one is _parse_target_date() (loaders.py:76), which regex-parses question_text with resolution_criteria as fallback. Not a viable basis here: _prepare_questions() (pipeline.py:108) never calls it, and it has known gaps on the BFG question set. An explicit column makes the resolution date data rather than an inference.

Secondary gap: our own forecasts carry no timestamp

Dwell times need a real timestamp on every forecast row. Of the five forecast CSVs, only bioscancast_forecasts.csv (both the root and mock_forecasts/ copies) lacks forecast_date; the human, llm_baseline and both perplexity files have it. _prepare_forecasts() (pipeline.py:171-176) stamps the current run date when the column is absent — harmless for a categorical axis, but it would collapse every BioScanCast forecast onto a single date and yield wrong dwell times.

Proposed changes

1. Add a resolution_date column to the three question CSVs and read it in loaders.py / _prepare_questions(). Keep _parse_target_date() as a backfill for blank rows; treat the column as authoritative.
2. Emit horizon_days (resolution_date - forecast_date) from score_all_forecasts() into question_level_metrics.csv. This alone makes score-vs-horizon answerable without touching a plot.
3. Add forecast_date to bioscancast_forecasts.csv and to whatever writes it.
4. Plot against horizon_days, descending with 0 at the right, on a real numeric axis rather than np.arange slots — which also fixes the uneven spacing and makes questions with different resolution dates directly comparable.
5. Dwell-weighted aggregate: per (question, source), sort forecasts by date, take each forecast's dwell as the gap to the next version (the last one running to the resolution date), and compute the weighted time-average score. Suggested home is a weights= parameter on _summarise_group(), which keeps the raw per-question Brier scores untouched and proper and makes the weighting a reporting-layer choice.

Worth deciding explicitly: whether the dwell-weighted average replaces the latest-only box plot or sits alongside it. They answer different questions — how good is the system now, versus how good across the whole window — and both seem worth keeping.

Aside: quantile bands were replaced by standard-error bands

Adjacent to the above rather than part of it. 714cf65 drew a q1–q3 band around the median; e83a3f3 swapped it for mean ± standard error (sem = std / sqrt(n), visualisation.py:171-183). compare.py still computes q1_* and q3_* — the plot simply stopped reading them. The two say different things: spread across questions versus precision of the mean. Happy to split this out into its own issue.

Related

#32 — the eval-stage redesign the current plotting code comes from.