Skip to content

Phase 5:feat(benchmark): add validation framework tutorial vignette (#4076) - #4087

Merged
dlebauer merged 16 commits into
PecanProject:developfrom
ayushman1210:phase-5
Sep 18, 2026
Merged

dlebauer merged 16 commits into
PecanProject:developfrom
ayushman1210:phase-5

Conversation

@ayushman1210

Copy link
Copy Markdown
Contributor

Summary

Resolves #4076 (Phase 5 GSoC Benchmarking & Validation Milestone) by creating a comprehensive tutorial vignette for the PEcAn.benchmark package. The vignette explains and demonstrates the complete end-to-end benchmarking workflow, data intake schemas, statistical error model formulations, temporal alignment, metric evaluation, diagnostic visualizations, and automated Quarto scorecard rendering.


Key Changes

1. Tutorial Vignette (modules/benchmark/vignettes/validation_framework_tutorial.Rmd)

  • Data Intake & Schemas: Explains the standardized EFI Long Format (datetime, site_id, parameter, variable, prediction) and observation formats.
  • Statistical Error Model: Formulates the statistical observation error model used for generating simulated observation fixtures with known parameters:
    $$o_{v,t} = m_{v,t} + S_v(\beta_v + \epsilon_{v,t}), \qquad \epsilon_{v,t} \sim N(0, \tau_v^2)$$
  • Reshaping & Alignment: Demonstrates extracting ensemble member matrices via efi_long_to_array() and temporal alignment via align_by_time().
  • Metric Suite Evaluation: Explains when and why each metric is used, distinguishing point metrics on the ensemble mean (RMSE, BIAS, MAE, R2) from spread metrics on the full ensemble (COVERAGE, CRPS).
  • Visualizations: Demonstrates member spaghetti lines under $q_{05}/q_{95}$ ribbons (metric_timeseries_plot), scatter plots (metric_scatter_plot), and residual plots (metric_residual_plot).
  • Custom Data Guide: Instructs users on substituting custom model output CSVs and observation datasets.

2. Package Configuration (modules/benchmark/DESCRIPTION)

  • Added VignetteBuilder: knitr.
  • Added knitr and rmarkdown to Suggests.

3. Unit Test Coverage (modules/benchmark/tests/testthat/test-vignette_execution.R)

  • Added deterministic test verifying that vignette workflow code blocks execute cleanly during package test runs.

Verification & Testing

  1. Vignette Compilation:
    Rendered validation_framework_tutorial.Rmd using rmarkdown::render() without any errors or warnings.
  2. Unit Test Suite:
    Ran devtools::test("modules/benchmark"):
    [ FAIL 0 | WARN 0 | SKIP 1 | PASS 136 ] (All 136 package tests passing).

Related Issue

Closes #4076

@divine7022 divine7022 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @ayushman1210 ! took first pass; vignette reads well, schema section and the mean vs ensemble split are the parts I would keep as is. but CI still fails probable fix I have dropped below.

Comment thread modules/benchmark/DESCRIPTION
Comment thread modules/benchmark/DESCRIPTION
Comment thread modules/benchmark/vignettes/validation_framework_tutorial.Rmd
Comment thread modules/benchmark/inst/extdata/generate_simulated_fixture.R Outdated
Comment thread modules/benchmark/vignettes/validation_framework_tutorial.Rmd Outdated

@divine7022 divine7022 left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is much closer, thanks.

Comment thread modules/benchmark/inst/extdata/simulated_observations.csv Outdated
Comment thread modules/benchmark/vignettes/validation_framework_tutorial.Rmd Outdated
Comment thread modules/benchmark/vignettes/validation_framework_tutorial.Rmd Outdated

@divine7022 divine7022 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for putting this together!,
one thing i never actually dropped on you, sorry, of examples/benchmarks/ameriflux/,
real ensemble is the upstream dependency and i think that's still where we are. can this run_ameriflux_benchmark.R come out of this PR and come back with the ensemble ? the simulated fixtures stay either way since the vignette uses them now.
fine as a follow up -- just reporting here
i will defer this to @dlebauer

@dlebauer
dlebauer added this pull request to the merge queue Sep 18, 2026
Merged via the queue into PecanProject:develop with commit a3fd7bb Sep 18, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

Phase 5 benchmarking and validation tutorial vignette

3 participants