R scripts to generate city-level forecasts
This repository contains R code to generate city-level forecasts for submission to the flu-metrocast hub. All models are exploratory/preliminary, though we will regularly update this document to describe the latest mathematical model used in the submission.
All outputs submitted to the Hub will be archived in this repository, along with additional model metadata (such as the model definition associated with a submission and details on any additional data sources used or decisions made in the submission process). If significant changes to the model are made during submission, we will rename the model in the submission file.
Since the Hub now solicits both local level and aggregate (typically state-level) forecasts of the specified target (typically the percent of ED visits due to flu), we will:
- run each set of local jurisdictions within an aggregate location jointly
- run each aggregate location (typically state) independently using the aggregate data
A table of all the locations being solicited is available at the flu-metrocast hub GitHub.
Most forecast targets are the percent of ED visits due to flu. California's target is flu hospital admissions (counts), which uses a negative binomial observation model instead (see Model definition).
Except for California, all data is available publicly, so the forecasts generated should be completely reproducible for any given forecast date.
We use the mvgam package, which is a an R package that leverages both mgcv and brms formula interface to fit Bayesian Dynamic Generalized Additive Models (GAMs).
These packages use metaprogramming to produce Stan files, and we also include the Stan code generated by the package.
To produce forecasts each week we follow the following workflow:
- Run the targets pipeline (
targets::tar_make()) for the forecast date. This populatesoutput/cityforecasts/{forecast_date}. - Push the forecasts to a
forecast/{forecast_date}branch, publish a report for review on a public GitHub pages site at https://epiforecasts.io/cityforecasts/{forecast_date}, and open a PR intomainlinking to the report. - Once the forecasts have been reviewed, open a PR into the flu-metrocast hub GitHub under our team name
epiforecasts.
Steps 1-2 are done by bin/run_weekly_forecast.sh, which runs every week on a schedule on a local machine (see Weekly schedule), as the California data is only available locally.
Step 3 is automated too, but only runs once a person has reviewed the forecasts. Merging a Forecasts for {forecast_date} PR into main triggers the Submit to hub workflow. It commits the forecast to a new branch off the hub's main, pushes it to the kaitejohnson/flu-metrocast fork and opens a PR into the hub. To submit (or resubmit) a forecast that is already on main, run the workflow manually from the Actions tab with its forecast date, or run FORECAST_DATE={forecast_date} bin/submit_to_hub.sh locally (DRY_RUN=1 commits without pushing).
California's historical data is private and isn't in this repository or on the hub.
It is read from the California data folder, flu_metrocast_california_data, which must sit next to this repository:
github_repos/
├── cityforecasts/
└── flu_metrocast_california_data/
The pipeline combines it with the California data on the hub (using the hub's values where both have a week). If the folder isn't there, California is skipped with a warning and all other locations are forecast as usual, so the rest of the forecasts can still be reproduced without it.
The private data is never published: the report on GitHub Pages only shows data from the hub.
The raw data plots in output/figures/ do show it, but that folder is not committed.
All commands are run in a terminal from the cityforecasts folder, e.g.
cd ~/Documents/github_repos/cityforecastsYou need:
- R with the packages listed in
.github/workflows/weekly-forecast.yaml, installed in your usual R library (the renv library isn't used), and CmdStan (cmdstanr::install_cmdstan()). - The GitHub CLI, logged in (
gh auth login), with push access to this repository. - The California data folder next to this repository (see California data).
To run the weekly forecast now, with the code currently checked out:
bin/run_weekly_forecast.shThis does steps 1-2 above: it waits until the hub data has been updated for the forecast date (checking every 10 minutes, for up to 3 hours by default), runs the pipeline, then pushes the forecasts, publishes the report and opens the PR. It skips the run if the forecast for that date already exists. It is configured with environment variables, e.g.
# Rerun a forecast that already exists
FORCE=1 bin/run_weekly_forecast.sh
# Run for a past forecast date with the hub data as it was then (a
# flu-metrocast commit SHA), without pushing anything or opening a PR
FORECAST_DATE=2026-10-07 HUB_REF=<flu-metrocast SHA> PUSH=0 OPEN_PR=0 bin/run_weekly_forecast.shSee the top of bin/run_weekly_forecast.sh for all the options.
The forecast date defaults to the most recent Wednesday, so running it on a Thursday still produces that week's (Wednesday's) forecast.
To only run the pipeline, without waiting for data or pushing anything, start R in this folder and run targets::tar_make() (set the FORECAST_DATE and HUB_REF environment variables first to run for another date).
The weekly forecast runs automatically on a Mac using launchd (macOS's built-in scheduler), by default every Wednesday at 18:00 UK time (1 PM US Eastern). The hub data is usually updated later in the afternoon US time, so the run keeps checking for it every 10 minutes until it arrives, for up to 6 hours (until midnight UK time). The Mac only needs to be open and connected at 18:00 and while the run is going. It runs in GitHub Actions, as it used to, only if started manually: it can't access the California data, so it skips California.
To set up the schedule (once):
bin/install_local_schedule.shAt the scheduled time, bin/scheduled_local_run.sh runs. It:
- resets a separate copy of this repository,
../cityforecasts-scheduled(a git worktree), to the latestorigin/main, so the merged code is always used, whatever branch or uncommitted changes you have in your own copy; - runs
bin/run_weekly_forecast.shthere (checking for updated hub data every 10 minutes, for up to 6 hours); - writes a log to
logs/scheduled_{timestamp}.logand shows a notification when it has finished or failed.
If the Mac is asleep at the scheduled time, the run starts as soon as it wakes. If it is shut down, that week's run is skipped, so run bin/scheduled_local_run.sh by hand instead.
The hour is in the Mac's local time zone: 18:00 in the UK is 1 PM US Eastern for most of the year, but an hour off for the few weeks in spring and autumn when only one of the two has changed its clocks (the run still waits for the data).
On a Mac in another time zone, pass the local hour, e.g. bin/install_local_schedule.sh wednesday 13 for US Eastern.
Useful commands:
| To… | Run |
|---|---|
| Move the run to another day, e.g. Thursday for one week | bin/install_local_schedule.sh thursday |
| Go back to Wednesdays at 18:00 | bin/install_local_schedule.sh |
| Run the scheduled job now | launchctl kickstart gui/$(id -u)/io.epiforecasts.cityforecasts.weekly |
| Do exactly what the scheduled job does, by hand | bin/scheduled_local_run.sh |
| Check the schedule | launchctl print gui/$(id -u)/io.epiforecasts.cityforecasts.weekly | grep -A4 descriptor |
| Remove the schedule | bin/install_local_schedule.sh --uninstall |
Each of these replaces the previous schedule, so after moving a run to Thursday, run bin/install_local_schedule.sh once that week's forecast is done to go back to Wednesdays.
launchd runs a copy of bin/scheduled_local_run.sh, so if you change that script, rerun bin/install_local_schedule.sh (with the same day) to update the copy.
Reports from every automated run are kept on GitHub Pages, so forecasts from past forecast dates can be reviewed at any time:
- The index page, https://epiforecasts.io/cityforecasts/, lists every report by forecast date, newest first.
- Each report is at https://epiforecasts.io/cityforecasts/{forecast_date}/, where
{forecast_date}is the Wednesday forecast date inYYYY-MM-DDformat (e.g. https://epiforecasts.io/cityforecasts/2026-10-14/). - Each weekly
Forecasts for {forecast_date}PR also links to its report.
The reports are stored on the gh-pages branch. A rerun for the same forecast date replaces that date's report.
The below describes the preliminary model used:
For the forecasts of flu hospital admissions (counts, California only), we assume a negative binomial observation process, as counts are overdispersed relative to a Poisson:
For the forecasts of the percent of ED visits due to flu, we assume a lognormal observation process on the proportion of ED visits due to flu:
The lognormal assumes constant relative (multiplicative) observation noise, so that changes at low proportions (e.g. early-season upticks) are attributed to the latent state rather than treated as noise, as a Beta observation model with constant precision does. Because the reported proportions are rounded to 0.0001, observed zeros (true value < 0.00005) are replaced with 0.00005 before fitting.
We model the latent state with a hierarchical GAM component to capture shared seasonality and a univariate autoregressive component to capture deviations from the seasonal pattern within each location.
When a location is fit on its own (e.g. aggregate/state locations), there is no hierarchy:
For the forecasts of flu hospital admissions (counts),
For the forecasts of percent of ED visits due to flu,