ops(railway): shorten the merge-to-build lag tail by triggering seeder deploys from CI #6142

Why

Railway's per-service Watch Paths filter silently refuses pushes that plainly match its glob. The rejection is recorded only as a deployment record with status SKIPPED carrying meta.commitHash, which nothing reads, and the service keeps running the previous image indefinitely. That is #6141.

Measured against production on 2026-08-04 across the 77 Railway services built from this repository:

The two exceptions on the unfiltered side are both explained: seed-military-flights had been recovered with a railway up upload, which carries no commit SHA at all, and umami is the separate webhook outage in #6064.

Narrowing the filter does not help. seed-conflict-intel shipped an exact 24-path dependency closure — the most careful in the fleet — and had the worst skip rate at 51% over its last 500 deployments.

Correction on the shape of the defect. Railway builds the full tree at a SHA, so a refused commit's changes ride along on the next build that fires. This is a lag tail, not a loss rate — hours from merge to a build containing the change are p50 0h everywhere, but p90 19.0h and max 62.6h (seed-conflict-intel). Lag is harmless for a copy tweak and an outage when the delayed commit fixes an active crash loop, and nothing inside the repository distinguishes the two: unmonitored lag is indistinguishable from loss. That is why detection shipped first and why this issue is about the trigger, not about making every push build.

Why not just clear the filters

That was #6141's proposal and it works: every unfiltered service tracks head. It was rejected on cost. Measured from real buildOnly deployment records across 67 services:

• median build 13s, mean 59s, slowest five 337–567s (seed-consumer-prices, seed-supply-chain-trade, seed-grocery-basket, seed-bundle-static-ref, digest-notifications)
• ~75 build-minutes per main push across the fleet
• ~30 commits/day on main over the last 14 days → ~2,250 build-minutes/day

and three filtered services are always-on rather than cron — ais-relay, notification-relay, scenario-worker — so a rebuild is a restart. ais-relay would drop its AIS websocket connections on every merge to main. (seed-iran-events also pins no cron, but it runs its script once and exits, so a rebuild is its only trigger rather than an interruption.)

Proposed work

Move the change detection out of Railway and into CI, where we own it and can test it.

1. A GitHub Action on push to main computes, per service, whether that service's runtime dependency closure changed in the pushed range. The closures already exist in scripts/railway-services.json and the import-graph walk that keeps them complete already exists in tests/nixpacks-seeder-import-graph.test.mjs and the closure contract test.
2. For each service whose closure changed, call railway redeploy (or railway up) explicitly.
3. Leave Railway's own watch paths alone — with an explicit trigger, whether Railway also decides to build is no longer load-bearing. Deleting them is explicitly not the plan: the filter does a legitimate job, and clearing it rebuilds every seeder on each unrelated push without shortening the tail that matters.

Note railway redeploy rebuilds the deployment it already has, so it cannot advance a service to a newer commit. The trigger must be a deploy-latest-commit call or a railway up from a clean CI checkout at the target SHA.

The difference that matters is not the width of the glob; it is that the matching happens in code with tests rather than in Railway's matcher, which is the component that proved unreliable.

Acceptance

• A merge touching a service's dependency closure always produces a build for that service.
• node scripts/check-railway-deploy-drift.mjs reports zero REJECTED_PUSH and zero BEHIND services, and scripts/railway-deploy-drift-baseline.json is emptied of the entries this issue owns.
• Build minutes stay within the same order of magnitude as today.
• The p90 merge-to-build lag drops from 19h to under one build cycle.

Already shipped

Detection landed ahead of the fix, so the failure is no longer silent while this is open:

• scripts/check-railway-deploy-drift.mjs compares each service's running commit against main's head and reports REJECTED_PUSH / BEHIND / BUILD_FAILED / UNKNOWN_SOURCE and four other non-healthy verdicts.
• It runs every 15 minutes in .github/workflows/seed-freshness-monitor.yml.
• The currently-stale services are acknowledged in scripts/railway-deploy-drift-baseline.json against this issue, with an expiry so the suppression cannot outlive its cause.

Related

• ops(railway): merged seeder fixes can sit undeployed for hours (p90 19h, max 62h) — Watch Paths SKIPs matching pushes and nothing alarms on the drift #6141 — the finding and the detection.
• ops(railway): umami silently stopped auto-deploying — three consecutive merges produced no deployment record, and nothing detects the drift #6064 — the webhook-side sibling: umami has empty watch paths and is stale anyway, which is why the drift check is deliberately agnostic about cause.