fix(analytics): judge collector health against its own baseline instead of a fixed 0.5 (#6026)

[Yigtwxx]

Summary

Fixes #6026.

api/analytics-health.js:47-49 decided the whole thing on a raw quotient:

export function shouldEmitAggregateAlert(writes, failures) {
  return writes >= MIN_WRITES && failures / writes >= MIN_FAILURE_RATE;
}

Two separate reasons that cannot separate a dead collector from ordinary attrition.

The denominator carried no weight. failures / writes reads identically for 5/5 and 5000/5000, and those are not the same claim. At the MIN_WRITES = 5 floor the 95% Wilson half-width at the worst case p = 0.5 is 0.368 — the rate was not resolved at all, so the floor was not measuring anything.

The threshold was absolute, and the quantity it measures is not. 0.5 is a property of this audience's ad-blocker baseline, not of collector health. The issue's own evidence is the proof: 21:00 UTC on 2026-08-01 was the busiest hour of the day with zero gap, 67,394 events landed, and the alert still pegged at the once-per-window ceiling of 60. No sample-size fix reaches that case — the window was large and its rate really was above 0.5. It was simply normal.

What this changes

1. Rate judgements read the low end of a Wilson score interval, not the point estimate. Sample size becomes part of the claim instead of being invisible. 17/31 is 54.8% and clears the old gate; its lower bound is 0.4035 and it no longer does.

2. MIN_WRITES is derived rather than picked. The half-width of the interval at p = 0.5 is z * sqrt(0.25 / n), so n = 31 is the smallest denominator that resolves the rate to within ±0.15 (0.1477 — n = 30 gives 0.1502). That is the resolution needed to tell an ad-blocker baseline from a collector that has stopped accepting writes. The comment records the derivation so the next person can re-derive it rather than inherit another round number.

3. A window is compared against that cohort's own observed baseline. Per-cohort day-scoped counters accumulate alongside the window counters, and the previous — complete — day supplies the baseline. The alert fires only when the window's lower bound clears the baseline's upper bound, which is the issue's second suggested direction ("calibrate it from observed healthy-hour rates rather than a round 0.5") with no operator number required. Against the 21:00 hour:

	lower	upper
window, 3,000/5,000 (60%)	0.5886
baseline, 610,000/1,000,000 (61%)		0.6108

0.5886 > 0.6108 is false, so it stays silent. A genuine collector death — 190/200 — has a lower bound of 0.9181 and clears it immediately.

4. Three consecutive breached windows are required before Sentry hears about it. The stored value is count:bucket, so a gap of two or more windows resets the run rather than letting alternating breached/healthy windows accumulate into a false streak, and re-entering the same bucket is idempotent (every isolate in a window reads the same prior value and computes the same successor). Per the issue's own timings this costs at most two extra minutes — the real outage was surfaced in 9.

5. The Sentry payload now carries what the floors would have to be tuned against: failureRateLowerBound, baselineFailureRate, baselineFailureRateUpperBound, baselineWriteCount, consecutiveBreachedWindows, and the two constants in force.

Cost is one Redis round trip, unchanged. The window counters, the day accumulator, the previous day's baseline and the streak all ride the existing pipeline (6 commands to 13). A healthy window still makes exactly one call — there is a test for that, because the endpoint is on the hot path.

Design decisions left for you

• MIN_FAILURE_RATE is deliberately still 0.5. It is now a backstop rather than the primary gate — it only stops an alert on a deployment whose baseline is so low that a statistically real excursion is still operationally uninteresting. Raising it is the one part of fix(analytics): collector aggregate-health alert fires during healthy peak traffic (MIN_WRITES=5 / MIN_FAILURE_RATE=0.5) #6026 that needs the measured ad-block baseline, and picking a number for it here would be inventing a calibration from data I do not have. Happy to set it if you have the figure.
• MIN_CONSECUTIVE_BREACHED_WINDOWS = 3 trades two minutes of detection latency for immunity to a single noisy window. If you would rather have the minute back, 2 still kills the single-window case.
• The baseline is the previous calendar day, not a rolling 24h window. A rolling window needs either EXPIRE ... NX or hourly sub-buckets; a day-scoped key means re-arming its TTL on every write cannot extend the window it measures, which seemed worth more than the boundary alignment. A long outage moves a full day's rate by a few points, so it does not blind the alert; the alternative — using the previous hour — would, since an outage hour would become the baseline for the next one.
• The Sentry message string changed to describe what now triggers it. The explicit fingerprint is untouched, so existing grouping is preserved.
• Cold start: until a full baseline day exists, conditions (1) and (2) carry the decision, which is the pre-fix(analytics): collector aggregate-health alert fires during healthy peak traffic (MIN_WRITES=5 / MIN_FAILURE_RATE=0.5) #6026 behaviour with an honest denominator. A deployment therefore never starts louder than it does today.

Verification

api/analytics-health.test.mjs    23 pass, 0 fail   (4 tests before this PR)
npm run test:sidecar            329 pass, 0 fail   (306 on origin/main)

Every guard is mutation-proven:

Mutant	Red
MIN_WRITES back to the pre-fix 5	3
rate judged on the point estimate instead of the interval	1
baseline comparison dropped	2
consecutive-window requirement dropped	2
streak no longer resets on a gap	2
streak double-counts inside one window	1
under-resolved baseline accepted	1

No survivors.

The fixtures are built through a pipelineResults() helper shaped exactly like the reply the endpoint reads, so a test cannot assert against a Redis state the real call could not produce. Two of the cases are the incident itself rather than synthetic values: an hour of baseline-matching windows that must stay silent for all 60 of them, and a three-window run that must stay silent for the first two.

Other gates:

npm run typecheck        clean
npx biome check          clean (3 files)
npm run lint:boundaries  no violations
check-unicode-safety     2651 files scanned, clean
esbuild edge bundle      api/analytics-health.js bundles clean
                         (--bundle --format=esm --platform=browser)

npm run test:data: identical failure set to origin/main — 47 failing test names on both, comm diff empty in both directions.

One thing found along the way

api/analytics-health.test.mjs was not run by anything. It is in neither test:data (which globs tests/ plus a fixed list) nor test:sidecar, and no workflow picks it up, so the four assertions it already carried had never gated a merge. This PR adds it to test:sidecar next to the other api/*.test.mjs entries — that is the 306 to 329 above.

Out of scope

• src/services/analytics-collector-transport.ts:396 (ENVIRONMENT_NOISE_MIN_WRITES = 5) is a separate per-page fallback floor on the client, not the cross-user aggregate this issue is about. Left alone deliberately; changing it would alter what the browser reports rather than how the server judges it.
• The queue-overflow kind being downstream of network/timeout rather than independent, noted at the end of fix(analytics): collector aggregate-health alert fires during healthy peak traffic (MIN_WRITES=5 / MIN_FAILURE_RATE=0.5) #6026. That is a taxonomy change in the transport, with its own blast radius.
• The underlying collector OOM (fix(analytics): umami collector OOMs under peak load — 144 min of silent analytics blackout on 2026-08-01 #6024) and the alerting-target change (ops(analytics): alert on umami memory, not Railway deployment status — status has masked two silent collector deaths #6025). This PR only stops the existing alert from crying wolf.

Type of change

• Bug fix
• New feature
• New data source / feed
• New map layer
• Refactor / code cleanup
• Documentation
• CI / Build / Infrastructure

Affected areas

• Map / Globe
• News panels / RSS feeds
• AI Insights / World Brief
• Market Radar / Crypto
• Desktop app (Tauri)
• API endpoints (/api/*) — /api/analytics-health; request and response contracts are unchanged, only the internal alert decision
• Config / Settings
• Other: package.json test wiring

Checklist

• Tested on worldmonitor.app variant — N/A. The change is entirely inside the server-side alert decision; there is no user-visible surface, and reproducing it in production would mean waiting for another collector outage. Verified through the exported decision functions with mutation proof that each guard has teeth.
• Tested on tech.worldmonitor.app variant (if applicable) — N/A, no variant-specific behaviour.
• New RSS feed domains added to api/rss-proxy.js allowlist (if adding feeds) — N/A, no feeds added.
• No API keys or secrets committed
• TypeScript compiles without errors (npm run typecheck)

Documentation Alignment Checklist

N/A — this PR does not publish or change a documentation claim. It changes when an internal Sentry warning is emitted; no methodology, API/MCP contract, generated doc or example changes. Listed for completeness:

• Claim ledger attached or linked — N/A, no documented claim changes.
• All required Audit Council role signoffs attached — N/A, no methodology or contract change.
• Generated docs regenerated from proto where applicable — N/A, no proto change.
• Fixture-backed examples recomputed — N/A, no published example depends on this endpoint.
• Redis writers/readers enumerated for every documented key — the endpoint gains two key families under the existing analytics:collector-health:v1:{env}: prefix, neither of which is documented anywhere: …:day:{dayIndex}:{cohort}:{writes,failures} (written and read here only, TTL 48h) and …:{cohort}:streak (written and read here only, TTL 180s). The existing …:{bucket}:{cohort}:{writes,failures,reported} keys are unchanged in shape and TTL.

[vercel]

@Yigtwxx is attempting to deploy a commit to the World Monitor Team on Vercel.

A member of the Team first needs to authorize it.

[koala73]

Thanks for this @Yigtwxx — the diagnosis in the description is right, and the move from a raw quotient to a sample-size-aware interval is the correct shape. The Wilson helper, the derived floor, and the per-cohort baseline are all good ideas. Reviewed it in depth; there's one blocker and one design question, plus some test gaps that hid both.

Blocker: a saturated baseline makes the alarm permanently unsatisfiable

shouldEmitAggregateAlert ends in:

return observed > wilsonBounds(baseline.failures, baseline.writes).upper;

wilsonBounds clamps upper with Math.min(1, …). So once a baseline approaches 100% failure, that comparison can never be true. Measured against a 600000/600000 baseline:

real outage    190/200      lower=0.9181   alert=FALSE
real outage  5000/5000      lower=0.9995   alert=FALSE
real outage 100000/100000   lower=1.0000   alert=FALSE

There are two ways to reach that state, and the first is self-inflicted:

1. The outage writes its own next-day baseline. The day accumulators are INCRBY'd unconditionally on every report (api/analytics-health.js:226-227), including reports from windows the alarm has already judged breached. A collector that stays dead across a UTC midnight publishes a ~100%-failure baseline, and at 00:00 UTC that becomes the thing the next day's windows must exceed. The alarm goes quiet precisely while the collector stays dead — which is the bug(analytics): Umami collector dead since 2026-07-20 17:05Z — OOM crash-loop, 502, ~4 days of product analytics lost (blinds #5534 funnel) #5565 failure mode (dead collector, ~4 days, unnoticed) this alert exists to catch.

2. It's reachable by anyone. bucket is server-derived, but the counters are not. At the endpoint's own limits — 60 req/min and MAX_COUNTER_DELTA 10,000 — one rate-limit bucket writes 600,000 baseline writes/minute; MIN_BASELINE_WRITES (620) clears in 0.06s. About a minute of unauthenticated traffic disables the alarm for 24h.

The same lever works in reverse, incidentally: three requests/day of {writes: 10000, failures: 1} drag a 61% baseline's upper bound from 0.6108 to 0.5930, at which point a genuinely healthy peak hour starts alerting again — i.e. it re-creates the exact #6026 noise this PR removes.

Suggested handling: give the baseline a veto ceiling. Above some upper-bound threshold (0.9 works) the baseline is no longer describing "normal", so it should lose its veto and let conditions (1) and (2) decide alone. That's strictly the safe direction — a saturated baseline can only ever have suppressed alerts. Worth pairing with excluding already-breached windows from the day accumulator, so the outage stops feeding the thing that later exonerates it.

Design question: does the baseline gate actually fix #6026?

A full day of counters collapses the baseline's Wilson interval onto its point estimate:

n=5,000      upper-mean = 0.01128
n=500,000    upper-mean = 0.00113
n=2,000,000  upper-mean = 0.00057

So condition (3) degenerates into window lower bound > day mean + ~0.001, and the only remaining slack is the window's own CI. Against a 61% day mean at n=1e6:

peak 62%, window n=20,000 -> ALERTS
peak 63%, window n= 5,000 -> ALERTS
peak 64%, window n= 1,000 -> ALERTS

The perverse part is that higher traffic tightens the window CI, so the gate gets more trigger-happy as volume rises — weakest during the busiest hour, which is exactly the #6026 incident condition. The 3-consecutive-window requirement doesn't help either, since a peak hour spans ~60 windows.

This isn't necessarily wrong — it depends on whether this deployment's peak-hour ad-block rate actually runs above its daily mean, which is your call and needs the measured data. But the current test only pins the 60%-vs-61% case, where peak sits below the mean, so it doesn't exercise the situation that caused the incident. Options: compare against the same hour-of-day from the baseline period, or widen the baseline interval by the between-hour variance rather than treating a whole day as one binomial sample.

The tests can't see either problem

Running mutants against the suite as it stands (23/23 green baseline):

• Six wiring mutants survive, because drive()'s fake dispatches purely on commands.length === WINDOW_COMMANDS and never inspects a command, while pipelineResults() returns values positionally. Survivors include reading the baseline from TODAY instead of yesterday (self-referential), swapping the window GETs, swapping the day INCRBY targets, and dropping the baseline TTL to 1s. pipelineResults compounds it by putting the same writes/failures values at offsets 0, 1, 4, 5, 6 and 7 — and counterResult accepts both the raw INCRBY number and the GET string, so those slots are mutually indistinguishable.

• Swapping .upper for .lower on the baseline comparison leaves all 23 tests green — as does relaxing > to >=. Every baseline fixture uses n = 100,000 or 1,000,000, where the interval is 0.0016–0.0051 wide, so the two endpoints are numerically indistinguishable. The PR's headline comparison is unverified by its own suite. A deliberately thin baseline (e.g. {writes: 620, failures: 380}, interval [0.5803, 0.6445]) with a window landing between the endpoints separates them.

To be clear, the suite is not toothless — the decision-function mutants your description lists really are killed (point-estimate instead of Wilson, streak 3→1, dropping the MIN_BASELINE_WRITES floor). The blind spots are the two areas that table never probed: the Redis wiring and the interval endpoint.

Smaller items

• The streak SET result is never checked. The claim pipeline issues SET streakKey (index 0) and SET reportedKey NX (index 1), but only claim?.[1] is inspected. A per-command {error} on the streak write is invisible, so the streak silently fails to persist and the next window restarts the three-minute detection clock — during the outage.
• {error} and a miss are conflated. counterResult/stringResult both map a per-command error and an absent key to null. For the window's own counters that means an unreadable Redis answers 204 (client reads it as accepted); for the streak read it means "no run in progress". Both should fail closed.
• advanceBreachStreak resets a valid run when the stored bucket is newer. advanceBreachStreak('9:501', 500) === 1 — a straggler that crossed the boundary discards a 9-window run. Note that returning the count alone isn't sufficient: the caller then persists ${count}:${bucket} with its own older bucket and the next window sees a gap anyway, so the function needs to control which bucket gets stored.
• MIN_WRITES is per-window with no accumulation across the streak, so a cohort under 31 writes/minute can be 100% dead and never alert (30/30 all-failing is silent), with nothing reporting that it's unmonitored. Worth checking real per-window volume for identify and critical-event before merging.
• The derivation comment names Wilson but quotes Wald. z * sqrt(0.25 / n) is the normal approximation; the quoted 0.1477 at n=31 and 0.1502 at n=30 are its values to 4 decimals. Under the Wilson formula the code actually uses, the same ±0.15 criterion gives n = 28. Keeping 31 is fine (it's the more conservative of the two) — the comment just needs to say which formula it is, so the next person can re-derive it.
• shouldEmitAggregateAlert's docstring says "three independent conditions", but they're short-circuited conjunctions with if (!baseline) return true; in the middle.
• package.json conflicts with main on the test:sidecar line; main has since added scripts/ais-relay-ingestion.test.cjs there. Worth resolving carefully so that entry isn't dropped.

Also worth saying: wiring api/analytics-health.test.mjs into test:sidecar is a real catch — it genuinely gated nothing before this PR.

Getting the fixes

I have all of the above except the two design items implemented and verified — 30 tests passing (was 23), with all nine mutants above confirmed killed, biome clean, and the file still bundling for the edge runtime. I couldn't push it here because "Allow edits by maintainers" is unchecked on this PR, so the branch is only writable by you.

If you tick that box I'll push the branch directly. Otherwise happy to attach the patch or open it as a separate branch you can pull from — whichever you prefer.

[Yigtwxx]

Box is ticked — the branch is writable by you now, push whenever suits you.

I rebased onto 028ecc229 first so your push lands on a clean base. The package.json conflict is resolved keeping both entries: main's scripts/ais-relay-ingestion.test.cjs stays where main put it, and api/analytics-health.test.mjs goes in before api/security/report.test.mjs. npm run test:sidecar is 338/338 locally.

The blocker

Confirmed, and it's not just "can never be true" — it's exactly unsatisfiable, independent of Math.min. At p = 1 the Wilson upper bound evaluates to exactly 1 on its own: centre = 1 + z²/2n, margin = z²/2n, so the numerator is (1 + z²/n) over a denominator of (1 + z²/n). Meanwhile the observed lower bound at p = 1 is 1/(1 + z²/n), strictly below 1 for every finite n. So no window — no denominator, no failure count, not even a perfect 100000/100000 — can clear a saturated baseline. Your measured table is the general case, not a numerical artifact.

The 0.9 veto ceiling is the right shape. Worth noting the pairing you suggest isn't optional: the ceiling alone only rescues the fully saturated case. A collector failing at 85% still publishes an 85% baseline the next day, which sits under the ceiling and keeps its veto, so the outage still exonerates itself — just quietly. Excluding already-breached windows from the day accumulator is the part that actually closes that path.

The derivation comment

Confirmed, and your n is right. z * sqrt(0.25 / n) is the normal approximation, not Wilson. Under the Wilson form the code actually uses, the ±0.15 criterion first holds at n = 28; n = 27 gives 0.1509. Keeping 31 and fixing the comment to name the formula is the right call.

The design question

This is the one I can't settle from the code, and I think the framing points somewhere slightly different from either option you listed.

The day aggregate's Wilson interval measures sampling error only. It treats a day as one homogeneous binomial, when the quantity that actually moves is the audience mix by hour — so the interval collapses toward ~0.001 while the real between-hour dispersion stays exactly where it was. That's the mechanism behind the perverse part you spotted: more samples buy a tighter estimate of a quantity that was never the source of the variation, so the gate tightens as volume rises.

Which suggests the statistical machinery on the baseline side isn't earning its place at day scale however it's sliced. The cheapest honest version is probably to stop treating (3) as an interval comparison at all and give it an explicit effect size — window lower bound > baseline rate + delta, or > baseline rate × k — with delta or k set from your measured peak-to-mean spread. Hour-of-day baselines are the more precise answer, but they cost 24 keys per cohort and a day longer to warm; and the between-hour variance option needs per-hour counters anyway to estimate that variance, so it arrives at the same storage shape by a longer route.

All three want the same input from you that MIN_FAILURE_RATE does, along with the real per-window volume for identify and critical-event for the MIN_WRITES accumulation point. Tell me which shape you want and I'll implement it on top of whatever you push.