Problem
The cross-user aggregate alert added in #5998 fires during healthy peak traffic, not just during outages.
api/analytics-health.js:21-22:
const MIN_WRITES = 5;
const MIN_FAILURE_RATE = 0.5;
A 5-sample denominator inside a 60-second window cannot separate a dead collector from ordinary ad-blocker attrition — and this audience's baseline block rate sits right at the 0.5 floor.
Evidence from the 2026-08-01 incident
WORLDMONITOR-Y6 / Y7 sat pegged at exactly 60 events/hour from 19:00–23:00 UTC. That is the ceiling: the SET NX latch permits one alert per 60s window, so the floor was being crossed in every window for five consecutive hours.
Mapping the alert against the outages confirmed in Postgres (19:55:56 → 20:51:34 and 22:35:47 → 00:04:15):
| Hour (UTC) |
Alert |
Reality |
| 19h |
60 |
real — outage began 19:56 |
| 20h |
59 |
real — outage until 20:51 |
| 21h |
60 |
false — 67,394 events landed, the day's busiest hour, zero gap |
| 22h |
60 |
real — outage began 22:36 |
| 23h |
60 |
real — full blackout |
The alert did earn its keep: it surfaced a genuine, previously-silent outage within 9 minutes. But it also pegged at maximum through the single healthiest hour of the day, and it cannot tell an operator which is which.
Why this matters
The next real outage lands in an inbox already saturated with false positives. The alert's value depends entirely on its silence meaning something.
Suggested direction
- Raise
MIN_WRITES well above 5 so the denominator can actually support a rate judgement
- Lift
MIN_FAILURE_RATE above the measured ad-block baseline, or calibrate it from observed healthy-hour rates rather than a round 0.5
- Consider requiring N consecutive breached windows before emitting, so a single noisy minute cannot trip it
Related: the underlying collector OOM, and the fact that queue-overflow is downstream of network/timeout rather than an independent failure (dominant client-side kind at 664/978 in WORLDMONITOR-Y3).
Problem
The cross-user aggregate alert added in #5998 fires during healthy peak traffic, not just during outages.
api/analytics-health.js:21-22:A 5-sample denominator inside a 60-second window cannot separate a dead collector from ordinary ad-blocker attrition — and this audience's baseline block rate sits right at the 0.5 floor.
Evidence from the 2026-08-01 incident
WORLDMONITOR-Y6 / Y7 sat pegged at exactly 60 events/hour from 19:00–23:00 UTC. That is the ceiling: the
SET NXlatch permits one alert per 60s window, so the floor was being crossed in every window for five consecutive hours.Mapping the alert against the outages confirmed in Postgres (
19:55:56 → 20:51:34and22:35:47 → 00:04:15):The alert did earn its keep: it surfaced a genuine, previously-silent outage within 9 minutes. But it also pegged at maximum through the single healthiest hour of the day, and it cannot tell an operator which is which.
Why this matters
The next real outage lands in an inbox already saturated with false positives. The alert's value depends entirely on its silence meaning something.
Suggested direction
MIN_WRITESwell above 5 so the denominator can actually support a rate judgementMIN_FAILURE_RATEabove the measured ad-block baseline, or calibrate it from observed healthy-hour rates rather than a round 0.5Related: the underlying collector OOM, and the fact that
queue-overflowis downstream ofnetwork/timeoutrather than an independent failure (dominant client-side kind at 664/978 in WORLDMONITOR-Y3).