Skip to content

fix(analytics): collector aggregate-health alert fires during healthy peak traffic (MIN_WRITES=5 / MIN_FAILURE_RATE=0.5) #6026

Description

@koala73

Problem

The cross-user aggregate alert added in #5998 fires during healthy peak traffic, not just during outages.

api/analytics-health.js:21-22:

const MIN_WRITES = 5;
const MIN_FAILURE_RATE = 0.5;

A 5-sample denominator inside a 60-second window cannot separate a dead collector from ordinary ad-blocker attrition — and this audience's baseline block rate sits right at the 0.5 floor.

Evidence from the 2026-08-01 incident

WORLDMONITOR-Y6 / Y7 sat pegged at exactly 60 events/hour from 19:00–23:00 UTC. That is the ceiling: the SET NX latch permits one alert per 60s window, so the floor was being crossed in every window for five consecutive hours.

Mapping the alert against the outages confirmed in Postgres (19:55:56 → 20:51:34 and 22:35:47 → 00:04:15):

Hour (UTC) Alert Reality
19h 60 real — outage began 19:56
20h 59 real — outage until 20:51
21h 60 false — 67,394 events landed, the day's busiest hour, zero gap
22h 60 real — outage began 22:36
23h 60 real — full blackout

The alert did earn its keep: it surfaced a genuine, previously-silent outage within 9 minutes. But it also pegged at maximum through the single healthiest hour of the day, and it cannot tell an operator which is which.

Why this matters

The next real outage lands in an inbox already saturated with false positives. The alert's value depends entirely on its silence meaning something.

Suggested direction

  • Raise MIN_WRITES well above 5 so the denominator can actually support a rate judgement
  • Lift MIN_FAILURE_RATE above the measured ad-block baseline, or calibrate it from observed healthy-hour rates rather than a round 0.5
  • Consider requiring N consecutive breached windows before emitting, so a single noisy minute cannot trip it

Related: the underlying collector OOM, and the fact that queue-overflow is downstream of network/timeout rather than an independent failure (dominant client-side kind at 664/978 in WORLDMONITOR-Y3).

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium priority, schedule when capacity allowsarea: infrastructureCables, pipelines, power grids, cyber threatsbugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions