fix: damp waypoint HPA scale-down (PICCHU-INV-MESH-9)

[ddbenson]

Draft. The rendered spec is tested and the mechanism is well evidenced, but the runtime arithmetic in the "Needs a reviewer's eyes" section below should be confirmed before this merges — it changes scaling behaviour for every ambient app on all four production clusters at once.

Problem

EnsureWaypointHPA builds the waypoint HPA but never sets spec.behavior, so Kubernetes defaults apply:

	default
scaleUp.stabilizationWindowSeconds	0
scaleUp policies	100% / +4 pods per 15s
scaleDown.stabilizationWindowSeconds	300
scaleDown policy	100% per 15s

Scale up instantly, then five minutes later drop up to everything in one step. That is a sawtooth generator.

Every waypoint pod removed terminates the gRPC connections routed through it. A waypoint is a shared L7 proxy, so one aggressive step is felt by every caller of the namespace simultaneously:

rpc error: code = Unavailable desc = upstream connect error or disconnect/reset
before headers. reset reason: connection termination

Evidence from production

• 129 waypoint ScalingReplicaSet events fleet-wide in 30 minutes (09:45–10:15 UTC, Sept 8).

• 88 scale-downs of the 4→3 transition alone for pushnotifications-production on production-yolk-c, Sept 5–8.

• Near-lockstep across all four clusters — same global traffic curve, so the HPAs move together:

  21:19:16  yolk-b  scaled up   3 → 4
  21:19:27  yolk-a  scaled up   4 → 6
  21:19:39  yolk-d  scaled up   3 → 5
                                ← 5m 20s ≈ scaleDown.stabilizationWindowSeconds: 300
  21:24:39  yolk-b  scaled down 4 → 2
  21:24:48  yolk-a  scaled down 6 → 3
  21:25:02  yolk-d  scaled down 5 → 3
  

That 5m20s gap being exactly the 300s default is what pinned this down.

Downstream, this surfaces as notifications / users gRPC latency SLO burn alerts in #eng-watch-alerts several times a day, plus matching errors on notifications → pushnotifications / auth and no healthy upstream on users → postservice.

The change

spec.behavior.scaleDown with two policies and SelectPolicy: Max:

• Percent 25% / 60s — damping at higher replica counts.
• Pods 1 / 60s — keeps the HPA able to make progress at low counts. A percent-only rule rounds down to zero pods there (at 3 replicas, ceil(3 × 0.75) = 3) and would pin the waypoint above its floor indefinitely.

scaleUp is deliberately left on the Kubernetes defaults. Scaling up terminates no connections, so damping it buys nothing for this failure mode and would only slow the waypoint's response to a traffic spike. Setting only ScaleDown is safe — the HPA controller fills in scaleUp defaults when the side is nil.

Needs a reviewer's eyes

1. SelectPolicy: Max semantics. I read this as "allow the policy permitting the larger change", so at 3 replicas the pods policy (remove 1) wins over the percent policy (remove 0), and at 100 replicas the percent policy (remove 25) wins over pods (remove 1). If that is backwards, the pods policy becomes the binding constraint everywhere and scale-down from high replica counts gets very slow — cost, not availability. Worth confirming against the HPA controller source rather than taking my word for it.
2. Rate at the top of the range. Apps with maxReplicas: 100 coming back down to their floor take roughly 8–10 minutes under 25%/60s. That seems right for a proxy fleet but is a judgement call.
3. Interaction with PICCHU-INV-MESH-7. The waypoint PDB sets minAvailable = MinReplicas - 1. Slower scale-down means more sustained pods, which should only make evictions easier, but I have not exercised the two together.

Testing

Adds controllers/plan/ensureWaypointHPA_test.go — three tests covering the rendered behavior, the min/max/CPU defaults (including the MinReplicas >= 2 floor from PICCHU-INV-MESH-3), and the nil-spec no-op.

This closes half of the gap AGENTS.md records: "ensureWaypointHPA.go and ensureWaypointPDB.go have ZERO test files." ensureWaypointPDB.go still has none. Note the test pins the rendered spec only — it does not exercise the HPA controller's runtime interpretation of the policies, which is the part flagged above.

go build ./...                  ok
go vet ./controllers/plan/      ok
go test ./controllers/plan/     ok

Not in this PR

• Graceful drain on waypoint termination (terminationGracePeriodSeconds + preStop, or EXIT_ON_ZERO_ACTIVE_CONNECTIONS). This is the higher-value fix — it makes scale-downs harmless regardless of frequency, where this PR only reduces how often they happen. Separate PR, separate release.
• Orphaned waypoint HPAs. Apply returns early when p.HPA == nil and DeleteWaypointHPA only runs when ambient is switched off, so removing waypointHPA from an app.yml strands the HPA. writerstats has no waypointHPA in any target yet has one actively scaling it (2→3, 53 times since Sept 4).
• Default constant desync. AGENTS.md already flags that min/max/cpu defaults are duplicated between syncer.go and ensureWaypointHPA.go. This PR does not touch those, but the behavior constants are deliberately defined in one place only.

Related

Medium/mono#109365 — raises pushnotifications waypointHPA.minReplicas 2→6, sets targetCPUUtilizationPercentage and waypointResources. That is the app-level canary and can land first; this PR is the fleet-wide root-cause fix and should not land in the same observation window.

🤖 Generated with Claude Code