Skip to content

fix(operator): retry shared Gateway listener patches on conflict and after a failed pass - #261

Open
dcmcand wants to merge 1 commit into
mainfrom
fix/shared-listener-conflict-retry
Open

dcmcand wants to merge 1 commit into
mainfrom
fix/shared-listener-conflict-retry

Conversation

@dcmcand

@dcmcand dcmcand commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Slice 5: Shared listener patches survive a write conflict

Found during the live pre-flight for #233 on a fresh NIC cluster, where the internal and external endpoints share nebari-gateway. The operator patched llm-https, then failed on llm-internal-https:

patching internal Gateway listener: updating Gateway envoy-gateway-system/nebari-gateway: Operation cannot be fulfilled on gateways.gateway.networking.k8s.io "nebari-gateway": the object has been modified

It retried only on the 5 minute tick, so for 5 minutes every internal route was Accepted=False/NoMatchingParent. The Gateway has other writers (its managedFields list argocd-controller and envoy-gateway); which one conflicted here is inferred, not observed.

Contracts touched: the shared listener patch and the singleton's retry after a failed reconcile (internal behavior, no API or value change).
Depends on: none. Independent of the #173 / #258 stack and of #260.

What changes

  • patchGatewayListener runs Get, merge and Update inside retry.RetryOnConflict(retry.DefaultBackoff, ...), so every attempt re-reads the Gateway. DefaultBackoff makes 4 tries with sleeps of 10, 50 and 250 ms. NotFound still returns nil, an unchanged listener still skips the Update, and non-conflict errors fail at once.
  • Start retries a failed reconcile after RetryInterval (new field, default 10 s), doubling on each further failure up to Interval (5 min), and returns to Interval after a success. Zero or negative intervals use the defaults. cmd/main.go is unchanged.

Validation

Step Result
make test exit 0
TestClusterTLSSingleton_PatchListenerConflict (4 rows) and TestClusterTLSSingleton_RetryAfterFailure (7 rows) all PASS with -race; each row was shown to fail with its code path removed
golangci-lint v2.4.0 under Go 1.24.5 (as lint.yaml) 0 issues.

Live, on a NIC Hetzner cluster: the in-cluster operator was scaled to 0 and the operator binary run against the cluster with webhooks off. Each run removed both pack listeners, then started 16 loops annotating nebari-gateway (about 16 writes per second, sustained). That flood is far heavier than a real cluster sees; it is there to make conflicts reproducible.

Binary Runs Both listeners on the first pass Outcome otherwise
main 3 0 3 fell back to the 5 minute tick (one was the original symptom)
conflict retry only, DefaultRetry 5 4 1 fell back to the tick
conflict retry only, DefaultBackoff 12 10 2 fell back to the tick
this PR (conflict retry + fast retry after failure) 6 4 2 used "will retry in 10s/20s/40s"; one patched llm-internal-https 34 s after start while the flood was still running; none waited for the tick
this PR, final build, no flood 2 2 no "will retry in"
this PR, final build, flood 3 3 none waited for the tick

No fixed conflict-retry budget wins against a sustained flood, which is why a failed pass is now retried after 10 s instead of 5 minutes.

…after a failed pass

The shared-TLS singleton patched the llm-https and llm-internal-https
listeners with a single Get then Update, and retried a failed reconcile
only on the next 5 minute tick. When another writer touched the shared
Gateway in between, the Update failed with a conflict and the listener
stayed missing for up to 5 minutes, leaving its routes NoMatchingParent.

patchGatewayListener now runs the Get, merge and Update inside
retry.RetryOnConflict(retry.DefaultBackoff, ...), so each attempt
re-reads the Gateway. Other errors still fail at once.

Start now retries a failed reconcile after RetryInterval (default 10s),
doubling on each further failure up to Interval, and returns to Interval
after a success. Zero or negative intervals use the defaults instead of
making the timer spin.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant