Repository navigation
Conversation
…after a failed pass The shared-TLS singleton patched the llm-https and llm-internal-https listeners with a single Get then Update, and retried a failed reconcile only on the next 5 minute tick. When another writer touched the shared Gateway in between, the Update failed with a conflict and the listener stayed missing for up to 5 minutes, leaving its routes NoMatchingParent. patchGatewayListener now runs the Get, merge and Update inside retry.RetryOnConflict(retry.DefaultBackoff, ...), so each attempt re-reads the Gateway. Other errors still fail at once. Start now retries a failed reconcile after RetryInterval (default 10s), doubling on each further failure up to Interval, and returns to Interval after a success. Zero or negative intervals use the defaults instead of making the timer spin.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slice 5: Shared listener patches survive a write conflict
Found during the live pre-flight for #233 on a fresh NIC cluster, where the internal and external endpoints share
nebari-gateway. The operator patchedllm-https, then failed onllm-internal-https:It retried only on the 5 minute tick, so for 5 minutes every internal route was
Accepted=False/NoMatchingParent. The Gateway has other writers (its managedFields listargocd-controllerandenvoy-gateway); which one conflicted here is inferred, not observed.Contracts touched: the shared listener patch and the singleton's retry after a failed reconcile (internal behavior, no API or value change).
Depends on: none. Independent of the #173 / #258 stack and of #260.
What changes
patchGatewayListenerruns Get, merge and Update insideretry.RetryOnConflict(retry.DefaultBackoff, ...), so every attempt re-reads the Gateway.DefaultBackoffmakes 4 tries with sleeps of 10, 50 and 250 ms. NotFound still returns nil, an unchanged listener still skips the Update, and non-conflict errors fail at once.Startretries a failed reconcile afterRetryInterval(new field, default 10 s), doubling on each further failure up toInterval(5 min), and returns toIntervalafter a success. Zero or negative intervals use the defaults.cmd/main.gois unchanged.Validation
make testTestClusterTLSSingleton_PatchListenerConflict(4 rows) andTestClusterTLSSingleton_RetryAfterFailure(7 rows)-race; each row was shown to fail with its code path removedlint.yaml)0 issues.Live, on a NIC Hetzner cluster: the in-cluster operator was scaled to 0 and the operator binary run against the cluster with webhooks off. Each run removed both pack listeners, then started 16 loops annotating
nebari-gateway(about 16 writes per second, sustained). That flood is far heavier than a real cluster sees; it is there to make conflicts reproducible.mainDefaultRetryDefaultBackoffllm-internal-https34 s after start while the flood was still running; none waited for the tickNo fixed conflict-retry budget wins against a sustained flood, which is why a failed pass is now retried after 10 s instead of 5 minutes.