Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion docs/src/content/docs/architecture.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,17 @@ policy writes set `ApplyFailed` and trigger controller retries. A missing
while policy creation is retried every 15 seconds. `Applied` means the policies
were written, not that Envoy has finished applying them to traffic.

Once the model is serving, the reconciler applies its `InferencePool` and
`AIGatewayRoute`s and reports each outcome: `InferencePoolReady`,
`ExternalEndpointReady` and `InternalEndpointReady`. On an `LLMModel` the
endpoint conditions cover the route only; the policies are reported by
`SecurityPoliciesReady`. A disabled endpoint reports `False` with
`EndpointDisabled`, which is not a failure. A failed apply does not stop the
reconcile: the condition reads `GatewayCRDUnavailable` (retried every 15
seconds) or `ApplyFailed` (retried every minute), and a model that would
otherwise be `Ready` reports `Degraded`, because its pods serve but nothing
can route to them.

### Internal endpoint

For JupyterLab, in-cluster chat UIs, notebooks.
Expand Down Expand Up @@ -307,7 +318,7 @@ Semantics worth knowing:
- **`modelsOwnedBy` is set on both routes.** Each endpoint's `/v1/models` aggregates only the routes declaring its own hostname, so each endpoint carries its own catalog without depending on the other endpoint being enabled.
- **Catch-all conflicts are not validated.** Two catch-all PassthroughModels on the same gateway race for unclaimed model ids the same way two identical HTTPRoutes would; the webhook validates per-CR only, consistent with the LLMModel webhook dropping cross-CR collision checks. Document one catch-all per cluster as the supported shape.
- **Names are unique across both kinds.** Both an LLMModel and a PassthroughModel derive their api-keys Secret name as `<name>-api-keys`, so sharing a name would make the two controllers fight over one Secret. Both webhooks reject a CR whose name is already taken by the other kind in the same namespace. The key-manager cache also keeps kind-prefixed entries as defense in depth.
- **Status.** Phase is `Ready`/`Error` with conditions `BackendConfigured`, `ExternalEndpointReady`, and `InternalEndpointReady`. Secret-backed providers also report the informational `UpstreamCredentialResolved` condition without blocking `Ready`: `False` with `SecretNotFound` or `APIKeyMissing` means the Secret or its non-empty `apiKey` entry is missing; `Unknown` with `LookupFailed` means the read failed and is retried after one minute. Workload identity has no upstream Secret, so this condition is omitted. Missing AI Gateway CRDs surface as `GatewayCRDUnavailable` with a 15-second requeue; real apply failures report `ApplyFailed` with a one-minute requeue. Both controllers share the reason string and the interval, but the LLMModel reconciler applies this convention to its `SecurityPolicy` only: a failed `AIGatewayRoute` apply is logged and skipped with no condition and no requeue, so a served model can report `Ready` while unroutable. Converging the remaining LLMModel applies is tracked in [#109](https://github.com/nebari-dev/llm-serving-pack/issues/109).
- **Status.** Phase is `Ready`/`Error` with conditions `BackendConfigured`, `ExternalEndpointReady`, and `InternalEndpointReady`. Secret-backed providers also report the informational `UpstreamCredentialResolved` condition without blocking `Ready`: `False` with `SecretNotFound` or `APIKeyMissing` means the Secret or its non-empty `apiKey` entry is missing; `Unknown` with `LookupFailed` means the read failed and is retried after one minute. Workload identity has no upstream Secret, so this condition is omitted. Missing AI Gateway CRDs surface as `GatewayCRDUnavailable` with a 15-second requeue; real apply failures report `ApplyFailed` with a one-minute requeue. Both controllers share the reasons and the intervals. On a PassthroughModel each endpoint condition covers its route and its policy together.
- **Generated gateway resources are reconciled on-change-only.** Because the AI Gateway kinds (`Backend`, `BackendTLSPolicy`, `AIServiceBackend`, `BackendSecurityPolicy`, `AIGatewayRoute`, `SecurityPolicy`) are applied as unstructured objects, `SetupWithManager` registers `For(&PassthroughModel{})`, a watch on operator-managed api-keys Secrets, and a targeted watch that maps provider credential Secret events to the PassthroughModels that reference them. It does not `Owns(...)` the generated gateway resources. If a generated resource is deleted out-of-band, it is not recreated until the next PassthroughModel edit, api-keys Secret change in the namespace, or referenced provider credential Secret change.

---
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -363,7 +363,7 @@ The operator writes these fields to `status`; they are read-only.
| `status.endpoints.external` | string | Actual external endpoint URL once provisioned. |
| `status.endpoints.internal` | string | Actual internal endpoint URL once provisioned. |
| `status.modelSize` | string | Actual model size after download. |
| `status.conditions` | []Condition | Standard Kubernetes conditions (type, status, reason, message). |
| `status.conditions` | []Condition | `SecurityPoliciesReady`, `InferencePoolReady`, `ExternalEndpointReady` and `InternalEndpointReady` (type, status, reason, message). A failed gateway apply turns `Ready` into `Degraded`; see [Architecture](/architecture/#dual-endpoint-auth). |

---

Expand Down
137 changes: 137 additions & 0 deletions operator/internal/controller/conditions.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
/*
Copyright 2026.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controller

import (
"time"

"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
)

// Status condition types shared by the LLMModel and PassthroughModel
// reconcilers.
//
// Both controllers apply gateway resources whose CRDs are installed
// separately from this pack (section 4 of the install runbook). A missing CRD
// does not fail a reconcile: everything the pack owns still converges, and the
// applies succeed on a later pass once the CRDs exist. It must be visible,
// though, because a route that was never created shows up only as an HTTP 404
// or 500 at request time.
const (
// CondBackendConfigured covers a PassthroughModel's provider plumbing:
// Backend, BackendTLSPolicy, AIServiceBackend, BackendSecurityPolicy.
CondBackendConfigured = "BackendConfigured"
// CondExternalEndpointReady covers the external endpoint. On a
// PassthroughModel that is the route and its apiKeyAuth policy; on an
// LLMModel it is the route only, because the policy is applied earlier and
// reported by SecurityPoliciesReady.
CondExternalEndpointReady = "ExternalEndpointReady"
// CondInternalEndpointReady covers the internal endpoint: the route and its
// JWT policy on a PassthroughModel, the route only on an LLMModel.
CondInternalEndpointReady = "InternalEndpointReady"
// CondInferencePoolReady covers the InferencePool an LLMModel needs. It is
// separate from the endpoint conditions because it comes from a different
// CRD bundle: a cluster can have the AI Gateway CRDs and not the Inference
// Extension's, and the llm-d End-Point Picker crashloops without
// InferencePool while routing looks fine.
CondInferencePoolReady = "InferencePoolReady"
)

const (
reasonApplied = "Applied"
reasonDisabled = "EndpointDisabled"
// applyFailedRequeue is how long to wait before retrying a gateway apply
// that failed for a reason other than a missing CRD.
applyFailedRequeue = time.Minute
)

// conditionFor turns the outcome of an apply into a status condition. A nil
// err reports Applied with okMessage. A missing CRD reports
// GatewayCRDUnavailable; any other error reports ApplyFailed. Both carry the
// error text, which is where a reader learns which resource failed.
func conditionFor(condType string, err error, okMessage string) metav1.Condition {
if err != nil {
reason := reasonApplyFailed
if meta.IsNoMatchError(err) {
reason = reasonGatewayCRDUnavailable
}
return metav1.Condition{
Type: condType,
Status: metav1.ConditionFalse,
Reason: reason,
Message: err.Error(),
}
}
return metav1.Condition{
Type: condType,
Status: metav1.ConditionTrue,
Reason: reasonApplied,
Message: okMessage,
}
}

// disabledCondition reports an endpoint the spec switched off. It is False
// like a failure, but for a deliberate reason, and hasGatewayFailure does not
// count it.
func disabledCondition(condType string) metav1.Condition {
return metav1.Condition{
Type: condType,
Status: metav1.ConditionFalse,
Reason: reasonDisabled,
Message: "endpoint disabled in spec",
}
}

// hasGatewayFailure reports whether any condition records a gateway apply
// that failed or is waiting on a missing CRD.
func hasGatewayFailure(conds []metav1.Condition) bool {
return gatewayRequeue(conds) > 0
}

// gatewayRequeue returns how soon to retry after the gateway applies recorded
// in conds, or 0 when none failed. A missing CRD retries on the shorter CRD
// interval, since installing the CRD is enough for the next pass to succeed.
func gatewayRequeue(conds []metav1.Condition) time.Duration {
var requeue time.Duration
for _, c := range conds {
if c.Status != metav1.ConditionFalse {
continue
}
switch c.Reason {
case reasonGatewayCRDUnavailable:
return gatewayCRDRequeue
case reasonApplyFailed:
requeue = applyFailedRequeue
}
}
return requeue
}

// phaseWithGatewayFailure downgrades a Ready model to Degraded when one of
// its gateway resources could not be applied. The workload is serving but
// nothing can route to it, which is what Degraded describes; Error would
// overstate it. Other phases already say something more specific and are
// left alone.
func phaseWithGatewayFailure(phase llmv1alpha1.LLMModelPhase, conds []metav1.Condition) llmv1alpha1.LLMModelPhase {
if phase == llmv1alpha1.PhaseReady && hasGatewayFailure(conds) {
return llmv1alpha1.PhaseDegraded
}
return phase
}
183 changes: 183 additions & 0 deletions operator/internal/controller/conditions_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
/*
Copyright 2026.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controller

import (
"errors"
"testing"
"time"

"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime/schema"

llmv1alpha1 "github.com/nebari-dev/nebari-llm-serving-pack/operator/api/v1alpha1"
)

func TestConditionFor(t *testing.T) {
noMatch := &meta.NoKindMatchError{GroupKind: schema.GroupKind{Group: "aigateway.envoyproxy.io", Kind: "AIGatewayRoute"}}

tests := []struct {
name string
err error
wantStatus metav1.ConditionStatus
wantReason string
wantMessage string
}{
{
name: "no error reports Applied with the ok message",
err: nil,
wantStatus: metav1.ConditionTrue,
wantReason: reasonApplied,
wantMessage: "route applied",
},
{
name: "missing CRD reports GatewayCRDUnavailable",
err: noMatch,
wantStatus: metav1.ConditionFalse,
wantReason: reasonGatewayCRDUnavailable,
wantMessage: noMatch.Error(),
},
{
name: "other errors report ApplyFailed and carry the message",
err: errors.New("admission webhook denied the request"),
wantStatus: metav1.ConditionFalse,
wantReason: reasonApplyFailed,
wantMessage: "admission webhook denied the request",
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := conditionFor(CondExternalEndpointReady, tt.err, "route applied")
if got.Type != CondExternalEndpointReady {
t.Errorf("Type = %q, want %q", got.Type, CondExternalEndpointReady)
}
if got.Status != tt.wantStatus {
t.Errorf("Status = %q, want %q", got.Status, tt.wantStatus)
}
if got.Reason != tt.wantReason {
t.Errorf("Reason = %q, want %q", got.Reason, tt.wantReason)
}
if got.Message != tt.wantMessage {
t.Errorf("Message = %q, want %q", got.Message, tt.wantMessage)
}
})
}
}

func TestGatewayRequeue(t *testing.T) {
applied := metav1.Condition{Type: CondExternalEndpointReady, Status: metav1.ConditionTrue, Reason: reasonApplied}
failed := metav1.Condition{Type: CondInternalEndpointReady, Status: metav1.ConditionFalse, Reason: reasonApplyFailed}
crdMissing := metav1.Condition{Type: CondInferencePoolReady, Status: metav1.ConditionFalse, Reason: reasonGatewayCRDUnavailable}
disabled := disabledCondition(CondInternalEndpointReady)

tests := []struct {
name string
conds []metav1.Condition
wantFailed bool
wantRequeue time.Duration
}{
{name: "no conditions", conds: nil, wantFailed: false, wantRequeue: 0},
{name: "all applied", conds: []metav1.Condition{applied}, wantFailed: false, wantRequeue: 0},
{
// A disabled endpoint is False, but deliberately so. Treating it as a
// failure would degrade every model that turns an endpoint off.
name: "disabled endpoint is not a failure",
conds: []metav1.Condition{applied, disabled},
wantFailed: false,
wantRequeue: 0,
},
{name: "apply failure retries after a minute", conds: []metav1.Condition{applied, failed}, wantFailed: true, wantRequeue: time.Minute},
{name: "missing CRD retries on the CRD interval", conds: []metav1.Condition{crdMissing}, wantFailed: true, wantRequeue: gatewayCRDRequeue},
{
// A missing CRD is the faster case to recover from (install it and
// the next pass succeeds), so it sets the interval when both occur.
name: "missing CRD wins over an apply failure",
conds: []metav1.Condition{failed, crdMissing},
wantFailed: true,
wantRequeue: gatewayCRDRequeue,
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := hasGatewayFailure(tt.conds); got != tt.wantFailed {
t.Errorf("hasGatewayFailure() = %v, want %v", got, tt.wantFailed)
}
if got := gatewayRequeue(tt.conds); got != tt.wantRequeue {
t.Errorf("gatewayRequeue() = %v, want %v", got, tt.wantRequeue)
}
})
}
}

func TestPhaseWithGatewayFailure(t *testing.T) {
failed := []metav1.Condition{{Type: CondExternalEndpointReady, Status: metav1.ConditionFalse, Reason: reasonApplyFailed}}
crdMissing := []metav1.Condition{{Type: CondInferencePoolReady, Status: metav1.ConditionFalse, Reason: reasonGatewayCRDUnavailable}}
ok := []metav1.Condition{{Type: CondExternalEndpointReady, Status: metav1.ConditionTrue, Reason: reasonApplied}}

tests := []struct {
name string
phase llmv1alpha1.LLMModelPhase
conds []metav1.Condition
want llmv1alpha1.LLMModelPhase
}{
{
// The pod is up but nothing can route to it. That is what Degraded
// means; Error would overstate it.
name: "Ready degrades when an apply failed",
phase: llmv1alpha1.PhaseReady,
conds: failed,
want: llmv1alpha1.PhaseDegraded,
},
{
name: "Ready degrades when a gateway CRD is missing",
phase: llmv1alpha1.PhaseReady,
conds: crdMissing,
want: llmv1alpha1.PhaseDegraded,
},
{
name: "Ready stays Ready when every apply succeeded",
phase: llmv1alpha1.PhaseReady,
conds: ok,
want: llmv1alpha1.PhaseReady,
},
{
// Degraded from determinePhase (some replicas down) already says
// the model is not fully healthy.
name: "Degraded stays Degraded",
phase: llmv1alpha1.PhaseDegraded,
conds: failed,
want: llmv1alpha1.PhaseDegraded,
},
{
name: "Error is not softened to Degraded",
phase: llmv1alpha1.PhaseError,
conds: failed,
want: llmv1alpha1.PhaseError,
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := phaseWithGatewayFailure(tt.phase, tt.conds); got != tt.want {
t.Errorf("phaseWithGatewayFailure(%q) = %q, want %q", tt.phase, got, tt.want)
}
})
}
}
Loading
Loading