Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Models can be loaded from HuggingFace (default) or mounted as OCI/modelcar image
- **Envoy Gateway installed and configured for AI Gateway integration** - `extensionApis.enableBackend`, `extensionManager` pointing at the AI Gateway controller service, and `backendResources` allowing `inference.networking.k8s.io/InferencePool`. This is a **hard requirement**; without it, the routing layer 404s at runtime. Apply it by committing [`examples/envoy-gateway-overlay.yaml`](examples/envoy-gateway-overlay.yaml) to your NIC config repo as `values/envoy-gateway/overlays/20-ai-gateway.yaml`, which survives `nic deploy --regen-apps`. ([`examples/envoy-gateway.yaml`](examples/envoy-gateway.yaml) is the whole-Application form, for clusters NIC does not manage.) See the [Installation guide](https://packs.nebari.dev/llm-serving-pack/installation/#6-reconfigure-envoy-gateway-with-ai-gateway-extension-wiring) for details.
- Envoy AI Gateway 1.1.0 installed. **Note**: the `envoyAIGateway.install` flag in this chart is not yet implemented - tracked in [#44](https://github.com/nebari-dev/llm-serving-pack/issues/44). Until that is done, install it manually as an ArgoCD app (see [examples/envoy-ai-gateway.yaml](examples/envoy-ai-gateway.yaml)).
- [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension) (GIE) CRDs installed - v1.5.0 validated. The pack needs `InferencePool`; the llm-d End-Point Picker crashloops without it.
- A cert-manager `ClusterIssuer` the operator can use for the shared-hostname Certificate. Default expected name is `letsencrypt-production`; override with `platform.tls.clusterIssuer` in the chart values.
- A cert-manager `ClusterIssuer` the operator can use for the shared-hostname Certificate. Default expected name is `letsencrypt-issuer`, the one NIC creates; override with `platform.tls.clusterIssuer` in the chart values.
- DNS for `llm.<baseDomain>` and `llm-internal.<baseDomain>` resolving to the shared Gateway's load balancer (a wildcard CNAME on the base domain is the simplest way). Required for HTTP-01 issuance on the shared Certificate.
- A StorageClass for model storage (EFS, EBS gp3, or any CSI-backed storage that can provision PVCs large enough for your models).

Expand Down Expand Up @@ -68,7 +68,7 @@ spec:
# Must name a cert-manager ClusterIssuer that already
# exists on the cluster. HTTP-01 is the assumed challenge
# type; no wildcards required.
clusterIssuer: letsencrypt-production
clusterIssuer: letsencrypt-issuer

defaults:
storage:
Expand Down Expand Up @@ -196,7 +196,7 @@ response = client.chat.completions.create(
| `platform.baseDomain` | Base domain for the Nebari deployment (required) | `""` |
| `platform.gateway.external.name` | Name of the external Gateway resource | `nebari-gateway` |
| `platform.gateway.external.namespace` | Namespace of the external Gateway | `envoy-gateway-system` |
| `platform.gateway.internal.name` | Name of the internal Gateway resource | `nebari-internal-gateway` |
| `platform.gateway.internal.name` | Name of the internal Gateway resource | `nebari-gateway` |
| `platform.gateway.internal.namespace` | Namespace of the internal Gateway | `envoy-gateway-system` |
| `auth.oidc.issuerURL` | OIDC issuer URL (static value, or read from Secret if empty) | `""` |
| `auth.oidc.groupsClaim` | JWT claim containing group memberships | `groups` |
Expand Down
14 changes: 10 additions & 4 deletions charts/nebari-llm-serving/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,11 @@ platform:
external:
name: nebari-gateway
namespace: envoy-gateway-system
# The defaults are the Gateway NIC creates. Both endpoints share it;
# the operator adds a separate listener for each hostname. Point
# internal at another Gateway to split the traffic.
internal:
name: nebari-internal-gateway
name: nebari-gateway
namespace: envoy-gateway-system
# manageSharedListeners controls whether the operator patches HTTPS
# listeners for llm.<baseDomain> and llm-internal.<baseDomain> onto
Expand All @@ -37,9 +40,12 @@ platform:
# clusterIssuer - the cert-manager ClusterIssuer used to issue the
# shared TLS certificate covering llm.<baseDomain> and
# llm-internal.<baseDomain>. HTTP-01 is the assumed challenge type;
# wildcard SANs are not supported. Override when running against a
# staging issuer or a different production issuer.
clusterIssuer: letsencrypt-production
# wildcard SANs are not supported. The default is the ClusterIssuer
# NIC creates today; override it for a staging issuer or any other
# issuer. Once NIC publishes its issuer to packs (platform values,
# nebari-dev/nebari-infrastructure-core#781), the chart will read it
# from there.
clusterIssuer: letsencrypt-issuer
# secretName - when set, bring-your-own-certificate mode: the operator
# does NOT create a cert-manager Certificate (and deletes one it
# previously managed). The shared Gateway listeners terminate TLS with
Expand Down
4 changes: 2 additions & 2 deletions docs/src/content/docs/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,10 @@ Pass these with `--set key=value` or in a `values.yaml` override file.
| `platform.baseDomain` | `""` | **Required.** Base domain for the Nebari deployment (e.g. `your-cluster.example.com`). |
| `platform.gateway.external.name` | `nebari-gateway` | Name of the external Envoy Gateway resource. |
| `platform.gateway.external.namespace` | `envoy-gateway-system` | Namespace of the external Envoy Gateway. |
| `platform.gateway.internal.name` | `nebari-internal-gateway` | Name of the internal Envoy Gateway resource. |
| `platform.gateway.internal.name` | `nebari-gateway` | Name of the internal Envoy Gateway resource. The default shares NIC's Gateway with the external endpoint; the operator adds a separate listener for each hostname. |
| `platform.gateway.internal.namespace` | `envoy-gateway-system` | Namespace of the internal Envoy Gateway. |
| `platform.gateway.manageSharedListeners` | `true` | When `true`, the operator patches HTTPS listeners for `llm.<baseDomain>` and `llm-internal.<baseDomain>` onto the shared Gateways. Set to `false` if a cluster admin owns the listeners out-of-band. |
| `platform.tls.clusterIssuer` | `letsencrypt-production` | The cert-manager `ClusterIssuer` used to issue the shared TLS certificate. Override when using a staging issuer. |
| `platform.tls.clusterIssuer` | `letsencrypt-issuer` | The cert-manager `ClusterIssuer` used to issue the shared TLS certificate. The default is the issuer NIC creates; override it for a staging issuer or any other issuer. |
| `platform.tls.secretName` | `""` | Bring-your-own-certificate mode. When set, no cert-manager Certificate is created; the operator expects a pre-provisioned `kubernetes.io/tls` Secret of this name in the operator namespace. Use for air-gapped or private-CA clusters where ACME issuance is not possible. cert-manager is not required when this is set. |

### Auth / OIDC
Expand Down
9 changes: 4 additions & 5 deletions docs/src/content/docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,9 @@ In addition you need:
HTTP-01 ACME challenges will run against `llm.<baseDomain>` and
`llm-internal.<baseDomain>`, so both must resolve before you start.
- **A Cluster Issuer** that cert-manager can use for HTTP-01. NIC
ships one; the validated install used `letsencrypt-issuer`
(the chart's default `letsencrypt-production` is a different name -
you will set `platform.tls.clusterIssuer` to match in section 8).
ships `letsencrypt-issuer`, which is the chart's default
`platform.tls.clusterIssuer`. If your cluster uses a different
issuer, set it in section 8.
- **A cluster-config git repo** that ArgoCD's AppProject is configured
to read from. New `Application` manifests in this runbook will be
committed there, under a path of your own choosing (shown throughout
Expand Down Expand Up @@ -1071,8 +1071,7 @@ spec:
namespace: envoy-gateway-system
manageSharedListeners: true
tls:
# NIC ships ClusterIssuer "letsencrypt-issuer", not the
# chart default "letsencrypt-production".
# NIC's ClusterIssuer, and the chart default.
clusterIssuer: letsencrypt-issuer
defaults:
storage:
Expand Down
7 changes: 3 additions & 4 deletions docs/src/content/docs/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ For a full production-grade setup (air-gapped clusters, HuggingFace token secret
- **Envoy Gateway** configured for AI Gateway integration - `extensionApis.enableBackend`, `extensionManager` pointing at the AI Gateway controller service, and `backendResources` allowing `inference.networking.k8s.io/InferencePool`. Commit [`examples/envoy-gateway-overlay.yaml`](https://github.com/nebari-dev/llm-serving-pack/blob/main/examples/envoy-gateway-overlay.yaml) to your NIC config repo as `values/envoy-gateway/overlays/20-ai-gateway.yaml`. ([Installation &sect;6](/installation/#6-reconfigure-envoy-gateway-with-ai-gateway-extension-wiring) for full wiring details)
- **Envoy AI Gateway** v1.1.0 installed. **Note**: the `envoyAIGateway.install` chart flag is not yet implemented ([#44](https://github.com/nebari-dev/llm-serving-pack/issues/44)). Install manually via ArgoCD (see [`examples/envoy-ai-gateway.yaml`](https://github.com/nebari-dev/llm-serving-pack/blob/main/examples/envoy-ai-gateway.yaml)). ([Installation &sect;5](/installation/#5-install-the-envoy-ai-gateway-controller))
- [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension) CRDs, v1.5.0 validated - `InferencePool` is the one the pack requires ([Installation &sect;4](/installation/#4-install-ai-gateway-and-inference-extension-crds))
- A cert-manager `ClusterIssuer` for the shared TLS certificate (default name: `letsencrypt-production`; override with `platform.tls.clusterIssuer`) ([Installation &sect;2.4](/installation/#24-confirm-the-cluster-issuer-is-ready))
- A cert-manager `ClusterIssuer` for the shared TLS certificate (default name: `letsencrypt-issuer`, the one NIC creates; override with `platform.tls.clusterIssuer`) ([Installation &sect;2.4](/installation/#24-confirm-the-cluster-issuer-is-ready))
- DNS for `llm.<baseDomain>` and `llm-internal.<baseDomain>` pointing at the shared Gateway load balancer ([Installation &sect;2.3](/installation/#23-confirm-dns-resolves-to-the-gateway-lb))
- A StorageClass that can provision PVCs large enough for your models (EFS, EBS gp3, or equivalent) ([Installation &sect;1](/installation/#1-what-this-runbook-assumes); [sizing guidance](/shared-storage/))

Expand Down Expand Up @@ -48,8 +48,7 @@ spec:
platform:
baseDomain: "your-cluster.example.com"
# Gateway names below must match the Envoy Gateways in your cluster.
# This example points both endpoints at one shared gateway; the chart
# default for the internal gateway is "nebari-internal-gateway".
# These are the chart defaults: both endpoints share NIC's gateway.
gateway:
external:
name: nebari-gateway
Expand All @@ -59,7 +58,7 @@ spec:
namespace: envoy-gateway-system
manageSharedListeners: true
tls:
clusterIssuer: letsencrypt-production
clusterIssuer: letsencrypt-issuer

defaults:
storage:
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@ kubectl -n envoy-gateway-system logs \
| grep CERTIFICATE_VERIFY_FAILED
```

`<gateway>` is the Gateway the request came in through, named in the chart's `platform.gateway` values: by default `nebari-gateway` for `llm.<baseDomain>` and `nebari-internal-gateway` for `llm-internal.<baseDomain>`. Keep `--tail=-1`: with a label selector, `kubectl logs` otherwise prints only the last 10 lines of each pod.
`<gateway>` is the Gateway the request came in through, named in the chart's `platform.gateway` values: by default `nebari-gateway` for both `llm.<baseDomain>` and `llm-internal.<baseDomain>`. Keep `--tail=-1`: with a label selector, `kubectl logs` otherwise prints only the last 10 lines of each pod.

By default the gateway accepts only the public CA roots in the Envoy proxy image, so a certificate from a private or corporate CA needs `spec.provider.tls.caConfigMapName` (see [Self-hosted servers](/self-hosted/)). When that field is set, check the policy status:

Expand Down
26 changes: 26 additions & 0 deletions docs/src/content/docs/upgrading.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,32 @@ the separate AI Gateway CRD chart. The pack does not install those controllers.

Newest first. A release with no entry needs only the routine steps.

### Upgrading from 0.1.4 or earlier: gateway and issuer defaults

Two defaults now match what NIC creates:

| Value | Old default | New default |
|-------|-------------|-------------|
| `platform.gateway.internal.name` | `nebari-internal-gateway` | `nebari-gateway` |
| `platform.tls.clusterIssuer` | `letsencrypt-production` | `letsencrypt-issuer` |

If you set both values, nothing changes. If you left one unset and your
cluster relies on the old name, **set that one explicitly before
upgrading**. Add only the value you left unset; the example shows both:

```yaml
platform:
gateway:
internal:
name: nebari-internal-gateway
tls:
clusterIssuer: letsencrypt-production
```

Without it, the operator moves the internal endpoint's route and listener
to `nebari-gateway`, or points the shared Certificate at an issuer your
cluster may not have, and it can no longer be renewed.

### Upgrading from 0.1.4 or earlier: the pack's namespace

The pack's namespace no longer follows `fullnameOverride` or `nameOverride`.
Expand Down
4 changes: 2 additions & 2 deletions operator/internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ type OperatorConfig struct {
// the shared-TLS certificate covering llm.<baseDomain> and
// llm-internal.<baseDomain>. HTTP-01 is the assumed challenge type;
// wildcards are not supported by this pack.
ClusterIssuerName string // LLM_CLUSTER_ISSUER_NAME (default: "letsencrypt-production")
ClusterIssuerName string // LLM_CLUSTER_ISSUER_NAME (default: "letsencrypt-issuer")
// TLSSecretName, when non-empty, switches shared TLS to
// bring-your-own-certificate mode: the operator does NOT create a
// cert-manager Certificate (and deletes the one it previously
Expand Down Expand Up @@ -137,7 +137,7 @@ func LoadFromEnv() (*OperatorConfig, error) {
DefaultStorageClassName: os.Getenv("LLM_DEFAULT_STORAGE_CLASS_NAME"),
APIKeysNamespace: os.Getenv("LLM_API_KEYS_NAMESPACE"),
OperatorNamespace: os.Getenv("POD_NAMESPACE"),
ClusterIssuerName: getEnvOrDefault("LLM_CLUSTER_ISSUER_NAME", "letsencrypt-production"),
ClusterIssuerName: getEnvOrDefault("LLM_CLUSTER_ISSUER_NAME", "letsencrypt-issuer"),
TLSSecretName: os.Getenv("LLM_TLS_SECRET_NAME"),
ManageSharedListeners: getEnvBool("LLM_MANAGE_SHARED_LISTENERS", true),
UsageMeteringEnabled: getEnvBool("LLM_USAGE_METERING_ENABLED", false),
Expand Down
6 changes: 3 additions & 3 deletions operator/internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ func TestLoadFromEnv(t *testing.T) {
DefaultEPPImage: "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0",
APIKeysNamespace: "",
OperatorNamespace: "llm-serving",
ClusterIssuerName: "letsencrypt-production",
ClusterIssuerName: "letsencrypt-issuer",
ManageSharedListeners: true,
},
},
Expand Down Expand Up @@ -161,7 +161,7 @@ func TestLoadFromEnv(t *testing.T) {
DefaultEPPImage: "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0",
APIKeysNamespace: "",
OperatorNamespace: "llm-serving",
ClusterIssuerName: "letsencrypt-production",
ClusterIssuerName: "letsencrypt-issuer",
ManageSharedListeners: true,
},
},
Expand Down Expand Up @@ -196,7 +196,7 @@ func TestLoadFromEnv(t *testing.T) {
DefaultServingImage: "ghcr.io/llm-d/llm-d-cuda:v0.7.0",
DefaultEPPImage: "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0",
OperatorNamespace: "",
ClusterIssuerName: "letsencrypt-production",
ClusterIssuerName: "letsencrypt-issuer",
ManageSharedListeners: true,
},
},
Expand Down
Loading