Repository navigation
feat: support kubernetes as a new provider [multi-cluster] - #140
Conversation
Signed-off-by: kerthcet <kerthcet@gmail.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
2b54eb3 to
ad390f2
Compare
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Cherry-picked from hiverge/nebula c0c90be, without chart/values-prod.yaml, which only exists in that fork. Signed-off-by: ky-hiverge <kante@hiverge.ai>
…fresh Namespace isolation across clusters, on once every cluster in the table has cidrs and one is marked host: - Provision fences each namespace before the real pod exists: an Admin ClusterNetworkPolicy on the host lets X reach only X's own remote pods, and in the target an Admin fence plus a NetworkPolicy admit only X and the host. - The host's default deny of the targets' ranges ships as config/kubernetes-provider/cluster-network-policy.yaml, off until enabled in its kustomization; keep its networks equal to the non-host clusters' cidrs. - The webhook refuses hostNetwork Pods and keeps the opt-in label on a placed Pod, as isolation selects the instance by it. Failover: - A Karpenter refusal of the Pod's own spec (an untolerated taint, incompatible requirements) blocks the candidate for that workload only (DenyWorkload, keyed by util.WorkloadKey); any other refusal blocks it for everyone. - A lapsed block that nobody probes within 2 minutes is dropped. Also: refresh the EKS token's credentials before it outlives them and drop it on a 401; nebula_provider_auth_failures_total and nebula_provider_auth_failing from the poll loop; Nebula's own labels are never copied to the real pod; $(VAR) in env values stays literal; docs/multi-cluster.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Admitted Pods with unsupported user volumes can run without their mounts, silently losing configuration or storage; these volumes must be rejected before provisioning while allowing the injected service-account projection.
1 open issue
What changed in this PR
Adds an EKS-backed Kubernetes provider to Nebula’s placement system, supporting the multi-cluster work in #141. Workloads run as real pods in target clusters while host-cluster placeholders track their status.
Changes:
- Adds cluster configuration, EKS authentication, Karpenter integration, and namespace isolation policies.
- Extends lifecycle handling, workload-scoped failover, secret-aware environment delivery, and provider-auth metrics.
- Adds deployment configuration, documentation, tests, and instance-ownership checks during teardown.
| File | Description |
|---|---|
pkg/vnode/status.go |
Maps provisioning, unready, and classified failure states. |
pkg/vnode/metrics_test.go |
Updates GPU test fixtures. |
pkg/vnode/handler.go |
Handles asynchronous failures, readiness deadlines, and polling races. |
pkg/vnode/handler_test.go |
Tests new lifecycle and blocklisting behavior. |
pkg/vnode/env.go |
Separates configuration and Secret-derived environment values. |
pkg/vnode/env_test.go |
Tests environment splitting and precedence. |
pkg/util/pod.go |
Adds workload identity and renames the terminal-phase helper. |
pkg/util/pod_test.go |
Tests workload keys and terminal phases. |
pkg/util/accelerator.go |
Requires GPU type and resource count together. |
pkg/util/accelerator_test.go |
Tests rejection of incomplete GPU requests. |
pkg/provider/runpod/runpod.go |
Adopts revised environment and block-scope APIs. |
pkg/provider/runpod/runpod_test.go |
Updates environment and classification tests. |
pkg/provider/provider.go |
Extends provider contracts for regional capabilities and failures. |
pkg/provider/modal/modal.go |
Shortens provisioning timeout and adopts shared APIs. |
pkg/provider/modal/modal_test.go |
Tests timeout classification and updated contracts. |
pkg/provider/modal/client.go |
Classifies credential-mint deadlines as capacity shortages. |
pkg/provider/kubernetes/pod.go |
Builds target pods and environment Secrets. |
pkg/provider/kubernetes/node_autoscaler.go |
Defines the node-autoscaler interface. |
pkg/provider/kubernetes/netpol.go |
Creates and repairs namespace isolation policies. |
pkg/provider/kubernetes/netpol_test.go |
Tests isolation configuration and policy creation. |
pkg/provider/kubernetes/karpenter.go |
Adds node selection and scheduling-refusal classification. |
pkg/provider/kubernetes/eks.go |
Adds EKS clients, token caching, and regional offerings lookup. |
pkg/provider/kubernetes/eks_test.go |
Tests token format, refresh, and invalidation. |
pkg/provider/kubernetes/eks_offerings_test.go |
Tests regional GPU-family filtering. |
pkg/provider/kubernetes/config.go |
Loads and validates the cluster table. |
pkg/provider/errors.go |
Adds request/spec classifications and capacity-tier context. |
pkg/provider/errors_test.go |
Tests revised error classification. |
pkg/provider/aws/aws.go |
Adopts revised environment and block-scope APIs. |
pkg/provider/aws/aws_test.go |
Updates environment and classification tests. |
pkg/metrics/placement.go |
Uses the shared provider-label constant. |
pkg/metrics/labels.go |
Centralizes the provider-label name. |
pkg/metrics/auth.go |
Adds provider-auth failure metrics. |
pkg/metrics/auth_test.go |
Tests auth metric transitions. |
pkg/failover/blocklist.go |
Adds explicit deny policies and single-probe retries. |
pkg/failover/blocklist_test.go |
Tests scoped blocks and retry-probe behavior. |
internal/webhook/v1/pod_webhook.go |
Tightens GPU, host-network, and opt-out validation. |
internal/webhook/v1/pod_webhook_test.go |
Tests the new admission restrictions. |
internal/controller/samples_provisionable_test.go |
Accounts for Kubernetes GPU packing in sample checks. |
internal/controller/pod_placement_helpers.go |
Filters regional capabilities and admits workload-aware probes. |
internal/controller/pod_placement_controller.go |
Updates the blocklist interface for probe admission. |
internal/controller/pod_placement_controller_test.go |
Tests regional filtering and workload-scoped failover. |
internal/controller/nodeclaim_controller.go |
Verifies instance ownership before teardown. |
internal/controller/nodeclaim_controller_test.go |
Tests rejection of forged instance IDs. |
go.sum |
Updates AWS dependency checksums. |
go.mod |
Adds EKS and STS dependencies and updates AWS modules. |
docs/README.md |
Links the multi-cluster guide. |
docs/multi-cluster.md |
Documents setup, placement, isolation, and troubleshooting. |
docs/metrics.md |
Documents auth metrics and GPU validation behavior. |
docs/architecture.md |
Updates environment and asynchronous failover documentation. |
config/samples/nodepool.yaml |
Adds a Kubernetes placement example. |
config/rbac/role.yaml |
Allows Kubernetes virtual-node impersonation. |
config/manager/manager.yaml |
Mounts the optional cluster table and supplies its flag. |
config/kubernetes-provider/remote-rbac.yaml |
Defines target-cluster permissions. |
config/kubernetes-provider/kustomization.yaml |
Generates the cluster ConfigMap. |
config/kubernetes-provider/clusters.yaml |
Provides cluster configuration and isolation examples. |
config/kubernetes-provider/cluster-network-policy.yaml |
Supplies the host-side default-deny policy. |
config/default/kustomization.yaml |
Adds an optional Kubernetes-provider configuration entry. |
cmd/main.go |
Registers the provider and exposes cluster configuration. |
Review effort: Balanced
| // but the service-account token, which is simply not mounted here. | ||
| spec.ServiceAccountName, spec.DeprecatedServiceAccount = "", "" | ||
| spec.AutomountServiceAccountToken = ptrTo(false) | ||
| spec.Volumes = nil |
|
/lgtm |
InftyAI-Agent
left a comment
There was a problem hiding this comment.
Approved: PR has both lgtm and approved labels

What this PR does / why we need it
Which issue(s) this PR fixes
Fixes #141
Special notes for your reviewer
Does this PR introduce a user-facing change?