Skip to content

feat: support kubernetes as a new provider [multi-cluster] - #140

Merged
InftyAI-Agent merged 22 commits into
InftyAI:mainfrom
kerthcet:feat/kubernetes-provider
Nov 12, 2026
Merged

InftyAI-Agent merged 22 commits into
InftyAI:mainfrom
kerthcet:feat/kubernetes-provider

Conversation

@kerthcet

@kerthcet kerthcet commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

What this PR does / why we need it

Which issue(s) this PR fixes

Fixes #141

Special notes for your reviewer

Does this PR introduce a user-facing change?


Signed-off-by: kerthcet <kerthcet@gmail.com>
@InftyAI-Agent InftyAI-Agent added needs-triage Indicates an issue or PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. do-not-merge/needs-kind Indicates a PR lacks a label and requires one. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 22, 2026
@coderabbitai

coderabbitai Bot commented Oct 22, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
@kerthcet
kerthcet force-pushed the feat/kubernetes-provider branch from 2b54eb3 to ad390f2 Compare October 23, 2026 10:33
kerthcet and others added 13 commits October 23, 2026 18:05
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
Cherry-picked from hiverge/nebula c0c90be, without chart/values-prod.yaml,
which only exists in that fork.

Signed-off-by: ky-hiverge <kante@hiverge.ai>
@kerthcet kerthcet changed the title feat: support kubernetes as a new provider feat: support kubernetes as a new provider [multi-cluster] Nov 2, 2026
…fresh

Namespace isolation across clusters, on once every cluster in the table has
cidrs and one is marked host:
- Provision fences each namespace before the real pod exists: an Admin
  ClusterNetworkPolicy on the host lets X reach only X's own remote pods, and
  in the target an Admin fence plus a NetworkPolicy admit only X and the host.
- The host's default deny of the targets' ranges ships as
  config/kubernetes-provider/cluster-network-policy.yaml, off until enabled in
  its kustomization; keep its networks equal to the non-host clusters' cidrs.
- The webhook refuses hostNetwork Pods and keeps the opt-in label on a placed
  Pod, as isolation selects the instance by it.

Failover:
- A Karpenter refusal of the Pod's own spec (an untolerated taint,
  incompatible requirements) blocks the candidate for that workload only
  (DenyWorkload, keyed by util.WorkloadKey); any other refusal blocks it for
  everyone.
- A lapsed block that nobody probes within 2 minutes is dropped.

Also: refresh the EKS token's credentials before it outlives them and drop it
on a 401; nebula_provider_auth_failures_total and nebula_provider_auth_failing
from the poll loop; Nebula's own labels are never copied to the real pod;
$(VAR) in env values stays literal; docs/multi-cluster.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kerthcet
kerthcet marked this pull request as ready for review November 12, 2026 19:56
Copilot AI balanced review requested due to automatic review settings November 12, 2026 19:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Admitted Pods with unsupported user volumes can run without their mounts, silently losing configuration or storage; these volumes must be rejected before provisioning while allowing the injected service-account projection.

1 open issue
What changed in this PR

Adds an EKS-backed Kubernetes provider to Nebula’s placement system, supporting the multi-cluster work in #141. Workloads run as real pods in target clusters while host-cluster placeholders track their status.

Changes:

  • Adds cluster configuration, EKS authentication, Karpenter integration, and namespace isolation policies.
  • Extends lifecycle handling, workload-scoped failover, secret-aware environment delivery, and provider-auth metrics.
  • Adds deployment configuration, documentation, tests, and instance-ownership checks during teardown.
File Description
pkg/​vnode/​status.go Maps provisioning, unready, and classified failure states.
pkg/​vnode/​metrics_test.go Updates GPU test fixtures.
pkg/​vnode/​handler.go Handles asynchronous failures, readiness deadlines, and polling races.
pkg/​vnode/​handler_test.go Tests new lifecycle and blocklisting behavior.
pkg/​vnode/​env.go Separates configuration and Secret-derived environment values.
pkg/​vnode/​env_test.go Tests environment splitting and precedence.
pkg/​util/​pod.go Adds workload identity and renames the terminal-phase helper.
pkg/​util/​pod_test.go Tests workload keys and terminal phases.
pkg/​util/​accelerator.go Requires GPU type and resource count together.
pkg/​util/​accelerator_test.go Tests rejection of incomplete GPU requests.
pkg/​provider/​runpod/​runpod.go Adopts revised environment and block-scope APIs.
pkg/​provider/​runpod/​runpod_test.go Updates environment and classification tests.
pkg/​provider/​provider.go Extends provider contracts for regional capabilities and failures.
pkg/​provider/​modal/​modal.go Shortens provisioning timeout and adopts shared APIs.
pkg/​provider/​modal/​modal_test.go Tests timeout classification and updated contracts.
pkg/​provider/​modal/​client.go Classifies credential-mint deadlines as capacity shortages.
pkg/​provider/​kubernetes/​pod.go Builds target pods and environment Secrets.
pkg/​provider/​kubernetes/​node_autoscaler.go Defines the node-autoscaler interface.
pkg/​provider/​kubernetes/​netpol.go Creates and repairs namespace isolation policies.
pkg/​provider/​kubernetes/​netpol_test.go Tests isolation configuration and policy creation.
pkg/​provider/​kubernetes/​karpenter.go Adds node selection and scheduling-refusal classification.
pkg/​provider/​kubernetes/​eks.go Adds EKS clients, token caching, and regional offerings lookup.
pkg/​provider/​kubernetes/​eks_test.go Tests token format, refresh, and invalidation.
pkg/​provider/​kubernetes/​eks_offerings_test.go Tests regional GPU-family filtering.
pkg/​provider/​kubernetes/​config.go Loads and validates the cluster table.
pkg/​provider/​errors.go Adds request/spec classifications and capacity-tier context.
pkg/​provider/​errors_test.go Tests revised error classification.
pkg/​provider/​aws/​aws.go Adopts revised environment and block-scope APIs.
pkg/​provider/​aws/​aws_test.go Updates environment and classification tests.
pkg/​metrics/​placement.go Uses the shared provider-label constant.
pkg/​metrics/​labels.go Centralizes the provider-label name.
pkg/​metrics/​auth.go Adds provider-auth failure metrics.
pkg/​metrics/​auth_test.go Tests auth metric transitions.
pkg/​failover/​blocklist.go Adds explicit deny policies and single-probe retries.
pkg/​failover/​blocklist_test.go Tests scoped blocks and retry-probe behavior.
internal/​webhook/​v1/​pod_webhook.go Tightens GPU, host-network, and opt-out validation.
internal/​webhook/​v1/​pod_webhook_test.go Tests the new admission restrictions.
internal/​controller/​samples_provisionable_test.go Accounts for Kubernetes GPU packing in sample checks.
internal/​controller/​pod_placement_helpers.go Filters regional capabilities and admits workload-aware probes.
internal/​controller/​pod_placement_controller.go Updates the blocklist interface for probe admission.
internal/​controller/​pod_placement_controller_test.go Tests regional filtering and workload-scoped failover.
internal/​controller/​nodeclaim_controller.go Verifies instance ownership before teardown.
internal/​controller/​nodeclaim_controller_test.go Tests rejection of forged instance IDs.
go.sum Updates AWS dependency checksums.
go.mod Adds EKS and STS dependencies and updates AWS modules.
docs/​README.md Links the multi-cluster guide.
docs/​multi-cluster.md Documents setup, placement, isolation, and troubleshooting.
docs/​metrics.md Documents auth metrics and GPU validation behavior.
docs/​architecture.md Updates environment and asynchronous failover documentation.
config/​samples/​nodepool.yaml Adds a Kubernetes placement example.
config/​rbac/​role.yaml Allows Kubernetes virtual-node impersonation.
config/​manager/​manager.yaml Mounts the optional cluster table and supplies its flag.
config/​kubernetes-provider/​remote-rbac.yaml Defines target-cluster permissions.
config/​kubernetes-provider/​kustomization.yaml Generates the cluster ConfigMap.
config/​kubernetes-provider/​clusters.yaml Provides cluster configuration and isolation examples.
config/​kubernetes-provider/​cluster-network-policy.yaml Supplies the host-side default-deny policy.
config/​default/​kustomization.yaml Adds an optional Kubernetes-provider configuration entry.
cmd/​main.go Registers the provider and exposes cluster configuration.

Review effort: Balanced

// but the service-account token, which is simply not mounted here.
spec.ServiceAccountName, spec.DeprecatedServiceAccount = "", ""
spec.AutomountServiceAccountToken = ptrTo(false)
spec.Volumes = nil
@kerthcet

Copy link
Copy Markdown
Member Author

/lgtm
/kind feature

@InftyAI-Agent InftyAI-Agent added lgtm Looks good to me, indicates that a PR is ready to be merged. feature Categorizes issue or PR as related to a new feature. and removed do-not-merge/needs-kind Indicates a PR lacks a label and requires one. labels Nov 12, 2026

@InftyAI-Agent InftyAI-Agent left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved: PR has both lgtm and approved labels

@InftyAI-Agent
InftyAI-Agent merged commit 3c2b103 into InftyAI:main Nov 12, 2026
25 of 26 checks passed
@kerthcet
kerthcet deleted the feat/kubernetes-provider branch November 12, 2026 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. feature Categorizes issue or PR as related to a new feature. lgtm Looks good to me, indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a label and requires one. needs-triage Indicates an issue or PR lacks a label and requires one.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support kubernetes multi-cluster

4 participants