You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds an opt-in lambda provider for Lambda Cloud On-Demand GPU instances. Nebula launches one VM per workload, starts a public container with cloud-init, and reports readiness after the container responds to /healthz. The initial catalog includes one A10 SKU.
Records launch attempts on NodeClaims so a lost API response can be recovered without starting a second instance. The adapter confirms termination before the NodeClaim finalizer releases.
make test, race tests for the Lambda adapter and virtual kubelet, and lint passed.
On a K3s cluster, a real Lambda A10 instance in us-west-1 ran the sample container and returned its GPU model from /healthz. The manager adopted the same VM after a restart. Deleting the smoke-test Pod removed its NodeClaim, and the Lambda API confirmed termination.
A separate one-replica Deployment reached 1/1 Ready on a real A10 instance.
kubectl logs and kubectl exec are deferred for the Lambda provider.
Does this PR introduce a user-facing change?
Nebula can run single-container On-Demand GPU workloads on Lambda Cloud using the opt-in `lambda` provider. Configure the Lambda credentials and workspace settings in `docs/deploy.md` to enable it.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Lambda workload restrictions prevent failover to compatible providers, and filtering terminated VM records can leave Pods Pending indefinitely after a lost launch response.
The reason will be displayed to describe this comment to others. Learn more.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Cold Pod adoption repeatedly probes the entire VM fleet, causing N² health requests after a manager restart and allowing slow probes to delay unrelated Pod adoption.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Avoid fleet-wide health probes on each cold pod adoption
pkg/provider/lambda/lambda.go:508
Each cold GetPod calls instanceByClaim in pkg/vnode/handler.go, which invokes List for the entire provider. This loop then probes every active VM, so adopting N Pods after a manager restart performs N² /healthz requests rather than N; slow probes also delay adoption of unrelated Pods. Please use a claim-scoped, non-probing lookup for cold adoption and reserve fleet-wide health checks for the poll loop, or share a safely coalesced observation snapshot across these calls.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Active polling races can apply stale observations to newer Pod entries or drop previously observed VMs, causing Pods to fail incorrectly or remain Pending indefinitely.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
featureCategorizes issue or PR as related to a new feature.needs-priorityIndicates a PR lacks a label and requires one.needs-triageIndicates an issue or PR lacks a label and requires one.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it
Adds an opt-in
lambdaprovider for Lambda Cloud On-Demand GPU instances. Nebula launches one VM per workload, starts a public container with cloud-init, and reports readiness after the container responds to/healthz. The initial catalog includes one A10 SKU.Records launch attempts on NodeClaims so a lost API response can be recovered without starting a second instance. The adapter confirms termination before the NodeClaim finalizer releases.
Which issue(s) this PR fixes
Fixes #152
Special notes for your reviewer
make test, race tests for the Lambda adapter and virtual kubelet, and lint passed.us-west-1ran the sample container and returned its GPU model from/healthz. The manager adopted the same VM after a restart. Deleting the smoke-test Pod removed its NodeClaim, and the Lambda API confirmed termination.1/1 Readyon a real A10 instance.kubectl logsandkubectl execare deferred for the Lambda provider.Does this PR introduce a user-facing change?