Repository navigation
feat: skip a provider whose instance can't hold the Pod's ephemeral-storage - #147
Conversation
…torage A Modal sandbox can't request disk: it gets a fixed 512 GiB quota (the Python client drops ephemeral_disk for sandboxes, and the Go SDK has no field). A Pod asking more used to reach Modal and fail at Provision, and as that error blocks nothing, its controller's replacement went back to Modal and failed again, never trying a provider that could serve it. Capabilities.MaxDiskGiB (0 = no known limit) declares the most disk one instance can have, and placement skips a provider past it, recording disk_unsupported, so the Pod lands on the pool's next provider or stays gated as no_candidate. Modal sets 512; the others keep 0. Modal's own refusal stays, for a Pod placed by an older manager. The virtual node advertises MaxDiskGiB per Pod, summed over its 1k Pods (500 TiB on nebula-modal): the scheduler checks the node's total, so a node cannot cap one Pod's disk, and a flat 512Gi would admit a single such Pod. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the change to filter providers by Pod ephemeral-storage requirements.
No open issues 🎉
What changed in this PR
This PR makes placement skip providers that cannot satisfy a Pod’s ephemeral-storage needs, preventing repeated provisioning failures on Modal and allowing fallback to another provider.
Changes:
- Adds
MaxDiskGiBandServesDisk, with Modal advertising its fixed 512 GiB limit. - Filters placement candidates by disk capacity and adjusts virtual-node storage totals.
- Adds boundary and fallback tests, plus a documented
disk_unsupportedmetric reason.
| File | Description |
|---|---|
pkg/vnode/node.go |
Derives advertised storage from the provider’s per-instance limit and virtual Pod capacity. |
pkg/vnode/node_test.go |
Tests limited and unlimited virtual-node storage totals. |
pkg/provider/provider.go |
Defines the disk capability and eligibility check. |
pkg/provider/modal/modal.go |
Advertises Modal’s 512 GiB limit and explains the existing provisioning backstop. |
pkg/provider/modal/modal_test.go |
Checks agreement between the advertised limit and provisioning refusal. |
pkg/provider/capabilities_test.go |
Tests disk eligibility boundaries and unknown limits. |
pkg/metrics/placement.go |
Defines the disk_unsupported skip reason. |
internal/controller/pod_placement_helpers.go |
Skips providers whose disk limit is below the Pod’s needs. |
internal/controller/pod_placement_controller_test.go |
Tests boundary values, decimal-unit rounding, fallback, and retained gating. |
internal/controller/nodeclaim_controller_test.go |
Adds configurable disk limits to the shared fake provider. |
docs/metrics.md |
Documents the new disk-related skip reason. |
Review effort: Balanced
It is checked against one Pod's ephemeral-storage, and each Pod gets its own instance, so two Pods at the limit both fit; the old name read as a total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
AWS refuses ephemeral-storage past awsMaxDiskGiB (gp3's 16 TiB less the 30 GiB AMI root, 16354 GiB) at launch, but left MaxDiskGiBPerPod at zero, so placement read it as unlimited: a GPU Pod asking 16Ti was ungated onto AWS and failed there, and as that error blocks nothing, its replacement did the same. AWS now declares the limit, so such a Pod skips AWS for the next provider or stays gated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
|
/lgtm |
InftyAI-Agent
left a comment
There was a problem hiding this comment.
Approved: PR has both lgtm and approved labels
InftyAI-Agent
left a comment
There was a problem hiding this comment.
Approved: PR has both lgtm and approved labels
Problem
A Modal sandbox can't request disk. It gets a fixed per-container quota of 512 GiB: Modal's Python client sets
ephemeral_disk = Nonefor sandboxes, and the Go SDK'sSandboxCreateParamshas no disk field, though the backend protocol hasephemeral_disk_mb.A Pod asking for more
ephemeral-storagereached Modal and failed at Provision withmodal: N GiB disk exceeds the 512 GiB Modal provides. That error matches no failover category, so it blocked nothing. A controller's replacement Pod went back to Modal and failed again, and never tried a provider that could serve it, such as AWS, which sizes gp3 to the request up to 16354 GiB. AWS had the same gap: it refuses disk pastawsMaxDiskGiB(gp3's 16 TiB less the 30 GiB AMI root) at launch, with the same churn.Change
provider.Capabilities.MaxDiskGiBPerPodis the most local disk the provider can give one Pod; each Pod gets its own, and 0 means no known limit.ServesDisksits next toServesCapacityTierandServesEgress. Modal sets 512 and AWS setsawsMaxDiskGiB(16354), each matching the line its launch already refuses at. RunPod and Kubernetes keep 0: neither refuses disk today.ephemeral-storage(first container's limit, else its request, rounded up to GiB, as before) and skips a provider past its limit. The skip is recorded asdisk_unsupported. The Pod lands on the pool's next provider, or stays gated asno_candidateif none fits.MaxDiskGiBPerPodper Pod, summed over its 1k Pods: 500 TiB onnebula-modal, 16Pi elsewhere as before. Node capacity is a total the scheduler checks across all bound Pods, so a node can't cap one Pod's disk; a flat512Giwould admit a single 512 GiB Pod at a time. Placement enforces the per-Pod limit.sandboxSpecFromPodstays as a backstop for a Pod placed by an older manager.disk_unsupportedis added to the skip reasons indocs/metrics.md.Units: the limit is 512 GiB.
512Gistays on Modal;513Gi, and decimal550G(~512.2 GiB, rounded up to 513), skip it.Testing
TestPlacement_DiskPastAProvidersMaxSkipsIt:512Gistays on Modal;513Giand550Gfall through to the next provider;513Giwith no other provider stays gated. WithServesDiskforced totrue, the three over-limit cases fail.TestPlacement_GPUPodPastAWSDiskSkipsIt: a GPU Pod asking16Tiskips AWS for the next provider, or stays gated.TestServesDisk,TestVirtualCapacity_DiskFollowsMaxDiskGiBPerPod, andTestDiskLimit_PlacementAndRefusalAgreefor Modal and AWS (each one's advertised limit equals the line its launch refuses at).go test ./...(non-e2e) andmake lintpass.🤖 Generated with Claude Code