fix(model): --wait-for-hash must wait for a deployable version (STO-441) - #337
Merged
jebenexer merged 1 commit intoSep 9, 2026
Conversation
The api assigns a `ph-`-prefixed placeholder hash the moment a model version exists, while its status is still NEEDS_HASH, and myModels does not filter those out. The wait treated any nonempty hash as ready, so it returned immediately with a model url no endpoint would accept -- and --delete-my-model-files-after-upload deleted the local files on that false confirmation. Readiness is now status POD_READY/READY plus a non-placeholder hash, the same set the api's endpoint validator accepts. A version that failed server-side ends the wait with `conflict` instead of burning the whole --hash-timeout budget, and a content-deduped upload resolves through metadata.dedupedToCanonicalVersion to the canonical hash rather than waiting forever on a uuid the api deprecated. Also bumps golang.org/x/crypto to v0.56.0 to clear GO-2026-6354 / GO-2026-6355 (ssh.Dial deadlock DoS) flagged by govulncheck in CI.
jebenexer
force-pushed
the
benjaminbrannaka/sto-441-runpodctl-wait-for-hash-must-wait-for-a-deployable-model
branch
from
September 8, 2026 18:42
e1b7ec2 to
7d8962d
Compare
rutvik-runpod
approved these changes
Sep 9, 2026
|
bugbot run |
Collaborator
Author
|
@cursor review |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 7d8962d. Configure here.
jebenexer
deleted the
benjaminbrannaka/sto-441-runpodctl-wait-for-hash-must-wait-for-a-deployable-model
branch
September 9, 2026 18:35
Promptless documentation updates
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STO-441:
runpodctl --wait-for-hashmust wait for a deployable model versionhttps://linear.app/runpod/issue/STO-441
--wait-for-hashreturned as soon as the version had any hash. the api assigns aph-<32 hex>placeholder the moment the version exists, while its status is stillNEEDS_HASH, andmyModels(what the cli polls) does not filter those out — so the wait ended immediately and printed a model url that the endpoint validator rejects with "Model Repo reference is not ready for endpoint deployment".--delete-my-model-files-after-uploaddeleted the local files on that false confirmation. it now only fires once the version is genuinely deployable.statusin {POD_READY,READY} plus a non-ph-hash — the exact set the api'sREADY_MODEL_REPO_STATUSESgate accepts.FAILED/POD_REMOVED/DEPRECATEDwith no canonical pointer) now ends the wait immediately with codeconflict, instead of burning the whole--hash-timeoutbudget and reportingtimeout(which tells an agent the work is still running and to poll rather than re-upload).metadata.dedupedToCanonicalVersion. the wait follows that pointer and reports the canonical hash.waiting for model to be deployable.docs/runpodctl_model_add.md, READMEconflictrow, and an AGENTS.md pitfall for the placeholder-hash and dedupe-deprecation behavior (neither is discoverable from the cli code alone).golang.org/x/cryptoto v0.56.0 to clear thegovulncheckCI failure (GO-2026-6354 / GO-2026-6355, anssh.Dialdeadlock DoS reachable fromcmd/project/ssh.go).Test plan:
runpod-in-a-boxstack (real GraphQL server built from actualRunPodsource, real MySQL, real LocalStack S3, no cloud credentials needed). Forced a freshly-uploaded version to a terminalPOD_REMOVEDstatus while its hash was still theph--prefixed placeholder: the pre-fix binary printedmodel is ready to deploywith the placeholder-hash url (the exact bug), the fixed binary correctly failed fast with{"error":"...is pod_removed, so it will never become deployable...","code":"conflict"}. Happy path (real async hashing via the local model-repo-hasher poller) succeeds identically on both binaries with the real non-placeholder hash — no regression.node/graphql/schema/model.ts's sharedModel.versionsfilter (visibleModelVersionsWhere) hidesNEEDS_HASH,FAILED, andDEPRECATEDversions frommyModelsonmaintoday (confirmed byte-identical resolver code).POD_REMOVEDis not filtered and was confirmed reachable live;FAILEDand the dedupe-follow path are correct here but currently unreachable throughmyModelsas it exists onmain— not a regression (the pre-fix code has the same blind spot), but worth a follow-up if fast-detection for those two paths is wanted.the two pre-existing wait tests now start from the real bug state (
ph-…+NEEDS_HASH) rather than an empty hash, so they fail against the old code. new coverage: the 9-case readiness table, terminal fast-exit (assertsconflictand that it never polls again), dedupe→canonical resolution, timeout message content, and "local files survive when only a placeholder hash ever appears".Note
Medium Risk
Changes model-upload wait and post-upload local deletion gates; wrong logic could delete files early or mislead agents on errors, though behavior is heavily unit-tested.
Overview
Fixes
model add --wait-for-hashso it no longer treats “any nonempty hash” as success. The API assigns aph-placeholder while status isNEEDS_HASH, which previously ended the wait immediately and could print a deploy URL the endpoint validator rejects;--delete-my-model-files-after-uploadcould delete locals on that false positive.Readiness is now
POD_READYorREADYplus a non-placeholder hash, matching deploy-time rules. Deduped uploads followmetadata.dedupedToCanonicalVersionand succeed with the canonical hash even when the new UUID staysDEPRECATED. Terminal statuses (FAILED,POD_REMOVED, bareDEPRECATED) fail fast with codeconflictinstead of burning--hash-timeoutand reportingtimeout. Timeout messages name the last status andrunpodctl model list --name …; stderr progress says “waiting for model to be deployable”.Docs (README, AGENTS.md, generated
model addhelp) andapi.ModelVersionStatusconstants were extended accordingly. Tests cover placeholder polling, terminal exit, dedupe resolution, and timeout copy.golang.org/x/crypto(and related x/ modules) were bumped for govulncheck CI.Reviewed by Cursor Bugbot for commit 7d8962d. Configure here.