Skip to content

feat(ai): graduate video generation with start, status, result and cancel - #63657

Merged
rekram1-node merged 2 commits into
devfrom
video-api
Nov 13, 2026
Merged

rekram1-node merged 2 commits into
devfrom
video-api

Conversation

@rekram1-node

@rekram1-node rekram1-node commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Issue for this PR

No issue. Follows #63634, which removed the dead Gemini Veo and Runway video protocols.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Video generation in @opencode/ai moves out of experimental and becomes a job you start and then check on yourself, instead of a call that blocks and polls for you. That's what opencode core needs: it decides when to poll, and it can pick a job back up after a restart from a stored token.

const job = yield* ai.video.start({ model, prompt, start, end, references, duration: "8 seconds", aspect, resolution, sound })
// job = { id, token }   token is plain JSON, store it with the model

yield* ai.video.status(model, job.token) // one read: { status, progress? }
yield* ai.video.result(model, job.token) // { videos, usage?, providerMetadata? }
yield* ai.video.cancel(model, job.token) // typed UnsupportedOperation where the provider can't cancel

The Promise client has the same calls under ai.video. Every call after start takes (model, token), so recovering a job is the same as reading it normally.

What changed in the request:

  • frames: { first, last } is now start and end.
  • references takes any media kind; protocols route images, video and audio to their own fields.
  • durationSeconds: 10 is now duration: "10 seconds" (an Effect duration string, seconds only for now).
  • aspect also accepts "auto" (follow the start image), resolution documents 480p | 720p | 1080p | 4k and passes anything else through, and audio is now sound.
  • n, seed, negativePrompt and the source video field are gone from the shared request. They're on too few providers; use providerOptions.

Removed: generate, stream, resume and the live handle (await, events, refresh) for video. There's no built-in polling loop and no queue position, since no video provider reports one. Other modalities keep Generation as is.

Enforcement: the package only rejects locally what a protocol has no way to send, such as a reference video on xAI. Values a provider may reject (durations, resolutions, reference counts, mixing frames and references) go to the provider and its error comes back typed. Spelling and routing differences are handled silently by each protocol.

How it works:

  • MediaRoute.queued gets attach(model, token, execute), which decodes a token into the route's status, result and cancel operations without a status read first. resume now uses it too.
  • VideoClient is its own small service (start, status, result, cancel) on top of that, and VideoModel only accepts queued routes.
  • xAI maps start to image, end to last_frame, image references to reference_images, audio references to reference_audios, sound to generate_audio, and leaves aspect_ratio out for "auto" when a start frame sets the shape. Edit and extend stay reachable with providerOptions: { mode, video: { url } }.
  • Media.Asset gets stream(), and Media.write and the Promise ai.write now stream URL downloads to disk instead of buffering whole files.

Trade-offs:

  • Callers that only want a one-liner now write a short polling loop. I'd rather add a helper later than bake a loop into the API core can't control.
  • A request's headers/http overlay only applies to start. Status and result reads are rebuilt from the token, so they use the model's deployment settings, which is what a resumed job in another process gets anyway.
  • ai.experimental.video is removed without aliases, since it was experimental. The release notes should mention that ai.video graduated with this shape.

No feature flag: this is a library API with no opencode runtime behavior yet.

How did you verify your code works?

  • bun typecheck and bun run typecheck:tsc in packages/ai
  • bun test in packages/ai: 2073 pass, 0 fail
  • bun run check at the root
  • The recorded xAI test now drives start, then status until terminal, then result, and still replays the existing cassette: the request body is unchanged.
  • New unit tests cover the xAI field mapping, "auto" aspect, edit and extend via provider options, rejecting reference videos and non-second durations before sending, token round trips and foreign tokens, status mapping and progress, result on pending, failed, expired and moderated jobs, unsupported cancel, read retries (503, 429 retry-after, no retry on 400) and Media.Asset.stream.
  • The Promise tests that exercised the generic generation handle through video now use AssemblyAI transcription, which still has the handle.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Added by the opencode bot (AI-generated).

Public API: graduates video to ai.video (start/status/result/cancel), removing ai.experimental.video and renaming the request fields (breaking for experimental callers)

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new video job API looks sound to me: start returns a token, then status, result and cancel each read the job from that token, and queued routes get a new attach. Unit tests and type checks pass. There are no xAI credentials, so I couldn't run the flow against the real provider.

One regression, in streamed file writes (inline comment): when a download fails, the target file has already been truncated. On the base, a 404 left an existing file untouched. With this PR the file ends up empty. I checked Media.write against both branches. The Promise ai.write opens the file with "w" before downloading, so it should behave the same, though I didn't run it.

Public API impact (@opencode/ai). Breaking for anyone using ai.experimental.video; it was experimental:

  • Removed: ai.experimental.video in both the Effect and Promise clients, along with generate, stream, resume and the live handle for video. Removed from @opencode/ai/experimental: Video, VideoClient, VideoAspectRatio, VideoModel, VideoModelSchema, VideoResponse and the video types. VideoEvent and VideoFrames are gone entirely.
  • Added: ai.video.{request,start,status,result,cancel} in both clients. New exports: VideoClient (Effect entry), VideoDuration, VideoJob, VideoStatus, plus the video exports above moved to the main entry. Also Media.Asset.stream() and a required attach on MediaRoute.QueuedRoute.
  • Changed request fields:
    • frames.first/frames.last → start/end
    • durationSeconds: number → duration: "N seconds"
    • audio → sound
    • aspect now also accepts "auto"
    • references accepts any media kind
    • n, seed, negativePrompt and video are removed (use providerOptions)
  • xAI edit and extend now come from providerOptions.mode plus providerOptions.video, instead of the request's video field.
  • Media.write and Promise ai.write now stream URL downloads to disk instead of buffering them; see the regression above.

Comment thread packages/ai/src/media.ts Outdated
.writeFile(path, data)
.pipe(Effect.mapError((cause) => invalid(`Failed to write media file ${path}`, cause)))
yield* asset.stream().pipe(
Stream.run(fs.sink(path)),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fs.sink(path) opens the target with "w" before the download starts. A failed download (for example a 404 from an expired video URL) or a mid-stream error now truncates an existing file. I checked this: on the base a 404 left the file's bytes alone, and with this PR it ends up empty. The Promise ai.write in src/index.ts calls open(path, "w") before streaming too, so it should behave the same. Streaming into a temporary file next to path and renaming it into place on success would keep the old behaviour without buffering.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fc6528a: both Media.write and the Promise ai.write now stream into a temporary file next to the target and rename it into place only after the download finishes, removing it on failure. A 404 leaves the existing file untouched; tests cover both writers.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked fc6528a and the fix is correct.

  • Media.write (media.ts ~376–392): it writes to ${path}.<uuid>.tmp, then renames that file onto path. Effect.onError deletes the temp file on any failure, including interruption, and ignores errors from that delete. The original AIError (for example the 404) is still passed through.
  • Promise ai.write (index.ts ~207–224): it works the same way. The file handle is closed in the inner finally before the rename or the delete runs. rm(..., { force: true }) is safe even when open failed. Abort reasons and AIError are rethrown as they are, and anything else is wrapped as a file error.
  • Temp file location: it sits next to the target, so the rename stays on one filesystem and replaces the old file in a single step.

What I ran: the shared checkout is still on 3f25c4b, so I ran test/media.test.ts and test/promise.test.ts in a temporary worktree of pr/head. All 28 tests passed, including the new 404 cases. I didn't run the full suite or typecheck.

Left alone: if the process is killed during a download (for example SIGKILL), a stray *.tmp file can stay behind. That's normal for this pattern and I don't think it's worth handling.

I didn't change anything. The temporary worktree is removed, and the bun.lock modification in repo/ was already there, so I left it.

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your new commits fix the file-write problem from my last review. Media.write and the Promise ai.write now stream into a temporary file next to the target, rename it into place only once the download finishes, and delete it on failure. A failed download leaves an existing file as it was, and the two new tests cover that. The touched test files and the type checks pass. I found nothing new.

The public API impact is the same as in my earlier review. These commits only change how the two write functions save the file.

@rekram1-node
rekram1-node merged commit 5eac280 into dev Nov 13, 2026
10 checks passed
@rekram1-node
rekram1-node deleted the video-api branch November 13, 2026 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant