Repository navigation
feat: report finished segments during pack() and accept URL sources - #3
Merged
Merged
Conversation
M3u8::read() and Mpd::read() take an optional ledger of segment sizes by name. A segment named by the manifest but gone from disk is described with the recorded size instead of failing the package as missing; one that is neither on disk nor in the ledger still fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Packager::SEGMENT fires for each segment ffmpeg has finished while the rest are still encoding, with its name, path and size. A listener may upload and delete it; the package still describes it from that size. A segment counts as finished once a higher-numbered sibling exists in its sequence, and HLS now writes `.tmp` until rename (temp_file is a default flag). Init segments and the last of each sequence are reported when the run ends, manifests never, and nothing that was in the directory, unchanged, before the run. Without a listener nothing is watched. Sources may be an http(s) URL, which ffmpeg and ffprobe read with range requests, so a presigned object storage link is probed, packed and tiled without a local copy. Every other scheme is refused, and so is a local name that starts like a protocol: is_file() followed stream wrappers, so file:///etc/passwd passed the old check and ftp:// reached the network. Messages name a URL source without its query string. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shipping E2E records progress next to segments and asserts it stays monotonic, reaches 100, and is still reported after the first segment has left. Seen red with mid-run progress suppressed while watching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ffmpeg writes out_time_us=N/A for a progress block with no output timestamp yet, which a float cast read as 0, so progress fell from ~77% back to 0 before reaching 100 (seen on DASH and CMAF under ffmpeg 8.1.2). Non-numeric and negative times are now skipped. A consumer writing progress to storage would otherwise publish that 0. Also: tests no longer use GLOB_BRACE, which musl does not define, and the watcher documents the one rewrite it cannot tell apart (same second, same size, reused inode). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Appwrite videos worker runs on an
emptyDirscratch volume. Today one encode holds the whole source plus the whole rendition on that disk until the end. This lets a consumer ship each segment as soon as it is finished, and give ffmpeg the source as a URL instead of a local copy.What changes
Packager::SEGMENTfires duringpack()for each finished segment (Written:file,path,size). The listener may upload and delete the file; the returnedPackagestill describes it, with that size..tmpuntil rename (temp_filejoins the default flags).M3u8::read(),Mpd::read()) take an optional ledger of shipped sizes, so a shipped segment is not "missing".http(s)://URL; ffmpeg and ffprobe read it with range requests.tile(),grab()andsubtitle()accept it too.is_file()follows PHP stream wrappers, sofile:///etc/passwdpassed the old check andftp://…made a network call. Now only a plain local path or http(s) is accepted, and a local name starting like a protocol (concat:) is refused.Tests
composer lint,composer checkRemoteTestserves the sources from a local range-capable server and checks that ffmpeg actually sendsRangerequests.Not covered here
Release
appwrite/appwrite
__stream_v8requiresdev-feat-shipped-segmentsuntil this is merged and tagged (next after 0.0.2).🤖 Generated with Claude Code