feat(oss): add streaming range reads with integration CI by wylswz · Pull Request #32 · langgenius/dify-cloud-kit

wylswz · 2026-11-05T05:24:16Z

Summary

Add optional streaming reads and native byte ranges together with the integration checks that validate the feature. Feature and test infrastructure ship in one PR, so the implementation cannot land separately from its tests.

This PR consolidates the previous stack (#33 + #32); it targets main and includes both commits. #33 is superseded. There is no remaining stacked-PR dependency.

Part of ENG-1423

Changes

Add optional StreamingOSS.Range(ctx, key, *ByteRange) (io.ReadCloser, error) for Local FS, S3, Aliyun OSS, and Azure Blob without changing the existing OSS interface.
Add SupportsStreaming, OpenStream, and ErrStreamingNotSupported for explicit capability detection; no implicit full-buffer fallback.
Add optional Preparer.Prepare(ctx) with SupportsPrepare, oss.Prepare, and ErrPrepareNotSupported. Azure implements idempotent container creation; only ContainerAlreadyExists is treated as success, and preparation is never called implicitly. Other providers retain their existing behavior.
Validate bounded and offset-to-EOF ranges, prevent integer overflow, support cancellation, and verify backend Content-Range responses.
Preserve buffered legacy Load behavior while closing response bodies correctly; keep Azure resumable reads with SDK-signed Range options.
Include hermetic unit tests and emulator integration tests for full/ranged reads, EOF clamping, empty objects, errors, retries, cancellation, and resource closure.
Add pinned LocalStack 4.0.3 and Azurite 3.35.0 with loopback-only ports and isolated test resources.
Add PR CI, failure-log artifacts, and unconditional container cleanup; no cloud credentials are required.
Keep the real-Azure read-only test separately opt-in. Enterprise caller integration is outside this PR.

Validation

Local preparation unit tests and Azurite create/repeated/concurrent preparation checks pass; the existing streaming integration suites remain green.
Local go test -race -count=1 ./oss/... passes.
Local make test-integration passes against LocalStack and Azurite using the combined feature and infrastructure.
The previous stacked revision passed the GitHub-hosted Storage Integration check.
Confirm CI on this consolidated PR targeting main.

Real Azure live validation is not claimed: the attempted fixture upload was denied by data-plane permissions and no blob was created. Emulator and unit tests do not validate cloud RBAC or managed identity.

Merge Gate

The integration workflow is included in this PR. Configure Storage Integration as a required branch-protection/ruleset check if merge enforcement is desired; this PR does not change repository policy.

…gration tests Add Range (streaming byte-range reads) to Google GCS, Tencent COS, Huawei OBS, and Volcengine TOS, closing the gap with the existing Local/S3/Aliyun/Azure implementations. 416 responses map to ErrRangeNotSatisfiable and ranged reads verify the server honored the Range header via Content-Range. Huawei OBS resolves open-ended and single-byte ranges with a HEAD first because its SDK cannot emit "bytes=N-" or "bytes=N-N". Add explicit, idempotent Prepare (bucket creation) to S3 and GCS alongside the existing Azure implementation, so provisioning also goes through the library interface. Rewrite the emulator integration tests to exercise only the library's own interface (factory.Load, Prepare, Save, Range/OpenStream, Load, Exists, State, Delete) instead of raw cloud SDK clients, and add a fake-gcs-server emulator to the compose stack. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…ration suite Extend the shared streaming checks with single-byte, last-byte, and open-ended unsatisfiable range cases, and verify Prepare idempotency in the S3/GCS emulator tests. GCS now maps an empty-object open-ended range to ErrRangeNotSatisfiable even when the server answers 200 with an empty body instead of 416 (fake-gcs-server behavior). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Copilot

🟡 Changes recommended

Unresolved defects break OBS cancellation and last-byte streaming, prevent missing S3 buckets from being recreated outside us-east-1, and incorrectly treat S3/GCS bucket-name collisions as successful preparation.

5 open issues

What changed in this PR

Adds optional streaming and byte-range reads to the storage abstraction without changing the existing OSS interface. This consolidates the feature and its test infrastructure into one PR.

Changes:

Adds streaming capabilities, shared range validation, and provider implementations while preserving buffered Load behavior.
Adds explicit storage preparation for Azure Blob, S3, and GCS.
Adds provider unit tests, emulator integration tests, CI cleanup and diagnostics, and usage documentation.

File	Description
`tests/integration/s3_test.go`	Exercises S3 against LocalStack.
`tests/integration/gcs_test.go`	Exercises GCS against fake-gcs-server.
`tests/integration/fixture_test.go`	Adds deterministic fixtures and bounded read verification.
`tests/integration/common_test.go`	Shares streaming and legacy interface checks.
`tests/integration/azure_test.go`	Exercises Azure reads and preparation against Azurite.
`README.md`	Documents capabilities, provider limitations, and testing.
`oss/volcenginetos/tos.go`	Adds TOS streaming and closes buffered-read bodies.
`oss/volcenginetos/tos_stream_test.go`	Tests TOS range behavior.
`oss/tencentcos/cos.go`	Adds COS streaming and response validation.
`oss/tencentcos/cos_stream_test.go`	Tests COS range behavior.
`oss/stream.go`	Defines streaming capabilities and range validation.
`oss/stream_test.go`	Tests range semantics and capability helpers.
`oss/s3/s3.go`	Adds S3 streaming and explicit bucket preparation.
`oss/s3/s3_stream_test.go`	Tests S3 ranges, cancellation, and closure.
`oss/s3/s3_prepare_test.go`	Tests bucket preparation and error handling.
`oss/prepare.go`	Defines optional preparation capabilities and helpers.
`oss/prepare_test.go`	Tests preparation detection and delegation.
`oss/oss.go`	Adds the GCS project setting for preparation.
`oss/local/localfile.go`	Adds file-section streaming with context checks.
`oss/local/localfile_stream_test.go`	Tests local ranges and reader lifecycle.
`oss/huaweiobs/obs.go`	Adds OBS streaming and size-based range handling.
`oss/huaweiobs/obs_stream_test.go`	Tests OBS ranges and SDK limitations.
`oss/gcsblob/gcs.go`	Adds GCS streaming and bucket preparation.
`oss/gcsblob/gcs_stream_test.go`	Tests GCS ranges and response validation.
`oss/gcsblob/gcs_prepare_test.go`	Tests GCS preparation and configuration requirements.
`oss/azureblob/blob.go`	Adds Azure ranged retry readers and container preparation.
`oss/azureblob/blob_stream_test.go`	Tests Azure ranges, retries, cancellation, and closure.
`oss/azureblob/blob_prepare_test.go`	Tests preparation idempotency, concurrency, and errors.
`oss/azureblob/blob_live_test.go`	Adds opt-in, read-only real-Azure validation.
`oss/aliyun/aliyun.go`	Adds Aliyun streaming with range verification.
`oss/aliyun/aliyun_stream_test.go`	Tests Aliyun ranges, cancellation, and closure.
`Makefile`	Adds emulator lifecycle and integration-test targets.
`compose.integration.yaml`	Defines pinned emulators with loopback-only ports.
`.github/workflows/integration.yml`	Runs tests, captures diagnostics, and cleans up services.

_{Review effort: Balanced}

Prepare treated S3 BucketAlreadyExists and any GCS 409 as success, so a bucket name taken by another account/project reported ready and only failed later on the first write. On a create conflict, now re-check that the bucket is accessible and fail otherwise. GCS also checks access before creating, so repeated Prepare no longer hits the rate-limited create API. The check lists at most one object, which needs only storage.objects.list instead of storage.buckets.get. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…ation # Conflicts: # oss/huaweiobs/obs.go

Huawei OBS: the SDK's RangeStart/RangeEnd cannot express "bytes=N-" or "bytes=N-N", so single-byte and last-byte-to-EOF reads (including any read of a one-byte object) failed Content-Range verification. Send the Range header verbatim instead, which also drops the extra HEAD request. The SDK takes no per-request context, so cancellation now abandons the in-flight GET and closes the body on ctx done, letting blocked reads return ctx.Err(). S3: bucket creation outside us-east-1 failed without a LocationConstraint. Set it from the region for AWS, in both Prepare and the legacy constructor; S3-compatible endpoints keep their default. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

The OBS SDK cannot express "bytes=N-" or single-byte ranges through its typed fields and takes no per-request context, so a cancelled request can only be abandoned, not aborted. Rather than ship a Range with these caveats, OBS no longer implements StreamingOSS: OpenStream fails with ErrStreamingNotSupported and callers use the buffered Load, which is restored to the main-branch implementation. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

wylswz added 2 commits November 5, 2026 13:23

feat(oss): add optional streaming and range reads

045040e

test(oss): add emulator integration checks for pull requests

1239ec7

wylswz mentioned this pull request Nov 5, 2026

feat(oss): add optional streaming and range reads #33

Closed

3 tasks done

wylswz added this pull request to stack #34 November 5, 2026 05:28

wylswz removed this pull request from stack #34 November 5, 2026 13:04

wylswz changed the title ~~test(oss): add emulator-backed integration checks to PR CI~~ feat(oss): add streaming range reads with integration CI Nov 5, 2026

wylswz changed the base branch from feature/streaming-range to main November 5, 2026 13:04

wylswz and others added 5 commits November 5, 2026 21:06

ci: gate streaming feature on unit and integration tests

786f362

refactor(oss): rename streaming Open method to Range

acde563

feat(oss): add explicit preparation capability for Azure containers

9c95263

wylswz requested a balanced review from Copilot November 9, 2026 09:12

Copilot started reviewing on behalf of wylswz November 9, 2026 09:12

View session

Copilot AI reviewed Nov 9, 2026

View reviewed changes

Comment thread oss/gcsblob/gcs.go Resolved

Comment thread oss/huaweiobs/obs.go Resolved

Comment thread oss/huaweiobs/obs.go Resolved

Comment thread oss/s3/s3.go Resolved

Comment thread oss/s3/s3.go Resolved

wylswz and others added 4 commits November 9, 2026 17:40

Merge remote-tracking branch 'origin/main' into test/streaming-integr…

89206ad

…ation # Conflicts: # oss/huaweiobs/obs.go

wylswz marked this pull request as ready for review November 9, 2026 10:18

wylswz requested a review from 41tair November 9, 2026 10:18

Navigation Menu

Repository navigation

feat(oss): add streaming range reads with integration CI - #32

feat(oss): add streaming range reads with integration CI#32
wylswz wants to merge 11 commits into
mainfrom
test/streaming-integration

wylswz commented Nov 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

wylswz commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Merge Gate

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wylswz commented Nov 5, 2026 •

edited

Loading