Skip to content

[POC] Adding initial comparison b/w rust and python for standard bucket - #1017

Draft
raj-prince wants to merge 12 commits into
fsspec:mainfrom
raj-prince:with_rust_sdk
Draft

raj-prince wants to merge 12 commits into
fsspec:mainfrom
raj-prince:with_rust_sdk

Conversation

@raj-prince

Copy link
Copy Markdown
Collaborator

No description provided.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an optional Rust-based read backend for gcsfs using the official google-cloud-storage SDK via PyO3 to improve object read performance. It includes the PyO3 extension bindings, a Python wrapper, integration into GCSFileSystem, and benchmark tools. The code review identified several critical issues and optimization opportunities: the global static Rust client ignores instance-specific configurations and should be refactored into a per-instance class; negative offsets are not resolved before being passed to Rust, which will cause conversion errors; missing object errors (404) should be mapped to FileNotFoundError to maintain compatibility with fsspec; read_backend needs fail-fast validation during initialization; memory allocations in Rust can be optimized using Vec::with_capacity; the double-thread-hop overhead should be eliminated by using PyO3's async integration; and a potential division-by-zero panic in the benchmark tool must be prevented.

Comment thread rust/gcsfs_rust/src/lib.rs Outdated
Comment thread gcsfs/core.py
Comment thread rust/gcsfs_rust/src/lib.rs Outdated
Comment thread gcsfs/core.py
Comment thread rust/gcsfs_rust/src/lib.rs Outdated
Comment thread gcsfs/rust_backend.py
Comment thread rust/bench_rust_read/src/main.rs
@codecov

codecov Bot commented Aug 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 56.52174% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.98%. Comparing base (211f484) to head (3bc06c5).
⚠️ Report is 18 commits behind head on main.

Files with missing lines Patch % Lines
gcsfs/rust_backend.py 58.82% 7 Missing ⚠️
gcsfs/core.py 50.00% 3 Missing ⚠️

❌ Your patch check has failed because the patch coverage (56.52%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1017      +/-   ##
==========================================
- Coverage   90.05%   89.98%   -0.08%     
==========================================
  Files          16       17       +1     
  Lines        3641     3764     +123     
==========================================
+ Hits         3279     3387     +108     
- Misses        362      377      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@raj-prince raj-prince changed the title Adding initial comparison b/w rust and python for standard bucket [POC] Adding initial comparison b/w rust and python for standard bucket Aug 24, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant