[POC] Adding initial comparison b/w rust and python for standard bucket - #1017
raj-prince wants to merge 12 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces an optional Rust-based read backend for gcsfs using the official google-cloud-storage SDK via PyO3 to improve object read performance. It includes the PyO3 extension bindings, a Python wrapper, integration into GCSFileSystem, and benchmark tools. The code review identified several critical issues and optimization opportunities: the global static Rust client ignores instance-specific configurations and should be refactored into a per-instance class; negative offsets are not resolved before being passed to Rust, which will cause conversion errors; missing object errors (404) should be mapped to FileNotFoundError to maintain compatibility with fsspec; read_backend needs fail-fast validation during initialization; memory allocations in Rust can be optimized using Vec::with_capacity; the double-thread-hop overhead should be eliminated by using PyO3's async integration; and a potential division-by-zero panic in the benchmark tool must be prevented.
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (56.52%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #1017 +/- ##
==========================================
- Coverage 90.05% 89.98% -0.08%
==========================================
Files 16 17 +1
Lines 3641 3764 +123
==========================================
+ Hits 3279 3387 +108
- Misses 362 377 +15 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
No description provided.