fix: read error response body in _get_file_request to enable 401 retry

[Yonghui-Lee]

Fixes #1082

In _get_file_request, validate_response(r.status, None, rpath) was called before reading the response body. When GCS returned an intermittent HTTP 401 with {"error": {"code": 401, "message": "Invalid Credentials"}}, validate_response received content=None and raised HttpError({"code": 401, "message": ""}). Because is_retriable only retries 401 when "Invalid Credentials" is in exception.message, file downloads failed immediately without retrying.

Changes

• gcsfs/core.py (_get_file_request):
  • Read the response body when r.status >= 400 so error details reach validate_response and retry_request, while preserving chunked streaming to disk for 2xx responses.
  • Default callback = callback or NoOpCallback() to avoid AttributeError when _get_file_request is invoked without a callback.
  • Remove the redundant validate_response(r.status, data, rpath) call after the read loop.
• gcsfs/retry.py (validate_response):
  • Exclude status == 401 from the elif status != 401 and "invalid" in str(msg): ValueError check so 401 responses containing lowercase "invalid" (e.g., "Invalid Credentials: invalid_token") surface as HttpError and remain retriable.
  • Ensure HttpError({"code": status, **error}) preserves the HTTP status code if "code" is missing from the parsed JSON "error" dictionary.
  • Safely handle non-UTF-8 error bytes (errors="replace") and valid JSON payloads that do not match the {"error": {"message": ...}} structure.
• gcsfs/tests/test_retry.py:
  • Add test_get_file_request_retries_401_invalid_credentials and test_validate_response_401_and_json_edge_cases.

[gemini-code-assist]

Code Review

This pull request improves error handling and retry logic for GCS requests in gcsfs. Specifically, it updates _get_file_request to validate responses immediately upon receiving a status code of 400 or higher by reading the response content. In validate_response, it enhances robustness when parsing JSON error payloads, decodes content using UTF-8 with replacement fallback, and ensures that 401 authentication errors containing 'invalid' are raised as retriable HttpErrors rather than ValueErrors. Corresponding unit tests have been added to verify these edge cases and retry behaviors. There are no review comments to address.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.39%. Comparing base (9290d6f) to head (3bb332d).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1084      +/-   ##
==========================================
+ Coverage   90.25%   90.39%   +0.14%     
==========================================
  Files          16       16              
  Lines        3755     3811      +56     
==========================================
+ Hits         3389     3445      +56     
  Misses        366      366              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rafaellott]

@Yonghui-Lee @zhixiangli
Thanks for merging this! Do you have an estimate for when a PyPI release containing this fix will be available? We're tracking it so we can update our pinned gcsfs version.

[Yonghui-Lee]

Hi @rafaellott. A new release is scheduled in a week. I hope that doesn't block you.

[rafaellott]

Hi @Yonghui-Lee , That sounds great to me. Thank you!