feat(io): refresh vended storage credentials before they expire by plusplusjiajia · Pull Request #892 · apache/iceberg-cpp

plusplusjiajia · 2026-08-19T03:43:08Z

Vended storage credentials expire, so new file operations on a reused table can eventually fail. ArrowS3FileIO now refreshes them on demand, starting five minutes before the earliest applicable s3.session-token-expires-at-ms.

Builds on the StorageCredentialProvider API merged in #899:

InitializeStorageCredentials() installs the initial credentials and provider before first use. Later SetStorageCredentials() calls retain the provider.
Each vended S3 client signs through a VendedCredentialsProvider (an Aws::Auth::AWSCredentialsProvider set as Arrow's S3Options::credentials_provider), which the AWS SDK asks before every request. It returns the keys the FileIO currently vends for the client's prefix and refreshes them once due, as Java's VendedCredentialsProvider does, so handles and streams opened before a refresh use the new keys. A refresh that changes only keys keeps the existing clients.
A request cannot build clients: Arrow holds its client registry for the duration of a request, and building a client locks it exclusively. A refresh started from a request that needs new clients is left to the next file operation, and requests never wait on another refresh.
A refresh is installed only if every S3 credential is usable: a session token needs a parseable expiry that has not passed. Failed or unusable refreshes keep the current credentials and back off for up to 30 seconds. Concurrent operations share one refresh; file operations with expired credentials wait up to 10 seconds for one in flight. A concurrent credential install supersedes any older refresh result.
Once a prefix's credentials have expired and cannot be refreshed, creating a handle or deleting under it fails with TokenExpired.
HttpClient::Get takes an optional timeout; the REST credentials request uses 10 seconds.

Tests cover refresh timing, concurrency, unusable and failed replacements, expiry errors, provider initialization and the HTTP timeout, plus real S3 round trips: through REST FileIO and ResolvingFileIO, and reading through a handle opened before a refresh, both for a keys-only refresh and for one that replaces the client.

wgtmac · 2026-08-23T16:20:24Z

Thanks for adding this feature! I think it is too large to review which may incur a long delay. Perhaps let's split it into smaller ones so we can review it one by one?

plusplusjiajia · 2026-08-24T07:50:15Z

Thanks for adding this feature! I think it is too large to review which may incur a long delay. Perhaps let's split it into smaller ones so we can review it one by one?

@wgtmac Thanks — split into a stack, smallest first:

test(io): stop probing the EC2 metadata service in S3 tests #897 — S3 tests stop probing the EC2 metadata service (54s → 0.3s).
refactor(io): hand out S3 delegates by shared_ptr and build them off the lock #898 — ArrowS3FileIO gets refactor(io): make FileIO resolution registry-driven #889's concurrency treatment. No behavior change.
feat(rest): load table credentials from the LoadCredentials endpoint #899 (draft until refactor(io): hand out S3 delegates by shared_ptr and build them off the lock #898 merges) — LoadCredentials plumbing, not yet invoked.
This PR shrinks to just the refresh policy.

Each builds and passes the full suite standalone; the diff here narrows as each one merges.

wgtmac · 2026-09-24T06:04:57Z

Is this ready for review?

plusplusjiajia · 2026-09-29T05:42:56Z

Is this ready for review?

@wgtmac Thanks! Not yet. #899 is ready for review now, and this one is stacked on it, so I'll mark it ready once #899 lands.

plusplusjiajia · 2026-10-09T05:29:46Z

@wgtmac Thanks for the provider refactor in #899! Rebased on main and adapted the refresh policy and tests to StorageCredentialProvider and InitializeStorageCredentials. Ready for review.

wgtmac · 2026-10-13T03:06:22Z

+  /// \brief Replace the credentials once they are close to expiring.
+  ///
+  /// Called before each handle is created; a handle keeps the delegate it was
+  /// built from, so I/O on an open one is not re-checked. A failure keeps the


Can we separate credential refresh from filesystem replacement? Existing handles and streams keep the old token, even after FileIO refreshes. A refreshable provider on the S3 client, as in Java, would cover those requests and avoid rebuilding clients for token-only changes.

Thanks! Done. Each vended S3 client now signs through a VendedCredentialsProvider set as Arrow's S3Options::credentials_provider. The AWS SDK asks it before every request: it returns the keys the FileIO currently vends for the client's prefix and refreshes them once due, so existing handles and streams pick up new tokens. A token-only refresh keeps the existing clients; only a change of prefixes or settings builds new ones. One caveat: Arrow holds its client registry while a request runs, and building a client locks it exclusively, so a refresh started from a request that needs new clients is left to the next file operation.

wgtmac · 2026-10-13T03:06:22Z

+    }
+    // Expired credentials leave nothing to proceed with, so wait instead.
+    if (!refresh_lock.try_lock_for(kExpiredCredentialWait)) {
+      return;


If this wait times out, the caller still uses credentials we know are expired. A failed refresh does the same. Return an error after expiry instead of creating a handle with unusable credentials.

Thanks! Done. Each prefix's delegate now tracks its own expiry. Once it has passed and the refresh failed or the wait timed out, creating a handle or deleting under that prefix returns TokenExpired. Locations outside vended prefixes keep the default credentials.

wgtmac · 2026-10-13T03:06:22Z

+
+  // `delegates`/`*refreshed` take the retired generation; both are declared
+  // before the lock, so it destructs only after the lock releases.
+  InstallCredentials(*refreshed, delegates);


This installs a returned token even if it is already expired or its expiry is missing or invalid. It can replace a still-valid token and then suppress retries for 30 seconds. Validate temporary credentials before installing them and keep the old set on failure.

Thanks! Done. A refreshed set is installed only if every S3 credential is usable: a session token needs a parseable expiry that has not passed, and the set must contain an S3 prefix. Otherwise the current set is kept and the refresh backs off as for any failed refresh, never past the current expiry. A token without a usable expiry at the initial install still triggers an immediate refresh, but is no longer treated as expired.

wgtmac · 2026-10-13T03:06:22Z

+  // Outside `mutex_`: both are slow and must not block readers.
+  Status status;
+  DelegatesByPrefix delegates;
+  auto refreshed = provider->Load();


The 10-second limit only covers waiting for the refresh lock. The thread running Load() can block indefinitely in the REST request, even while the old credentials are still valid. Add a timeout to the refresh request itself.

Thanks! Done. HttpClient::Get takes an optional timeout, and RestStorageCredentialProvider::Load() uses 10 seconds, so a stalled catalog no longer blocks the refreshing operation indefinitely; a timeout counts as a failed refresh.

Vended storage credentials expire, so new file operations on a reused table could eventually fail. ArrowS3FileIO now refreshes them through the StorageCredentialProvider installed by InitializeStorageCredentials, starting five minutes before the earliest applicable s3.session-token-expires-at-ms. Later SetStorageCredentials calls keep the provider. Each vended S3 client signs through a VendedCredentialsProvider, which the AWS SDK asks before every request: it refreshes once due and returns the keys the FileIO currently vends for the client's prefix, as Java's VendedCredentialsProvider does. Handles and streams opened before a refresh therefore use the new keys, including those on clients a later refresh replaced. A refresh that changes only keys keeps the existing clients. A request cannot build clients, since Arrow holds its client registry for the request and building one locks it exclusively. A refresh it starts that needs new clients is left to the next file operation, and it never waits on another refresh. A refresh is installed only if every S3 credential is usable: a session token must carry a parseable expiry that has not passed. Otherwise, or if loading fails, the current set is kept and the refresh backs off for up to 30 seconds. Concurrent operations share one refresh, and operations with expired credentials wait up to 10 seconds for one in flight. A concurrent credential install supersedes any older refresh result. Once a delegate's credentials have expired and could not be refreshed, creating a handle or deleting under its prefix fails with TokenExpired instead of handing out credentials known to be unusable. The LoadCredentials request had no deadline, so a stalled catalog could block a file operation indefinitely. HttpClient::Get takes an optional timeout, after which the request fails with an IOError, and RestStorageCredentialProvider uses 10 seconds. The bundle now links aws-cpp-sdk-core when S3 is enabled.

wgtmac · 2026-10-14T07:54:03Z

Could you rebase onto main and resolve the conflicts?

wgtmac · 2026-10-14T07:53:51Z

+  if (!refresh_lock.try_lock()) {
+    // Another operation is already fetching; normally just use what we have.
+    if (from_request) {
+      return;


This also skips the refresh when the token has expired. Another request on an open stream then signs with that expired token. Can we wait for a request-safe credential refresh, or fail explicitly instead of returning expired keys?

wgtmac · 2026-10-14T07:53:51Z

+  }
+  auto result = delegates;
+  for (size_t i = 0; i < result.size(); ++i) {
+    if (!result[i].vended_keys || !DiffersOnlyInKeys(*current_s3[i], *refreshed_s3[i])) {


Can we match these credentials by prefix instead of array position? The server may return the same prefixes in a different order. That makes a key-only refresh look like a client change, so an existing stream keeps its old token.

wgtmac · 2026-10-14T07:53:51Z

    return {};
  }
+  // No response to hand the error handler.
+  if (response.error.code == cpr::ErrorCode::OPERATION_TIMEDOUT) {


Check the transport error before accepting the HTTP status. A server can send 200 headers and then stall during the body. CPR keeps status 200 when the transfer times out, so this returns success instead of an IOError.

plusplusjiajia force-pushed the feat-vended-credential-refresh branch 4 times, most recently from d92ef4d to 7088aed Compare August 21, 2026 15:56

plusplusjiajia mentioned this pull request Aug 24, 2026

feat(rest): load table credentials from the LoadCredentials endpoint #899

Merged

plusplusjiajia marked this pull request as draft August 24, 2026 09:56

plusplusjiajia force-pushed the feat-vended-credential-refresh branch from 7088aed to b06186d Compare August 24, 2026 10:49

plusplusjiajia force-pushed the feat-vended-credential-refresh branch 2 times, most recently from afe009b to c15c6e3 Compare September 15, 2026 06:37

plusplusjiajia force-pushed the feat-vended-credential-refresh branch from c15c6e3 to 27d9bab Compare September 26, 2026 05:53

plusplusjiajia mentioned this pull request Sep 26, 2026

refactor(io): hand out S3 delegates by shared_ptr and build them off the lock #898

Merged

plusplusjiajia force-pushed the feat-vended-credential-refresh branch from 27d9bab to bf50711 Compare September 29, 2026 04:19

plusplusjiajia force-pushed the feat-vended-credential-refresh branch from f0f0f83 to a1cf5ef Compare October 9, 2026 05:29

plusplusjiajia marked this pull request as ready for review October 9, 2026 05:29

wgtmac requested changes Oct 13, 2026

View reviewed changes

plusplusjiajia force-pushed the feat-vended-credential-refresh branch from a1cf5ef to 6f8cfe1 Compare October 13, 2026 07:46

wgtmac requested changes Oct 14, 2026

View reviewed changes

Conversation

plusplusjiajia commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wgtmac commented Aug 23, 2026

Uh oh!

plusplusjiajia commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wgtmac commented Sep 24, 2026

Uh oh!

plusplusjiajia commented Sep 29, 2026

Uh oh!

plusplusjiajia commented Oct 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Oct 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

plusplusjiajia commented Aug 19, 2026 •

edited

Loading

plusplusjiajia commented Aug 24, 2026 •

edited

Loading