Repository navigation
fix(io): attempt every S3 delete and report how many failed - #966
Merged
wgtmac merged 11 commits intoOct 13, 2026
Merged
Conversation
plusplusjiajia
force-pushed
the
fix-s3-delete-files-attempt-all
branch
from
September 26, 2026 06:30
fee511b to
1f5fb83
Compare
zhjwpku
approved these changes
Sep 26, 2026
zhjwpku
reviewed
Sep 27, 2026
ArrowS3FileIO::DeleteFiles grouped locations by credential prefix and returned at the first group that failed, so files in later groups were never attempted. Java's S3FileIO attempts every batch, logs each failed path and reports the failure count. Delete each file through its delegate, log each failure, and return "Failed to delete N of M files" at the end. This sends the same requests as before, since Arrow's S3 DeleteFiles also deletes one file at a time. ResolvingFileIO stays fail-fast across delegates, as it is in Java.
Java's S3FileIO runs its deletes on an executor sized by s3.delete.num-threads, which defaults to the number of processors. DeleteFiles now does the same, with the calling thread taking part. Java also packs keys into DeleteObjects batches. Arrow has no public batch delete for S3, so each thread still deletes one file at a time, as Arrow's own DeleteFiles does.
std::jthread needs -fexperimental-library with libc++ 18 and 19, which the Clang 18+ requirement covers, so use std::async futures instead; they also wait for their thread if an exception unwinds. Helper threads now bind the caller's logger, as logger.h prescribes for thread pools. Without it their warnings went to the global logger while the returned error only carries the failure count.
wait() leaves an exception stored in a helper's future, so a delete that threw was neither counted nor reported, and DeleteFiles could return success. get() rethrows it on the calling thread, as the sequential loop did.
plusplusjiajia
force-pushed
the
fix-s3-delete-files-attempt-all
branch
from
September 29, 2026 04:11
41ff77e to
492c6cb
Compare
Each DeleteFiles call started its own threads, so s3.delete.num-threads bounded one call rather than the FileIO: concurrent calls could run many times that number. The FileIO now owns an Arrow thread pool of that size, created with the FileIO but starting workers on the first delete, and runs each batch on it through TaskGroup, as Java's S3FileIO does with its executor. Failures are still logged per file and reported as a count.
wgtmac
requested changes
Oct 10, 2026
plusplusjiajia
force-pushed
the
fix-s3-delete-files-attempt-all
branch
from
October 10, 2026 09:02
f319cff to
7db5868
Compare
Use a shared caller-owned executor, defaulting to sequential deletion. Bound tasks, count delete exceptions, and drain accepted work on failure.
plusplusjiajia
force-pushed
the
fix-s3-delete-files-attempt-all
branch
from
October 10, 2026 09:11
7db5868 to
e121314
Compare
A rejected or throwing Submit returned its own error: with no worker accepted, no file was attempted, and otherwise the accepted workers still deleted every file while the failure count was lost. Log the rejection and let the calling thread take the files no accepted worker has, then report the count as usual. A rejected worker's future is still waited for, since Submit may accept a task before failing. Also name the worker bound, forward-declare Executor in arrow_io_util.h, and make the Executor docs say callbacks throughout.
plusplusjiajia
force-pushed
the
fix-s3-delete-files-attempt-all
branch
from
October 10, 2026 11:28
c79a6e9 to
4b042f8
Compare
wgtmac
approved these changes
Oct 12, 2026
wgtmac
left a comment
wgtmac
left a comment
Member
There was a problem hiding this comment.
Thanks for your quick update!
I pushed a small follow-up: executor initialization is now immutable to make its lifetime contract clearer, and s3.delete.num-threads limits workers per call without creating a pool. (Sorry for my confusing comment which suggested to remove this flag exactly). BTW, submission failures are returned after draining accepted work, rather than falling back to the caller and bypassing the executor’s concurrency limit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #898 (comment).
ArrowS3FileIO::DeleteFilesreturned at the first credential prefix whose delete failed, so files under the remaining prefixes were never attempted. It now attempts every file unless task submission fails, logs deletion failures, and returnsFailed to delete N of M files. Exceptions thrown by individual deletes are counted too.InitializeS3FileIODeleteExecutor(Executor*)initializes a caller-owned executor shared by all S3 FileIOs, including registry-created instances. Initialize it before the first delete; otherwise, the first delete freezes the defaultnullptr. Repeating the same pointer is allowed, but replacing it is not. The executor must remain alive until the last delete completes.s3.delete.num-threadslimits worker tasks per call, defaulting to the hardware thread count. Workers share a file index, keeping task/future storage bounded by the configured parallelism. The shared executor controls total concurrency; this property does not create or resize a thread pool.Arrow has no public S3 batch-delete API, so each request still deletes one file.
ResolvingFileIOkeeps its existing fail-fast behavior across delegates.Tests cover deletion across allowed/denied prefixes, thrown deletes, rejected/throwing submissions, bounded task counts and concurrency, immutable executor initialization, and executor configuration for both direct and registry-created FileIOs.