[python] Defer BLOB payload reads until after filtering

[XiaoHongbo-Hope]

Purpose

Filtered data-evolution scans previously materialized projected BLOB payloads before applying residual filters and limits. This made filtered-out rows still incur payload I/O.

Read eligible BLOB fields as descriptors through merge, filter and limit, then materialize only surviving rows. Predicates referencing the BLOB field keep the eager path.

Tests

• deferred_blob_resolve_test.py
• data_evolution_deletion_vector_test.py
• test_limit_pushdown.py
• DedicatedFormatWriterTest#test_update_blob_column
• DedicatedFormatWriterTest#test_array_blob_column_write_read_and_update
• DedicatedFormatWriterTest#test_map_blob_column_write_read_and_update

[Copilot]

🟡 Not ready to approve

The newly introduced use of AuthFilterReader in the data-evolution chain can drop file_io metadata (since AuthFilterReader doesn’t adopt it), which can break descriptor-mode blob access and should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR improves PyPaimon data-evolution read efficiency by deferring scalar BLOB payload materialization until after row selection (filters/auth filters/limits), so filtered-out rows do not incur payload I/O; it also adds an option to restore eager behavior.

Changes:

• Add a deferred BLOB materialization stage for data-evolution reads and integrate it into the split-read pipeline.
• Adjust limit handling and parallel read decisions to avoid materializing BLOB payloads from splits that would be discarded by LIMIT.
• Add read.defer-blob-resolve option, documentation, and new tests validating deferred/eager behavior across predicates, limits, auth filters, and null payloads.

File summaries

File	Description
paimon-python/pypaimon/tests/deferred_blob_resolve_test.py	New tests asserting deferred BLOB reads with predicate/limit/auth filters and verifying eager fallback cases.
paimon-python/pypaimon/read/table_read.py	Wires remaining-limit per split, adds parallelism gating when deferred BLOB + LIMIT may prune, and passes auth-filter information into split reads.
paimon-python/pypaimon/read/split_read.py	Computes deferred BLOB field set, reads deferred scalar BLOBs as descriptors, applies post-merge auth filtering, and adds DeferredBlobResolveReader.
paimon-python/pypaimon/read/reader/deferred_blob_resolve_reader.py	New reader that materializes deferred BLOB payloads into Arrow batches after row selection.
paimon-python/pypaimon/common/options/core_options.py	Adds read.defer-blob-resolve option and accessor.
docs/docs/pypaimon/blob.md	Documents deferred materialization behavior and the opt-out configuration.

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 3
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Human review recommended

It changes core read-path semantics (filter/limit/auth interactions plus concurrency behavior) and warrants final human validation despite strong test additions.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[JingsongLi]

Why need read.defer-blob-resolve=false to restore the previous behavior?

[XiaoHongbo-Hope]

Why need read.defer-blob-resolve=false to restore the previous behavior?

Removed