Skip to content

One parquet engine - #2744

Merged
norberttech merged 3 commits into
1.xfrom
one-parquet-engine
Nov 17, 2026
Merged

norberttech merged 3 commits into
1.xfrom
one-parquet-engine

Conversation

@norberttech

Copy link
Copy Markdown
Member

Resolves: #xxx

Change Log


Added

  • flow-php/flow-php-ext - Parquet engine FlowPhpParquetEngine, counted by the memory limit of a run
  • flow-php/flow-php-ext - shared crate flow-parquet-engine with the Parquet Rust code
  • flow-php/etl-adapter-parquet - CI runs the adapter suites with each extension and with both

Fixed

  • flow-php/flow-php-ext - freed large buffers return to the system on Linux
  • flow-php/etl-adapter-parquet - a worker loads the extension of its engine
  • flow-php/etl - redundant variable warnings in ExternalSort

Changed

  • flow-php/etl-adapter-parquet - default engine is FlowPhpParquetEngine with flow_php, else PhpParquetEngine
  • flow-php/flow-php-ext - reads and writes Parquet without arrow
  • flow-php/flow-php-ext - arrow-* crates 59 to 60
  • flow-php/arrow-ext - Parquet code moved to the shared crate
  • flow-php/arrow-ext - FLOW_ARROW_ABI 2 to 3
  • flow-php/arrow-ext - release build is stripped

Removed

  • flow-php/arrow-ext - RustBatchReader, RustParquetBatch, RustArrowSchema
  • flow-php/arrow-ext - RustParquetFileWriter::writeArrowBatch()
  • flow-php/flow-php-ext - RustColumnsBatch and the crate flow-arrow-c

Deprecated

Security

…h flow_php

- crate flow-parquet-engine, linked into both extensions
- flow_php on arrow-* 60
- split, PIE archive and Nix follow the links
- CI runs the Parquet adapter suites per extension lane
- arrow-ext release build stripped
…e memory limit

- FlowPhpParquetEngine, the adapter default with flow_php
- no Arrow C hop between the extensions
- ETL without flow_php reads and writes on the PHP engine
- arrow-ext without the batch classes, FLOW_ARROW_ABI 3
- redundant variable warnings in ExternalSort
…on Linux

- mmap threshold fixed at 128 KiB at startup, the environment still wins
- phpt 001 no longer depends on the extension load order
@norberttech norberttech added this to the 0.48.0 milestone Nov 17, 2026
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.17647% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.07%. Comparing base (cb97b82) to head (7a99d32).
⚠️ Report is 1 commits behind head on 1.x.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@             Coverage Diff              @@
##                1.x    #2744      +/-   ##
============================================
- Coverage     91.08%   91.07%   -0.01%     
- Complexity    32248    32255       +7     
============================================
  Files          2238     2238              
  Lines         93175    93192      +17     
============================================
+ Hits          84865    84879      +14     
- Misses         8310     8313       +3     
Components Coverage Δ
etl 96.23% <100.00%> (ø)
cli 87.40% <ø> (ø)
lib-array-dot 96.52% <ø> (ø)
lib-azure-sdk 67.98% <ø> (ø)
lib-doctrine-dbal-bulk 94.44% <ø> (ø)
lib-filesystem 89.08% <ø> (ø)
lib-types 94.64% <ø> (ø)
lib-parquet 82.80% <ø> (ø)
lib-parquet-viewer 83.15% <ø> (ø)
lib-snappy 89.86% <ø> (ø)
lib-dremel 0.00% <ø> (ø)
lib-postgresql 90.43% <ø> (ø)
lib-telemetry 86.85% <ø> (ø)
bridge-filesystem-async-aws 96.72% <ø> (ø)
bridge-filesystem-azure 95.34% <ø> (ø)
bridge-filesystem-sftp 90.29% <ø> (ø)
bridge-monolog-http 96.84% <ø> (ø)
bridge-monolog-telemetry 92.85% <ø> (ø)
bridge-openapi-specification 91.95% <ø> (ø)
symfony-http-foundation 80.76% <ø> (ø)
bridge-symfony-console 98.01% <ø> (ø)
bridge-psr18-telemetry 100.00% <ø> (ø)
bridge-psr3-telemetry 98.95% <ø> (ø)
bridge-psr7-telemetry 100.00% <ø> (ø)
bridge-telemetry-otlp 90.11% <ø> (ø)
bridge-symfony-http-foundation-telemetry 92.85% <ø> (ø)
bridge-symfony-filesystem-bundle 93.15% <ø> (ø)
bridge-symfony-filesystem-cache 98.18% <ø> (ø)
bridge-symfony-postgresql-bundle 93.72% <ø> (ø)
bridge-symfony-postgresql-cache 94.41% <ø> (ø)
bridge-symfony-postgresql-messenger 98.80% <ø> (ø)
bridge-symfony-postgresql-session 93.65% <ø> (ø)
bridge-symfony-telemetry-bundle 90.76% <ø> (ø)
adapter-chartjs 82.46% <ø> (ø)
adapter-csv 97.68% <ø> (ø)
adapter-doctrine 95.14% <ø> (ø)
adapter-google-sheet 97.53% <ø> (ø)
adapter-http 87.40% <ø> (ø)
adapter-json 90.62% <ø> (ø)
adapter-logger 47.05% <ø> (ø)
adapter-parquet 96.95% <90.62%> (-0.26%) ⬇️
adapter-text 97.68% <ø> (ø)
adapter-xml 98.01% <ø> (ø)
adapter-avro 6.25% <ø> (ø)
adapter-excel 96.79% <ø> (ø)
adapter-postgresql 97.35% <ø> (ø)
adapter-seal 90.00% <ø> (ø)
bridge-phpunit-postgresql 75.30% <ø> (ø)
bridge-phpunit-telemetry 87.36% <ø> (ø)
bridge-phpstan-types 0.00% <ø> (ø)
bridge-postgresql-valinor 100.00% <ø> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@norberttech
norberttech merged commit bcb7314 into 1.x Nov 17, 2026
31 checks passed
@norberttech
norberttech deleted the one-parquet-engine branch November 17, 2026 01:32
@github-project-automation github-project-automation Bot moved this from Todo to Done in Roadmap Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant