Skip to content

Commit a15ec2d

Browse files
abimaelmartellabimaelmartell
authored
docs(readme): refresh local parser benchmark results (#192)
* docs(readme): refresh parser benchmark results * docs(readme): include stored parser results * docs(readme): refresh remaining parser results * docs: sync benchmark references
1 parent 7b7960e commit a15ec2d

5 files changed

Lines changed: 34 additions & 31 deletions

File tree

‎README.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,17 +28,17 @@ Evaluated on the [opendataloader-bench](https://github.com/opendataloader-projec
2828

2929
| Engine | Overall | Reading Order (NID) | Tables (TEDS) | Headings (MHS) | Speed (200 docs) |
3030
|---|---|---|---|---|---|
31-
| pdf-inspector | **0.875** | **0.915** | **0.814** | 0.788 | **2.8s** |
32-
| liteparse | 0.870 | 0.908 | 0.693 | **0.811** | 13.9s |
33-
| opendataloader | 0.843 | 0.912 | 0.489 | 0.760 | 9.8s |
34-
| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 15.5s |
35-
| markitdown | 0.583 | 0.879 | 0.000 | 0.000 | 6.7s |
31+
| pdf-inspector | **0.875** | **0.915** | **0.814** | 0.788 | **0.470s** |
32+
| liteparse | 0.873 | 0.913 | 0.693 | **0.811** | 0.750s |
33+
| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 2.569s |
34+
| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 17.117s |
35+
| markitdown | 0.589 | 0.844 | 0.273 | 0.000 | 16.165s |
3636

37-
Results were refreshed on July 16, 2026, on an Apple M4 Pro. Engine versions were pdf-inspector 0.1.6, LiteParse 2.6.0, OpenDataLoader 2.1.1, PyMuPDF4LLM 0.2.0, and MarkItDown 0.1.4. Speed is the median of three complete corpus runs.
37+
Results were refreshed on July 31, 2026, on an Apple M4 Pro. Engine versions were pdf-inspector 0.2.6, LiteParse 2.10.1, OpenDataLoader 2.2.1, PyMuPDF4LLM 0.2.0, and MarkItDown 0.1.5. Speed is the median of five alternating or rotating complete corpus runs after an excluded warm-up run, with each parser processing documents sequentially in a single process.
3838

39-
For context, engines that use OCR or model-based document parsing (docling, marker, mineru) score 0.83-0.88 overall but take 2-180 minutes on the same corpus — pdf-inspector reaches the top of that range without either, in 2.8 seconds.
39+
The complete parser configuration, per-document predictions, evaluator output, and generated charts are available in the [reproducible results branch](https://github.com/firecrawl/opendataloader-bench/tree/abi/pdf-parser-benchmark-results).
4040

41-
**Best fit:** Native-text PDFs where speed, reading order, and table structure matter. pdf-inspector delivered the highest overall, reading-order, and table scores, along with the fastest complete run in this benchmark. That makes it a strong local default for reports, research papers, financial documents, invoices, and legal PDFs that need clean, structured Markdown without adding OCR latency or infrastructure.
41+
**Best fit:** Native-text PDFs where speed, reading order, and table structure matter. In this comparison, pdf-inspector delivered the higher overall, reading-order, and table scores, along with the fastest complete run. That makes it a strong local default for reports, research papers, financial documents, invoices, and legal PDFs that need clean, structured Markdown without adding OCR latency or infrastructure.
4242

4343
Use the [paired benchmark harness](docs/benchmarking.md) to compare two local builds against the exact same corpus and evaluator revision.
4444

‎docs/benchmarking.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,14 @@ overwrite one another.
3030

3131
## Published comparison protocol
3232

33-
The public benchmark table was refreshed on July 16, 2026, on an Apple M4 Pro
34-
using pdf-inspector 0.1.6, LiteParse 2.6.0, OpenDataLoader 2.1.1,
35-
PyMuPDF4LLM 0.2.0, and MarkItDown 0.1.4. Every engine processed the same 200
36-
PDFs with OCR disabled. Reported speed is the median of three complete corpus
37-
runs; quality scores come from the benchmark evaluator over all 200 outputs.
33+
The public benchmark table was refreshed on July 31, 2026, on an Apple M4 Pro
34+
using pdf-inspector 0.2.6, LiteParse 2.10.1, OpenDataLoader 2.2.1,
35+
PyMuPDF4LLM 0.2.0, and MarkItDown 0.1.5. Every engine processed the same 200
36+
PDFs sequentially in a single process with OCR disabled. Reported speed is the
37+
median of five alternating or rotating complete corpus runs after an excluded
38+
warm-up run; quality scores come from the benchmark evaluator over all 200
39+
outputs. Raw timings, predictions, evaluations, and charts are available in the
40+
[results branch](https://github.com/firecrawl/opendataloader-bench/tree/abi/pdf-parser-benchmark-results).
3841

3942
## Optional backend evidence probe
4043

‎docs/python.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,13 @@ Built by [Firecrawl](https://firecrawl.dev) to handle text-based PDFs locally in
1818

1919
| Engine | Overall | Reading order | Tables (TEDS) | Headings | Speed |
2020
|---|---|---|---|---|---|
21-
| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | **2.8s** |
22-
| liteparse | 0.870 | 0.908 | 0.693 | **0.811** | 13.9s |
23-
| opendataloader | 0.843 | 0.912 | 0.489 | 0.760 | 9.8s |
24-
| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 15.5s |
25-
| markitdown | 0.583 | 0.879 | 0.000 | 0.000 | 6.7s |
21+
| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | **0.470s** |
22+
| liteparse | 0.873 | 0.913 | 0.693 | **0.811** | 0.750s |
23+
| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 2.569s |
24+
| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 17.117s |
25+
| markitdown | 0.589 | 0.844 | 0.273 | 0.000 | 16.165s |
2626

27-
Refreshed July 16, 2026, on Apple M4 Pro; speed is the median of three complete corpus runs. Full methodology and versions are in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
27+
Refreshed July 31, 2026, on Apple M4 Pro; speed is the median of five complete corpus runs after an excluded warm-up. Full methodology and versions are in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark), with raw timings and artifacts in the [results branch](https://github.com/firecrawl/opendataloader-bench/tree/abi/pdf-parser-benchmark-results).
2828

2929
## Install
3030

‎docs/rust-api.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,13 @@ Built by [Firecrawl](https://firecrawl.dev) to handle text-based PDFs locally in
1818

1919
| Engine | Overall | Reading order | Tables (TEDS) | Headings | Speed |
2020
|---|---|---|---|---|---|
21-
| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | **2.8s** |
22-
| liteparse | 0.870 | 0.908 | 0.693 | **0.811** | 13.9s |
23-
| opendataloader | 0.843 | 0.912 | 0.489 | 0.760 | 9.8s |
24-
| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 15.5s |
25-
| markitdown | 0.583 | 0.879 | 0.000 | 0.000 | 6.7s |
21+
| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | **0.470s** |
22+
| liteparse | 0.873 | 0.913 | 0.693 | **0.811** | 0.750s |
23+
| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 2.569s |
24+
| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 17.117s |
25+
| markitdown | 0.589 | 0.844 | 0.273 | 0.000 | 16.165s |
2626

27-
Refreshed July 16, 2026, on Apple M4 Pro; speed is the median of three complete corpus runs. Full methodology and versions are in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
27+
Refreshed July 31, 2026, on Apple M4 Pro; speed is the median of five complete corpus runs after an excluded warm-up. Full methodology and versions are in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark), with raw timings and artifacts in the [results branch](https://github.com/firecrawl/opendataloader-bench/tree/abi/pdf-parser-benchmark-results).
2828

2929
## Install
3030

‎napi/README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,13 @@ Built by [Firecrawl](https://firecrawl.dev) for hybrid OCR pipelines — extract
1818

1919
| Engine | Overall | Reading order | Tables (TEDS) | Headings | Speed |
2020
|---|---|---|---|---|---|
21-
| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | **2.8s** |
22-
| liteparse | 0.870 | 0.908 | 0.693 | **0.811** | 13.9s |
23-
| opendataloader | 0.843 | 0.912 | 0.489 | 0.760 | 9.8s |
24-
| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 15.5s |
25-
| markitdown | 0.583 | 0.879 | 0.000 | 0.000 | 6.7s |
21+
| **pdf-inspector** | **0.875** | **0.915** | **0.814** | 0.788 | **0.470s** |
22+
| liteparse | 0.873 | 0.913 | 0.693 | **0.811** | 0.750s |
23+
| opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 2.569s |
24+
| pymupdf4llm | 0.735 | 0.886 | 0.401 | 0.424 | 17.117s |
25+
| markitdown | 0.589 | 0.844 | 0.273 | 0.000 | 16.165s |
2626

27-
Refreshed July 16, 2026, on Apple M4 Pro; speed is the median of three complete corpus runs. Full methodology and versions are in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark).
27+
Refreshed July 31, 2026, on Apple M4 Pro; speed is the median of five complete corpus runs after an excluded warm-up. Full methodology and versions are in the [repo README](https://github.com/firecrawl/pdf-inspector#benchmark), with raw timings and artifacts in the [results branch](https://github.com/firecrawl/opendataloader-bench/tree/abi/pdf-parser-benchmark-results).
2828

2929
## Install
3030

0 commit comments

Comments
 (0)