Skip to content

Commit 99db218

Browse files
abimaelmartellabimaelmartell
committed
update readme
1 parent 0f2f890 commit 99db218

1 file changed

Lines changed: 61 additions & 232 deletions

File tree

‎README.md‎

Lines changed: 61 additions & 232 deletions
Original file line numberDiff line numberDiff line change
@@ -2,256 +2,85 @@
22

33
Fast Rust library for PDF inspection, classification, and text extraction. Intelligently detects scanned vs text-based PDFs to enable smart routing decisions.
44

5-
## Features
6-
7-
- **Smart Detection** - Detects scanned vs text-based PDFs in ~10-50ms by sampling content streams for text operators (`Tj`/`TJ`) without loading the full document
8-
- **Direct Extraction** - Text extraction using [lopdf](https://github.com/J-F-Liu/lopdf) with no external dependencies
9-
- **Structure Detection** - Headers (by font size), lists, code blocks (monospace fonts)
10-
- **CLI Tools** - `detect-pdf` and `pdf2md` binaries included
11-
12-
## Installation
13-
14-
Add to your `Cargo.toml`:
15-
16-
```toml
17-
[dependencies]
18-
pdf-inspector = { git = "https://github.com/firecrawl/pdf-inspector" }
19-
```
20-
21-
## Usage
22-
23-
### Quick Start
24-
25-
The simplest way to convert a PDF to Markdown:
26-
27-
```rust
28-
use pdf_inspector::process_pdf;
29-
30-
fn main() -> Result<(), Box<dyn std::error::Error>> {
31-
let result = process_pdf("document.pdf")?;
32-
33-
match result.pdf_type {
34-
pdf_inspector::PdfType::TextBased => {
35-
println!("Markdown:\n{}", result.markdown.unwrap());
36-
}
37-
pdf_inspector::PdfType::Scanned => {
38-
println!("PDF is scanned - OCR required");
39-
}
40-
_ => {}
41-
}
42-
43-
Ok(())
44-
}
45-
```
46-
47-
### PDF Type Detection
48-
49-
Quickly detect if a PDF is text-based or scanned without full extraction:
50-
51-
```rust
52-
use pdf_inspector::{detect_pdf_type, PdfType};
53-
54-
fn main() -> Result<(), Box<dyn std::error::Error>> {
55-
let result = detect_pdf_type("document.pdf")?;
56-
57-
println!("Type: {:?}", result.pdf_type);
58-
println!("Pages: {}", result.page_count);
59-
println!("Confidence: {:.0}%", result.confidence * 100.0);
60-
61-
if let Some(title) = result.title {
62-
println!("Title: {}", title);
63-
}
64-
65-
match result.pdf_type {
66-
PdfType::TextBased => println!("Ready for text extraction"),
67-
PdfType::Scanned => println!("Needs OCR"),
68-
PdfType::ImageBased => println!("Mostly images"),
69-
PdfType::Mixed => println!("Mix of text and images"),
70-
}
71-
72-
Ok(())
73-
}
74-
```
75-
76-
### Text Extraction
77-
78-
Extract plain text from a PDF:
79-
80-
```rust
81-
use pdf_inspector::extract_text;
82-
83-
fn main() -> Result<(), Box<dyn std::error::Error>> {
84-
let text = extract_text("document.pdf")?;
85-
println!("{}", text);
86-
Ok(())
87-
}
88-
```
89-
90-
### Extract Text with Position Information
91-
92-
Get text items with position data for advanced processing:
93-
94-
```rust
95-
use pdf_inspector::{extract_text_with_positions, TextItem};
96-
use pdf_inspector::extractor::group_into_lines;
97-
98-
fn main() -> Result<(), Box<dyn std::error::Error>> {
99-
let items = extract_text_with_positions("document.pdf")?;
100-
101-
for item in &items {
102-
println!("'{}' at ({}, {}) size={}",
103-
item.text, item.x, item.y, item.font_size);
104-
}
105-
106-
// Group items into lines
107-
let lines = group_into_lines(items);
108-
for line in lines {
109-
println!("Line: {}", line.text());
110-
}
111-
112-
Ok(())
113-
}
114-
```
115-
116-
### Custom Markdown Conversion
117-
118-
Convert text to Markdown with custom options:
119-
120-
```rust
121-
use pdf_inspector::{to_markdown, MarkdownOptions};
122-
123-
fn main() {
124-
let text = "• First item\n• Second item\n\nconst x = 5;";
125-
126-
// With all detection enabled (default)
127-
let md = to_markdown(text, MarkdownOptions::default());
128-
println!("{}", md);
129-
130-
// Disable code detection
131-
let opts = MarkdownOptions {
132-
detect_headers: true,
133-
detect_lists: true,
134-
detect_code: false,
135-
base_font_size: None,
136-
};
137-
let md = to_markdown(text, opts);
138-
println!("{}", md);
139-
}
140-
```
141-
142-
### Processing from Memory
143-
144-
All functions have memory buffer variants for processing PDFs already in memory:
145-
146-
```rust
147-
use pdf_inspector::{process_pdf_mem, detector::detect_pdf_type_mem};
148-
use pdf_inspector::extractor::{extract_text_mem, extract_text_with_positions_mem};
149-
150-
fn main() -> Result<(), Box<dyn std::error::Error>> {
151-
let buffer = std::fs::read("document.pdf")?;
152-
153-
// Process from memory
154-
let result = process_pdf_mem(&buffer)?;
155-
156-
// Or detect only
157-
let detection = detect_pdf_type_mem(&buffer)?;
158-
159-
// Or extract text
160-
let text = extract_text_mem(&buffer)?;
161-
162-
Ok(())
163-
}
164-
```
165-
166-
### Custom Detection Configuration
167-
168-
Fine-tune the detection algorithm:
169-
170-
```rust
171-
use pdf_inspector::detector::{detect_pdf_type_with_config, DetectionConfig};
172-
173-
fn main() -> Result<(), Box<dyn std::error::Error>> {
174-
let config = DetectionConfig {
175-
max_pages_to_sample: 10, // Sample more pages
176-
min_text_ops_per_page: 5, // Require more text operators
177-
text_page_ratio_threshold: 0.8, // Stricter text classification
178-
};
179-
180-
let result = detect_pdf_type_with_config("document.pdf", config)?;
181-
println!("{:?}", result.pdf_type);
182-
183-
Ok(())
184-
}
185-
```
5+
## Supported Features
6+
7+
| Category | Feature | Description |
8+
|----------|---------|-------------|
9+
| **Detection** | Fast Classification | ~10-50ms by sampling content streams |
10+
| | PDF Types | TextBased, Scanned, ImageBased, Mixed |
11+
| | Confidence Scoring | 0.0-1.0 scale for classification certainty |
12+
| | Configurable Thresholds | Tune sampling depth and detection sensitivity |
13+
| | Metadata Extraction | Document title from PDF Info dictionary |
14+
| **Text Extraction** | Plain Text | Direct extraction from text-based PDFs |
15+
| | Position-Aware | Text with X/Y coordinates, font info, page numbers |
16+
| | Multi-Column Support | Automatic detection and proper reading order |
17+
| | Text Encoding | UTF-16BE, UTF-8, and Latin-1 |
18+
| **Headers** | Auto Detection | H1-H4 based on font size ratios |
19+
| **Lists** | Bullet Points | `•`, `-`, `*`, `○`, `●`, `◦` |
20+
| | Numbered Lists | `1.`, `1)`, `(1)` |
21+
| | Letter Lists | `a.`, `a)`, `(a)` |
22+
| **Code Blocks** | Monospace Fonts | Courier, Consolas, Monaco, Menlo, Fira Code, JetBrains Mono |
23+
| | Keyword Detection | Language keywords and syntax patterns |
24+
| **Tables** | Region Detection | Automatic table boundary identification |
25+
| | Column/Row Detection | Position clustering for structure |
26+
| | Markdown Output | Proper alignment and formatting |
27+
| | Footnotes | Extraction and formatting |
28+
| **Text Processing** | Subscript/Superscript | Font size and Y-offset detection |
29+
| | Hyphenation Fixing | Rejoins words broken across lines |
30+
| | Page Number Filtering | Removes isolated page numbers |
31+
| | URL Formatting | Converts URLs to markdown links |
32+
| | Drop Cap Merging | Handles large initial letters |
33+
34+
## Output Formats
35+
36+
| Format | Description |
37+
|--------|-------------|
38+
| Markdown | Headers, lists, code blocks, tables, page breaks |
39+
| Plain Text | Basic text extraction |
40+
| JSON | Metadata with type, confidence, page count, timing |
41+
| Positioned Items | Low-level text with coordinates and font info |
18642

18743
## CLI Tools
18844

189-
### pdf2md
190-
191-
Convert a PDF to Markdown:
192-
193-
```bash
194-
# Output to stdout
195-
pdf2md document.pdf
196-
197-
# Output to file
198-
pdf2md document.pdf output.md
45+
| Tool | Description |
46+
|------|-------------|
47+
| `pdf2md` | Convert PDF to Markdown (supports `--json` output) |
48+
| `detect-pdf` | Detect PDF type without conversion (supports `--json` output) |
19949

200-
# JSON output with metadata
201-
pdf2md document.pdf --json
202-
```
50+
## API Overview
20351

204-
### detect-pdf
52+
### Functions
20553

206-
Detect PDF type without conversion:
54+
| Function | Description |
55+
|----------|-------------|
56+
| `process_pdf` / `process_pdf_mem` | Detect, extract, and convert to markdown |
57+
| `detect_pdf_type` / `detect_pdf_type_mem` | Fast type detection only |
58+
| `extract_text` / `extract_text_mem` | Plain text extraction |
59+
| `extract_text_with_positions` | Text with coordinates |
60+
| `to_markdown` | Convert text to markdown |
20761

208-
```bash
209-
# Human-readable output
210-
detect-pdf document.pdf
62+
### Types
21163

212-
# JSON output
213-
detect-pdf document.pdf --json
214-
```
64+
| Type | Description |
65+
|------|-------------|
66+
| `PdfType` | `TextBased`, `Scanned`, `ImageBased`, `Mixed` |
67+
| `PdfProcessResult` | Full result with text, markdown, and metadata |
68+
| `PdfTypeResult` | Detection result with type, confidence, page count |
69+
| `TextItem` | Text with position, font info, and page number |
70+
| `TextLine` | Grouped items on the same line |
71+
| `MarkdownOptions` | Configuration for markdown conversion |
72+
| `DetectionConfig` | Configuration for PDF type detection |
73+
| `PdfError` | `Io`, `Parse`, `Encrypted`, `InvalidStructure` |
21574

21675
## How Detection Works
21776

218-
Instead of loading the entire PDF, we:
219-
22077
1. Load only metadata (xref table, trailer, page count)
22178
2. Sample first ~5 pages' content streams
22279
3. Scan raw bytes for `Tj`/`TJ` (text) and `Do` (image) operators
22380
4. Classify based on text operator presence
22481

22582
This allows detecting 300+ page PDFs in milliseconds.
22683

227-
## API Reference
228-
229-
### Types
230-
231-
| Type | Description |
232-
|------|-------------|
233-
| `PdfType` | Enum: `TextBased`, `Scanned`, `ImageBased`, `Mixed` |
234-
| `PdfProcessResult` | Full processing result with text, markdown, and metadata |
235-
| `PdfTypeResult` | Detection result with type, confidence, and page count |
236-
| `TextItem` | Text with position (x, y), font info, and page number |
237-
| `TextLine` | Group of `TextItem`s on the same line |
238-
| `MarkdownOptions` | Configuration for markdown conversion |
239-
| `DetectionConfig` | Configuration for PDF type detection |
240-
| `PdfError` | Error type: `Io`, `Parse`, `Encrypted`, `InvalidStructure` |
241-
242-
### Functions
243-
244-
| Function | Description |
245-
|----------|-------------|
246-
| `process_pdf(path)` | High-level: detect, extract, and convert |
247-
| `process_pdf_mem(buffer)` | Same as above, from memory |
248-
| `detect_pdf_type(path)` | Fast type detection |
249-
| `detect_pdf_type_mem(buffer)` | Type detection from memory |
250-
| `extract_text(path)` | Extract plain text |
251-
| `extract_text_mem(buffer)` | Extract text from memory |
252-
| `extract_text_with_positions(path)` | Extract text with coordinates |
253-
| `to_markdown(text, options)` | Convert text to markdown |
254-
25584
## License
25685

25786
MIT

0 commit comments

Comments
 (0)