vgi

vgi

Add your own functions and tables to DuckDB — written in Rust, shipped as one binary.
No C++ extension to compile, no linking against DuckDB, no version coupling.

---

A VGI worker is a small Rust program that DuckDB talks to over Apache Arrow IPC. It can expose scalar / table / aggregate functions and whole catalogs (schemas, tables, views) that behave like native DuckDB objects. DuckDB launches your worker for you when a query needs it — you never run a server by hand.

vgi is the Rust SDK for building those workers. It is byte-for-byte wire-compatible with the canonical Python SDK, so a Rust worker drops in behind the same ATTACH ... (TYPE vgi). Built on vgi-rpc; stock arrow-rs 59.x, MSRV 1.97.

Why a worker instead of a C++ extension?

Traditional DuckDB extension	VGI worker
Written in C/C++, compiled and linked against DuckDB	Written in Rust, one standalone binary
Must be rebuilt for each DuckDB version	Version independent
Complex build / signing / release cycle	cargo build, ship the binary
Runs in-process	Process isolation

Reach for it when you want to: call REST APIs from SQL, run ML inference, expose an external database / API / filesystem as a queryable catalog, or ship domain-specific functions to your team as a single binary.

Your first worker

1. Create a project and add the dependencies (these are exactly what the example below needs):

# Cargo.toml
[dependencies]
vgi = "0.48"
vgi-rpc = "0.32"
arrow-array = "60"
arrow-schema = "60"

2. Write a function and serve it:

// src/main.rs
use std::sync::Arc;

use arrow_array::{cast::AsArray, ArrayRef, RecordBatch, StringArray};
use arrow_schema::DataType;
use vgi::{ArgumentMonotonicity, ArgSpec, FunctionMetadata, ProcessParams, ScalarFunction, Worker};
use vgi_rpc::{Result, RpcError};

/// `upper_case(s)` — uppercase a string column.
struct UpperCase;

impl ScalarFunction for UpperCase {
    fn name(&self) -> &str {
        "upper_case"
    }

    fn metadata(&self) -> FunctionMetadata {
        FunctionMetadata {
            description: "Convert string values to uppercase".into(),
            return_type: Some(DataType::Utf8),
            argument_monotonicity: Some(vec![ArgumentMonotonicity::Unknown]),
            ..Default::default()
        }
    }

    fn argument_specs(&self) -> Vec<ArgSpec> {
        vec![ArgSpec::column("value", 0, "varchar", "String to uppercase")]
    }

    fn process(&self, params: &ProcessParams, batch: &RecordBatch) -> Result<RecordBatch> {
        let col = batch.column(0).as_string::<i32>();
        let upper: StringArray = col.iter().map(|v| v.map(str::to_uppercase)).collect();
        let out: ArrayRef = Arc::new(upper);
        RecordBatch::try_new(params.output_schema.clone(), vec![out])
            .map_err(|e| RpcError::runtime_error(e.to_string()))
    }
}

fn main() {
    let mut worker = Worker::new();
    worker.register_scalar(UpperCase);
    worker.run(); // serves stdio (default), --unix <path>, or --http
}

argument_monotonicity is optional and scalar-only. A present vector has one entry per ordered argument declaration. Fixed, defaulted, and constant arguments each occupy one slot; a vararg declaration occupies one slot even when a call expands it. Named call syntax does not change this order.

3. Build it:

cargo build --release

4. Call it from a DuckDB engine that has the vgi extension. The vgi extension currently ships with Query Farm's Haybarn DuckDB distribution, which starts with no install via uvx haybarn-cli. From your project directory:

-- Haybarn ships the `vgi` extension. DuckDB LAUNCHES the worker for you;
-- LOCATION is the command it runs, and the alias 'demo' is what you
-- qualify functions with in SQL.
ATTACH 'demo' (TYPE vgi, LOCATION './target/release/my-worker');

SELECT demo.main.upper_case(name) FROM (VALUES ('alice'), ('bob')) t(name);
-- ALICE
-- BOB

-- Or drop the prefix:
USE demo;
SELECT main.upper_case('hello');   -- HELLO

LOCATION gotcha: the path is resolved relative to the DuckDB process's working directory, not your project. If the worker isn't found, use an absolute path (e.g. LOCATION '/abs/path/to/target/release/my-worker').

That's it — a native-speed SQL function, shipped as one static binary, with no extension to compile.

Iterating

Change your Rust, rebuild, and re-attach. DuckDB pools the worker process per attachment, so the reliable way to pick up a new build is to re-ATTACH (or start a fresh session):

cargo build --release

DETACH demo;
ATTACH 'demo' (TYPE vgi, LOCATION './target/release/my-worker');

Troubleshooting

• ATTACH can't find the worker — LOCATION is resolved relative to DuckDB's working directory, not your project. Use an absolute path.
• Catalog Error: ... upper_case does not exist — qualify with the attach alias (demo.main.upper_case) or run USE demo; first.
• A runtime error in your function — anything you return as RpcError (or any panic) surfaces in DuckDB's error message; return descriptive errors from process to make debugging easy.
• Type mismatch at the call site — argument_specs is validated at bind time, so a wrong-typed column fails fast with a clear message before any rows flow.

Function types

Register any mix of these via the typed traits in vgi:

Type	Trait	SQL pattern	Use case
Scalar	ScalarFunction	SELECT f(col) FROM t	Per-row transforms (1:1)
Table	TableFunction	SELECT * FROM f(args)	Generate / scan data
Table-In-Out	TableInOutFunction	SELECT * FROM f((SELECT …))	Streaming transforms
Table-Buffering	TableBufferingFunction	SELECT * FROM f((SELECT …))	Aggregate-then-emit (sink → combine → source)
Aggregate	AggregateFunction	SELECT f(col) … GROUP BY …	Grouped / window / streaming aggregates

Each trait is small: name, metadata, argument_specs, an on_bind to resolve the output schema, and process (or the buffering / aggregate lifecycle methods). See filter ownership and typed pruning for safe external-data scans using FilterHandling::Sdk and partial-column candidate masks.

Projection & filter pushdown, ORDER BY / TABLESAMPLE hints, settings, secrets (two-phase bind), bearer auth, and a cross-process state store are handled for you.

Beyond functions: full catalogs

Worker::set_catalog exposes a complete catalog — schemas, function-backed tables, views, and macros — with constraints, column statistics, time travel (AT), and secondary catalogs attachable by name:

ATTACH 'external_db' (TYPE vgi, LOCATION './my-catalog-worker');

SELECT * FROM external_db.main.users;            -- a function-backed table
SELECT * FROM external_db.analytics.daily_view;  -- a view
SELECT external_db.main.transform(col) FROM t;   -- a function

A worker can act as a bridge — databases, APIs, filesystems — presented to DuckDB as native catalogs.

Transports

Worker::run picks the transport from argv:

• stdio (default) — DuckDB spawns the worker per query. Nothing to configure.
• Unix socket (--unix <path>) — one long-lived worker (the launcher contract).
• HTTP (--http) — Arrow-IPC over HTTP with AEAD-sealed stateless stream tokens and optional bearer auth.

Testing your worker

Use vgi-client with features = ["testing"] in your dev dependencies. TestWorker exercises attach, bind, and scan against your own binary, while ScanResult compares typed schemas and unordered rows with duplicate counts and NULLs preserved. Build typed filters with FilterExpr and compare pushed scans with independently filtered full scans using check_pushdown.

The testing guide shows fixture construction, filter-only projections, exact integer checks, and CI commands. The external-data example provides a complete worker with subprocess tests.

Where to go next

• API docs (docs.rs) — every trait and type.
• Example worker — a fixture worker exercising every function kind and full catalogs.

License

Query Farm Source-Available License v1.0 — see LICENSE. Copyright © 2025, 2026 Query Farm LLC.