wayruntime

Open source under Apache-2.0 — the whole repository, core included. Use it in production, commercially, in closed-source software; no separate license to buy.

A self-contained CPU inference runtime for GGUF language models. One static library, one public header, one CLI. It loads a GGUF file with a hostile-input posture, tokenizes, runs the transformer decode with a per-session KV cache, samples, and streams tokens back — with zero third-party code and zero network code in the tree.

Lineage. wayruntime is extracted from the AI stack of an unreleased, from-scratch operating system, where local inference is a built-in system service rather than an application dependency. This is that engine, matured and ported to Linux and Windows. The OS ships later; this runs today.

Why

• No dependency surface. The library is 100% first-party C11: no BLAS, no vendored parsers, no HTTP client — the tree contains no network code at all (grep -ri socket src include comes back empty). What you audit is what runs.
• Determinism as a feature. Parallel matmul is bit-exact: output is identical for any worker-thread count. Batched decode is bit-exact versus serial stepping. Samplers are seeded and replay deterministically. Greedy output is byte-identical between the Linux and Windows builds of the same model.
• Verified against the reference. Greedy decode agrees with llama.cpp 20/20 on teacher-forced argmax over the same GGUF (Qwen3-0.6B); byte-level tokenization is canonical BPE with the Qwen2-family pretokenizer, scoring 78/78 exact id-sequence matches against llama.cpp's tokenizer on a mixed corpus.
• Errors, never guesses. Unsupported architectures and tensor dtypes are refused explicitly. Context overflow is an error, not a silent truncation. A requested SIMD level the host cannot bind is an error, never a silent fallback.

What's inside

• GGUF loader, hardened: header version gate, tensor bounds and overlap validation against the real file size, hostile-input posture throughout (see docs/SECURITY.md)
• BPE tokenizer with byte-level and SentencePiece modes
• Transformer decode for llama-family, Qwen3, and Gemma-class (experimental) models: GQA, flash-attention tiling, RoPE, F16 KV cache
• Quantized weights: F32, F16, BF16, Q4_0, Q8_0, Q4_K, Q5_K, Q6_K compute paths; Q4_1/Q5_0/Q5_1 are refused explicitly rather than mis-executed
• Runtime SIMD dispatch: scalar, AVX2, AVX-512, NEON — probed at engine creation, switchable at runtime
• Persistent worker pool with bit-exact parallel matmul
• Batched decode: up to 16 sessions stepped together, their LM-head projections coalesced into one GEMM, bit-exact versus serial
• Sampler: greedy, temperature, top-k, top-p, repetition penalty; seeded, deterministic replay
• Streaming token callback and grammar/token-mask constrained decoding (see examples/grammar_mask.c)
• Chat-template autodetect: ChatML variants including Qwen3 /no_think handling, Gemma turn markers

Quick start

make && make test          # offline gate: gcc, make, python3; zero warnings
make check                 # + integration suite; needs one small GGUF (docs/TESTING.md)
./build/posix/wayrt verify  model.gguf     # load, print metadata, unload
./build/posix/wayrt generate --prompt "The capital of France is" model.gguf
./build/posix/wayrt chat    model.gguf     # interactive, streaming
./build/posix/wayrt bench   model.gguf     # tokens/s + engine counters

Windows: make WIN=1 cross-compiles build/win/wayrt.exe (mingw-w64). Same commands, same output — greedy generation is byte-identical to the Linux build.

Model weights use a read-only file mapping by default, reducing weight copies and startup memory. If mapping is unavailable the loader falls back to streamed reads automatically. Use the global --no-mmap flag when an environment requires copied weight storage.

Using the library

Everything is behind one header, include/wayruntime/wayruntime.h (Apache-2.0, freestanding, C11): engine → model → session → generate.

#include <stdio.h>
#include <wayruntime/wayruntime.h>

int main(int argc, char **argv)
{
    (void)argc;
    wr_engine *eng; wr_model *mdl; wr_session *ses;
    if (wr_engine_create(NULL, &eng) != WR_OK) return 1;
    if (wr_model_load(eng, argv[1], NULL, &mdl) != WR_OK) return 1;
    if (wr_session_create(mdl, NULL, &ses) != WR_OK) return 1;
    wr_generate_params p = { .prompt = "The capital of France is",
                             .max_tokens = 32, .stop_token = -1 };
    wr_generate_result out;
    if (wr_generate(ses, &p, &out) == WR_OK) { printf("%s\n", out.text); wr_free(out.text); }
    wr_session_destroy(ses); wr_model_free(mdl); wr_engine_destroy(eng);
    return 0;
}

gcc -std=c11 -O2 -Wall -Wextra -Iinclude hello.c build/posix/libwayruntime.a -lpthread -lm
./a.out model.gguf

This example compiles warning-free and runs, exactly as shown, against the built library. Lower-level control (prefill/step loops, raw logits, batched stepping, token masks) is the same header; the wr_generate facade is just the short path.

Passing NULL model parameters (as above) prefers mmap with a safe streamed fallback. To opt out explicitly, zero-initialize wr_model_params, set use_mmap = WR_MMAP_DISABLED, and pass its address to wr_model_load.

Status — what works, what doesn't

Works today, and is covered by layered tests (unit + golden numeric self-tests, an offline integration suite, real-model CLI runs, a differential suite against llama.cpp, and native Windows runs — see docs/TESTING.md for how to run each layer and what it proves):

• zero-warning C11 build on gcc and mingw-w64 (-Wall -Wextra -Wshadow -Wvla)
• 57/57 unit checks, including 10 golden numeric self-tests, plus a 77-check integration battery (hostile-GGUF refusals and a concurrent-sessions-vs-sequential gate included)
• greedy decode verified 20/20 teacher-forced argmax agreement with llama.cpp on the same GGUF (Qwen3-0.6B)
• tokenization verified against llama.cpp's tokenizer on the same GGUF: 78/78 exact id-sequence matches on a mixed corpus (prose, code, multi-space runs, CJK, contractions, special tokens)
• byte-identical greedy output, Linux vs Windows
• no libm dependency in the default build's hot path (first-party polynomial math; make MATH_APPROX=0 selects libm instead)

Not yet (deliberately, v0.1):

• CPU only — no GPU backends
• library + CLI only — no server, no streaming HTTP endpoint
• little-endian hosts only (big-endian is refused at engine creation, not mis-run)
• gcc / mingw-w64 only: the kernels use GCC vector extensions; MSVC is out of scope by design
• the byte-level pretokenizer classifies Unicode with compact range tables, not the full Unicode database: scripts outside the tables can split at different boundaries than upstream (bytes are never altered); the SentencePiece mode has no pretokenizer, matching the origin engine it was validated against
• Gemma-class model support is experimental and not yet real-model-tested
• default context cap is 4096 tokens (raise per model via wr_model_params.max_context, up to the compiled attention cap)
• no external security audit — see docs/SECURITY.md

Layout

include/wayruntime/     the public C API — one freestanding header
src/core/               engine, SIMD kernels, quant codecs, GGUF loader,
                        tokenizer, model graph, sessions, batch, sampler
src/platform/posix,win/ threads, file mapping, CPU feature probes
src/cli/                the wayrt command-line tool
examples/               Apache-2.0 SDK examples (grammar-masked decoding)
test/                   unit + golden, integration, real-model suites
docs/                   SECURITY.md, TESTING.md

Security

The trust boundary is the GGUF file: it is treated as hostile input until validated. docs/SECURITY.md describes the threat model and how to report vulnerabilities privately — do not open public issues for security reports.

License

Apache License 2.0 — every first-party file, core included. The public header, the runtime, the platform layers, the CLI, the examples, the tests, and the docs are all Apache-2.0. Embedded SPDX identifiers and REUSE.toml are authoritative per file; LICENSING.md explains the whole picture.

You may use wayruntime in production, commercially, and in closed-source software, subject to Apache-2.0's notice and attribution requirements. Linking libwayruntime.a carries no obligation beyond Apache-2.0's own terms. Attribution for redistributions is in NOTICE.

Earlier releases carried a BUSL-1.1 core with an Apache-2.0 API surface. The repository was relicensed in full to Apache-2.0 on 2026-09-02; the Change Date and commercial-license route are gone.

Contributions are accepted for the whole repository under Apache-2.0 with DCO 1.1 sign-off, and no CLA. See CONTRIBUTING.md and DCO. What the binaries link against is inventoried in THIRD-PARTY-NOTICES.md.