Huncho

Huncho is a Rust server for decision models. Give it text or JSON and a set of questions, and it returns structured answers with calibrated probabilities. For example, you can ask which team should handle a support ticket or whether a customer is requesting a refund.

It runs Jev-style "System One" models: models that score possible answers in a forward pass, with no text generation or token-by-token decode loop.

state + questions  ->  Huncho  ->  answers + probabilities

Huncho implements the Jev /v1/systemone API. You can use the TypeSafe Python SDK by pointing its base URL at your Huncho server.

Installation

Fedora and EPEL 9

On supported x86_64 systems, install from the Huncho COPR repository:

sudo dnf copr enable quadsdev/huncho
sudo dnf install huncho

The RPM includes ONNX and native CPU/CUDA runtimes. Kev and Clef use a compatible NVIDIA GPU when available and otherwise run on CPU. CPU hosts need no NVIDIA packages. See GPU setup for driver and runtime requirements, or RPM packaging to build your own package.

From source

You'll need a current stable Rust toolchain with Cargo, Git, a C/C++ compiler, pkg-config, and OpenSSL development headers.

git clone https://github.com/grafuls/huncho.git
cd huncho
cargo install --locked --path crates/huncho-cli --features onnx,clef

This installs the CPU runtimes for ONNX, Laya, Kev, and Clef, including Hub downloads and native tokenization. Cargo installs huncho in ~/.cargo/bin; make sure that directory is on your PATH:

export PATH="$HOME/.cargo/bin:$PATH"

Add that line to your shell's startup file to keep it for future terminals. The first installation downloads dependencies and a prebuilt ONNX Runtime. For a mock-only installation, omit --features onnx,clef. CUDA builds also require the NVIDIA CUDA toolkit; see the build guide and backend guide for optional runtime features.

Check that Huncho is installed:

huncho --version

Quick start

After installing Huncho, you'll need curl for the example requests.

Start a server with the built-in mock model:

huncho serve --mock --bind 127.0.0.1:8080

The mock needs no model downloads and returns deterministic test values. Use it to try the API and check your integration; its answers don't reflect the meaning of your input.

In another terminal, check which models are available:

curl -s http://127.0.0.1:8080/v1/models

Then ask two questions about a customer message:

curl -s -X POST http://127.0.0.1:8080/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "state": "The customer wants a refund because the shoes are too small.",
    "model": "mock",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which team handles this?",
        "criteria": { "returns": "Refund request", "billing": "Charging issue" }
      },
      "is_refund": {
        "type": "noul",
        "instructions": "Is the customer requesting a refund?"
      }
    }
  }'

The mock returns:

{
  "model": "mock",
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": { "billing": 0.5034228, "returns": 0.4965772 },
      "confidence": 0.0068455935
    },
    "is_refund": { "type": "noul", "noul": 0.4965772 }
  },
  "usage": { "input_tokens": 54, "output_tokens": 0 }
}

Each answer uses the question ID you supplied. The API supports these types:

Type	Use it for	Result
choice	Choosing among named options, such as a support team.	Chosen label, probabilities for all options, and confidence.
noul	A yes/no question.	Probability that the answer is yes.
score	Rating something on ordered levels, such as low/medium/high.	Per-level probabilities, a probability-weighted score, and confidence.

See the API guide for request formats, error responses, and optional response details.

Run a real model

Huncho can load a local model package or resolve one from Hugging Face. A package's huncho-model.json describes its weights, decision head, prompt format, and calibration settings. See the package format if you're preparing your own model.

The RPM and source installation above include the runtimes for the examples below. Check the setup guide for your model:

Model or runtime	Setup guide
ONNX package	ONNX backend
Laya / ModernBERT or Kev / Qwen safetensors	Native backends
Cloudflare Clef	Clef
Optional CPU llama.cpp or vLLM runtime	llama.cpp, vLLM

The optional llama.cpp and vLLM backends require the installation steps in their guides.

Choose a model

huncho models lists the curated shortlist — which models ship reference goldens, which serve immediately, and which need calibration first:

huncho models

The quickest real-model start is the pinned Kev-0.8B package; see Kev from Hugging Face below. The first load downloads the adapter, tokenizer, pointer head, and pinned base checkpoint (~2 GB). It serves in trial mode: the upstream repository ships no golden file, so its status is unqualified until you pin a labeled golden.

Serving modes: trial (default) and strict

With a calibrated model, events assigned an 80% probability should happen about 80% of the time. Huncho applies the model's calibration settings and checks that the serving runtime preserves its reference behavior.

By default Huncho serves in trial mode. A model whose package ships a golden file is qualified automatically at startup; models without one still serve, with the gap stated plainly rather than hidden:

• every model's status — qualified, numerical, unqualified, or unfitted — appears in startup logs, /health, /v1/models, and (with x-huncho-extensions) each response;
• a model with no fresh qualification logs a loud UNQUALIFIED warning: its probabilities are unverified;
• a golden that fails still refuses to start, in every mode, and pending or refit-required calibration is refused in every mode. Trial mode only skips qualification that was never supplied — it never waves through a model that was checked and rejected.

For production, opt into strict mode:

huncho serve --require-qualified

Strict mode refuses to start unless every model holds a fresh complete labeled qualification for its exact execution profile (backend, dtype, device, options). A model that only passed the numerical gates (probability fidelity without observed outcomes) is also refused, with a message saying exactly what's missing.

Qualification tiers and huncho qualify

Qualification comes in two tiers:

• numerical — the golden's reference vectors pass the probability-delta and argmax gates. A package whose repository ships a golden.json reaches this tier automatically at startup.
• full (qualified) — the golden also carries observed outcomes for every question, so the ECE/Brier calibration gates run. Strict mode demands this tier.

huncho qualify runs the gates for one model and reports the tier it earned:

huncho qualify --model jaredpalmer/kev-4b

If the golden has no observed labels, add --template labeled-golden.json to get a fill-in file, record what actually happened for each case, and re-run — that upgrades the model to full qualification. Keep this evaluation set separate from the data used to fit calibration.

In strict mode (or to override a package golden), pin the suite explicitly at serve time:

--qualification-golden 'MODEL_NAME=/path/to/pinned-labeled-golden.json'

Huncho checks probability differences, agreement on the chosen answer, and calibration error drift. A manifest marked fit doesn't skip these checks. Some batching and prefix optimizations also require comparisons against independent forward passes. The calibration guide explains fitting, evaluation files, and acceptance limits. Numerical-only conform and bench runs are available for analysis, but don't establish that a model is ready for strict serving.

Local safetensors package

For a prepared Laya package:

huncho serve --manifest my-laya/huncho-model.json

It serves in trial mode. To serve it strictly, pin a labeled golden:

huncho serve --manifest my-laya/huncho-model.json --require-qualified \
  --qualification-golden 'MODEL_NAME=/path/to/pinned-labeled-golden.json'

If you have a Laya checkpoint checkout, follow the local checkpoint conversion guide to prepare the package with huncho convert.

Kev from Hugging Face

huncho serve --model jaredpalmer/kev-0.8b

The first load downloads the adapter, tokenizer, pointer head, and pinned base checkpoint. Use "model": "kev-0.8b" in requests. The native path defaults to FP32 on CPU and FP16 on CUDA; --dtype overrides it. Use --revision <commit> to select a specific Hub revision (the curated pin is bf75a6a8848ea6960ff2ed108d9ed44c2941174f).

Larger Kev packages (kev-4b, kev-9b) serve the same way. Their repositories ship no golden, so strict mode needs your own labeled suite (see huncho qualify). Current limit: the retained Kev-4B CPU FP32 held-out profile fails the probability difference check and is not accepted for serving. See Kev support and validation before choosing an execution profile.

Cloudflare Clef

huncho serve --model Cloudflare/clef

Clef uses a native Rust/Candle runtime and scores all questions together. It needs no Python runtime or model conversion. It starts with pending calibration: fit it before serving (see the Clef guide for memory requirements, fitting steps and current validation limits); a pending variant is refused in every mode.

Huncho selects the backend from each model's package. Use --backend to override that choice; if the required runtime is missing, the CLI tells you which build feature to enable. The supported decision heads are F1 (option-marker), F2 (pointer), F3 (candidate-logit), F4 (slot), and F5 (joint schema), all behind the same API.

Performance

Huncho scores answers without generating tokens. Depending on the model and backend, you can also reduce repeated work or serve requests concurrently:

• Batch questions: combine compatible rows in one model call, with bounded padding where supported. See batch controls and cross-request batches.
• Reuse shared state: native CPU Kev can process a state once and reuse it across question branches or requests. See Kev prefix batches.
• Cache repeated inputs: opt-in caches reuse tokenization, prompts, and complete responses within configured memory budgets. The HTTP server can also combine identical in-flight requests. See cache controls.
• Run CPU replicas: supported runtimes share immutable weights across independent serving contexts. See replicas and shared bases across adapters.
• Tune CPU execution: optional OpenBLAS and family-specific head and attention optimizations reduce computation and allocations. See backend profiles.

Most tuning controls are opt-in, and each serving context must pass the calibration checks. Performance depends on your workload. The optimization roadmap records measurements, implemented features, and remaining qualification work. Experimental CPU Q8_0/Q4_0 packages require a calibration refit and fresh labeled conformance; see quantization.

For inference in the browser, the separate CPU WASM package supports compatible F1 ONNX graphs and dedicated workers that move inference off the page thread. It has its own packaging and qualification requirements.

Development

Run these checks from the root of a source checkout. The default build can run without model weights:

cargo test

# Compare the example package with its reference vectors using the mock backend.
huncho conform \
  --manifest examples/mock-model/huncho-model.json --backend mock \
  --golden examples/mock-model/golden.json

To check optional backends and the native tokenizer:

cargo test -p huncho-backend --features onnx
cargo test -p huncho-backend --features candle
cargo test -p huncho-core --features tokenizers --test hf_tokenizer

The mock model package includes an ONNX fixture for testing numerical agreement. It doesn't establish real-model calibration.

Crate	Responsibility
huncho-core	Request/response types, model packages, prompts, heads, calibration, and conformance.
huncho-backend	Mock and optional inference runtimes.
huncho-api	HTTP endpoints, authentication, and metrics.
huncho-hub	Hugging Face model package resolution.
huncho-cli	serve, convert, calibrate, conform, and bench commands.

Deployment and further reading

With the RPM package, set HUNCHO_DEVICE=cpu to force CPU, or cuda / cuda:N to require a GPU. See GPU setup.

The repository also includes a Dockerfile and systemd service examples. The operations guide covers deployment, environment variables, authentication, health checks, and Prometheus metrics.

Guide	Use it to
API	Build requests and interpret answers.
Model packages	Prepare and inspect model artifacts.
Backends	Choose a runtime and check its limits.
Calibration	Fit temperatures and qualify a model for serving.
Operations	Configure, deploy, and monitor Huncho.

License

Huncho's own code is licensed under Apache-2.0 (see LICENSE). The distribution bundles components that keep their own licenses — MIT-licensed runtimes and a patched, dual-licensed Candle kernel — so binary packages carry the expression Apache-2.0 AND MIT. See THIRD-PARTY-NOTICES.md.