VEP-bench is a public benchmark for measuring the native genetic variant effect prediction capabilities of language models. Models answer without internet access or tools, and every response can be inspected in the public results explorer.
Explore the leaderboard and responses
VEP-bench uses deterministic, versioned question sets built from biological reference data. Models receive only the model-visible prompt; answer keys stay local and scoring is deterministic.
The benchmark is intentionally transparent:
- questions and reference answers are public development data;
- prompts use strict, machine-readable final-answer formats;
- complete responses and provider-exposed reasoning are preserved when available;
- evaluations use one OpenRouter integration; and
- results are published to a static explorer with no database or backend.
VEP-bench measures current model capability. Temporal task cohorts can reduce direct source leakage but do not guarantee absence from training data.
| Task | Model input | Target | Questions |
|---|---|---|---|
| OpenSplice SNV | Complete three-exon minigene cassette, assay context, and a 50-SNV panel | Signed change in exon inclusion | 20 |
| satMutMPRA | Physical reporter context, the complete mutagenized insert, and a 50-variant panel | Signed reporter-activity effects | 16 |
| Saturation genome editing | Gene and endogenous assay context, one exon with 100 bp flanks, and a 50-SNV panel | Continuous functional damage | 15 |
Each task has versioned sources, a prompt, methodology, limitations, and results.
The default environment contains only the evaluator. It fetches and verifies published questions on demand; website, publication, and task-preparation dependencies are separate private workspace projects.
uv sync --locked
export OPENROUTER_API_KEY=...
uv run --no-sync vepbench evaluate --model provider/model-idEvaluation is an explicit, potentially paid local action. See the evaluation guide for versioned profiles, direct runs, batch collection, and local question files.
The explorer provides task-specific leaderboards and lets readers inspect the exact prompt given to a model, its complete response, available reasoning, and deterministic score. Interpret each score alongside the task's methodology and limitations.
- Benchmark task catalog
- Contributor documentation
- Architecture and data contracts
- Development setup
- Evaluation
- Publishing
VEP-bench is available under the Apache License 2.0.