Skip to content

About

Production-grade evaluation harness for LLMs, tool-using agents, trajectories, regressions, cost, and CI quality gates

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Agent Eval Harness

An extensible evaluation platform for language models and tool-using agents.

agent-eval-harness is being built to answer the questions that ordinary unit tests cannot:

  • Did the model produce the right answer in the required structure?
  • Did the agent choose the right tools and arguments?
  • Was its trajectory efficient, grounded, and free of loops?
  • Did a prompt or model change improve quality without unacceptable latency or cost?
  • Should a pull request pass the AI quality gate?

Project status

This repository is under incremental development. The architecture, typed contracts, deterministic metrics, async runner, agent pipeline, provider abstraction, model evaluator, and structured LLM judge are in place. Reporting, regression comparison, and end-to-end commands will follow in focused pull requests.

See DESIGN.md for the goals, system boundaries, contracts, metric strategy, and delivery plan.

Planned capabilities

  • Model and agent evaluation through distinct evaluators
  • Observable agent trajectory tracing
  • Exact-match and structured-output checks
  • Tool precision, recall, argument accuracy, and unknown-tool detection
  • Loop, failure-rate, and step-efficiency metrics
  • Configurable LLM-as-a-Judge evaluation
  • Repeated runs with score variance and latency percentiles
  • Token usage and estimated cost tracking
  • Baseline-versus-candidate regression comparison
  • Terminal, Markdown, and JSON reporting
  • Deterministic good-agent and bad-agent examples
  • Docker-based local execution
  • GitHub Actions quality gates that do not require paid model credentials

Implemented foundations

  • Strict, version-ready Pydantic contracts for test cases, execution results, trajectories, metrics, evaluation runs, and regressions
  • Validated example datasets for model and agent evaluation
  • A generic typed metric interface
  • Exact-match evaluation with optional text normalization
  • JSON parsing, JSON Schema validation, and required-field accuracy
  • Tool precision, recall, argument accuracy, and unknown-tool detection
  • Trajectory efficiency, repeated-call loop detection, and tool-failure scoring
  • Concurrency-bounded async execution with per-case timeouts and exception isolation
  • Ordered observable trajectory recording with defensive snapshots
  • Deterministic good and bad agents that exercise the real evaluation architecture
  • Agent evaluation with per-test scores, thresholds, and run-level pass aggregates
  • Provider-neutral model requests and normalized responses
  • LiteLLM integration across hosted and local model backends
  • Repeated model evaluation with isolated provider failures
  • Structured LLM judging for faithfulness, goal completion, relevance, and hallucination
  • Token totals and available cost aggregates with explicit missing-price state

Current agent-evaluation API

The deterministic pipeline can be exercised without provider credentials:

import asyncio
import json
from pathlib import Path

from harness.evaluators import AgentEvaluator
from harness.schemas import AgentTestCase
from target_agent import GoodMockAgent

payload = json.loads(Path("datasets/agent_golden_dataset.json").read_text())
cases = [AgentTestCase.model_validate(item) for item in payload]
run = asyncio.run(AgentEvaluator(threshold=0.85).evaluate(cases, GoodMockAgent()))

assert run.aggregates["pass_rate"] == 1.0

Current model-evaluation API

Hosted providers use the same evaluator contract as local test doubles:

import asyncio
import json
from pathlib import Path

from harness.evaluators import ModelEvaluator
from harness.providers import LiteLLMProvider
from harness.schemas import ModelTestCase

payload = json.loads(Path("datasets/model_golden_dataset.json").read_text())
cases = [ModelTestCase.model_validate(item) for item in payload]
run = asyncio.run(
    ModelEvaluator(
        LiteLLMProvider(),
        model="openai/gpt-4.1-mini",
        repeats=3,
    ).evaluate(cases)
)

Pass an LLMJudge to ModelEvaluator to add repeated structured judge scores. Judge calls run at temperature zero; malformed and failed runs remain visible, and judge scores are explicitly marked as probabilistic.

Architecture at a glance

dataset + configuration
          |
          v
  asynchronous runner
      /           \
model evaluator   agent evaluator -> observable trace
      \           /
       metric engine
            |
 aggregation + regression comparison
            |
 terminal / Markdown / JSON / CI gate

Providers and target agents produce normalized execution results. Metrics evaluate those results independently, keeping provider SDKs and agent frameworks out of the core evaluation logic.

Development setup

The current milestone provides the project shell and a versioned CLI entry point. Evaluation commands will be added in subsequent milestones.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[dev]'

python cli.py --version
pytest -q

Copy .env.example to .env before configuring hosted providers. The deterministic local path will not require provider credentials.

Runtime settings use the AGENT_EVAL_ prefix and are validated at startup:

Variable Default Constraint
AGENT_EVAL_LOG_LEVEL INFO DEBUG, INFO, WARNING, ERROR, or CRITICAL
AGENT_EVAL_MAX_CONCURRENCY 5 Positive integer
AGENT_EVAL_TEST_TIMEOUT_SECONDS 60 Positive number of seconds

Provider credentials remain optional and are never included in container build contexts.

The same shell can be verified in Docker:

docker compose run --rm eval

Evaluation philosophy

The harness prioritizes deterministic evidence whenever ground truth exists. LLM judges are useful for open-ended qualities such as faithfulness and goal completion, but their scores are probabilistic signals—not objective truth.

Agent evaluation uses observable events such as tool calls, arguments, results, errors, retries, timing, and final answers. It does not depend on hidden chain-of-thought.

Planned command-line interface

The following commands describe the intended developer experience and will be enabled as their milestones land:

python cli.py model-eval \
  --dataset datasets/model_golden_dataset.json \
  --model openai/gpt-4.1-mini \
  --runs 3

python cli.py agent-eval \
  --dataset datasets/agent_golden_dataset.json \
  --agent mock-good \
  --threshold 0.85 \
  --json-output artifacts/current.json

python cli.py compare \
  --baseline artifacts/baseline.json \
  --candidate artifacts/current.json

The CLI will use stable exit codes:

  • 0: quality gate passed
  • 1: quality gate failed
  • 2: configuration or execution error

Delivery milestones

  1. Architecture and initial design
  2. Python package and container skeleton
  3. Typed schemas and deterministic metrics
  4. Async runner, trajectory tracer, and mock-agent evaluation
  5. Model providers and LLM-as-a-Judge
  6. Regression engine, reporters, and CLI
  7. CI quality gates and release hardening

Each milestone will be delivered through a focused pull request. Tests and validation will grow with the implementation.

Guiding constraints

  • No paid API key is required for the default test and CI path.
  • One failed case must not terminate a complete evaluation run.
  • Missing cost or usage data must remain explicit rather than silently becoming zero.
  • Stored evaluation artifacts must be versioned and reproducible.
  • Provider and agent integrations must depend on stable interfaces.

Contributing

The project is currently establishing its core architecture. Keep changes focused, typed, testable, and provider-neutral. New behavior should include deterministic tests, and architectural tradeoffs should be recorded in DESIGN.md.

License

A license will be selected before the first stable release.

About

Production-grade evaluation harness for LLMs, tool-using agents, trajectories, regressions, cost, and CI quality gates

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages