Skip to content

feat(api): migrate experiments to the API client - #682

Closed
Abhijeet Prasad (AbhiPrasad) wants to merge 1 commit into
mainfrom
abhi-api-experiment-service
Closed

feat(api): migrate experiments to the API client#682
Abhijeet Prasad (AbhiPrasad) wants to merge 1 commit into
mainfrom
abhi-api-experiment-service

Conversation

@AbhiPrasad

@AbhiPrasad Abhijeet Prasad (AbhiPrasad) commented Aug 12, 2026

Copy link
Copy Markdown
Member

ref #673
supercedes #640

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve #639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve #639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.
@AbhiPrasad

Copy link
Copy Markdown
Member Author

Closing in favor of the generated API client — tracked in #683.

Why: this PR returns frozen dataclasses (ExperimentRecord, ExperimentComparison, …) with
from_dict validation and a raw verbatim-body passthrough. #683 generates TypedDicts from
braintrust-openapi. Merging this would put
two incompatible return conventions in braintrust.api, so the convention question is settled up
front instead of after the fact: generated TypedDicts, with additive-field tolerance coming from
non-closed TypedDicts rather than a raw field. Recorded as open decision 7 in #683.

Nothing here is wasted — the branch stays readable and is the reference implementation for the redo.
#683 records these as requirements it must satisfy:

  • get_base() returning None on HTTP 400 (a fresh project's first eval always takes this path, so
    generated bindings must not flatten it into a generic 4xx error);
  • get() / get_base() / compare() classified RetryMode.SAFE_READ — the concrete case of the
    call-site audit's POST-based-logical-read class;
  • additive backend fields surviving round-trips;
  • SummarySuccess / SummarySkipped in logger.py;
  • the coverage shape — real-backend VCR cassettes plus a local http.server test asserting exact
    request targets and additive-field preservation, plus type tests — as the template for every
    subsequent resource migration.

Note on #639: this PR was its only fix in flight, and that bug is user-facing — a failed
comparison read currently looks identical to a successful experiment with no scores. #683 tracks two
ways to unpark it: re-land only the product-level half now (the strict-by-default semantics and the
discriminated SummarySuccess/SummarySkipped/SummaryFailed result in logger.py, which are
eval-summary types rather than REST wire types and so do not touch the convention decision), or wait
for generated Experiments bindings in step 3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Experiment.summarize() silently discards scores/metrics on any fetch failure instead of surfacing or retrying

1 participant