APIFlow-Bench

LLM benchmark for real API development work — auth chains, schema repair, multi-step stateful workflows, error recovery. Industry-contract action space, evaluated under Inspect AI. The 1.0 task bank is REST; GraphQL and MCP adapters ship in the harness, with task banks planned for a future release.

Status: public release — eval harness, the frozen 1.0 task bank, scoring, and the leaderboard site builder. Release provenance is pinned in registry.json.

Results & data:

• Leaderboard — https://www.postman.com/ai/apiflow-leaderboard/
• Raw trial transcripts (all 44,362 trials) — https://github.com/postmanlabs/apiflow-bench-transcripts
• Per-model transcript bundles (tar.gz) — this repo's releases page

Headline result — on the leaderboard's default view (the hardest slice: the full chain-1to20 tasks), the top of the 19-model panel is:

Rank	Model	Pass rate
1	gpt-5.5	72.7% (40/55)
2 (tie)	claude-opus-4-8	69.1% (38/55)
2 (tie)	deepseek-v4-flash	69.1% (38/55)

Other slices (all tasks, per-axis, per-difficulty) order differently — see the leaderboard for every view.

What's in this repo

• Harness (src/apiflow_bench/) — a neutral Inspect AI scaffold: 7 polymorphic tools over typed EntityRef handles, mock REST/GraphQL/MCP backends, deterministic validators.
• Task bank (tasks/v1.0) — the frozen 1.0 bank: 467 tasks (241 solo + 226 chain).
• Scoring (src/apiflow_bench/scoring/) — binary pass/fail per task, bootstrap CIs, and the 5-layer multi-dimensional leaderboard (capability / reliability / efficiency / robustness / generalization).
• Leaderboard site builder (scripts/leaderboard_publish/) — builds the full-transparency static leaderboard site (aggregate views + per-trial transcript pages), with an answer-leak scanner and a published-site checker. CF-Pages dynamic build: docs/leaderboard_cf_dynamic.md.

Setup

conda create -n apiflow-bench python=3.13 -y
conda activate apiflow-bench
pip install -e .              # installs the `apiflow` CLI

cp .env.example .env          # then fill in your API keys (table below)

apiflow auto-loads .env on startup. Set whichever providers you need:

Provider	.env variable
Anthropic	ANTHROPIC_API_KEY
OpenAI	OPENAI_API_KEY
Google	GOOGLE_API_KEY
Fireworks	FIREWORKS_API_KEY

(Running a single model only needs that one provider's key.)

Running APIFlow-Bench 1.0 (current bank)

1.0 is the current frozen bank: 467 tasks — 241 solo + 226 chain — across 13 independent worlds (tasks/v1.0). Each world was generated as one forward-loop chain of up to 20 segments — random axis + random difficulty per segment, sparse ledger coupling — and frozen as per-segment solo tasks plus cumulative chain-1to<k> prefix tasks. Eleven worlds carry the full 20 segments; two were truncated during generation (v56-w06 at 5 replay-verified segments, v56-w18 at 14). Each truncated world additionally ships one built-but-unverified solo segment (v56-w06-sub06, v56-w18-sub15), giving 241 solo tasks. See the bank-local manifest tasks/v1.0/registry.json.

Terminology used throughout:

• world — one procedurally generated mock-API environment with a multi-segment storyline.
• segment — one step of a world's storyline; each segment freezes into one solo task.
• solo vs chain-1to<k> — a solo task covers a single segment in isolation; a chain-1to<k> task requires completing segments 1 through k cumulatively in one session.
• epoch / trial — one full pass over a task set; the published panel ran 5 epochs, i.e. 5 independent trials per (model, task) pair.
• canary — a GUID marker line at the top of every task.yaml (BIG-bench convention) so benchmark data can be filtered from training corpora and contamination detected.
• provenance-gated grading — instead of hiding expected answers, grading keys on task-unique canary-derived values that an agent can only produce by actually driving the mock backends; transcripts are therefore published unredacted.

# Cheap first run — one task, one model, one trial:
apiflow run --task v56-w01-sub01 -m anthropic/claude-haiku-4-5-20251001

Cost warning: the full-bank command below runs 467 tasks × 5 epochs = 2,335 trials per model. On the published panel this cost roughly $8–$730 in API fees per model, depending on price tier (see the leaderboard's Cost / trial column), plus several hours of wall-clock time per model.

# One model over the whole bank (the published panel ran 5 epochs):
apiflow run-all \
  --roots tasks/v1.0 \
  --epochs 5 \
  --log-dir logs/v1.0-eval \
  -m anthropic/claude-sonnet-4-6

# Score → pass rate ± 90% CI (the leaderboard bootstraps the CI over worlds):
apiflow score logs/v1.0-eval

# Build the drill-down leaderboard site:
apiflow leaderboard --full logs/v1.0-eval --no-llm-review
open logs/v1.0-eval/leaderboard_full/overview.html

The published 19-model panel is listed in configs/v1.0_eval_models.txt (the initial 13 models) + configs/v1.0_eval_models_addendum.txt (3 models added 2026-07-01) + configs/v1.0_eval_models_addendum2.txt (3 models added 2026-07-18) — pass each spec with -m. How the panel was selected — and why no Google or xAI models appear — is covered in Limitations & conflicts of interest below. Epoch count (5) and the bank fingerprint are pinned in registry.json. Sampling temperature is not pinned in this repository's configs; the published panel ran at temperature 1.0, as documented on the published leaderboard.

Bank integrity & replay. Verify your checkout of the bank byte-for-byte, and replay reference trajectories against their validators:

python scripts/bank_sha256.py tasks/v1.0     # must equal bank_content_sha256 in registry.json
python scripts/golden_replay.py tasks/v1.0   # forward-PASS / negative-FAIL for every world

465 of the 467 tasks are replay-verified. Per-task reference.json files ship with 239 of the 241 solo tasks; chain tasks carry no file of their own — they replay by composing their segments' references. The two solo tasks without one — v56-w06-sub06 and v56-w18-sub15 — were part of the evaluated panel runs and are included in the frozen bank for completeness, but they are excluded from replay verification (hence tasks_golden_replay_pass: 465 in the bank-local manifest).

Provenance note: task files carry historical internal identifiers (v56-* world/task ids) frozen at evaluation time — they match the published leaderboard transcripts verbatim. This release bank is the evaluated bank with internal annotation fields (review-workflow labels such as public_release_status and release_blockers) removed from the evaluation: metadata block: instructions, fixtures, validators and expected answers are byte-identical to the evaluated bank, and equivalence was verified by re-running golden replay (465/467) after the strip. The bank content sha256 pinned in registry.json fingerprints this annotation-stripped release bank — content-identical to the evaluated bank, not byte-identical. The bank-local tasks/v1.0/registry.json was normalized the same way for release, and its tasks_golden_replay_pass: 465 counts the replay-verified subset described above.

Design

• Spec-first: action space, capability axes, scoring, governance defined before any task is authored.
• 7-tool polymorphic action space: read / write / edit / search / execute / clarify / report_blocked over typed EntityRef handles. The same 7 tools express every operation; protocol differences are parameters, not new tools.
• Validator-first: every task ships a deterministic Python validator; reference solutions are replay-verified for 465 of the 467 bank tasks (see Bank integrity & replay). Pass/fail is binary; LLM verifiers are non-ranking (clean/suspect labels only).
• Neutral scaffold: Inspect AI's built-in react() agent loop. No custom planner / summarizer / retry logic — measuring models, not scaffolds.
• Deterministic local mocks: agents drive mock backends running locally (in-process or as local subprocesses) — no real network calls and no containers. Task code (validator.py, mock_overrides.py) executes in the harness process, so treat third-party task banks like any code you run.
• Anti-contamination from day 1: BIG-bench-style canary line (unique GUID) in every task.yaml; see Limitations & conflicts of interest for the contamination horizon of a fully public release.
• Frozen releases: every benchmark version is pinned in registry.json (bank path, content sha256, epoch count, price-table version), so running 1.0 next year reproduces 1.0 even as the repo grows.

Limitations & conflicts of interest

Task generation and the oracle. Claude Opus 4.8 generated every world in this bank — instructions, mock environments, chain summaries, metadata annotation — and the per-segment solvability gate (pass@10 ≥ 3) ran on Claude Opus 4.7 (chosen over 4.8 because 4.8 passed too easily to leave headroom). Opus 4.8 is also a ranked model on the leaderboard. Mitigations, stated plainly: pass/fail is decided by deterministic Python validators plus golden replay — the LLM verifier (also Opus 4.8) attaches non-ranking clean/suspect labels and never determines a score; the solvability gate filters for solvable, not oracle-easy; and a generator-family advantage still cannot be excluded — the full harness, bank, and all 44,362 raw transcripts are public precisely so anyone can test for it. On the default (hardest) leaderboard slice, chain-1to20, the leader is a non-Anthropic model (gpt-5.5); Anthropic models lead some other slices.

A fully public 1.0, and the contamination horizon. As of this release, every task, every expected answer, and every transcript is public. All published scores were produced before release (runs June 29 – July 18, 2026), so they cannot be contaminated by this repository — but any evaluation run after models train on this data can be. Every task carries a unique GUID (the BIG-bench-style canary line), which supports recitation probes; a refreshed bank with a private held-out split is planned to anchor future rankings.

Why mock environments. Tasks run against deterministic local mocks, not live services: that buys byte-for-byte replayability, zero cost and zero API keys to reproduce, and precise difficulty control. The mocks' quirks are modeled on real API failure patterns — expiring tokens, 429s with Retry-After, versioned-route deprecations, partial writes — so a high score means an agent survives these failure modes under the 7-tool contract; it does not mean the agent can operate arbitrary production systems. The 1.0 bank is REST-only.

Known bank defects. In some chain tasks, the instruction-rewriting step dropped mid-chain requirements that the success signature still gates on, making those chain cells unpassable as written: chain families v56-w08 and v56-w10 at most lengths, v56-w16 at length 19, and v56-w04 at length 11 — roughly 8% of chain trials. All models face the same wall, so aggregate rankings compress rather than reorder, but per-chain-length slices should be read with care (the leaderboard's data notes carry the caveats). Golden replay did not catch this class because chains replay by composing per-segment references, which individually pass. A bank fix ships in the next bank release.

Panel selection. The panel was drawn to span providers, price tiers, and open- vs closed-weight models; the shipped 19 are the models for which the full 467-task × 5-epoch grid completed under the identical harness during the run window (a handful of trials are missing as harness errors). API access and serving stability were the binding constraints; no model was dropped because of its score. Some notable models (including Google's) were not run in this window and are planned for a future refresh.

Baseline configs (frozen snapshots)

Note: the few configs/baseline_*.yaml files that ship in this repo are historical snapshots from pre-1.0 internal history — their task ids and pinned commits are not part of this public snapshot, so they are not runnable against this tree. They document the freeze format; freeze your own baselines with apiflow baseline freeze.

A "baseline" is one full run of one set of models against one set of tasks. Each versioned baseline gets a frozen yaml under configs/; the yaml is the source of truth and lists every task ID explicitly plus the git commit at freeze time:

• Set reproducibility: re-running a baseline runs exactly its frozen task IDs, no matter how many tasks were added since.
• Content reproducibility: git checkout <commit> reproduces byte-identical task definitions.

Most "scoring drift" between baseline runs is set drift, so the explicit list is the load-bearing part. Inspired by Terminal-Bench's registry.json, which uses the same hybrid (commit_hash + task_id_subset).

apiflow baseline freeze   --name <ver> -m <m1> -m <m2> ...   # walk tasks/, capture git HEAD
apiflow baseline validate --config configs/baseline_<ver>.yaml
apiflow baseline run      --config configs/baseline_<ver>.yaml
apiflow baseline score    --config configs/baseline_<ver>.yaml

Scenario	What baseline run does
Task in yaml + same task in folder	Runs it (normal case).
Task in folder NOT in yaml	Skipped silently (not part of this version).
Task in yaml but DELETED from folder	Refuses to run with an error listing the missing IDs.
git_commit doesn't match HEAD	Warns "drift" but still runs — content may differ from freeze time.

Never hand-edit a frozen baseline yaml — re-freeze a new version instead.

Repository layout

src/apiflow_bench/    # the package
├── action_space/     # EntityRef + entity kinds + 7 polymorphic tools (+ tool-surface profiles)
├── adapters/         # rest / graphql / mcp adapters
├── harness/          # neutral solver, task loader, sandbox
├── scoring/          # validator runner, bootstrap CI, multi-dim leaderboard, LLM verifier
├── generator/        # procedural world/chain generation pipelines
├── governance/       # real-$ cost table, pool lint, release/freeze flow
├── baseline.py       # frozen-snapshot config: load / freeze / validate / run
├── canary.py         # BIG-bench-style canary line for tasks
└── cli.py            # `apiflow` entry: run, run-all, score, baseline, new-task, ...
src/mocks/            # FastAPI / strawberry / mcp mock loaders
configs/              # baseline yaml snapshots + eval model panels
tasks/                # the frozen 1.0 task bank (v1.0/)
scripts/              # run/eval helpers, golden_replay.py, bank_sha256.py, leaderboard_publish/
tests/                # harness self-tests (pytest)
registry.json         # per-release provenance manifest

CLI cheat sheet

apiflow list                                              # show all tasks
apiflow run --task <id> --model <m>                       # one-off single task
apiflow run-all --roots <bank> -m <m1> -m <m2>            # path-scoped multi-model run
apiflow score <log_dir>                                   # per-axis pass rate ± 90% CI
apiflow leaderboard --full <log_dir> --no-llm-review      # drill-down dashboard
apiflow new-task <axis> <protocol> --slug <slug>          # scaffold task with canary
apiflow validate-task <id>                                # lint task contract

# Inspect AI's interactive viewer
inspect view --log-dir <dir> --recursive

# Bank integrity / replay
python scripts/bank_sha256.py tasks/v1.0
python scripts/golden_replay.py tasks/v1.0

For the generation pipelines see docs/procedural_generation.md (legacy), docs/proven_spread_generation.md (Proven Spread), and docs/v5_design_spec.md (the V5 forward-loop chain pipeline).

Contributing

Tasks are versioned in registry.json. Author new tasks via apiflow new-task. Every task must:

1. Have a canary GUID in task.yaml header (auto-injected by the scaffold).
2. Keep task metadata (axis, protocol, difficulty, tags, expected pass-rate band) in task.yaml.
3. Provide a deterministic validator.py returning a Verdict.
4. Ship a reference_solution.md that the validator returns PASS on.
5. Pass apiflow validate-task <id>.

Never hand-edit a frozen bank or baseline yaml — freeze a new version instead. Run the self-tests with pytest (or PYTHONPATH=$PWD/src python -m pytest -q without an editable install) before opening a PR. See CONTRIBUTING.md.

Citation

@misc{apiflowbench2026,
  title  = {APIFlow-Bench: An Agentic API-Workflow Benchmark},
  author = {{Postman AI Team}},
  year   = {2026},
  url    = {https://github.com/postmanlabs/APIFlow-Bench}
}

Citation metadata also ships in CITATION.cff.

License

Apache License 2.0 — see LICENSE. Copyright 2026 Postman, Inc.