v0.2.0

Verifiers v0.2.0 Release Notes

Date: 07/10/2026

Highlights since v0.1.14

• A task-centric v1 API. Verifiers v1 now separates serializable per-row TaskData from the Task class that owns lifecycle hooks, tools, metrics, rewards, and judges. A Taskset loads typed tasks, a Harness defines how the model runs, and the two compose into an environment without mixing v1 and legacy APIs.
• Portable agent execution. The same taskset can run across built-in and custom harnesses on local subprocesses, Docker, or remote Prime and Modal sandboxes. Model traffic passes through the interception layer, which adapts the harness's provider dialect, records the trace as it happens, applies sampling configuration, and supports response and tool-result interception.
• Typed authoring, tools, and scoring. Strict Pydantic configs cover tasksets, tasks, harnesses, runtimes, and evaluation. Environment authors can scaffold packages with uv run init, expose taskset- or task-scoped MCP tools, attach user simulators, and score traces with rewards, metrics, group rewards, reusable scoring helpers, and configurable LLM judges.
• A complete v1 evaluation workflow. uv run eval supports TOML configs, dotted CLI overrides, dry runs, concurrency controls, rich progress and paging, resumable runs, and platform upload. Native validate, debug, and replay entrypoints cover environment checks, interactive diagnosis, and offline re-scoring of saved traces.
• First-class traces and observability. Traces capture the message graph, task identity and data, rewards, metrics, errors, token and timing information, runtime metadata, and per-message creation timestamps. Evaluation artifacts now use traces.jsonl, with improved streaming, memory use, and dashboard reporting for long or highly concurrent runs.
• Harbor and production environment support. v1 includes a Harbor taskset base, reusable built-in harnesses and tasksets, container-aware runtime configuration, and a companion catalog of production benchmarks in research-environments.

Selected changes included in v0.2.0

v1 framework and authoring

• Introduce the complete Verifiers v1 taskset, harness, runtime, trace, and CLI stack (#1825)
• Move per-task behavior from Taskset onto typed Task objects (#1948)
• Add native v1 debug and validate workflows (#1905)
• Add reusable scoring helpers and configurable reference/rubric judges (#1907, #1918)
• Add built-in Harbor support and reusable Lean taskset foundations (#1877, #1882, #1883)
• Refresh the v1 user documentation (#1916)

Evaluation, traces, and reliability

• Add offline trace replay and platform upload for eval runs (#1936, #1947)
• Rename the v1 evaluation artifact to traces.jsonl (#1956)
• Add trace record export, token-length attribution, runtime metadata, and message timestamps (#1844, #1875, #1959, #1960)
• Improve the rich dashboard with phase markers, usage totals, pending-rollout visibility, and paging (#1866, #1869, #1912, #1913)
• Make resumed runs restore their saved traces and complete only missing or errored work (#1892, #1962)
• Improve streaming, concurrency, and memory behavior for large v1 evaluations (#1793, #1807, #1815, #1838)

Full Changelog: v0.1.14...v0.2.0

What's Changed

• Require auth for interception endpoints by @xeophon in #1304
• feat(composable): render pre-compaction branches via harness hook by @snimu in #1291
• fix: mute httpcore/httpx DEBUG when env-worker json_logging sets root level by @rasdani in #1234
• Make sandbox worker caps configurable by @rasdani in #1305
• Support setup_state return-state compatibility by @willccbb in #1308
• Update v1 Taskset/Harness docs and opencode_harbor migration by @willccbb in #1309
• bump interception client size to handle trajectories with images by @eligotts in #1310
• Fix eval summary for heterogeneous metrics by @xeophon in #1295
• Refactor deep-agents envs to v1 taskset-harness and add Wikispeedia example by @willccbb in #1317
• Add SWE debug environment by @rasdani in #1306
• Move OpenCode config into reusable harness package by @willccbb in #1318
• Remove direct prime-sandboxes dependency from rlm-swe v1 by @willccbb in #1316
• Install sandbox fn program packages by @willccbb in #1319
• opencode: write config under XDG_CONFIG_HOME by @rasdani in #1320
• feat: surface swallowed response_future errors in interception proxy by @rasdani in #1196
• apt: harden sandbox bootstrap against transient archive.ubuntu.com flakes by @rasdani in #1284
• fix(renderer-client): wrap tools in OpenAI envelope to match training distribution by @hallerite in #1307
• fixes rlm compaction prompt so we still get tito hit by @eligotts in #1322
• Fix interception keepalive env override by @rasdani in #1321
• Pin OpenCode small model to intercepted provider by @rasdani in #1323
• Avoid holding sandbox workers with run_background_job() by @rasdani in #1328
• Skip OpenCode install when binary is pre-baked by @rasdani in #1176
• fix: narrow send_cancel BaseException catch to Exception by @rasdani in #1198
• Fix Harbor Hub task root handling by @xeophon in #1345
• Pin OpenCode small model in legacy configs by @rasdani in #1327
• Remove local training config scaffolding from Verifiers by @willccbb in #1348
• retry Read file timed out by @kcoopermiller in #1350
• chore: v0.1.15.dev0 dev release by @rasdani in #1352
• Tighten BugBot releasability and dependency rules by @willccbb in #1353
• Strengthen BugBot release and dependency rules by @willccbb in #1354
• Fix v1 lifecycle handler discovery by @xeophon in #1347
• Fix ZMQ env timeouts by @xeophon in #1355
• feat(renderer-client): thread multimodal sidecar through rollout + transport by @hallerite in #1346
• Map Anthropic reasoning effort by provider by @xeophon in #1338
• chore: v0.1.15.dev1 dev release by @xeophon in #1357
• Disable OpenCode title generation in harness configs by @rasdani in #1359
• chore: v0.1.15.dev2 dev release by @rasdani in #1360
• Use background jobs for v1 sandbox commands by @xeophon in #1364
• fix: treat OverlongPromptError as stop condition in interception proxy by @mikasenghaas in #1365
• chore: v0.1.15.dev3 dev release by @mikasenghaas in #1367
• [codex] Display eval cost from Prime pricing by @willccbb in #1368
• Add repo development best practices to AGENTS, docs, and skills by @willccbb in #1361
• Tighten v1 config, bindings, and handler typing by @willccbb in #1362
• Prepare v0.1.15.dev4 release notes and version bump by @willccbb in #1370
• Add Terminus2 harness by @xeophon in #1356
• chore: v0.1.15.dev5 dev release by @xeophon in #1372
• feat: consume vllm routed experts payloads by @S1ro1 in #1363
• fix: move routed experts helpers to response utils by @S1ro1 in #1373
• Tighten v1 EnvConfig typing by @willccbb in #1371
• perf(save): skip backward-byte scan on hot path by @mikasenghaas in #1376
• chore: v0.1.15.dev6 dev release by @willccbb in #1375
• Clarify managed Lab AGENTS guidance and add AGENTS.local.md overlay by @willccbb in #1383
• Add per-eval names for duplicate environment runs by @willccbb in #1384
• Document eval name labels in skill by @willccbb in https://github.com/PrimeInt...

v0.1.14

Verifiers v0.1.14 Release Notes

Date: 05/07/2026

Highlights since v0.1.13.dev8

• Composable v1 Taskset/Harness API. Adds the verifiers.v1 authoring surface around serializable Task/State data, composable Taskset and Harness objects, and the vf.Env(taskset, harness) adapter for existing eval and training workers. The release includes lifecycle decorators, typed config objects, endpoint routing, toolsets, MCP tools, sandbox/program utilities, nested harness support, v1 docs, migration notes, and several v1 example environments, including new OpenAI Agents, LangChain Deep Agents, and DSPy RLM harness examples.
• Consistent v1 environment configuration. Eval and RL/Hosted Training TOML now share the same public projection shape through [*.args], [*.taskset], and [*.harness] sections. v1 loaders accept both mapping and model-backed config objects through a common access helper, and strict child config parsing strips loader-local routing keys at the boundary.
• Model-family starter configs. Restructures bundled eval, RL, and GEPA starter configs around model families such as Qwen 3.5, Qwen 3.5 MoE, Nemotron 3, and Llama 3, with setup mirroring the new config set into Lab workspaces.
• New client and rendering paths. Adds an OpenAI Responses API client and a renderer-backed client path for exact token rendering and multi-turn bridge metrics. The renderer implementations now live in the external renderers package, exposed from Verifiers through an optional renderers extra and client integration. Renderer clients also forward preserve_all_thinking and preserve_thinking_between_tool_calls flags into the underlying renderer.
• More rollout observability and artifacts. Adds per-turn timing through eval outputs and TUI display, token-id preservation for Nemotron client responses, GEPA system-prompt artifact export plus path-based prompt loading, and Lean guard markers with tamper-aware LeanRubric scoring.
• Release and infrastructure hardening. Adds universal locks and a 7-day PyPI freshness cooldown, scopes Hub install freshness filtering to registry packages, skips secret-backed environment tests on fork PRs, points the composable RLM harness at rlm-harness, and routes opencode AGENT_WORKDIR per rollout.

Changes included in v0.1.14 (since v0.1.13.dev8)

Features and enhancements

• Taskset Harness (v1) (#1277)
• ApiEnv examples for OpenAI Agents, LangChain Deep Agents, and DSPy RLM (#1121)
• Refactor tau2 bench into a taskset-owned v1 environment (#1293)
• Restructure example configs around model families (#1297)
• add openai responses client (#1261)
• Renderer-backed client integration via the external renderers package (#1068, #1279, #1282)
• feat(renderer-client): forward preserve_*_thinking config flags (#1298)
• feat: per-turn timing (#1182)
• Add GEPA system prompt export and path-based prompt loading (#1268)
• feat(lean): lean-guard markers + tamper-aware LeanRubric (#1271)
• token id support for Nemotron client responses (#1231)

Fixes and maintenance

• Fix v1 env config projection and typed child loader boundaries (#1294)
• Skip secret-backed environment tests for fork PRs (#1292)
• Scope uv freshness filtering to PyPI for Hub installs (#1286)
• opencode harness: route AGENT_WORKDIR per-rollout instead of baked-in (#1280)
• chore: add 7-day supply chain cooldown via uv exclude-newer (#1274)
• chore: point DEFAULT_RLM_REPO_URL to rlm-harness (#1267)
• Update Lab workspace setup guidance (#1299)

Full Changelog: v0.1.13.dev8...v0.1.14

v0.1.13.dev8

Verifiers v0.1.13.dev8 Release Notes

Date: 04/28/2026

Highlights since v0.1.13.dev7

• Per-rollout wall-clock timeout for MultiTurnEnv. New timeout_seconds: float | None kwarg bounds total rollout via asyncio.wait_for; on fire, mark_timed_out(state) writes timed_out=True, is_completed=True, stop_condition="timeout_reached". New vf-eval --timeout SECONDS CLI flag injects timeout_seconds into the env constructor (and recognizes timeout = N at the top of [[eval]] TOML tables); wins over --extra-env-kwargs. CliAgentEnv derives sandbox timeout_minutes from this (rollout deadline + 60min scoring buffer, clamped to a 24h SDK ceiling) and exposes compute_sandbox_timeout_minutes on SandboxMixin for taskset overrides via SandboxSpec(timeout_minutes=None).
• Several smaller CliAgentEnv / composable / RLM fixes.

Changes included in v0.1.13.dev8 (since v0.1.13.dev7)

Features and enhancements

• feat: rollout timeout (#1258)
• Harness.keep_trajectory_step; rlm: rlm_max_depth, include_sub_rlm_trajectories (#1253)
• TITO: treat content='' / None as equal in prefix-match + warn on MITO fallback past turn 1 (#1259)
• Harness.environment_vars: per-rollout callable, rlm: rand threshold (#1248)
• TaskSet: accept DatasetBuilder for lazy dataset construction (#1251)
• cli_agent_env: bump default poll_interval from 1s to 5s (#1255)

Fixes and maintenance

• fix: avoid TOCTOU port race in CliAgentEnv interception server (#1264)
• rlm harness: remove sandbox-side git shim (replaced by rlm tool-level block, rlm#70) (#1262)
• fix: narrow math_verify BaseException catch to specific TimeoutException (#1197)
• fix: handle dict word_list in TextArenaEnv.ta_to_hf() (#1214)
• fix: correct timing accumulation in RubricGroup score_rollout and score_group (#1215)
• composable_env: skip caches/.git/.venv when tarring upload dirs (#1257)
• swe tasksets: default ds_num_proc to None for all SWE tasksets (#1256)
• git_checkout_cache: hold per-process in-use lock so concurrent resolves don't nuke active worktrees (#1252)
• fix: prepend vllm/ to slashless OPENAI_MODEL in composable opencode harness (#1250)
• rlm harness: stage git-refusal shim into $HOME/.local/bin so scoring can run git (#1244)
• rlm harness: remove dead RLM_KERNEL_PYTHON detection block (#1242)
• SWEBenchTaskSet.setup: symlink venv at /testbed/.venv matching WORKDIR (#1241)

Full Changelog: v0.1.13.dev7...v0.1.13.dev8

v0.1.13.dev7

Verifiers v0.1.13.dev7 Release Notes

Date: 04/24/2026

Highlights since v0.1.13.dev6

• rlm_harness swaps turn-based context caps for token-based auto-compaction: new summarize_at_tokens: int | None kwarg maps to RLM_SUMMARIZE_AT_TOKENS, while rlm_max_turns_in_context / RLM_MAX_TURNS_IN_CONTEXT are removed to match upstream rlm. summarize also drops out of the default rlm_tools set. Invalid shapes fail at harness-build time instead of deep inside the sandbox.
• Reverted TaskSet.filter / .take returning Self (originally #1232) — the change broke Python 3.10/3.11 compatibility. CI now exercises the 3.10 and 3.11 test matrices so the fix can be restored with confidence.

Changes included in v0.1.13.dev7 (since v0.1.13.dev6)

Features and enhancements

• rlm_harness: add summarize_at_tokens, drop rlm_max_turns_in_context (#1236)

Fixes and maintenance

• Revert "types: TaskSet.filter / .take return Self, not TaskSet (#1232)" (#1237)
• ci: add Python 3.10 and 3.11 to the test matrix (#1237)

Full Changelog: v0.1.13.dev6...v0.1.13.dev7

v0.1.13.dev6

Verifiers v0.1.13.dev6 Release Notes

Date: 04/23/2026

Highlights since v0.1.13.dev5

• rlm_harness is now the single source of truth for RLM_* sandbox env vars. New kwargs rlm_max_turns, rlm_max_turns_in_context, rlm_exec_timeout map 1:1 onto the matching env vars on Harness.environment_vars and merge into the sandbox via ComposableEnv.build_env_vars (harness-wins). Research envs can stop setting these via ComposableEnv(environment_vars=…) — pass them through as harness kwargs instead.
• TaskSet.filter / .take now return Self, not TaskSet, so subclass types survive taskset chaining for downstream typed consumers.

Changes included in v0.1.13.dev6 (since v0.1.13.dev5)

Features and enhancements

• rlm_harness: own RLM_MAX_TURNS / _IN_CONTEXT / _EXEC_TIMEOUT env vars (#1229)

Fixes and maintenance

• types: TaskSet.filter / .take return Self, not TaskSet (#1232)

Full Changelog: v0.1.13.dev5...v0.1.13.dev6

v0.1.13.dev5

Verifiers v0.1.13.dev5 Release Notes

Date: 04/22/2026

Highlights since v0.1.13.dev4

• Made the interception proxy's streaming response resilient to upstream cuts: 10s SSE keepalive comments keep idle streams warm, per-chunk asyncio.sleep(0) forces an event-loop yield so content and close can't race the transport flush under warmup-burst contention, and transport exceptions at prepare/write/write_eof are surfaced as StreamInterrupted into state["error"] so rollouts reschedule instead of looking like clean zero-turn completions.
• Added a new experimental mini_swe_agent composable harness (pip/uv install with SHA256-verified wheel download), exported alongside existing rlm and opencode harnesses.
• Extended SandboxMixin to cover VM sandboxes in addition to containers (including GPU VMs via CreateSandboxRequest), with documentation clarifying feature parity (file I/O, background jobs, cleanup) and container-only features (port exposure, SSH).

Changes included in v0.1.13.dev5 (since v0.1.13.dev4)

Latest changes from main

• Includes the latest main changes through the interception proxy streaming resilience fix (#1194), along with the mini_swe_agent harness (#1219) and SandboxMixin VM sandbox support/docs (#1222).

Features and enhancements

• Add mini-swe-agent harness (#1219)
• Update SandboxMixin (#1222)

Fixes and maintenance

• fix: make interception proxy streaming resilient to upstream cuts (#1194)

Full Changelog: v0.1.13.dev4...v0.1.13.dev5

v0.1.13.dev4

Verifiers v0.1.13.dev4 Release Notes

Date: 04/22/2026

Highlights since v0.1.13.dev3

• RLM harness: new rlm_tools kwarg sets both Harness.tool_names (for ToolMonitorRubric) and the sandbox RLM_TOOLS env var from a single source, plus new Harness.environment_vars field merged harness-wins-on-conflict by ComposableEnv.
• Refactored experimental RLM checkout caching; DEFAULT_RLM_BRANCH renamed to DEFAULT_RLM_REF and rlm_harness(..., rlm_branch=...) renamed to rlm_ref= to reflect that any git ref (branch, tag, sha) is accepted.
• Added SandboxTimeouts dataclass centralizing per-operation sandbox HTTP timeouts.
• Expanded task coverage with SWE-rebench-V2 and a multilingual SWESmith taskset, plus a filter_fn kwarg on all tasksets for ad-hoc row filtering.
• vf-eval: renamed -d/--debug to --disable-tui and --tui to --fullscreen for clearer intent.
• RLM rollout metrics (context tokens, programmatic tool calls) exposed to verifiers and auto-merged by the composable env.

Changes included in v0.1.13.dev4 (since v0.1.13.dev3)

Features and enhancements

• vf-eval: replace -d/--debug with --disable-tui, rename --tui to --fullscreen (#1183)
• Expose RLM metrics to verifiers (#1195)
• Add streaming observability + resume to TaskSet.validate() (#1169)
• Refactor experimental RLM checkout caching (#1202)
• feat: add filter_fn kwarg to all tasksets for ad-hoc row filtering (#1199)
• feat: add multilingual SWESmithTaskSet (#1186)
• feat: add SWE-rebench-V2 TaskSet (#1187)
• HarborMCPMixin (#1146)
• feat: SandboxTimeouts dataclass — centralize per-operation sandbox HTTP timeouts (#1207)
• Run SWE-Lego eval via dataset's canonical test_cmd (#1205)
• Authenticate interception server via INTERCEPTION_SECRET (#1180)
• feat: revert agent test edits at grading (swe_lego, swe_rebench_v2) (#1212)
• AgentError: rollout_id, sandbox_id, ... (#1218)
• Remove RLM_DEFAULT_TOOL_NAMES, accept rlm_tools (#1223)
• r2e_gym: add hide_tests_from_agent flag + expose instance_id/repo aliases (#1208)
• feat(rlm): upload a /usr/local/bin/git shim, gated by allow_git (#1225)

Fixes and maintenance

• Keep harness metrics merge inside experimental composable env (#1201)
• Propagate typed exceptions from SWE/Harbor validate_instance (#1204)
• fix: pass explicit 60s timeout to get_background_job in poll_job_completion (#1206)
• fix: bump opencode harness default release to v1.1.63-rl2 (#1184)
• validate(): extract resume-file parsing into a named helper (#1209)
• fix: SandboxTimeouts fields must be int (sidecar deserializes as u64) (#1210)
• fix: respect framework-injected OPENAI_API_KEY in RLM and opencode harnesses (#1213)
• fix: offload composable _upload_dir tar build to thread (#1224)

Full Changelog: v0.1.13.dev3...v0.1.13.dev4

v0.1.13.dev3

Verifiers v0.1.13.dev3 Release Notes

Date: 04/19/2026

Highlights since v0.1.13.dev2

• Propagated interception-stream write failures into rollout state as StreamInterrupted so truncated agent streams no longer surface as silent clean exits.
• Made RLM checkout resolution lazy in the composable harness, so loading RLM-based environments no longer clones the private checkout up front.

Changes included in v0.1.13.dev3 (since v0.1.13.dev2)

Features and enhancements

• make download of rlm lazy (#1192)

Fixes and maintenance

• fix: propagate interception-stream cuts into rollout state (#1191)

Full Changelog: v0.1.13.dev2...v0.1.13.dev3

v0.1.13.dev2

Verifiers v0.1.13.dev2 Release Notes

Date: 04/19/2026

Highlights since v0.1.13.dev1

• Added richer token usage reporting, including final-context token metrics, updated displays, and API docs.
• Expanded vf-tui compare mode so you can inspect any numeric metric with inline selection and adaptive bucketing.
• Improved composable/RLM harness integration with harness-owned upload dirs, cached local RLM checkouts, and auto-registered tool monitoring from harness.tool_names.
• Surfaced CLI agent crashes as infra errors even after prior turns, and now include full traces in agent error logs for debugging.
• Removed dead RLM tool config constants from the composable harness exports.

Changes included in v0.1.13.dev2 (since v0.1.13.dev1)

Features and enhancements

• Better token count metrics (#1108)
• vf-tui: compare for all metrics (#1117)
• harness.get_upload_dirs; reduce rlm github requests (#1178)
• tool-env: ToolMonitorRubric takes tool_names instead of tools (#1179)
• feat: auto-register ToolMonitorRubric from harness.tool_names (#1181)
• feat: include full trace in agent error logs (#1185)

Fixes and maintenance

• cli-agent: surface agent crashes as infra errors after any turn (#1177)
• Remove dead RLM tool config constants (#1189)

Full Changelog: v0.1.13.dev1...v0.1.13.dev2

v0.1.13.dev1

Verifiers v0.1.13.dev1 Release Notes

Date: 04/18/2026

Highlights since v0.1.12

• Added the SWELego-Real TaskSet (PrimeIntellect fork, filtered upstream) for broader SWE benchmark coverage.
• Added timeout_minutes kwarg to R2E, SWEBench, Multi-SWE, and OpenSWE tasksets for finer-grained per-task timeout control.
• Surfaced agent timeout as state['error'] in CliAgentEnv so timeouts are visible in eval results.
• Fixed CliAgentEnv poll loop to honor self.poll_interval consistently.
• Bumped prime-sandboxes to 0.2.20.

Changes included in v0.1.13.dev1 (since v0.1.12)

Features and enhancements

• feat: add SWELego-Real TaskSet (PrimeIntellect fork, filtered upstream) (#1149)
• feat: add timeout_minutes kwarg to R2E / SWEBench / Multi-SWE / OpenSWE tasksets (#1171)

Fixes and maintenance

• fix: surface agent timeout as state['error'] in CliAgentEnv (#1170)
• fix: honor self.poll_interval in CliAgentEnv poll loop (#1173)
• chore: bump prime-sandboxes to 0.2.20 (#1174)

Full Changelog: v0.1.12...v0.1.13.dev1