CHANGELOG.md

Changelog

Contract hash no longer depends on the process cwd or checkout path

_canon_mutable_trees previously RESOLVED the registered mutable-tree paths against the process cwd, folding the absolute checkout path into the contract hash — the same workspace hashed differently when evolve ran from a different directory, or after the workspace moved, and spuriously rolled its epoch. Paths are now normalized (never filesystem-resolved). BREAKING: every existing epoch's contract hash moves once; the workspace auto-rolls on the next evolve (the standard contract-change behavior). The parity CONTRACT-HASH golden is re-captured and is now checkout-independent.

All notable changes to zicato are recorded here. Format roughly follows Keep a Changelog.

[Unreleased]

⚠️ BREAKING DEFAULTS — noise-aware decision procedure (epochs auto-roll)

Existing workspaces whose contracts do not pin these knobs will AUTO-ROLL their epoch on the next evolve — the flipped defaults change the contract's canonical form, so the contract hash changes and a fresh epoch opens (generations across epochs are not directly comparable; this is the intended contract-roll semantics, not a migration bug). Pin the old values in scoring.json to keep an epoch's contract byte-stable ("tournament": {"params": {"replicates": 1}}, "proposer_quality": {"best_of_n": 1}, "overfitting": {"min_board_size_for_split": 8} — see examples/zicato_examples/target_0_convergence/scoring.json for the canonical pinned deterministic contract).

• Per-duel replicates default 1 → 2 for gauntlet / single-elim / double-elim / swiss (racing keeps 1 — its replication is intrinsic to the escalating board slices). The gauntlet's full-A/B path (run_tournament) now honors the structure's resolved replicates: two paired board runs, per-replicate cache slots, per-entry losses averaged before the gate. A duel decided by one paired run is decided by one noise draw; two averaged runs is the cheapest hedge.
• Proposer best_of_n default 1 → 3 with the self-critique pass: each propose-step samples a slate of three (each slot steered by a distinct edit-class hint) and selects the best. Pin best_of_n: 1 for scripted / deterministic proposers.
• overfitting.min_board_size_for_split default 8 → 6: boards of 6–7 entries now derive a holdout split.
• The Bradley–Terry evidence gate stays OPT-IN (promote_confidence_threshold absent ⇒ off) — deliberately. Measured (Tier-2 power harness): the gate blocks 100% of A/A false promotes, but on a two-contestant crowning pair its CIs separate only after an unbroken win streak of ~37 duels (mixed records never separate) — so a silent on-by-default gate with a small budget would freeze every true promotion at inconclusive, and a converging budget costs ~32×2×board fresh runs per crowning. The gate is a soundness device; decision power is bought with per-duel replication and a margin calibrated above the measured noise floor. When enabled it now reaches BOTH selection shapes: the multi-challenger driver AND the gauntlet crowning duel (previously unreachable under the default structure).
• Scaffolds enable the gate explicitly: zicato init writes the full effective recommended contract (racing field 4 / eta 2 / board_fraction 0.4, replicates 2, promote_confidence_threshold: 0.8, promote_confidence_replicates: 32) to the operator's live scoring.json (only when absent), and the tournament builder's blank draft opens on the same contract — the operator sees and prices the gate rather than inheriting a silent freeze or silent noise.
• A/A noise-floor calibration is the default soundness surface: zicato board audit (and the opt-in calibrate_noise_floor workspace knob) duels the champion against itself, persists the measured delta_scalar spread on the epoch record, and evolve WARNS loudly — with a margin_below_noise_floor health finding — when promote_margin sits inside the measured noise while the evidence gate is off (measured: a naive margin below the floor promotes pure noise — 20/60 A/A promotes in the Tier-2 harness).

Dashboard — decision-centric console (Variant T)

• Cross-epoch meta-loop ledger on the home view: the fleet of epochs and the champion lineage across them now lead the environment overview, replacing the full-width cross-epoch sparkline. Each contract change and champion-reign tick is anchored to the real generation position.
• Settings is a routed right-side drawer overlay, not a full page. Its Contract tab reuses the tournament builder's live preview, so the contract a run will use is previewed in place. Settings also carries a launcher into the standalone builder.
• The tournament builder is its own first-class view (#/builder), reachable from the Settings launcher and from the zicato builder CLI command (which boots the dashboard focused on the builder deep-link).
• Live racing hero redesigned: a metadata baseline strip, a full-width scalar track with a rung stepper, dense champion-gate rows, and a two-column WHAT'S RUNNING | LIVE ACTIVITY layout. Per-round racing models and the epoch/candidate views now build through one shared structure resolver, so non-gauntlet views render identically.
• S/M/L font-size control in the typeface picker (recalibrated ladder).
• Non-racing live-hero figures go responsive / full-width.
• Per-entry views surface a continuous outcome score plus precision / recall, alongside the existing pass/fail.

Dashboard — static-asset revalidation

• Static dashboard assets are served with an ETag derived from the file's identity and honor If-None-Match: an unchanged file revalidates to a bodyless 304 (no re-download), and editing a file changes its ETag so the browser always picks up CSS/JS edits.

Proposer

• The tool-using ADK agent is now the DEFAULT proposer (in builtin_default mode, bound to the auxiliary model and the read-only proposer tool registry). The skill-composed proposer is now the explicit opt-in (a configured proposers/<name>/ dir). The proposer is contract input #5: configuring a proposer dir — or editing one of its skills — rolls the epoch.
• A board-anonymized, train-slice-only failure-mode feedback channel to the proposer via the outcome_summarizer_spec operator hook: the proposer is fed marginal failure signal computed over the train slice only, with board contents anonymized, so it can target failure modes without memorizing the board.

Tournament / runtime

• Configurable racing wall-clock budgets: the racing strategy reads matchup_budget_seconds (caps every duel's total board-unit wall-clock) and final_rung_budget_seconds (overrides it for the final rung; defaults to the matchup budget). Both are opt-in — unset leaves a rung uncapped.
• Cost estimator uses per-structure default replicates: when params["replicates"] is unset, the builder's cost meter reads each strategy's own _default_replicates (swiss / single-elim / double-elim = 2; gauntlet / racing = 1) from the selection layer's single source of truth, instead of assuming a flat 1, so the meter cannot under-report the schedule a structure actually runs.

Gate / scoring

• Selectable pass-rate monotonicity scope. ScoringWeights pass_rate_monotonicity_scope selects per_entry (default — reject when any entry the parent passed regressed) vs aggregate (reject only when the child's overall pass-rate falls below the parent's beyond tolerance). The on/off switch remains the separate pass_rate_monotonicity bool. The scope is authored through the builder gate-config op (set_gate), surfaced in the analyzer report and the dashboard breakdown, and now carried through the subprocess-worker transport so an operator's aggregate choice can never silently downgrade to per_entry inside a worker.
• Continuous per-entry outcome scores. An expectation may return a float in [0, 1] recorded as a per-entry score plus precision / recall metrics, surfaced per entry in the dashboard.

Correctness

• per_judge_weights now survives the subprocess-worker transport (symmetric with the other scoring fields), so a worker scores with the operator's configured per-judge weights instead of resetting to the default.
• In-run process judges now grade the real tool-call ledger rather than the agent's narration (the file_findability judge was reading the agent's prose, not the actual tool calls), so a judgement reflects what the harness actually did, not what it claimed.

Storage

• Settled the storage design end-to-end and rewrote docs/design/STORAGE.md around it. The five data kinds get five fits: runtime state and the lineage/experiment/journal records → files (one record per file, through StorageBackend); telemetry → JSONL (goldfive's format, unchanged); generation source trees → a pluggable GenerationStore (directory snapshots today, git on the roadmap); the cross-run analytical index → a real DB (index.db, SQLite — DuckDB evaluated and deferred until scan latency is felt). The store of record stays files-canonical so concurrent subprocess runs stay lock-free and crash-isolated.
• Resolved the record-level vs generation-level seam fork: StorageBackend stays record-level (key→blob); generation source trees get a separate peer seam, GenerationStore, at the epoch/ domain layer — not an extension of StorageBackend. Forcing source-tree transactions into the record ABC would make it carry domain vocabulary and stop being an honest storage seam.
• zicato.epoch.genstore — the GenerationStore protocol plus the shipped DirectoryGenerationStore (the existing directory-snapshot mechanism, byte-for-byte). derive_generation is the generation-level transaction boundary the record seam could not express: copy the parent tree, apply the patch set all-or-nothing, return the child snapshot root. The orchestrator's snapshot / baseline-seed / patch-apply paths now route through this seam — the single site a git backend would later be substituted.
• epoch/ record I/O migrated onto StorageBackend (via the new zicato.epoch._storage adapter, mirroring runtime/._storage). experiment.json, the per-patch files, lineage.json, per-epoch config.json / scoring.json, and journal.md are now written with the same .tmp + fsync + rename atomicity the runtime layer already had — a crash mid-write can no longer leave a truncated record. No on-disk bytes moved; every epoch/ signature is unchanged.
• Continuous indexing promoted from an add-on to the stated design: the orchestrator's live dual-write keeps index.db current as the loop runs; zicato reindex is the batch rebuild / repair path. The canonical-file-first ordering rule is documented as load-bearing.

Robustness

• zicato evolve --max-wall-clock-seconds <S> — a total wall-clock budget for the whole evolve invocation. Previously only individual board entries had a wall_clock_budget_seconds; an N-entry × M-round invocation had no aggregate ceiling. The budget is enforced between rounds (the loop stops cleanly once it is spent) and within a round (a round that would overrun it is cancelled via asyncio.wait_for and recorded as an aborted round). It applies on top of — not instead of — each entry's per-entry budget. Also reads the ZICATO_MAX_WALL_CLOCK_SECONDS environment variable. When the loop stops on this budget, the evolve summary says so explicitly.

Dashboard

• The Files view gained a mutation-site browser. Below the generation file tree and patch set, an operator can now browse the mutation surface — every # zicato:mutable annotated span — and, for each site, see a line-level diff of the v0 baseline content against the patched content in any generation whose patch touched that mutation id. Sites no patch has touched just show their baseline content.
• New endpoints GET /api/mutations/{epoch} (the epoch's mutation surface, each site with file, role and the generations that patched it) and GET /api/mutations/{epoch}/{mutation_id} (one site's baseline content plus the per-generation patched content for a diff). Both are deterministic and backend-neutral: they re-enumerate each generation's materialised snapshot through enumerate_mutations and route patch-set reads through the GenerationStore seam, so the directory and git backends render identically. No re-apply — a generation's snapshot already IS its post-apply state.

[0.3.0] — 2026-05-15

Observability + analytics release. zicato grows a real dashboard, a SQLite analytical index, a tournament/competition view distinct from goldfive's execution view, and loop-health diagnostics. First real run against a live model landed in this cycle.

Live dashboard + supervisor

• zicato-supervisor Rust binary: watchdog (state-file monitoring + SIGTERM→SIGKILL escalation) + HTTP/SSE dashboard server, auto-spawned by zicato evolve.
• Multi-view dashboard (vanilla HTML/CSS/JS, bundled in the binary): Overview, Tree (cross-epoch lineage graph), Tournament (the bracket), Epoch (the contract). Fragment-routed; SSE-live.
• /api/epoch — the epoch's full evaluation contract (board, rubric, scoring, registered harness, mutation surface).
• /api/tournaments + /api/tournaments/:id — the gauntlet bracket and full per-matchup detail. /api/health-report — loop-health findings.
• Static assets served at document root (fixes the relative-path 404 that rendered the dashboard unstyled).

SQLite analytical index

• .zicato/index.db — a derived, queryable index of cross-run data (epochs / generations / experiments / patches / runs / loss_profiles / metric_counts / tournaments). Files stay canonical; the index is rebuildable via zicato reindex and dual-written live by the orchestrator. The Rust supervisor reads it via rusqlite.
• Generation source trees are NOT in SQLite (git, per the roadmap); per-run event capture stays JSONL. SQLite is the analytical layer only.

Tournament view — competition, not execution

• The dashboard Tournament view is a king-of-the-hill bracket: a champion lineage (winners spine) + discarded challengers, each matchup parent-vs-child over the board.
• Per-matchup detail: hypothesis, patches, per-entry A/B grid (improved/regressed/flat), scalar breakdown, gate verdict + reasoning.
• zicato/tournament/detail.py analytics: bracket assembly, A/B grid, hypothesis ledger (proposer calibration with explicit sign+magnitude match semantics), optimization trajectory + plateau detection, mutation heat map (win-correlation), tournament cost.
• Architectural split: harmonograf renders the execution view (the temporal trace of one run); the zicato dashboard owns the competition view. A per-run drill-down links the two.

Incremental board scoring

• The tournament runner now scores each board unit the instant its run(s) settle — on the same concurrency fan-out as the runs, not batched after every board finishes. As each unit completes its per-entry losses fold into a running partial aggregate (_IncrementalScorer) which is rewritten onto the live ActiveTournament (partial_parent_agg / partial_child_agg).
• A finished board's score is therefore available ASAP — while the sibling boards are still running — rather than only at round end. The dashboard's Partial aggregate panel now consumes the server-computed running aggregate (with a legacy fallback), so the scalar climbs as the tournament runs instead of sitting at 0.00.

Integration note

• This release folds together six dashboard / runner / proposer fixes developed in parallel: the route-driven Files view with a side-by-side changed-files diff; the Overview rebuilt as the environment home (with the queued-label fix); the Epoch view's narrative redesign; the Tournament view's harmonograf jump-off links (adk_session_id); incremental per-board scoring; and proposer/patch resilience (post-apply validation retry, full proposer content, surgical span apply). The partial-aggregate panel now lives in the Tournament view's hall — the Overview no longer renders the full board.

Loop-health diagnostics

• zicato/health/ — detects a toothless evaluation loop: degenerate scoring (consecutive zero-Δ tournaments), non-differentiating board entries, flat drift signal, missing expectations, stalled loop.
• zicato health CLI; the orchestrator writes a per-round health report, logs a loud warning on a critical finding, and stops the loop after sustained degeneracy (evolve --stop-on-degenerate).
• Motivated by the first real run — v0 and v1 scored identically (1.000000); the loop produced zero optimization signal and that was only discoverable by manual inspection.

Runtime + telemetry

• Orchestrator writes a populated heartbeat (epoch / generation / round / phase); per-run progress is bumped on each goldfive event so dashboard run cards animate.
• mutations.json dumped per epoch so the dashboard can show the mutation surface.
• HarmonografSink attached when ZICATO_HARMONOGRAF_URL is set — runs stream live to a harmonograf server; the dashboard deep-links each run to its harmonograf session.
• Progressive analysis.html — regenerated after every generation.

Harmonograf companion

• pedapudi/harmonograf#292 — a harmonograf-replay command ingests a zicato run's events.jsonl from disk, so any finished run is viewable in harmonograf.

Design docs

• New: TOURNAMENT.md, ANALYTICAL-INDEX.md, LOOP-HEALTH.md, RUNTIME.md, DASHBOARD.md, ROBUSTNESS.md, STORAGE.md.
• Updated ARCHITECTURE.md / CLI.md / EPOCHS-AND-JOURNALING.md.

Tests

• ~961 Python tests + the Rust supervisor suite (48 unit + 19 integration). The 5 pre-existing environment-dependent failures (goldfive editable-install drift) are unchanged.

[0.2.0] — 2026-05-15

Second alpha. Major surface expansion: drift-free objectives are first-class, the board API has a friendly Python builder, multi-objective scoring lands, regression-suite gating protects against breaking-change patches, and zicato now consumes goldfive's decision telemetry through a dedicated LLM analyzer.

Drift-free metrics

• MetricCount replaces drift-only counts. Arbitrary namespaces: drift:*, cost:*, latency:*, rubric:*, output:*, schema:*.
• LossProfile.metric_counts carries the unified view alongside drift_counts for back-compat. New first-class fields: tokens_spent, output_chars, schema_failures.
• HypothesisSpec.expected_metric_movements accepted alongside expected_drift_movements. Proposer schema validates either form.
• OutcomeRecord.metric_movements records actual deltas per namespace.
• New detectors: detect_metric_frequency(namespace=...), detect_cost_outliers, detect_rubric_score_movement. The original detect_drift_kind_frequency is now a thin wrapper.
• Analysis renderer: render_metric_movement_table(namespace_filter=...) with a "drift" filter producing byte-identical legacy output.

Multi-objective scoring

• ScoringWeights.namespace_weights — per-namespace weights with a sign convention (positive = higher-is-worse; negative = higher-is-better; zero = excluded from scalar).
• ScoringWeights.namespace_monotonicity — per-namespace monotonicity flags. Regression on a tracked namespace hard-rejects the candidate regardless of overall scalar improvement.
• aggregate_namespaced_metrics + scalar_components in the generation aggregate.
• evaluate_gate adds rule 3: per-namespace monotonicity check.

Regression-suite gate

• zicato.tournament.regression.run_regression_suite — asyncio subprocess pytest invoker with timeout + failed-id parsing.
• ScoringWeights.regression_gate_enabled / regression_test_command / regression_timeout_s. When enabled, any test failure in the candidate snapshot hard-rejects regardless of other metrics.
• tournament --skip-regression CLI flag.

Friendlier board API

• Board + Entry programmatic builder. Entry(id=..., input=..., evaluate=..., budget_s=...) auto-detects the kind from arguments (turns → multi-turn-scripted, persona → multi-turn-emulated, adversarial_agent_spec → synthetic-adversarial, etc.).
• Predicate factory family: Predicate.contains / .regex / .schema / .python.
• Rubric.judge(rubric_text, *, threshold, scale) — built-in LLM-as-judge matcher. Operator supplies the rubric text only; no separate Python module needed.
• New expectation kind "rubric" with evaluate_rubric_judge runtime.
• budget_s JSONL field as a back-compat alias for wall_clock_budget_seconds (preferred on write, accepted on read).

Decision-telemetry analyzer

• New zicato.analyzer module: DecisionEventSummary + aggregate_decision_events parse goldfive's new decision events (SteeringDecisionMade, LadderTransitionDecided, DetectorDispatchOrdered, PolicyApplied, RetryBudgetSpent).
• analyze_epoch_telemetry calls the auxiliary LLM with a structured prompt and writes epochs/{epoch}/insights/round_{N}.md with sections: Headline observations, Suspected over-intervention, Suspected under-intervention, Suggested next mutations.
• Orchestrator invokes the analyzer best-effort after every round; proposer embeds the latest insights into its user prompt so the next proposal is informed by the cross-run patterns.
• CLI: zicato analyze-telemetry [--workspace ...] [--epoch ...] [--round N].

Goldfive companion PRs (separate repo, awaiting merge)

• pedapudi/goldfive#440 — manifest expansion from 31 → 60 entries; new proto events LadderTransitionDecided, DetectorDispatchOrdered, PolicyApplied, RetryBudgetSpent at tags 40-43.
• pedapudi/goldfive#439 — pluggable Judge protocol with JudgeContext / JudgeVerdict; goldfive.wrap(judges=[...]); built-in detectors refactored as Judge instances; new JudgementEmitted proto event at tag 44.

Tests

• 644 → 813 passing (+169). Same 5 pre-existing environment failures on test_adapter_adk / test_synthetic_adversarial / test_telemetry_reducer_real_goldfive are unchanged (all environment-dependent on goldfive-editable-install drift).

Compatibility

• Every existing LossProfile, JSONL board, expected_drift_movements hypothesis, and tournament scoring config from 0.1 continues to work. The drift namespace is preserved verbatim; multi-objective scoring uses sensible defaults when namespace weights aren't configured.

[0.1.0] — 2026-05-15

First public alpha. End-to-end evolve loop functioning against both intended v0 dogfood targets. The library is usable; the API will still break in patch releases until v0.2.

Core loop

• zicato init / register / epoch new / mutations / propose / tournament / evolve / epoch close CLI surface auto-discovered via zicato.cli.commands.*.
• Annotated mutation surface (# zicato:mutable and # zicato:mutable:file) with AST resolution, applier, validator, and a mutations audit subcommand.
• Board format (single-turn, multi-turn-scripted, multi-turn-emulated, synthetic-adversarial, synthetic-clean) with five expectation kinds.
• Collusion-proof multi-turn emulator with the two-callable rule, sealed context construction, and a post-hoc answer-leak heuristic.
• Telemetry ingest via goldfive's JSONLPersistenceSink per run + a post-run reducer that produces LossProfile.
• Six pattern detectors (drift-kind frequency, hot tasks/agents, plan-revision instability, multi-turn memory failure, multi-turn context loss).
• LLM proposer with structured-output Experiment schema (hypothesis + patches) and retry-on-parse-failure.
• Tournament runner: full mode + fast inline keep/discard. Promote gate enforces a scalar margin AND strict pass-rate monotonicity.
• Epoch lifecycle with manual close + auto-close fallback; running journal.md; analysis pass at close producing both analysis.md (LLM-driven narrative) and a self-contained analysis.html (deterministic — inline SVG lineage, score trajectory, drift heatmap, per-experiment cards, dark-mode aware).
• Per-patch JSON files under generations/v{N}/patches/{patch_id}.json with patch_ids references in experiment.json (atomic write order: patches first, then experiment).

v1.1 robustness

• .zicato/runtime/ state file layer (heartbeat, active_runs, active_tournament, control, control_log, lock).
• Workspace lock with PID-aware stale-pid stealing.
• Heartbeat beater task lifecycle wired into evolve_n_rounds.
• Per-call aux_call_llm timeouts (ZICATO_AUX_CALL_TIMEOUT, default 120s) at every auxiliary call site (proposer, judge, emulator, analysis).
• Progressive analysis.html regeneration after each generation (in addition to the final LLM-driven analysis.md at epoch close).
• Tournament runner publishes active_tournament.json + per-run active_runs/{run_id}.json for the live dashboard.

Live dashboard

• Rust zicato-supervisor binary (single static binary, ~4.4 MB stripped). Auto-spawned by zicato evolve; URL printed to stdout. Opt out with --no-dashboard.
• Watchdog half: inotify-based file watching with SIGTERM → grace → SIGKILL signal escalation for stalled orchestrator + stalled runs.
• HTTP + Server-Sent Events server (default :7892). Endpoints: /api/state, /api/lineage, /api/active-runs, /api/active-tournament, /api/heartbeat, /events, /api/health, plus POST control endpoints (/api/control/pause, …/kill/{id}, …/promote/{gen}, …/reject/{gen}, …/rubric).
• Single-page browser UI (vanilla HTML/CSS/JS, no build step, ~77KB total) bundled with the binary via include_dir!. Includes lineage SVG, score trajectory, drift heatmap, tournament status panel with deterministic predicted-gate-verdict, active-runs strip, log tail, drill-down panels. Dark-mode aware. Action buttons present but disabled — v1.3 enables them via the control-file protocol.

Examples

• examples/target_1_presentation/ — vendored harmonograf reference agent with 9 mutation points + 7-entry board + rubric + scoring + deterministic mocks + RUN.md.
• examples/target_2_goldfive_steering/ — adversarial board against goldfive's optimization surface (pedapudi/goldfive PR #436) with LoopingAgent / HallucinatingAgent / RefusingAgent etc., a manifest_bridge translating goldfive's optimization manifest into zicato MutationPoints, mocks + RUN.md.

Design docs

• ARCHITECTURE.md, MUTATION-SURFACE.md, BOARD-FORMAT.md, EPOCHS-AND-JOURNALING.md, TELEMETRY.md, SCORING.md, EMULATOR.md, DOGFOOD-TARGETS.md, CLI.md, RATIONALE.md, VOCABULARY.md, RUNTIME.md, DASHBOARD.md, ROBUSTNESS.md, STORAGE.md — comprehensive coverage of every load-bearing decision.

Tooling

• hatchling-based packaging; uv + pip-editable installs work.
• pytest + pytest-asyncio test suite (~640 passing).
• ruff + mypy --strict clean on the Python tree.
• cargo build --release + cargo test + cargo clippy clean on the Rust supervisor binary.
• GitHub Actions CI workflow over Python 3.11/3.12.
• Makefile with install, test, lint, format, typecheck, check, supervisor, supervisor-test, install-supervisor, clean targets.

Known limitations (documented in RUN.md and the design docs)

• Mock-driven smoke runs produce byte-equal outputs across generations in v0; the tournament gate correctly rejects with "insufficient margin". Replace mocks with real LLMs for a meaningful loop.
• Multi-turn scripted/emulated drivers abort some examples with TypeErrors; the reducer treats those entries as zero-signal and the tournament continues. Cleanup is v1.0.1 work.
• Subprocess tournament workers (L3 robustness) and orchestrator control-file consumer (v1.3) are NOT yet implemented; see docs/design/ROBUSTNESS.md for the phasing plan.
• Git-backed generation storage (G1-G10) is documented but not yet implemented; v0 uses directory snapshots + per-patch JSON files.