You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The README was written before workspace separation, suggest-roles, JSON
sidecar emission, config auto-pickup, and --assume-untagged shipped. The
quickstart section in particular described an outdated three-step flow
that didn't reflect what the tool actually does today.
Restructured to walk users through the real flow:
1. Workspace setup (once): `init-judge` installs both bundled subagents.
2. Configure roles (once per target): `suggest-roles --apply` writes
directly into the workspace's ai_dev.yaml.
3. Run analyses (every time): `analyze TARGET --judge claude-cli`.
Config auto-loads, JSON sidecar auto-emits.
4. Handle missing trailers via `--assume-untagged`.
Other additions:
- Full CLI reference table (every command + every analyze flag) so users
can find the right knob without spelunking through --help output.
- JSON output structure section documenting the schema for downstream
consumers (Claude Cowork report generators, custom dashboards).
- Optional providers section noting that anthropic-api/openai/ollama are
stubbed at v0.2.
- Updated workspace layout diagram to show the new files
(roles-architect.md, ai_dev.yaml.bak from --apply, etc.).
No code changes. Verified all relative links in the README still resolve.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
> Measure how much AI co-programming actually accelerates software delivery — on any git repo.
4
4
5
-
`ai-dev-effectiveness` reads your git history, detects which commits were co-authored by AI coding agents (Claude, Copilot, Cursor, Codex, Aider, …), and produces an interactive HTML report comparing your real delivery against a hypothetical traditional team. It triangulates three independent estimators — top-down specialist roles, bottom-up per-commit formulas, and an AI-judge that reads each diff — so the productivity multipliers are credible, not just plausible.
5
+
`ai-dev-effectiveness` reads your git history, detects which commits were co-authored by AI coding agents (Claude, Copilot, Cursor, Codex, Aider, …), and produces an interactive HTML report (and a structured JSON sidecar) comparing your real delivery against a hypothetical traditional team. It triangulates three independent estimators — top-down specialist roles, bottom-up per-commit formulas, and an AIjudge that reads each diff — so the productivity multipliers are credible, not just plausible.
The analyzer **never writes inside the target repo** — the bundled subagent, the judgment cache, and all reports live in your local workspace. Each target gets its own subfolder so you can compare repos side by side.
28
+
That's the full happy-path. Each step is explained below.
29
+
30
+
The analyzer **never writes inside the target repo** — bundled subagents, the judgment cache, and all reports live in your workspace. Each target gets its own subfolder so you can compare repos side by side.
24
31
25
32
## What it measures
26
33
27
-
Three independent productivity multipliers:
34
+
Three independent productivity multipliers triangulating the same question:
28
35
29
36
1.**Top-down** — what would a traditional specialist team need? Person-months by role.
This installs both bundled subagents into `~/dev-effectiveness/.claude/agents/` (`effort-judge` for the per-commit AI judge, `roles-architect` for proposing specialist roles) plus a `settings.recommended.json` with the optional [ast-index](https://github.com/defendend/Claude-ast-index-search) MCP snippet.
52
+
53
+
### 2. Configure roles for each target (once per repo)
The `roles-architect` subagent surveys your codebase, identifies natural domain clusters, and writes calibrated specialist roles directly into `~/dev-effectiveness/ai_dev.yaml`. Takes 60–180 seconds, uses your existing Claude subscription, no diffs leave your machine. The original config (if any) is backed up as `ai_dev.yaml.bak`.
60
+
61
+
If you'd rather inspect the YAML before applying, drop `--apply` to print to stdout, or use `--out roles.yaml` to write to a file.
62
+
63
+
You can also write roles by hand — see [Configuring specialist roles](#configuring-specialist-roles) below.
The JSON contains everything needed for downstream report generation (Claude Cowork, custom dashboards, etc.) — three estimators side-by-side, cost comparison, per-agent / per-author / per-domain / per-package breakdowns, and an audit sample of judge rationales.
81
+
82
+
### 4. When trailers are missing
83
+
84
+
Real-world git histories often under-report AI assistance — GitHub squash-merges strip Co-Authored-By trailers, and some workflows commit without invoking the AI tool's commit flow. If you know your project was AI-paired but trailer detection looks too low:
36
85
37
86
```bash
38
-
cd~/dev-effectiveness # the workspace where init-judge ran
The judge uses your existing Claude Code CLI session — no API key, no metered billing, no diffs leaving your machine. If you have [ast-index](https://github.com/defendend/Claude-ast-index-search) on your `PATH`, the analyzer runs `ast-index rebuild` against the target before the judge starts so the bundled subagent can use `mcp__ast-index__*` tools for symbol-level lookups (much faster than grep-then-read for "find callers of this function").
92
+
Every untagged non-merge commit gets attributed to the named agent. Merge commits stay un-attributed (they don't represent direct authorship). Persist the choice in `ai_dev.yaml`:
43
93
44
-
Other providers (opt-in): `--judge anthropic-api` (separate API key), `--judge openai`, `--judge ollama` (local model).
94
+
```yaml
95
+
agents:
96
+
assume_untagged: "Claude Opus"
97
+
```
45
98
46
-
Disable ast-index for a run: `--no-ast-index`.
99
+
Run `ai-dev-effectiveness list-agents` to see valid agent names.
47
100
48
101
## Workspace layout
49
102
50
103
```
51
104
~/dev-effectiveness/ # your analyzer workspace
52
105
├── .claude/
53
-
│ ├── agents/effort-judge.md # ← installed by `init-judge`
106
+
│ ├── agents/effort-judge.md # bundled by init-judge
| `--no-ast-index` | Skip the `ast-index rebuild` step. |
143
+
| `--config FILE` | Use a non-standard config path (otherwise auto-picked). |
144
+
145
+
## Optional providers
72
146
73
-
[`examples/robotics-case-study/`](examples/robotics-case-study/) — anonymized analysis of a real 117K-LOC robotics project built by 1 dev + Claude over 10 months. Use it to see what a credible report looks like and to calibrate your own results.
147
+
The default AI judge uses your Claude Code CLI subscription (no API key, no metered billing, no diffs leaving your machine). Other providers exist for users who'd rather pay metered:
74
148
75
-
## Customize
149
+
- `--judge anthropic-api` — uses `ANTHROPIC_API_KEY` env var
150
+
- `--judge openai` — uses `OPENAI_API_KEY` env var
151
+
- `--judge ollama` — local model via Ollama (privacy-preserving)
152
+
- `--judge stub` — deterministic, used in CI tests
76
153
77
-
-`ai-dev-effectiveness init-config` — drops `ai_dev.yaml` into your workspace.
78
-
- Define your specialist roles, effort constants, agent signatures.
154
+
The `anthropic-api`, `openai`, and `ollama` providers are stubbed at v0.2 and will ship in a follow-up release.
155
+
156
+
## Case study
157
+
158
+
[`examples/robotics-case-study/`](examples/robotics-case-study/) — anonymized analysis of a real 117K-LOC ROS2 robotics project built by 1 dev + Claude over 10 months. Use it to see what a credible report looks like and to calibrate your own results.
79
159
80
160
## Configuring specialist roles
81
161
82
-
The top-down half of the analysis answers "what would a traditional team need?"
83
-
by summing your declared specialist roles. Without roles, the report skips the
84
-
top-down chart and the three-way reconciliation falls to two-way (formula +
85
-
judge only). Adding even three or four roles makes the comparison defensible.
162
+
The top-down half of the analysis answers "what would a traditional team need?" by summing your declared specialist roles. Without roles, the top-down chart is skipped and the three-way reconciliation falls to two-way (bottom-up + judge only).
Surveys the target codebase via a bundled Claude Code subagent
95
-
(`roles-architect`), identifies natural domain clusters, and prints a YAML
96
-
snippet ready to paste under your `ai_dev.yaml``roles:` key. Uses your
97
-
existing Claude subscription (no API key, no diffs leaving your machine).
98
-
Takes 60-180 seconds. Edit the result freely — the agent provides a starting
99
-
point grounded in your actual file layout, not the final word.
100
-
101
-
### Option B — write the roles yourself
170
+
Writes roles directly into your workspace's `ai_dev.yaml`. Drop `--apply` to print to stdout for inspection first.
102
171
103
-
Each role describes the slice of the codebase one specialist would own and
104
-
how many person-months a senior engineer (working without AI assistance)
105
-
would need to deliver it. Schema:
172
+
### Option B — write them by hand
106
173
107
174
```yaml
108
175
roles:
@@ -123,23 +190,42 @@ How to estimate `pm_low` / `pm_high`:
123
190
| Glue / config / IaC | 20-30 |
124
191
| ML training / distributed systems | 15-25 (incl. ramp) |
125
192
126
-
`pm = loc / (rate_per_day * 22 days_per_month)`. Use the lower rate for
127
-
`pm_high`, the higher rate for `pm_low`. The notebook averages the range to
128
-
produce the headline multiplier — wider spread = more uncertainty.
129
-
130
-
### A worked example
193
+
`pm = loc / (rate_per_day × 22 working_days_per_month)`. Use the lower rate for `pm_high`, the higher rate for `pm_low`. The tool averages the range — wider spread = more uncertainty.
194
+
195
+
[`examples/robotics-case-study/ai_dev.yaml`](examples/robotics-case-study/ai_dev.yaml) is a worked example covering a multi-domain ROS2 / CUDA / ML project.
0 commit comments