Skip to content

Commit 8267be0

Browse files
denn-gubskyclaude
andcommitted
README: rewrite around the v0.2.x flow
The README was written before workspace separation, suggest-roles, JSON sidecar emission, config auto-pickup, and --assume-untagged shipped. The quickstart section in particular described an outdated three-step flow that didn't reflect what the tool actually does today. Restructured to walk users through the real flow: 1. Workspace setup (once): `init-judge` installs both bundled subagents. 2. Configure roles (once per target): `suggest-roles --apply` writes directly into the workspace's ai_dev.yaml. 3. Run analyses (every time): `analyze TARGET --judge claude-cli`. Config auto-loads, JSON sidecar auto-emits. 4. Handle missing trailers via `--assume-untagged`. Other additions: - Full CLI reference table (every command + every analyze flag) so users can find the right knob without spelunking through --help output. - JSON output structure section documenting the schema for downstream consumers (Claude Cowork report generators, custom dashboards). - Optional providers section noting that anthropic-api/openai/ollama are stubbed at v0.2. - Updated workspace layout diagram to show the new files (roles-architect.md, ai_dev.yaml.bak from --apply, etc.). No code changes. Verified all relative links in the README still resolve. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent f47f81b commit 8267be0

1 file changed

Lines changed: 146 additions & 60 deletions

File tree

README.md

Lines changed: 146 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -2,107 +2,174 @@
22

33
> Measure how much AI co-programming actually accelerates software delivery — on any git repo.
44
5-
`ai-dev-effectiveness` reads your git history, detects which commits were co-authored by AI coding agents (Claude, Copilot, Cursor, Codex, Aider, …), and produces an interactive HTML report comparing your real delivery against a hypothetical traditional team. It triangulates three independent estimators — top-down specialist roles, bottom-up per-commit formulas, and an AI-judge that reads each diff — so the productivity multipliers are credible, not just plausible.
5+
`ai-dev-effectiveness` reads your git history, detects which commits were co-authored by AI coding agents (Claude, Copilot, Cursor, Codex, Aider, …), and produces an interactive HTML report (and a structured JSON sidecar) comparing your real delivery against a hypothetical traditional team. It triangulates three independent estimators — top-down specialist roles, bottom-up per-commit formulas, and an AI judge that reads each diff — so the productivity multipliers are credible, not just plausible.
66

77
## Quickstart
88

99
```bash
10+
# Install (macOS):
11+
brew install pipx && pipx ensurepath
1012
pipx install git+https://github.com/denn-gubsky/ai-dev-effectiveness
1113

12-
# Pick a folder you'll run analyses from. NOT one of your project repos.
14+
# Pick a folder DEDICATED to running analyses NOT one of your project repos.
1315
mkdir -p ~/dev-effectiveness && cd ~/dev-effectiveness
1416

15-
# Once: install the bundled judge subagent into THIS workspace.
17+
# One-time: install the bundled subagents (effort-judge + roles-architect).
1618
ai-dev-effectiveness init-judge
1719

18-
# Now analyze any repo you want. Reports go to ./<target_basename>/.
19-
ai-dev-effectiveness analyze /path/to/your/repo
20+
# One-time per target: have Claude survey the repo and propose specialist roles.
21+
ai-dev-effectiveness suggest-roles ~/work/your-repo --apply
22+
23+
# Every run: just go. Config auto-loads, JSON sidecar emits by default.
24+
ai-dev-effectiveness analyze ~/work/your-repo --judge claude-cli
2025
open ./your-repo/effectiveness-report.html
2126
```
2227

23-
The analyzer **never writes inside the target repo** — the bundled subagent, the judgment cache, and all reports live in your local workspace. Each target gets its own subfolder so you can compare repos side by side.
28+
That's the full happy-path. Each step is explained below.
29+
30+
The analyzer **never writes inside the target repo** — bundled subagents, the judgment cache, and all reports live in your workspace. Each target gets its own subfolder so you can compare repos side by side.
2431

2532
## What it measures
2633

27-
Three independent productivity multipliers:
34+
Three independent productivity multipliers triangulating the same question:
2835

2936
1. **Top-down** — what would a traditional specialist team need? Person-months by role.
3037
2. **Bottom-up** — per-commit effort using language-weighted formulas (base hours + per-LOC rate).
3138
3. **AI judge** — Claude reads each diff and emits a calibrated estimate via the bundled subagent.
3239

33-
When all three agree within 2x, the multiplier is defensible. When they diverge, the report flags it.
40+
When all three agree within 2×, the multiplier is defensible. When they diverge, the report flags it.
41+
42+
## End-to-end flow
43+
44+
### 1. Workspace setup (once)
45+
46+
```bash
47+
mkdir -p ~/dev-effectiveness && cd ~/dev-effectiveness
48+
ai-dev-effectiveness init-judge
49+
```
50+
51+
This installs both bundled subagents into `~/dev-effectiveness/.claude/agents/` (`effort-judge` for the per-commit AI judge, `roles-architect` for proposing specialist roles) plus a `settings.recommended.json` with the optional [ast-index](https://github.com/defendend/Claude-ast-index-search) MCP snippet.
52+
53+
### 2. Configure roles for each target (once per repo)
54+
55+
```bash
56+
ai-dev-effectiveness suggest-roles ~/work/your-repo --apply
57+
```
58+
59+
The `roles-architect` subagent surveys your codebase, identifies natural domain clusters, and writes calibrated specialist roles directly into `~/dev-effectiveness/ai_dev.yaml`. Takes 60–180 seconds, uses your existing Claude subscription, no diffs leave your machine. The original config (if any) is backed up as `ai_dev.yaml.bak`.
60+
61+
If you'd rather inspect the YAML before applying, drop `--apply` to print to stdout, or use `--out roles.yaml` to write to a file.
62+
63+
You can also write roles by hand — see [Configuring specialist roles](#configuring-specialist-roles) below.
64+
65+
### 3. Run analyses (every time)
3466

35-
## Optional: AI judge for diff-aware estimates
67+
```bash
68+
ai-dev-effectiveness analyze ~/work/your-repo --judge claude-cli
69+
```
70+
71+
What happens:
72+
73+
- **Workspace config auto-pickup**`<workspace>/ai_dev.yaml` is loaded automatically; you only need `--config` for non-standard paths.
74+
- **`ast-index rebuild`** runs against the target if `ast-index` is on `PATH`.
75+
- **Stratified sampling** of ~30–50 commits across (domain × size_bucket) for the judge.
76+
- **Reports written** to `<workspace>/<target_basename>/`:
77+
- `effectiveness-report.html` — interactive Plotly dashboard
78+
- `effectiveness-report.json` — structured AI-readable analysis
79+
80+
The JSON contains everything needed for downstream report generation (Claude Cowork, custom dashboards, etc.) — three estimators side-by-side, cost comparison, per-agent / per-author / per-domain / per-package breakdowns, and an audit sample of judge rationales.
81+
82+
### 4. When trailers are missing
83+
84+
Real-world git histories often under-report AI assistance — GitHub squash-merges strip Co-Authored-By trailers, and some workflows commit without invoking the AI tool's commit flow. If you know your project was AI-paired but trailer detection looks too low:
3685

3786
```bash
38-
cd ~/dev-effectiveness # the workspace where init-judge ran
39-
ai-dev-effectiveness analyze /path/to/your/repo --judge claude-cli
87+
ai-dev-effectiveness analyze ~/work/your-repo \
88+
--assume-untagged "Claude Opus" \
89+
--judge claude-cli
4090
```
4191

42-
The judge uses your existing Claude Code CLI session — no API key, no metered billing, no diffs leaving your machine. If you have [ast-index](https://github.com/defendend/Claude-ast-index-search) on your `PATH`, the analyzer runs `ast-index rebuild` against the target before the judge starts so the bundled subagent can use `mcp__ast-index__*` tools for symbol-level lookups (much faster than grep-then-read for "find callers of this function").
92+
Every untagged non-merge commit gets attributed to the named agent. Merge commits stay un-attributed (they don't represent direct authorship). Persist the choice in `ai_dev.yaml`:
4393

44-
Other providers (opt-in): `--judge anthropic-api` (separate API key), `--judge openai`, `--judge ollama` (local model).
94+
```yaml
95+
agents:
96+
assume_untagged: "Claude Opus"
97+
```
4598
46-
Disable ast-index for a run: `--no-ast-index`.
99+
Run `ai-dev-effectiveness list-agents` to see valid agent names.
47100

48101
## Workspace layout
49102

50103
```
51104
~/dev-effectiveness/ # your analyzer workspace
52105
├── .claude/
53-
│ ├── agents/effort-judge.md # ← installed by `init-judge`
106+
│ ├── agents/effort-judge.md # bundled by init-judge
107+
│ ├── agents/roles-architect.md
54108
│ ├── skills/effort-estimation/SKILL.md
55109
│ └── settings.recommended.json
56-
├── ai_dev.yaml # optional: custom config (init-config)
57-
├── repo-a/ # output for one analyzed target
110+
├── ai_dev.yaml # auto-loaded by `analyze`
111+
├── ai_dev.yaml.bak # written by `suggest-roles --apply`
112+
├── your-repo/
58113
│ ├── effectiveness-report.html
59-
│ ├── effectiveness-report.json
60-
│ └── .ai-dev-effectiveness-cache/
61-
└── repo-b/ # another target's outputs
114+
│ ├── effectiveness-report.json # auto-emitted by default
115+
│ └── .ai-dev-effectiveness-cache/ # location-stable judge cache
116+
└── another-repo/
62117
└── ...
63118
```
64119
65-
Useful flags:
120+
## CLI reference
66121
67-
- `--workspace PATH` — override the analyzer workspace (default: `$PWD`).
68-
- `--out-dir PATH` — override where this run's reports go.
69-
- `--judge-dry-run` — show what the judge would do (sample size, cost, paths) without invoking it.
122+
| Command | Purpose |
123+
|---|---|
124+
| `init-judge [WORKSPACE]` | Install bundled subagents into workspace's `.claude/`. |
125+
| `init-config [--out PATH]` | Drop a starter `ai_dev.yaml` template. |
126+
| `suggest-roles TARGET [--apply]` | Survey TARGET and propose specialist roles. With `--apply`, write directly into workspace's `ai_dev.yaml`. |
127+
| `list-agents` | Print the built-in AI-agent signature registry. |
128+
| `validate-config FILE` | Schema-check an `ai_dev.yaml` without running. |
129+
| `analyze TARGET [flags]` | Run the analysis. |
70130
71-
## Case study
131+
Useful `analyze` flags:
132+
133+
| Flag | Effect |
134+
|---|---|
135+
| `--judge claude-cli` | Enable the AI judge using your Claude subscription. |
136+
| `--judge-dry-run` | Show what the judge would do (sample size, paths) without running it. |
137+
| `--judge-all` | Judge every commit instead of stratified sampling. |
138+
| `--assume-untagged "Claude Opus"` | Attribute untagged non-merge commits to a named agent. |
139+
| `--workspace PATH` | Override the workspace (default `$PWD`). |
140+
| `--out-dir PATH` | Override report destination (cache stays at the default location). |
141+
| `--format html\|json\|both` | Output formats. Default `both` writes HTML + JSON. |
142+
| `--no-ast-index` | Skip the `ast-index rebuild` step. |
143+
| `--config FILE` | Use a non-standard config path (otherwise auto-picked). |
144+
145+
## Optional providers
72146
73-
[`examples/robotics-case-study/`](examples/robotics-case-study/) — anonymized analysis of a real 117K-LOC robotics project built by 1 dev + Claude over 10 months. Use it to see what a credible report looks like and to calibrate your own results.
147+
The default AI judge uses your Claude Code CLI subscription (no API key, no metered billing, no diffs leaving your machine). Other providers exist for users who'd rather pay metered:
74148
75-
## Customize
149+
- `--judge anthropic-api` — uses `ANTHROPIC_API_KEY` env var
150+
- `--judge openai` — uses `OPENAI_API_KEY` env var
151+
- `--judge ollama` — local model via Ollama (privacy-preserving)
152+
- `--judge stub` — deterministic, used in CI tests
76153
77-
- `ai-dev-effectiveness init-config` — drops `ai_dev.yaml` into your workspace.
78-
- Define your specialist roles, effort constants, agent signatures.
154+
The `anthropic-api`, `openai`, and `ollama` providers are stubbed at v0.2 and will ship in a follow-up release.
155+
156+
## Case study
157+
158+
[`examples/robotics-case-study/`](examples/robotics-case-study/) — anonymized analysis of a real 117K-LOC ROS2 robotics project built by 1 dev + Claude over 10 months. Use it to see what a credible report looks like and to calibrate your own results.
79159
80160
## Configuring specialist roles
81161
82-
The top-down half of the analysis answers "what would a traditional team need?"
83-
by summing your declared specialist roles. Without roles, the report skips the
84-
top-down chart and the three-way reconciliation falls to two-way (formula +
85-
judge only). Adding even three or four roles makes the comparison defensible.
162+
The top-down half of the analysis answers "what would a traditional team need?" by summing your declared specialist roles. Without roles, the top-down chart is skipped and the three-way reconciliation falls to two-way (bottom-up + judge only).
86163
87-
### Option A — let Claude propose the roles
164+
### Option A — let Claude propose them
88165
89166
```bash
90-
cd ~/dev-effectiveness
91-
ai-dev-effectiveness suggest-roles /path/to/your/repo
167+
ai-dev-effectiveness suggest-roles ~/work/your-repo --apply
92168
```
93169

94-
Surveys the target codebase via a bundled Claude Code subagent
95-
(`roles-architect`), identifies natural domain clusters, and prints a YAML
96-
snippet ready to paste under your `ai_dev.yaml` `roles:` key. Uses your
97-
existing Claude subscription (no API key, no diffs leaving your machine).
98-
Takes 60-180 seconds. Edit the result freely — the agent provides a starting
99-
point grounded in your actual file layout, not the final word.
100-
101-
### Option B — write the roles yourself
170+
Writes roles directly into your workspace's `ai_dev.yaml`. Drop `--apply` to print to stdout for inspection first.
102171

103-
Each role describes the slice of the codebase one specialist would own and
104-
how many person-months a senior engineer (working without AI assistance)
105-
would need to deliver it. Schema:
172+
### Option B — write them by hand
106173

107174
```yaml
108175
roles:
@@ -123,23 +190,42 @@ How to estimate `pm_low` / `pm_high`:
123190
| Glue / config / IaC | 20-30 |
124191
| ML training / distributed systems | 15-25 (incl. ramp) |
125192

126-
`pm = loc / (rate_per_day * 22 days_per_month)`. Use the lower rate for
127-
`pm_high`, the higher rate for `pm_low`. The notebook averages the range to
128-
produce the headline multiplier — wider spread = more uncertainty.
129-
130-
### A worked example
193+
`pm = loc / (rate_per_day × 22 working_days_per_month)`. Use the lower rate for `pm_high`, the higher rate for `pm_low`. The tool averages the range — wider spread = more uncertainty.
194+
195+
[`examples/robotics-case-study/ai_dev.yaml`](examples/robotics-case-study/ai_dev.yaml) is a worked example covering a multi-domain ROS2 / CUDA / ML project.
196+
197+
## JSON output structure
198+
199+
```jsonc
200+
{
201+
"tool": "ai-dev-effectiveness",
202+
"version": "0.2.0",
203+
"generated_at": "2026-05-05T...",
204+
"project": { "name": "...", "team_size": 1, ... },
205+
"headline": { "n_commits": 68, "n_ai_assisted": 56, "loc_total": 22277, ... },
206+
"estimators": {
207+
"top_down": { "roles": [...], "total_pm_mid": 23, "multiplier": 230 },
208+
"bottom_up": { "total_traditional_hours": 863, "multiplier": 10.8 },
209+
"judge": { "total_human_hours": ..., "multiplier": 2.13, ... }
210+
},
211+
"cost_comparison": { "traditional_cost_usd": 404800, "savings_multiplier": 227 },
212+
"by_agent": [...],
213+
"by_author": [...],
214+
"by_domain": [...],
215+
"by_package": [...],
216+
"by_complexity": [...],
217+
"author_agent_matrix": [...],
218+
"judge_samples": [...], // up to 15 judged commits with rationale
219+
"weekly": [...]
220+
}
221+
```
131222

132-
[`examples/robotics-case-study/ai_dev.yaml`](examples/robotics-case-study/ai_dev.yaml)
133-
is the actual config used to produce the case-study report — six roles
134-
covering a multi-domain ROS2 / CUDA / ML / depth-vision project. Use it as a
135-
template if your project has similar shape; otherwise let `suggest-roles`
136-
generate one tailored to your codebase.
223+
Every section a Claude Cowork report generator (or your own dashboard) needs to author a credible productivity narrative.
137224

138225
## Documentation
139226

140-
- [`docs/REPRODUCTION_GUIDE.md`](docs/REPRODUCTION_GUIDE.md) — full step-by-step
141-
walkthrough (legacy from the original notebook, covers manual notebook usage).
142-
- [`docs/config-reference.md`](docs/config-reference.md) — every YAML field documented.
227+
- [`docs/REPRODUCTION_GUIDE.md`](docs/REPRODUCTION_GUIDE.md) — full first-run walkthrough including troubleshooting.
228+
- [`docs/config-reference.md`](docs/config-reference.md) — every YAML field documented with type, default, and meaning.
143229

144230
## License
145231

0 commit comments

Comments
 (0)