Skip to content

Commit 8e1ce0a

Browse files
denn-gubskyclaude
andcommitted
Separate analyzer workspace from target repo
The analyzer was previously installing its bundled judge subagent inside the target repo's .claude/ and writing reports + cache there too. This mixed the analyzer's tooling with the target's project configuration and left behind artifacts (.ai-dev-effectiveness-cache/, effectiveness-report.html, the judge agent itself) in any repo it touched. New flow: mkdir ~/dev-effectiveness && cd ~/dev-effectiveness ai-dev-effectiveness init-judge # installs ./.claude/agents/effort-judge.md ai-dev-effectiveness analyze /some/target-repo --judge claude-cli # → reports + cache go to ./target-repo/ # → target repo is read-only Implementation: - analyze() takes new `workspace` and `out_dir` kwargs. Agent path and judge cache dir are resolved as absolute paths inside the workspace (overriding any YAML setting) so the target repo is guaranteed read-only. - AnalysisResult exposes `.workspace` and `.out_dir`; `.default_out_path(fmt)` returns the per-target output location. - Judge subprocess CWD remains the target (so Read/Grep/git show operate on target content) but the agent definition comes from the analyzer workspace via absolute --agent path. - New `ai_dev_effectiveness.ast_index` module: when `ast-index` is on PATH, runs `ast-index build` against the target before the judge starts. Best-effort; never fails the analysis. Disable with --no-ast-index or AI_DEV_EFFECTIVENESS_NO_AST_INDEX=1. - CLI: * `analyze TARGET` — TARGET defaults to '.', out goes to $PWD/<basename>/ when target != $PWD, else to $PWD (legacy single-repo mode). * `init-judge [WORKSPACE]` — WORKSPACE defaults to '.', help text explicitly says "NOT one of your project repos". * New flags: --workspace, --out-dir, --no-ast-index. - Tests: added test_workspace_target_separation (verifies nothing lands in target) and test_target_equals_workspace_keeps_legacy_layout. - README + case-study README rewritten around the new flow with a workspace-layout diagram. Verified end-to-end on the HockeyBot worktree: - Workspace at /tmp/ade-workspace/.claude/ - Output at /tmp/ade-workspace/musing-haslett-b9b799/ - Cache at /tmp/ade-workspace/musing-haslett-b9b799/.ai-dev-effectiveness-cache/ - Target /Users/denn/hockeybot_ws/.claude/worktrees/musing-haslett-b9b799/ contains zero analyzer artifacts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent f074b78 commit 8e1ce0a

7 files changed

Lines changed: 291 additions & 55 deletions

File tree

README.md

Lines changed: 43 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,19 @@
88

99
```bash
1010
pipx install git+https://github.com/denn-gubsky/ai-dev-effectiveness
11-
cd /path/to/your/repo
12-
ai-dev-effectiveness analyze .
13-
open effectiveness-report.html
11+
12+
# Pick a folder you'll run analyses from. NOT one of your project repos.
13+
mkdir -p ~/dev-effectiveness && cd ~/dev-effectiveness
14+
15+
# Once: install the bundled judge subagent into THIS workspace.
16+
ai-dev-effectiveness init-judge
17+
18+
# Now analyze any repo you want. Reports go to ./<target_basename>/.
19+
ai-dev-effectiveness analyze /path/to/your/repo
20+
open ./your-repo/effectiveness-report.html
1421
```
1522

16-
That's it. Zero config required for the first run — the tool auto-detects domains from your top-level dirs, classifies commits by language, and identifies AI co-authors from a built-in registry covering Claude (Opus/Sonnet/Haiku), GitHub Copilot, Cursor, Codex, Aider, Cody, Continue, Windsurf, Codeium, Tabnine, and Devin.
23+
The analyzer **never writes inside the target repo** — the bundled subagent, the judgment cache, and all reports live in your local workspace. Each target gets its own subfolder so you can compare repos side by side.
1724

1825
## What it measures
1926

@@ -28,23 +35,48 @@ When all three agree within 2x, the multiplier is defensible. When they diverge,
2835
## Optional: AI judge for diff-aware estimates
2936

3037
```bash
31-
ai-dev-effectiveness init-judge # installs the bundled subagent into .claude/
32-
ai-dev-effectiveness analyze . --judge claude-cli
38+
cd ~/dev-effectiveness # the workspace where init-judge ran
39+
ai-dev-effectiveness analyze /path/to/your/repo --judge claude-cli
40+
```
41+
42+
The judge uses your existing Claude Code CLI session — no API key, no metered billing, no diffs leaving your machine. If you have [ast-index](https://github.com/asgardiandevs/ast-index) on your `PATH`, the analyzer runs `ast-index build` against the target before the judge starts so the bundled subagent can use `mcp__ast-index__*` tools for symbol-level lookups (much faster than grep-then-read for "find callers of this function").
43+
44+
Other providers (opt-in): `--judge anthropic-api` (separate API key), `--judge openai`, `--judge ollama` (local model).
45+
46+
Disable ast-index for a run: `--no-ast-index`.
47+
48+
## Workspace layout
49+
50+
```
51+
~/dev-effectiveness/ # your analyzer workspace
52+
├── .claude/
53+
│ ├── agents/effort-judge.md # ← installed by `init-judge`
54+
│ ├── skills/effort-estimation/SKILL.md
55+
│ └── settings.recommended.json
56+
├── ai_dev.yaml # optional: custom config (init-config)
57+
├── repo-a/ # output for one analyzed target
58+
│ ├── effectiveness-report.html
59+
│ ├── effectiveness-report.json
60+
│ └── .ai-dev-effectiveness-cache/
61+
└── repo-b/ # another target's outputs
62+
└── ...
3363
```
3464

35-
The judge uses your existing Claude Code CLI session — no API key, no metered billing, no diffs leaving your machine. If you have [ast-index](https://github.com/defendend/Claude-ast-index-search) MCP configured, it's used for symbol-level lookups.
65+
Useful flags:
3666

37-
Other providers: `--judge anthropic-api` (separate API key), `--judge openai`, `--judge ollama` (local model).
67+
- `--workspace PATH` — override the analyzer workspace (default: `$PWD`).
68+
- `--out-dir PATH` — override where this run's reports go.
69+
- `--judge-dry-run` — show what the judge would do (sample size, cost, paths) without invoking it.
3870

3971
## Case study
4072

4173
[`examples/robotics-case-study/`](examples/robotics-case-study/) — anonymized analysis of a real 117K-LOC robotics project built by 1 dev + Claude over 10 months. Use it to see what a credible report looks like and to calibrate your own results.
4274

4375
## Customize
4476

45-
- `ai-dev-effectiveness init-config` — drops `ai_dev.yaml` into your repo
46-
- Define your specialist roles, effort constants, agent signatures
47-
- See [`docs/REPRODUCTION_GUIDE.md`](docs/REPRODUCTION_GUIDE.md) and [`docs/config-reference.md`](docs/config-reference.md)
77+
- `ai-dev-effectiveness init-config` — drops `ai_dev.yaml` into your workspace.
78+
- Define your specialist roles, effort constants, agent signatures.
79+
- See [`docs/REPRODUCTION_GUIDE.md`](docs/REPRODUCTION_GUIDE.md) and [`docs/config-reference.md`](docs/config-reference.md).
4880

4981
## License
5082

examples/robotics-case-study/README.md

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -74,10 +74,23 @@ The notebook walks through ten sections:
7474

7575
## Reproducing this on your own repo
7676

77-
1. `pipx install git+https://github.com/denn-gubsky/ai-dev-effectiveness`
78-
2. `cd /path/to/your/repo`
79-
3. `ai-dev-effectiveness init-config && $EDITOR ai_dev.yaml` — edit the
80-
`domains` and `roles` sections to match your project.
81-
4. `ai-dev-effectiveness analyze . --config ai_dev.yaml`
82-
5. Optionally: `ai-dev-effectiveness init-judge && ai-dev-effectiveness analyze . --config ai_dev.yaml --judge claude-cli`
83-
to add the AI judge as a third estimator.
77+
```bash
78+
pipx install git+https://github.com/denn-gubsky/ai-dev-effectiveness
79+
80+
# 1. Pick a workspace (NOT one of your project repos).
81+
mkdir -p ~/dev-effectiveness && cd ~/dev-effectiveness
82+
83+
# 2. One-time: install the bundled judge subagent into the workspace.
84+
ai-dev-effectiveness init-judge
85+
86+
# 3. Drop a config and edit the `domains` and `roles` sections to match
87+
# your project. (Use this case study's ai_dev.yaml as a starting point.)
88+
ai-dev-effectiveness init-config && $EDITOR ai_dev.yaml
89+
90+
# 4. Run the analysis. Reports land in ~/dev-effectiveness/<target_basename>/.
91+
ai-dev-effectiveness analyze /path/to/your/repo --config ai_dev.yaml \
92+
--judge claude-cli
93+
```
94+
95+
The analyzer is read-only with respect to your project — no `.claude/`, cache,
96+
or report files are ever created inside it.

src/ai_dev_effectiveness/__init__.py

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
from . import (
1919
agent_detector,
20+
ast_index,
2021
domain_classifier,
2122
effort_estimator,
2223
git_extractor,
@@ -46,6 +47,11 @@ class AnalysisResult:
4647
loc: pd.DataFrame
4748
metrics: _metrics.MetricsBundle
4849
figures: report_renderer.Figures
50+
workspace: Path = Path(".")
51+
out_dir: Path = Path(".")
52+
53+
def default_out_path(self, fmt: str) -> Path:
54+
return self.out_dir / f"effectiveness-report.{fmt}"
4955

5056
def to_html(self, path: str | Path | None = None) -> str:
5157
html = report_renderer.render_html(self.figures, self.config.project.name)
@@ -74,20 +80,38 @@ def to_json(self, path: str | Path | None = None) -> str:
7480
def analyze(
7581
repo: str | Path = ".",
7682
config: str | Path | Config | None = None,
83+
workspace: str | Path | None = None,
84+
out_dir: str | Path | None = None,
7785
) -> AnalysisResult:
7886
"""Run the full pipeline against `repo` and return an `AnalysisResult`.
7987
8088
Args:
81-
repo: path to a git repo. Walks up to find .git/.
89+
repo: path to the target git repo. Walks up to find .git/. Read-only.
8290
config: path to ai_dev.yaml, an already-loaded Config, or None for defaults.
91+
workspace: analyzer workspace where the bundled judge agent lives
92+
(`<workspace>/.claude/agents/effort-judge.md`). Defaults to cwd.
93+
out_dir: where to write the report and cache. Defaults to:
94+
- `cwd` if the resolved target is the same as cwd, OR
95+
- `cwd/<basename(target)>` if the target is elsewhere.
96+
Either way, NOTHING is ever written to the target repo.
8397
"""
8498
repo_path = _find_repo(Path(repo).resolve())
99+
workspace_path = Path(workspace).resolve() if workspace else Path.cwd().resolve()
100+
out_dir_path = _resolve_out_dir(out_dir, workspace_path, repo_path)
101+
out_dir_path.mkdir(parents=True, exist_ok=True)
85102

86103
if isinstance(config, Config):
87104
cfg = config
88105
else:
89106
cfg = _config.load(config) if config is not None else Config()
90107

108+
# Reroute judge artefacts to the analyzer workspace and the per-target
109+
# output dir, regardless of what the YAML config said. This guarantees
110+
# nothing lands inside the target repo.
111+
cfg.judge.cache_dir = str(out_dir_path / ".ai-dev-effectiveness-cache")
112+
cfg.judge.agent_path = str(workspace_path / ".claude/agents/effort-judge.md")
113+
cfg.judge.skill_path = str(workspace_path / ".claude/skills/effort-estimation/SKILL.md")
114+
91115
# 1. extract commits
92116
commits = git_extractor.extract_commits(repo_path)
93117
commits = git_extractor.add_week_columns(commits)
@@ -130,6 +154,11 @@ def analyze(
130154
# 7. optional AI judge — runs only if config.judge.enabled
131155
judge_summary_dict = None
132156
if cfg.judge.enabled:
157+
# Best-effort: build the ast-index for the target so the bundled
158+
# subagent can use mcp__ast-index__* tools for symbol-level lookups.
159+
ok, msg = ast_index.build(repo_path)
160+
print(f" ast-index: {msg}", flush=True)
161+
133162
judge_cfg = _judge.JudgeConfig(
134163
provider=cfg.judge.provider, model=cfg.judge.model,
135164
sample_size=cfg.judge.sample_size, judge_all=cfg.judge.judge_all,
@@ -163,6 +192,7 @@ def _progress(i, n, sha):
163192

164193
return AnalysisResult(
165194
config=cfg, commits=commits, loc=loc_df, metrics=bundle, figures=figures,
195+
workspace=workspace_path, out_dir=out_dir_path,
166196
)
167197

168198

@@ -175,6 +205,19 @@ def _find_repo(start: Path) -> Path:
175205
raise RuntimeError(f"No git repository found at or above {start}")
176206

177207

208+
def _resolve_out_dir(out_dir: str | Path | None, workspace: Path, target: Path) -> Path:
209+
"""Decide where to write reports and cache.
210+
211+
Default rule: workspace itself if target == workspace; else workspace/<basename(target)>.
212+
Explicit `out_dir` always wins.
213+
"""
214+
if out_dir is not None:
215+
return Path(out_dir).resolve()
216+
if target == workspace:
217+
return workspace
218+
return workspace / target.name
219+
220+
178221
def _project_months(commits: pd.DataFrame) -> float | None:
179222
if commits.empty:
180223
return None
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
"""Optional integration with ast-index (https://github.com/asgardiandevs/ast-index).
2+
3+
When `ast-index` is on PATH, we run `ast-index build` against the target repo
4+
before invoking the judge. The bundled effort-judge subagent's tool allowlist
5+
includes `mcp__ast-index__*`, so the agent can do symbol-level lookups during
6+
its judgment if the user has the ast-index MCP server configured.
7+
8+
This module is best-effort: every function returns gracefully when ast-index
9+
isn't installed or the build fails. The analysis pipeline never depends on it.
10+
"""
11+
from __future__ import annotations
12+
13+
import shutil
14+
import subprocess
15+
from pathlib import Path
16+
17+
18+
def is_installed() -> bool:
19+
"""True iff `ast-index` is on PATH."""
20+
return shutil.which("ast-index") is not None
21+
22+
23+
def build(target: Path, timeout_sec: int = 120) -> tuple[bool, str]:
24+
"""Run `ast-index build` in the target repo.
25+
26+
Args:
27+
target: path to the repo to index. ast-index walks it itself.
28+
timeout_sec: kill the build after this many seconds.
29+
30+
Returns:
31+
(success, message) — `success` is False when ast-index isn't installed,
32+
the build fails, or it times out. `message` is a short human-readable
33+
status suitable for printing to the user.
34+
"""
35+
import os
36+
if os.environ.get("AI_DEV_EFFECTIVENESS_NO_AST_INDEX") == "1":
37+
return (False, "ast-index disabled via AI_DEV_EFFECTIVENESS_NO_AST_INDEX=1.")
38+
if not is_installed():
39+
return (False, "ast-index not installed; skipping (install with `pipx install ast-index`).")
40+
41+
try:
42+
proc = subprocess.run(
43+
["ast-index", "build"],
44+
cwd=str(target),
45+
capture_output=True, text=True,
46+
timeout=timeout_sec, check=False,
47+
)
48+
except subprocess.TimeoutExpired:
49+
return (False, f"ast-index build timed out after {timeout_sec}s.")
50+
except OSError as e:
51+
return (False, f"ast-index build failed to start: {e}")
52+
53+
if proc.returncode == 0:
54+
return (True, "ast-index build OK")
55+
return (False, f"ast-index build exited {proc.returncode}: {proc.stderr.strip()[:200]}")

0 commit comments

Comments
 (0)