Skip to content

Add graph-job smoke tests for missing-ecosystem + coexists scenarios#523

Open
Nishnha wants to merge 6 commits into
mainfrom
nishnha/uv-graph-non-default-branch
Open

Add graph-job smoke tests for missing-ecosystem + coexists scenarios#523
Nishnha wants to merge 6 commits into
mainfrom
nishnha/uv-graph-non-default-branch

Conversation

@Nishnha

@Nishnha Nishnha commented Jun 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Add four smoke tests covering the two graph-job scenarios the base-ref graph dispatch in dependency-snapshots-api relies on. Both come straight from the three demo PRs in dsp-testing/uv-dependency-grapher:

Scenario Demo PR Smoke tests
Missing ecosystem on the base ref. PR introduces a new ecosystem versus its base branch → base graph job has no manifests for that ecosystem → CLI should return cleanly empty. #2 (head adds uv to an npm-only base), #3 (head adds npm to a uv-only base) smoke-uv-graph-missing-ecosystem.yaml, smoke-npm-graph-missing-ecosystem.yaml
Multiple ecosystems coexisting in the same directory. Head ref has both ecosystems' files at the same path → graph job for ecosystem X must scope its submission to X only, ignoring Y's manifests. #2 (head side) smoke-uv-graph-coexists.yaml, smoke-npm-graph-coexists.yaml

Test shapes

Missing-ecosystem tests lock in the CLI's clean "ecosystem absent" path: manifests: {} with status: skipped + reason: missing manifest files. Downstream consumers (the base-ref graph dispatcher) can rely on this combination to distinguish "ecosystem genuinely absent on this ref" from "ecosystem present but parsing failed".

Coexists tests lock in the per-ecosystem scoping: graph for uv on a dir with both uv.lock + package-lock.json returns only the uv submission (and vice versa for npm). Catches any regression where file discovery becomes greedy across ecosystems.

Fixtures

  • /npm - npm only (package.json + package-lock.json)
  • /uv - uv only (pyproject.toml + uv.lock)
  • /mixed-graph - new fixture, byte-for-byte copy of /npm + /uv side-by-side. Used only by the coexists tests so the missing-ecosystem tests can keep their single-ecosystem fixtures unchanged.

How this PR evolved

  1. Original version used dsp-testing/uv-dependency-grapher as an external fixture repo - failed in CI with job_repo_not_found because the smoke-tests runner can't clone external orgs.
  2. Initially called the multi-ecosystem cases *-multi-ecosystem.yaml - "multi-ecosystem" overloads a Dependabot-specific term (the multi-ecosystem updater feature). Renamed to coexists to focus on the per-call scoping behavior.
  3. Initially also included smoke-uv-graph-non-default-branch.yaml - dropped because the dependabot CLI is branch-agnostic. It clones by SHA and only echoes the branch name as metadata. The base-ref dispatch logic that actually depends on non-default-branch behavior lives in dependency-snapshots-api and is tested at that layer.

Regen technique

dependabot test -f tests/smoke-<name>.yaml --local . -o tests/smoke-<name>.yaml --timeout 10m

Then swap the locally-synthesized SHA for the real commit SHA where the fixture dir exists. --local bypasses the no-write-token check that blocks gh auth token from being used with script/regen.sh.

Gotcha I hit: local dependabot CLI 1.83.1 + cached docker images produced dependabot-core 0.371.0 output, while CI runs 0.381.0. The newer parser walks transitive deps and populates per-package dependencies: lists from the lockfile; the older one leaves them empty. Fix: brew upgrade dependabot + docker pull ghcr.io/dependabot/dependabot-updater-{npm,core}:latest.

Verification

All four tests pass in CI:

  • smoke (uv-graph-missing-ecosystem)
  • smoke (npm-graph-missing-ecosystem)
  • smoke (uv-graph-coexists)
  • smoke (npm-graph-coexists)

Re-ran each locally twice and diffed - deterministic at dependabot-core 0.381.0.

@Nishnha
Add smoke tests for graph jobs on non-default branches
aef5137 
Locks in graph command behavior for the scenarios the
dependency-snapshots-api base-ref graph job feature relies on
(github/dependency-snapshots-api#1611):

- smoke-uv-graph-non-default-branch.yaml: graphs uv on `release-v1`
  branch in dsp-testing/uv-dependency-grapher. Mirrors the base-ref code
  path's use case where snapshots-api dispatches a graph job pinned to a
  non-default branch.
- smoke-uv-graph-multi-ecosystem.yaml: graphs uv on a branch that also
  has package.json/package-lock.json. Covers "PR introduces a new
  ecosystem" where head has more ecosystems than base - the uv graph
  job must still find and submit only the uv manifest.
- smoke-npm-graph-multi-ecosystem.yaml: graphs npm on the same branch.
  Companion to the above - proves the other ecosystem's graph job
  submits its own manifest independently.

Fixtures live in dsp-testing/uv-dependency-grapher branches `release-v1`
and `smoke/multi-ecosystem` (mirrors what we used to demo the base-ref
feature on PR #1 there).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Nishnha Nishnha requested a review from a team as a code owner June 8, 2026 14:58
Nishnha and others added 2 commits June 10, 2026 13:57
@Nishnha
Replace multi-ecosystem tests with missing-ecosystem variant
e170435 
The previous "multi-ecosystem" tests ran a graph job on a branch
containing both uv and npm files and asserted that the response only
contained the requested ecosystem's manifest. That's an isolation
sanity check, not what we set out to test - and "multi-ecosystem"
overloads a term that means something specific in Dependabot
(the multi-ecosystem updater feature).

The scenario we actually want to guard against (and that base-ref
graph dispatch in github/dependency-snapshots-api depends on): asking
the CLI to graph an ecosystem that doesn't exist on the target branch
at all. That happens whenever a PR introduces a brand-new ecosystem
versus its base branch, where we kick off a graph job for the new
ecosystem against the base.

Replaces the two multi-ecosystem tests with:

- smoke-uv-graph-missing-ecosystem.yaml: graph uv on the npm-only
  release-npm-v1 branch
- smoke-npm-graph-missing-ecosystem.yaml: graph npm_and_yarn on the
  uv-only release-v1 branch

Both expect `manifests: {}` with `status: skipped` and
`reason: missing manifest files`, which is the CLI's clean
no-manifests-found path. Regenerated via:

    dependabot test -f tests/smoke-<name>.yaml \
        --local ../uv-dependency-grapher \
        -o tests/smoke-<name>.yaml --timeout 10m

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Nishnha
Re-target missing-ecosystem tests at in-repo fixtures, drop non-defau…
9c605f9 
…lt-branch

The previous iteration pointed at dsp-testing/uv-dependency-grapher, which
the smoke-tests CI runner can't clone (job_repo_not_found). Convention
across other graph tests in this repo (see PR #524 for python examples):
fixtures live in the smoke-tests repo itself, in per-ecosystem directories
referenced via 'directories:'.

Rewrote the missing-ecosystem tests against the existing /npm and /uv
fixture directories:

- smoke-uv-graph-missing-ecosystem.yaml: graph 'uv' against /npm
- smoke-npm-graph-missing-ecosystem.yaml: graph 'npm_and_yarn' against /uv

Both expect manifests: {} with status: skipped and reason: missing manifest
files - the CLI's clean no-manifests-found path. Regenerated via:

    dependabot test -f tests/smoke-<name>.yaml --local . \
        -o tests/smoke-<name>.yaml --timeout 10m

then swapped the synthesized local SHA for the actual main HEAD SHA where
both fixture dirs exist.

Also dropped smoke-uv-graph-non-default-branch.yaml. The dependabot CLI is
branch-agnostic - it clones by SHA and only echoes the branch name as
metadata. A smoke test for 'graph on a non-default branch' would just be a
test of 'graph on any commit', which the existing smoke-uv-graph.yaml
already covers. The base-ref dispatch logic that actually depends on
non-default-branch behavior lives in github/dependency-snapshots-api and
is tested at that layer.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Nishnha Nishnha changed the title Add smoke tests for graph jobs on non-default branches Add missing-ecosystem smoke tests for graph jobs Jun 10, 2026
Nishnha and others added 3 commits June 10, 2026 14:25
@Nishnha
Add mixed-graph fixture: pure copy of /npm + /uv side-by-side
4f1ba38 
Used by smoke-uv-graph-coexists.yaml and smoke-npm-graph-coexists.yaml
to verify that a graph job for one ecosystem ignores manifest files
belonging to other ecosystems in the same directory.

Files are byte-for-byte copies of npm/{package.json,package-lock.json}
and uv/{pyproject.toml,uv.lock} so the expected dependency tree is
identical to the single-ecosystem tests - only the path changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Nishnha
Add coexists graph tests: scope to one ecosystem when others share th…
a285d22 
…e dir

When a directory contains manifests for multiple ecosystems (e.g. a
codebase with both Python and JavaScript code under the same path),
each graph job invocation should produce a submission containing ONLY
the manifests matching its 'package-manager:' input. This is the
scenario from dsp-testing/uv-dependency-grapher#2 (head graph job runs
on a branch that has both uv and npm files; we expect uv graph to
return only uv.lock and npm graph to return only package-lock.json).

Two new tests over the new mixed-graph fixture (which is /npm + /uv
copied side-by-side):

- smoke-uv-graph-coexists.yaml: graph 'uv' on /mixed-graph
  → submission contains /mixed-graph/uv.lock only
- smoke-npm-graph-coexists.yaml: graph 'npm_and_yarn' on /mixed-graph
  → submission contains /mixed-graph/package-lock.json only

This is the head-side complement to the *-missing-ecosystem tests
(base-side coverage in the same scenario - graph for absent ecosystem
returns 'manifests: {}' with status: skipped).

Renamed from the original 'multi-ecosystem' framing because that name
overloads a Dependabot-specific term (the multi-ecosystem updater
feature). 'coexists' more precisely describes what the test exercises:
graph job correctly scopes its output when multiple ecosystems coexist
in the same directory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Nishnha
Regen npm coexists/missing tests against dependabot-core 0.381.0
b9f3ad2 
CI was failing on smoke-npm-graph-coexists because my local generated
the file against dependabot-core 0.371.0 (older brew install + stale
docker image) while CI runs 0.381.0. The 0.381.0 npm parser walks
transitive dependencies and populates each package's 'dependencies:'
list from the lockfile; 0.371.0 left them empty.

Re-regenned both npm graph tests after pulling the latest
ghcr.io/dependabot/dependabot-updater-npm + dependabot-updater-core
images and upgrading the brew dependabot CLI to 1.88.0.

Verified locally: dependabot test -f <file> --local . diff produces
empty (deterministic). The uv graph tests already matched 0.381.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Nishnha Nishnha changed the title Add missing-ecosystem smoke tests for graph jobs Add graph-job smoke tests for missing-ecosystem + coexists scenarios Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Nishnha