[dagster-dbt] Surface isolated dbt-fusion models in asset selection#33938
[dagster-dbt] Surface isolated dbt-fusion models in asset selection#33938Vamsi-klu wants to merge 2 commits into
Conversation
dbt-fusion manifests omit nodes with no parents or children from child_map, so a model with no source()/ref() calls was silently dropped from the asset graph. Add missing selectable node ids to the selection graph before running NodeSelector; this is a no-op for dbt-core manifests. Adds a regression test. closes dagster-io#33801
Greptile SummaryThis PR fixes a silent asset-drop in
Confidence Score: 5/5Safe to merge — the change is additive and explicitly a no-op for well-formed dbt-core manifests. The graph-backfill step only adds nodes; it never removes edges or rewrites existing graph data. For dbt-core manifests, No files require special attention.
|
| Filename | Overview |
|---|---|
| python_modules/libraries/dagster-dbt/dagster_dbt/utils.py | Adds isolated-node backfill to the DiGraph before passing it to graph_selector.Graph; well-commented, no-op for dbt-core manifests, and covers all known selectable manifest collections. |
| python_modules/libraries/dagster-dbt/dagster_dbt_tests/core/test_asset_selection.py | Two new parameterized tests cover the core regression (isolated model absent from child_map) and a mixed graph with both connected and isolated nodes; helper builders keep the fixtures readable. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[manifest_json] --> B[Build DiGraph from child_map]
B --> C{Any selectable unique_ids\nmissing from graph?}
C -- Yes\ndbt-fusion isolated nodes --> D[add_nodes_from missing ids]
C -- No\nnormal dbt-core manifest --> E[No-op]
D --> F[graph_selector.Graph]
E --> F
F --> G[NodeSelector]
A --> H[Manifest object\nnodes / sources / metrics / etc.]
H --> G
G --> I[Apply select / exclude / selector string]
I --> J[Return matched unique_ids]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[manifest_json] --> B[Build DiGraph from child_map]
B --> C{Any selectable unique_ids\nmissing from graph?}
C -- Yes\ndbt-fusion isolated nodes --> D[add_nodes_from missing ids]
C -- No\nnormal dbt-core manifest --> E[No-op]
D --> F[graph_selector.Graph]
E --> F
F --> G[NodeSelector]
A --> H[Manifest object\nnodes / sources / metrics / etc.]
H --> G
G --> I[Apply select / exclude / selector string]
I --> J[Return matched unique_ids]
Reviews (2): Last reviewed commit: "[dagster-dbt] Expand dbt fusion isolated..." | Re-trigger Greptile
|
Just marked this ready for review. It's a small dagster-dbt change that surfaces isolated dbt-fusion models in asset selection, with a focused test added. @smackesey when you get a chance, could you take a look or route it to the right dbt owner? CI is gated for fork PRs, so it'll need a maintainer to unblock it. cc @cmpadden for triage. Thanks! |
fb9a6ee to
9f7f404
Compare
Summary
DbtProjectComponent(and any selection that goes throughselect_unique_ids) silently drops dbt models that contain nosource()/ref()calls when the project is compiled with dbt-fusion (manifest schema v2+). The model is present andenabledinmanifest["nodes"], yet it never appears as a Dagster asset and no error is raised.Root cause
_select_unique_ids_from_manifestindagster_dbt/utils.pybuilds the selection graph solely frommanifest["child_map"]:dbt-core keys
child_mapby every node, but dbt-fusion omits any node that has neither parents nor children. A model with nosource()/ref()calls (and nothing referencing it) is therefore absent from theDiGraph.NodeSelectorintersects matched nodes with the graph's node set, so a node that is missing from the graph can never be selected — it is silently dropped.Fix
After constructing the graph from
child_map, add any selectableunique_ids from the manifest that are missing from the graph. This is a no-op for well-formed dbt-core manifests (wherechild_mapalready contains every node) and only repairs fusion manifests.Testing
Added
test_select_unique_ids_includes_isolated_fusion_models, which builds a manifest where an isolated model is absent fromchild_map(the exact dbt-fusion shape from the issue) and asserts the model is still selected. Verified the test fails without the fix and passes with it.ruffclean.closes: #33801
Was generative AI tooling used to author this PR?
Yes — Claude Code (Opus 4.8). Generated-by: Claude Code (Opus 4.8)