fix(dagster-dbt): handle dbt no-op and reused statuses as successful outcomes#33934
fix(dagster-dbt): handle dbt no-op and reused statuses as successful outcomes#33934cschanhniem wants to merge 1 commit into
Conversation
…outcomes In dbt-core v1.10+, a node can finish with status "no-op" when its upstream hasn't changed (cached/up-to-date). In v1.12+, "reused" is returned when results are reused from a previous run. Both are valid non-error outcomes. Previously, dagster-dbt only treated NodeStatus.Success as a successful materialization, silently dropping no-op and reused results. This caused assets to never produce outputs even though dbt ran successfully. This change defines a success status set that includes "no-op" and "reused" alongside "success", so refable nodes in these states yield proper materialization events. Fixes: dagster-io#33920 Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
Greptile SummaryThis PR fixes the dbt Cloud v2 run handler so that dbt nodes completing with
Confidence Score: 3/5The cloud v2 materialization fix is correct, but the non-cloud CLI path retains the same silent-drop behavior for no-op/reused nodes. The cloud v2 change itself is sound — result_status is a raw string from JSON and the frozenset comparison works correctly for all three values. However, dbt_cli_event.py still gates materialization on NodeStatus.Success alone, so CLI-based runs on dbt-core v1.10+ will continue to silently drop no-op/reused results. If the issue being fixed affects both execution paths, the fix is incomplete. python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_event.py — _is_node_successful_execution_event at line 275 still uses NodeStatus.Success only.
|
| Filename | Overview |
|---|---|
| python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/run_handler.py | Adds _DBT_SUCCESS_STATUSES frozenset and changes the materialization guard from NodeStatus.Success enum comparison to a string-set membership check, correctly handling no-op and reused statuses; leaves NodeStatus as an unused import. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[dbt Cloud run completes] --> B[run_results.json fetched]
B --> C{For each result node}
C --> D{resource_type in REFABLE_NODE_TYPES?}
D -- No --> E{resource_type == Test?}
D -- Yes --> F{result_status in _DBT_SUCCESS_STATUSES?}
F -- success/no-op/reused --> G{not ephemeral?}
F -- other --> H[Skipped - no event]
G -- Yes --> I{has_asset_def?}
G -- No --> H
I -- Yes --> J[yield Output]
I -- No --> K[yield AssetMaterialization]
E -- Yes --> L[yield AssetCheckResult or AssetCheckEvaluation]
E -- No --> H
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[dbt Cloud run completes] --> B[run_results.json fetched]
B --> C{For each result node}
C --> D{resource_type in REFABLE_NODE_TYPES?}
D -- No --> E{resource_type == Test?}
D -- Yes --> F{result_status in _DBT_SUCCESS_STATUSES?}
F -- success/no-op/reused --> G{not ephemeral?}
F -- other --> H[Skipped - no event]
G -- Yes --> I{has_asset_def?}
G -- No --> H
I -- Yes --> J[yield Output]
I -- No --> K[yield AssetMaterialization]
E -- Yes --> L[yield AssetCheckResult or AssetCheckEvaluation]
E -- No --> H
Comments Outside Diff (1)
-
python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/run_handler.py, line 23 (link)NodeStatusis no longer referenced in this file after the comparison was changed fromresult_status == NodeStatus.Successtoresult_status in _DBT_SUCCESS_STATUSES. It should be removed from the import to avoid lint errors and confusion.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Reviews (1): Last reviewed commit: "fix(dagster-dbt): handle dbt no-op and r..." | Re-trigger Greptile
Summary
In dbt-core v1.10+, a node can finish with status
no-opwhen its upstream hasn't changed (cached/up-to-date). In v1.12+,reusedis returned when results are reused from a previous run. Both are valid non-error outcomes defined indbt/artifacts/schemas/results.py.Previously, dagster-dbt's
to_default_asset_eventsonly treatedNodeStatus.Successas a successful materialization, silently dropping results withno-oporreusedstatus. This caused refable assets to never produceAssetMaterializationorOutputevents even though dbt ran successfully.Changes
_DBT_SUCCESS_STATUSESfrozenset with{"success", "no-op", "reused"}result_status == NodeStatus.Successtoresult_status in _DBT_SUCCESS_STATUSESSuccess, v1.10+ addsNoOp, v1.12+ addsReused)Fixes #33920
Test plan
no-oporreusedstatus now correctly yield materialization events instead of being silently dropped