Support DBT_ENGINE_ prefixed env variables

[yann1cks]

Summary & Motivation

This PR adds support for DBT_ENGINE_ prefixed env variables. These are the default since dbt version 1.11.

See the current dbt docs. It also describes that the old variables are still supported for backwards compatibility. Because of that it makes sense to just set both kinds of env variables in dagster to avoid any influence or variables already set.

Test Plan

Changelog

[greptile-apps]

Greptile Summary

This PR adds support for the DBT_ENGINE_-prefixed global config environment variables introduced as the default in dbt 1.11, ensuring Dagster-controlled dbt settings are respected regardless of which prefix convention the dbt version honours.

• A new _get_dbt_env_var helper in dbt_env.py reads DBT_ENGINE_<SUFFIX> first and falls back to the legacy DBT_<SUFFIX> name, correctly using an is not None guard so empty-string engine overrides are honoured; DBT_CLOUD_* variables are excluded from the transform.
• In resource.py, the env dict construction is refactored to collect all Dagster-owned settings into dbt_global_config_env and then mirror every entry with its DBT_ENGINE_* counterpart, ensuring Dagster's values always win over any user-set DBT_ENGINE_* variables already in the process environment.
• Six new tests cover the key edge cases: precedence, empty-string overrides, legacy fallback, cloud-var passthrough, indirect selection, and the full env-dict construction with conflicting user vars.

Confidence Score: 5/5

The change is well-scoped and backwards-compatible; the is not None guard in _get_dbt_env_var correctly handles empty-string engine overrides, and the env-dict ordering in resource.py ensures Dagster's required settings always override any user-supplied DBT_ENGINE_* values.

The core logic in _get_dbt_env_var and the env-dict construction is correct, and the six new tests cover the important edge cases. The only outstanding item is the missing guard in the comprehension, which is not a current bug given the fixed set of keys in dbt_global_config_env.

The DBT_ENGINE_* mirroring comprehension in resource.py (lines 690-693) would benefit from a defensive guard consistent with _get_dbt_env_var, but is not broken today.

Important Files Changed

Filename	Overview
python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_env.py	New helper _get_dbt_env_var checks DBT_ENGINE_* before DBT_*, correctly using is not None to honour empty-string overrides; DBT_CLOUD_* passthrough is correct.
python_modules/libraries/dagster-dbt/dagster_dbt/core/resource.py	Splits controlled env vars into dbt_global_config_env and mirrors each with DBT_ENGINE_*; comprehension has no guard for non-DBT_-prefixed or DBT_CLOUD_* keys, inconsistent with _get_dbt_env_var.
python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_invocation.py	Single call-site change: _get_dbt_target_path now routes through _get_dbt_env_var, honouring DBT_ENGINE_TARGET_PATH when set.
python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py	Single call-site change: indirect_selection read now respects DBT_ENGINE_INDIRECT_SELECTION precedence via _get_dbt_env_var.
python_modules/libraries/dagster-dbt/dagster_dbt_tests/core/test_resource.py	Six new tests cover precedence, empty-string override, legacy fallback, cloud-var passthrough, indirect-selection, and full env-dict construction with conflicting vars.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Call _get_dbt_env_var key, default] --> B{key starts with DBT_\nAND NOT DBT_CLOUD_?}
    B -- No --> C[os.getenv key, default]
    B -- Yes --> D[engine_value = os.getenv DBT_ENGINE_ + suffix]
    D --> E{engine_value is not None?}
    E -- Yes --> F[return engine_value\neven if empty string]
    E -- No --> G[return os.getenv key, default]

    H[DbtCliResource.cli] --> I[_get_unique_target_path\nvia _get_dbt_env_var]
    I --> J[Build dbt_global_config_env\nDBT_* keys with Dagster values]
    J --> K[Build final env dict]
    K --> L[Spread os.environ]
    L --> M[Override with dbt_global_config_env]
    M --> N[Override with DBT_ENGINE_* mirror comprehension]
    N --> O[Pass env to DbtCliInvocation.run]

Loading

_{Reviews (2): Last reviewed commit: "fix empty string issue" | Re-trigger Greptile}