From 60a41a9ac0688a126e1241a94bcd089559c904f7 Mon Sep 17 00:00:00 2001 From: Vamsi-klu Date: Sat, 20 Jun 2026 00:10:54 +0000 Subject: [PATCH] docs: replace TODO placeholders with real content and fix internal links Salvaged the conflict-free docs subset from #33263 onto current master as a focused, reviewable docs-only change. Replaces TODO/placeholder markers with valid prose, wires up PyObject API references, and corrects several internal links across the build/test guides and the GCP/Snowflake integration docs. --- .../rbac/user-roles-permissions.md | 10 +++--- .../dagster-plus/hybrid/kubernetes/setup.md | 3 +- .../etl-pipeline/6-create-a-sensor.md | 4 +-- .../sensors/testing-run-status-sensors.md | 2 +- .../assets/asset-versioning-and-caching.md | 2 +- .../troubleshooting-components.md | 15 ++++++++ .../partitioning-assets.md | 3 +- .../workspaces/migrating-workspace-yaml.md | 35 +++++++++++++++++-- .../test/running-a-subset-of-asset-checks.md | 2 +- .../testing-partitioned-config-and-jobs.md | 3 +- .../libraries/gcp/bigquery/reference.md | 4 +-- .../bigquery/using-bigquery-with-dagster.md | 3 +- .../libraries/snowflake/reference.md | 7 ++-- .../libraries/omni/dagster-omni.rst | 2 +- 14 files changed, 70 insertions(+), 25 deletions(-) diff --git a/docs/docs/deployment/dagster-plus/authentication-and-access-control/rbac/user-roles-permissions.md b/docs/docs/deployment/dagster-plus/authentication-and-access-control/rbac/user-roles-permissions.md index cd8530db729ff..800800f67eea5 100644 --- a/docs/docs/deployment/dagster-plus/authentication-and-access-control/rbac/user-roles-permissions.md +++ b/docs/docs/deployment/dagster-plus/authentication-and-access-control/rbac/user-roles-permissions.md @@ -99,7 +99,7 @@ To override a code location role for an individual user: 2. Click **Edit**. 3. Click the toggle to the left of the deployment to open a list of code locations. 4. Next to a code location, click **Edit user role**. -5. Select the user role for the code location: {/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/code-location-override.png" */} +5. Select the user role for the code location. 6. Click **Save**. #### Team members @@ -118,13 +118,11 @@ The above also applies to code locations and Branch Deployment roles. #### Viewing overrides -To view deployment-level overrides for a specific user, locate the user on the **Users** page and hover over a deployment: - {/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/user-overrides-popup.png" */} - -If there are code location-level overrides, a small **N override(s)** link will display beneath the user's deployment role. Hover over it to display the list of overrides: - {/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/code-location-override-popup.png" */} +To view deployment-level overrides for a specific user, locate the user on the **Users** page and hover over a deployment. A tooltip will display any overrides that apply at the deployment level. + +If there are code location-level overrides, a small **N override(s)** link will display beneath the user's deployment role. Hover over it to display the list of overrides. #### Removing overrides diff --git a/docs/docs/deployment/dagster-plus/hybrid/kubernetes/setup.md b/docs/docs/deployment/dagster-plus/hybrid/kubernetes/setup.md index b82222dbbe1fa..9c820d4edfcc6 100644 --- a/docs/docs/deployment/dagster-plus/hybrid/kubernetes/setup.md +++ b/docs/docs/deployment/dagster-plus/hybrid/kubernetes/setup.md @@ -84,9 +84,8 @@ helm --namespace dagster-cloud upgrade agent \ ## Troubleshooting -You can see basic health information about your agent in the Dagster+ UI: - [comment]: <> (TODO: Screenshot of Dagster+ Deployments agents tab) +You can see basic health information about your agent in the Dagster+ UI. ### View logs diff --git a/docs/docs/examples/full-pipelines/etl-pipeline/6-create-a-sensor.md b/docs/docs/examples/full-pipelines/etl-pipeline/6-create-a-sensor.md index 0bf01cdd72e29..b53bb5b1c8184 100644 --- a/docs/docs/examples/full-pipelines/etl-pipeline/6-create-a-sensor.md +++ b/docs/docs/examples/full-pipelines/etl-pipeline/6-create-a-sensor.md @@ -61,8 +61,8 @@ Now copy the following sensor code in the `sensors.py` file: 2. Navigate to the Automation page. 3. Turn on the `adhoc_request_sensor`. 4. Click on the `adhoc_request_sensor` details. - {/* TODO: Screenshot */} + You should see the sensor details view with recent ticks and the run history panel. 5. Create a `data/requests` directory in `dagster_tutorial`. Then include a `request.json` file: @@ -76,8 +76,8 @@ Now copy the following sensor code in the `sensors.py` file: ``` 6. Click on the green tick to see the run for this request. - {/* TODO: Screenshot */} + The run should appear in the runs list, and the tick details will show the run ID. ## Summary diff --git a/docs/docs/guides/automate/sensors/testing-run-status-sensors.md b/docs/docs/guides/automate/sensors/testing-run-status-sensors.md index 86e2420d39e07..8e9a94b6fcd28 100644 --- a/docs/docs/guides/automate/sensors/testing-run-status-sensors.md +++ b/docs/docs/guides/automate/sensors/testing-run-status-sensors.md @@ -34,7 +34,7 @@ Then we can execute this job and pull the attributes we need to build the `conte /> {/* TODO the methods and statuses below do not exist in API docs */} -{/* We have provided convenience functions and for retrieving `DagsterRunStatus.SUCCESS` and `DagsterRunStatus.FAILURE` events, respectively. If you have a run status sensor triggered on another status, you can retrieve all events from `result` and filter based on your event type. */} +To inspect the resulting status in tests, use the returned by `execute_in_process`. Its `dagster_run` and `all_events` properties let you filter for the status or event type you care about. We can use the same pattern to build the context for . If we wanted to test this run failure sensor: diff --git a/docs/docs/guides/build/assets/asset-versioning-and-caching.md b/docs/docs/guides/build/assets/asset-versioning-and-caching.md index 1ee3b80aa98fd..83fd26b24fcd1 100644 --- a/docs/docs/guides/build/assets/asset-versioning-and-caching.md +++ b/docs/docs/guides/build/assets/asset-versioning-and-caching.md @@ -18,8 +18,8 @@ Dagster's versioning system helps you determine ahead of time whether materializ Dagster has two versioning concepts to represent the code and input data used for each materialization: - **Code version.** A string that represents the version of the code that computes an asset. This is the `code_version` argument of . -- **Data version.** A string that represents the version of the data represented by the asset. This is represented as a `DataVersion` object. {/* TODO link `DataVersion` to API docs */} +- **Data version.** A string that represents the version of the data represented by the asset. This is represented as a object. By keeping track of code and data versions, Dagster can predict whether a materialization will change the underlying value. This allows Dagster to skip redundant materializations and instead return the previously computed value. In more technical terms, Dagster offers a limited form of [memoization](https://en.wikipedia.org/wiki/Memoization) for assets: the last-computed asset value is always cached. diff --git a/docs/docs/guides/build/components/building-pipelines-with-components/troubleshooting-components.md b/docs/docs/guides/build/components/building-pipelines-with-components/troubleshooting-components.md index 5f5aee80f2ce4..4d8f51a1ed2b7 100644 --- a/docs/docs/guides/build/components/building-pipelines-with-components/troubleshooting-components.md +++ b/docs/docs/guides/build/components/building-pipelines-with-components/troubleshooting-components.md @@ -6,3 +6,18 @@ unlisted: true --- TODO +This page lists common checks when a component fails to load or behaves unexpectedly. + +## Quick checks + +- **Validate the component type string**. Ensure the type name matches the registered component class and that the package providing it is installed in your environment. +- **Confirm configuration fields**. Component configuration is validated on load. If a field is missing or mis-typed, the error message will include the field path to update. +- **Verify workspace paths**. If you are using a workspace, confirm that `dg.toml` points at the correct project directories and that the project loads on its own. + +## Inspecting load errors + +If a component fails to load, the code location will show a load error in the UI and in CLI output. Use the stack trace there to locate the file and configuration entry that caused the failure. + +## Next steps + +For a refresher on how components are structured, see the [components overview](/guides/build/components). diff --git a/docs/docs/guides/build/partitions-and-backfills/partitioning-assets.md b/docs/docs/guides/build/partitions-and-backfills/partitioning-assets.md index 2c02f2a64f7f0..c32739b1dc25f 100644 --- a/docs/docs/guides/build/partitions-and-backfills/partitioning-assets.md +++ b/docs/docs/guides/build/partitions-and-backfills/partitioning-assets.md @@ -36,6 +36,7 @@ Sometimes you have a set of predefined categories for your data. For instance, y {/* TODO: Link to Backfill page to explain how to backfill regional sales data */} +To backfill historical partitions for regional sales data, see [Backfilling data](/guides/build/partitions-and-backfills/backfilling-data). ## Two-dimensional partitions \{#two-dimensional-partitions} @@ -97,4 +98,4 @@ After a partition has been successfully materialized, it will display as green i To view materializations by partition for a specific asset, navigate to the **Activity** tab of the asset's **Details** page: -![Asset activity section of asset details page](/images/guides/build/partitions-and-backfills/materialized-partitioned-asset-activity.png) \ No newline at end of file +![Asset activity section of asset details page](/images/guides/build/partitions-and-backfills/materialized-partitioned-asset-activity.png) diff --git a/docs/docs/guides/build/projects/workspaces/migrating-workspace-yaml.md b/docs/docs/guides/build/projects/workspaces/migrating-workspace-yaml.md index ac2e8a1f5bef4..2bf92823e125d 100644 --- a/docs/docs/guides/build/projects/workspaces/migrating-workspace-yaml.md +++ b/docs/docs/guides/build/projects/workspaces/migrating-workspace-yaml.md @@ -1,8 +1,39 @@ --- title: Migrating from workspace.yaml to dg.toml sidebar_position: 2000 -description: TODO +# description: TODO +description: Migrate legacy workspace.yaml configurations to a dg.toml workspace. unlisted: true --- -TODO \ No newline at end of file +TODO + +If you have an existing `workspace.yaml` file, you can migrate to a `dg.toml`-based workspace to align with the current Dagster OSS project and workspace structure. + +## Overview + +`workspace.yaml` directly lists code locations. A `dg.toml` workspace points to one or more Dagster projects, each with its own project configuration. The migration is therefore a move from "direct code locations" to "workspace + projects." + +## Step 1: Create a workspace root + +If you don't already have a workspace directory, create one by following [Creating workspaces](/guides/build/projects/workspaces/creating-workspaces). This will create a workspace root with a `dg.toml` file. + +## Step 2: Convert code locations into projects + +Each entry in `workspace.yaml` should map to a Dagster project in your workspace: + +1. For each code location, create a Dagster project using [Creating projects](/guides/build/projects/creating-projects). +2. Move or reference your existing definitions in that project. +3. Add each project path under `[[workspace.projects]]` in `dg.toml`. + +For the full `dg.toml` format, see the [dg.toml reference](/guides/build/projects/workspaces/dg-toml). + +## Step 3: Validate and remove workspace.yaml + +Run the workspace from the root directory: + +```shell +dg dev +``` + +If everything loads correctly, you can remove `workspace.yaml` and update any scripts to point to `dg.toml` (for example, `dg dev -w path/to/dg.toml`). diff --git a/docs/docs/guides/test/running-a-subset-of-asset-checks.md b/docs/docs/guides/test/running-a-subset-of-asset-checks.md index b9bc0762c0101..f7f32d0c17a1a 100644 --- a/docs/docs/guides/test/running-a-subset-of-asset-checks.md +++ b/docs/docs/guides/test/running-a-subset-of-asset-checks.md @@ -13,7 +13,7 @@ In this guide, we'll show you a few approaches to subsetting asset checks in decorator's `specs` and `can_subset` arguments, you can execute a subset of checks in a single op. {/* TODO link this to proper API doc */} -Inside the body of the function, we can use `AssetCheckExecutionContext.selected_asset_check_keys` to identify which computations to run. We can also set the decorator's `can_subset` parameter to `True` to execute a subset of the asset checks that the computation contains. +Inside the body of the function, we can use and its `selected_asset_check_keys` property to identify which computations to run. We can also set the decorator's `can_subset` parameter to `True` to execute a subset of the asset checks that the computation contains. As we don't know in advance which checks will be executed, we explicitly `yield` each asset check result that we're expected to create: diff --git a/docs/docs/guides/test/testing-partitioned-config-and-jobs.md b/docs/docs/guides/test/testing-partitioned-config-and-jobs.md index 72c85327de343..bcca224e7fc39 100644 --- a/docs/docs/guides/test/testing-partitioned-config-and-jobs.md +++ b/docs/docs/guides/test/testing-partitioned-config-and-jobs.md @@ -45,8 +45,7 @@ If you want to test that a : = '2023-01-02 00:00:00' - AND TIMESTAMP_SECONDS(TIME) < '2023-01-03 00:00:00'` + AND TIMESTAMP_SECONDS(TIME) < '2023-01-03 00:00:00' ``` @@ -185,11 +185,11 @@ If you want to store assets in different datasets, you can specify the dataset a You can also specify the dataset as part of the asset's asset key: {/* TODO add dedent=4 to CodeExample below */} - The dataset will be the last prefix before the asset's name. In this example, the `iris_data` asset will be stored in the `IRIS` dataset, and the `daffodil_data` asset will be found in the `DAFFODIL` dataset. diff --git a/docs/docs/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster.md b/docs/docs/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster.md index 1e6534ece2fa4..923203aab4012 100644 --- a/docs/docs/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster.md +++ b/docs/docs/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster.md @@ -147,13 +147,12 @@ When finished, your code should look like the following: ## Option 2: Using the BigQuery I/O manager -While using an I/O manager is not required, you may want to use an I/O manager to handle storing DataFrames as tables in BigQuery and loading BigQuery tables as DataFrames in downstream assets. You may want to use an I/O manager if: +While using an I/O manager is not required, you may want to use an I/O manager to handle storing DataFrames as tables in BigQuery and loading BigQuery tables as DataFrames in downstream assets. See [When to use I/O managers](/guides/build/io-managers) for guidance. You may want to use an I/O manager if: - You want your data to be loaded in memory so that you can interact with it using Python. - You'd like to have Dagster manage how you store the data and load it as an input in downstream assets. {/* TODO fix link: Using an I/O manager is not required, and you can reference [When to use I/O managers](/guides/build/io-managers#when-to-use-io-managers) to learn more. */} - This section of the guide focuses on storing and loading Pandas DataFrames in BigQuery, but Dagster also supports using PySpark DataFrames with BigQuery. The concepts from this guide apply to working with PySpark DataFrames, and you can learn more about setting up and using the BigQuery I/O manager with PySpark DataFrames in the [reference guide](/integrations/libraries/gcp/bigquery/reference). ### Step 1: Configure the BigQuery I/O manager diff --git a/docs/docs/integrations/libraries/snowflake/reference.md b/docs/docs/integrations/libraries/snowflake/reference.md index a1ae5f30e8085..12ba1b1967cd6 100644 --- a/docs/docs/integrations/libraries/snowflake/reference.md +++ b/docs/docs/integrations/libraries/snowflake/reference.md @@ -102,6 +102,7 @@ SELECT * When the `partition_expr` value is injected into this statement, the resulting SQL query must follow Snowflake's SQL syntax. Refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/constructs) for more information. {/* TODO fix link: When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/concepts/partitions-schedules-sensors/partitioning-assets#materializing-partitioned-assets). */} When materializing the above assets, a partition must be selected. In this example, the query used when materializing the `Iris-setosa` partition of the above assets would be: +When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets). In this example, the query used when materializing the `Iris-setosa` partition of the above assets would be: ```sql SELECT * @@ -130,6 +131,7 @@ SELECT * When the `partition_expr` value is injected into this statement, the resulting SQL query must follow Snowflake's SQL syntax. Refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/constructs) for more information. {/* TODO fix link: When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/concepts/partitions-schedules-sensors/partitioning-assets#materializing-partitioned-assets). */} When materializing the above assets, a partition must be selected. The `[partition_start]` and `[partition_end]` bounds are of the form `YYYY-MM-DD HH:MM:SS`. In this example, the query when materializing the `2023-01-02` partition of the above assets would be: +When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets). The `[partition_start]` and `[partition_end]` bounds are of the form `YYYY-MM-DD HH:MM:SS`. In this example, the query when materializing the `2023-01-02` partition of the above assets would be: ```sql SELECT * @@ -153,6 +155,7 @@ The Snowflake I/O manager can also store data partitioned on multiple dimensions Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in a downstream asset. For multi-partitions, Dagster concatenates the `WHERE` statements described in the above sections to craft the correct `SELECT` statement. {/* TODO fix link: When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/concepts/partitions-schedules-sensors/partitioning-assets#materializing-partitioned-assets). */} When materializing the above assets, a partition must be selected. For example, when materializing the `2023-01-02|Iris-setosa` partition of the above assets, the following query will be used: +When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets). For example, when materializing the `2023-01-02|Iris-setosa` partition of the above assets, the following query will be used: ```sql SELECT * @@ -173,21 +176,21 @@ You can specify the default schema where data will be stored as configuration to To store assets in different schemas, specify the schema as metadata: {/* TODO add dedent=4 prop to CodeExample below */} - You can also specify the schema as part of the asset's asset key: - {/* TODO add dedent=4 prop to CodeExample below */} In this example, the `iris_dataset` asset will be stored in the `IRIS` schema, and the `daffodil_dataset` asset will be found in the `DAFFODIL` schema. diff --git a/docs/sphinx/sections/integrations/libraries/omni/dagster-omni.rst b/docs/sphinx/sections/integrations/libraries/omni/dagster-omni.rst index 05d472cde528a..d3af4a81fcf9c 100644 --- a/docs/sphinx/sections/integrations/libraries/omni/dagster-omni.rst +++ b/docs/sphinx/sections/integrations/libraries/omni/dagster-omni.rst @@ -24,4 +24,4 @@ it to, and you will need to redeploy your code location after updating your meta The simplest way to update the stored state of your ``OmniComponent`` is to use the ``dg utils refresh-component-state`` command. When deploying your code location, this command should be executed in your CI/CD workflow (e.g. github actions). -.. TODO add link to state-backed component docs once they exist \ No newline at end of file +For more on state-backed components, see the `state-backed components guide `_.