Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ To override a code location role for an individual user:
2. Click **Edit**.
3. Click the toggle to the left of the deployment to open a list of code locations.
4. Next to a code location, click **Edit user role**.
5. Select the user role for the code location: {/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/code-location-override.png" */}
5. Select the user role for the code location.
6. Click **Save**.

#### Team members
Expand All @@ -118,13 +118,11 @@ The above also applies to code locations and Branch Deployment roles.

#### Viewing overrides

To view deployment-level overrides for a specific user, locate the user on the **Users** page and hover over a deployment:

{/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/user-overrides-popup.png" */}

If there are code location-level overrides, a small **N override(s)** link will display beneath the user's deployment role. Hover over it to display the list of overrides:

{/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/code-location-override-popup.png" */}
To view deployment-level overrides for a specific user, locate the user on the **Users** page and hover over a deployment. A tooltip will display any overrides that apply at the deployment level.

If there are code location-level overrides, a small **N override(s)** link will display beneath the user's deployment role. Hover over it to display the list of overrides.

#### Removing overrides

Expand Down
3 changes: 1 addition & 2 deletions docs/docs/deployment/dagster-plus/hybrid/kubernetes/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,8 @@ helm --namespace dagster-cloud upgrade agent \

## Troubleshooting

You can see basic health information about your agent in the Dagster+ UI:

[comment]: <> (TODO: Screenshot of Dagster+ Deployments agents tab)
You can see basic health information about your agent in the Dagster+ UI.

### View logs

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ Now copy the following sensor code in the `sensors.py` file:
2. Navigate to the Automation page.
3. Turn on the `adhoc_request_sensor`.
4. Click on the `adhoc_request_sensor` details.

{/* TODO: Screenshot */}
You should see the sensor details view with recent ticks and the run history panel.

5. Create a `data/requests` directory in `dagster_tutorial`. Then include a `request.json` file:

Expand All @@ -76,8 +76,8 @@ Now copy the following sensor code in the `sensors.py` file:
```

6. Click on the green tick to see the run for this request.

{/* TODO: Screenshot */}
The run should appear in the runs list, and the tick details will show the run ID.

## Summary

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Then we can execute this job and pull the attributes we need to build the `conte
/>

{/* TODO the methods and statuses below do not exist in API docs */}
{/* We have provided convenience functions <PyObject section="execution" module="dagster" object="ExecuteInProcessResult" method="get_job_success_event" /> and <PyObject section="execution" module="dagster" object="ExecuteInProcessResult" method="get_job_failure_event" /> for retrieving `DagsterRunStatus.SUCCESS` and `DagsterRunStatus.FAILURE` events, respectively. If you have a run status sensor triggered on another status, you can retrieve all events from `result` and filter based on your event type. */}
To inspect the resulting status in tests, use the <PyObject section="execution" module="dagster" object="ExecuteInProcessResult" /> returned by `execute_in_process`. Its `dagster_run` and `all_events` properties let you filter for the status or event type you care about.

We can use the same pattern to build the context for <PyObject section="schedules-sensors" module="dagster" object="run_failure_sensor" />. If we wanted to test this run failure sensor:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Dagster's versioning system helps you determine ahead of time whether materializ
Dagster has two versioning concepts to represent the code and input data used for each materialization:

- **Code version.** A string that represents the version of the code that computes an asset. This is the `code_version` argument of <PyObject section="assets" module="dagster" object="asset" decorator />.
- **Data version.** A string that represents the version of the data represented by the asset. This is represented as a `DataVersion` object.
{/* TODO link `DataVersion` to API docs */}
- **Data version.** A string that represents the version of the data represented by the asset. This is represented as a <PyObject section="assets" module="dagster" object="DataVersion" /> object.

By keeping track of code and data versions, Dagster can predict whether a materialization will change the underlying value. This allows Dagster to skip redundant materializations and instead return the previously computed value. In more technical terms, Dagster offers a limited form of [memoization](https://en.wikipedia.org/wiki/Memoization) for assets: the last-computed asset value is always cached.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,18 @@ unlisted: true
---

TODO
This page lists common checks when a component fails to load or behaves unexpectedly.

## Quick checks

- **Validate the component type string**. Ensure the type name matches the registered component class and that the package providing it is installed in your environment.
- **Confirm configuration fields**. Component configuration is validated on load. If a field is missing or mis-typed, the error message will include the field path to update.
- **Verify workspace paths**. If you are using a workspace, confirm that `dg.toml` points at the correct project directories and that the project loads on its own.

## Inspecting load errors

If a component fails to load, the code location will show a load error in the UI and in CLI output. Use the stack trace there to locate the file and configuration entry that caused the failure.

## Next steps

For a refresher on how components are structured, see the [components overview](/guides/build/components).
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ Sometimes you have a set of predefined categories for your data. For instance, y
<CodeExample path="docs_snippets/docs_snippets/guides/build/partitions_backfills/partitioning/static_partitioning.py" language="python" title="src/<project_name>/defs/assets.py"/>

{/* TODO: Link to Backfill page to explain how to backfill regional sales data */}
To backfill historical partitions for regional sales data, see [Backfilling data](/guides/build/partitions-and-backfills/backfilling-data).

## Two-dimensional partitions \{#two-dimensional-partitions}

Expand Down Expand Up @@ -97,4 +98,4 @@ After a partition has been successfully materialized, it will display as green i

To view materializations by partition for a specific asset, navigate to the **Activity** tab of the asset's **Details** page:

![Asset activity section of asset details page](/images/guides/build/partitions-and-backfills/materialized-partitioned-asset-activity.png)
![Asset activity section of asset details page](/images/guides/build/partitions-and-backfills/materialized-partitioned-asset-activity.png)
Original file line number Diff line number Diff line change
@@ -1,8 +1,39 @@
---
title: Migrating from workspace.yaml to dg.toml
sidebar_position: 2000
description: TODO
# description: TODO
description: Migrate legacy workspace.yaml configurations to a dg.toml workspace.
unlisted: true
---

TODO
TODO
Comment thread
Vamsi-klu marked this conversation as resolved.

If you have an existing `workspace.yaml` file, you can migrate to a `dg.toml`-based workspace to align with the current Dagster OSS project and workspace structure.

## Overview

`workspace.yaml` directly lists code locations. A `dg.toml` workspace points to one or more Dagster projects, each with its own project configuration. The migration is therefore a move from "direct code locations" to "workspace + projects."

## Step 1: Create a workspace root

If you don't already have a workspace directory, create one by following [Creating workspaces](/guides/build/projects/workspaces/creating-workspaces). This will create a workspace root with a `dg.toml` file.

## Step 2: Convert code locations into projects

Each entry in `workspace.yaml` should map to a Dagster project in your workspace:

1. For each code location, create a Dagster project using [Creating projects](/guides/build/projects/creating-projects).
2. Move or reference your existing definitions in that project.
3. Add each project path under `[[workspace.projects]]` in `dg.toml`.

For the full `dg.toml` format, see the [dg.toml reference](/guides/build/projects/workspaces/dg-toml).

## Step 3: Validate and remove workspace.yaml

Run the workspace from the root directory:

```shell
dg dev
```

If everything loads correctly, you can remove `workspace.yaml` and update any scripts to point to `dg.toml` (for example, `dg dev -w path/to/dg.toml`).
2 changes: 1 addition & 1 deletion docs/docs/guides/test/running-a-subset-of-asset-checks.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ In this guide, we'll show you a few approaches to subsetting asset checks in <Py
Using the <PyObject section="asset-checks" module="dagster" object="multi_asset_check" decorator /> decorator's `specs` and `can_subset` arguments, you can execute a subset of checks in a single op.

{/* TODO link this to proper API doc */}
Inside the body of the function, we can use `AssetCheckExecutionContext.selected_asset_check_keys` to identify which computations to run. We can also set the decorator's `can_subset` parameter to `True` to execute a subset of the asset checks that the computation contains.
Inside the body of the function, we can use <PyObject section="execution" module="dagster" object="AssetCheckExecutionContext" /> and its `selected_asset_check_keys` property to identify which computations to run. We can also set the decorator's `can_subset` parameter to `True` to execute a subset of the asset checks that the computation contains.

As we don't know in advance which checks will be executed, we explicitly `yield` each asset check result that we're expected to create:

Expand Down
3 changes: 1 addition & 2 deletions docs/docs/guides/test/testing-partitioned-config-and-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,7 @@ If you want to test that a <PyObject section="partitions" module="dagster" objec
## Testing partitioned jobs

{/* TODO fix the API docs so this can be a PyObject */}

To run a partitioned job in-process on a particular partition, supply a value for the `partition_key` argument of [`dagster.JobDefinition.execute_in_process`](/api/dagster/execution):
To run a partitioned job in-process on a particular partition, supply a value for the `partition_key` argument of <PyObject section="jobs" module="dagster" object="JobDefinition.execute_in_process" />:

<CodeExample
path="docs_snippets/docs_snippets/guides/build/partitions_backfills/partitioned_job_test.py"
Expand Down
4 changes: 2 additions & 2 deletions docs/docs/integrations/libraries/gcp/bigquery/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ When materializing the above assets, a partition must be selected, as described
SELECT *
WHERE SPECIES in ('Iris-setosa')
AND TIMESTAMP_SECONDS(TIME) >= '2023-01-02 00:00:00'
AND TIMESTAMP_SECONDS(TIME) < '2023-01-03 00:00:00'`
AND TIMESTAMP_SECONDS(TIME) < '2023-01-03 00:00:00'
```

</TabItem>
Expand All @@ -185,11 +185,11 @@ If you want to store assets in different datasets, you can specify the dataset a
You can also specify the dataset as part of the asset's asset key:

{/* TODO add dedent=4 to CodeExample below */}

<CodeExample
path="docs_snippets/docs_snippets/integrations/bigquery/reference/dataset.py"
startAfter="start_asset_key"
endBefore="end_asset_key"
dedent="4"
/>

The dataset will be the last prefix before the asset's name. In this example, the `iris_data` asset will be stored in the `IRIS` dataset, and the `daffodil_data` asset will be found in the `DAFFODIL` dataset.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -147,13 +147,12 @@ When finished, your code should look like the following:

## Option 2: Using the BigQuery I/O manager

While using an I/O manager is not required, you may want to use an I/O manager to handle storing DataFrames as tables in BigQuery and loading BigQuery tables as DataFrames in downstream assets. You may want to use an I/O manager if:
While using an I/O manager is not required, you may want to use an I/O manager to handle storing DataFrames as tables in BigQuery and loading BigQuery tables as DataFrames in downstream assets. See [When to use I/O managers](/guides/build/io-managers) for guidance. You may want to use an I/O manager if:

- You want your data to be loaded in memory so that you can interact with it using Python.
- You'd like to have Dagster manage how you store the data and load it as an input in downstream assets.

{/* TODO fix link: Using an I/O manager is not required, and you can reference [When to use I/O managers](/guides/build/io-managers#when-to-use-io-managers) to learn more. */}

This section of the guide focuses on storing and loading Pandas DataFrames in BigQuery, but Dagster also supports using PySpark DataFrames with BigQuery. The concepts from this guide apply to working with PySpark DataFrames, and you can learn more about setting up and using the BigQuery I/O manager with PySpark DataFrames in the [reference guide](/integrations/libraries/gcp/bigquery/reference).

### Step 1: Configure the BigQuery I/O manager
Expand Down
7 changes: 5 additions & 2 deletions docs/docs/integrations/libraries/snowflake/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ SELECT *
When the `partition_expr` value is injected into this statement, the resulting SQL query must follow Snowflake's SQL syntax. Refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/constructs) for more information.

{/* TODO fix link: When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/concepts/partitions-schedules-sensors/partitioning-assets#materializing-partitioned-assets). */} When materializing the above assets, a partition must be selected. In this example, the query used when materializing the `Iris-setosa` partition of the above assets would be:
When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets). In this example, the query used when materializing the `Iris-setosa` partition of the above assets would be:
Comment thread
Vamsi-klu marked this conversation as resolved.

```sql
SELECT *
Expand Down Expand Up @@ -130,6 +131,7 @@ SELECT *
When the `partition_expr` value is injected into this statement, the resulting SQL query must follow Snowflake's SQL syntax. Refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/constructs) for more information.

{/* TODO fix link: When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/concepts/partitions-schedules-sensors/partitioning-assets#materializing-partitioned-assets). */} When materializing the above assets, a partition must be selected. The `[partition_start]` and `[partition_end]` bounds are of the form `YYYY-MM-DD HH:MM:SS`. In this example, the query when materializing the `2023-01-02` partition of the above assets would be:
When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets). The `[partition_start]` and `[partition_end]` bounds are of the form `YYYY-MM-DD HH:MM:SS`. In this example, the query when materializing the `2023-01-02` partition of the above assets would be:

```sql
SELECT *
Expand All @@ -153,6 +155,7 @@ The Snowflake I/O manager can also store data partitioned on multiple dimensions
Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in a downstream asset. For multi-partitions, Dagster concatenates the `WHERE` statements described in the above sections to craft the correct `SELECT` statement.

{/* TODO fix link: When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/concepts/partitions-schedules-sensors/partitioning-assets#materializing-partitioned-assets). */} When materializing the above assets, a partition must be selected. For example, when materializing the `2023-01-02|Iris-setosa` partition of the above assets, the following query will be used:
When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets). For example, when materializing the `2023-01-02|Iris-setosa` partition of the above assets, the following query will be used:

```sql
SELECT *
Expand All @@ -173,21 +176,21 @@ You can specify the default schema where data will be stored as configuration to
To store assets in different schemas, specify the schema as metadata:

{/* TODO add dedent=4 prop to CodeExample below */}

<CodeExample
path="docs_snippets/docs_snippets/integrations/snowflake/schema.py"
startAfter="start_metadata"
endBefore="end_metadata"
dedent="4"
/>

You can also specify the schema as part of the asset's asset key:

{/* TODO add dedent=4 prop to CodeExample below */}

<CodeExample
path="docs_snippets/docs_snippets/integrations/snowflake/schema.py"
startAfter="start_asset_key"
endBefore="end_asset_key"
dedent="4"
/>

In this example, the `iris_dataset` asset will be stored in the `IRIS` schema, and the `daffodil_dataset` asset will be found in the `DAFFODIL` schema.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,4 @@ it to, and you will need to redeploy your code location after updating your meta
The simplest way to update the stored state of your ``OmniComponent`` is to use the ``dg utils refresh-component-state`` command. When
deploying your code location, this command should be executed in your CI/CD workflow (e.g. github actions).

.. TODO add link to state-backed component docs once they exist
For more on state-backed components, see the `state-backed components guide <https://docs.dagster.io/guides/build/components/state-backed-components>`_.
Loading