Skip to content

Fix flaky test_collection_metadata#9316

Draft
crusaderky wants to merge 3 commits into
dask:mainfrom
crusaderky:test_collection_metadata
Draft

Fix flaky test_collection_metadata#9316
crusaderky wants to merge 3 commits into
dask:mainfrom
crusaderky:test_collection_metadata

Conversation

@crusaderky

@crusaderky crusaderky commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Fix occasional test failure

__________________________ test_collections_metadata ___________________________

c = <Client: No scheduler connected>
s = <Scheduler 'tcp://127.0.0.1:50135', workers: 0, cores: 0, tasks: 0>
a = <Worker 'tcp://127.0.0.1:50136', name: 0, status: closed, stored: 0, running: 0/1, ready: 0, comm: 0, waiting: 0>
b = <Worker 'tcp://127.0.0.1:50139', name: 1, status: closed, stored: 0, running: 0/2, ready: 0, comm: 0, waiting: 0>

    @pytest.mark.filterwarnings("ignore:Dask annotations")
    @gen_cluster(client=True)
    async def test_collections_metadata(c, s, a, b):
        np = pytest.importorskip("numpy")
        pd = pytest.importorskip("pandas")
        dd = pytest.importorskip("dask.dataframe")
        df = pd.DataFrame(
            {"x": np.random.random(1000), "y": np.random.random(1000)},
            index=np.arange(1000),
        )
        ldf = dd.from_pandas(df, npartitions=10)
    
        with span("foo") as span_id:
            await c.compute(ldf)
    
        ext = s.extensions["spans"]
        span_ = ext.spans[span_id]
        assert span_
>       collections_meta = span_.metadata["collections"]
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       TypeError: 'NoneType' object is not subscriptable

distributed/tests/test_spans.py:840: TypeError

WIP unreviewed clanker output; please disregard for now

@crusaderky crusaderky added flaky test Intermittent failures on CI. tests Unit tests and/or continuous integration labels Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    19 files   -     21      19 suites   - 21   16h 34m 20s ⏱️ + 1h 33m 58s
 5 000 tests +   846   5 000 ✅ +1 025       0 💤  -   178  0 ❌  - 1 
95 000 runs  +14 159  85 000 ✅ +8 397  10 000 💤 +5 763  0 ❌  - 1 

Results for commit 6fa439b. ± Comparison against base commit ace5560.

This pull request removes 4154 and adds 5000 tests. Note that renamed tests count towards both.
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_allowlist
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_non_standard_ports
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_port_zero
distributed.cli.tests.test_dask_scheduler ‑ test_defaults
distributed.cli.tests.test_dask_scheduler ‑ test_hostport
distributed.cli.tests.test_dask_scheduler ‑ test_idle_timeout
distributed.cli.tests.test_dask_scheduler ‑ test_interface
distributed.cli.tests.test_dask_scheduler ‑ test_multiple_protocols
distributed.cli.tests.test_dask_scheduler ‑ test_multiple_workers
…
distributed.tests.test_spans ‑ test_collections_metadata[1-5000]
distributed.tests.test_spans ‑ test_collections_metadata[10-5000]
distributed.tests.test_spans ‑ test_collections_metadata[100-5000]
distributed.tests.test_spans ‑ test_collections_metadata[1000-5000]
distributed.tests.test_spans ‑ test_collections_metadata[1001-5000]
distributed.tests.test_spans ‑ test_collections_metadata[1002-5000]
distributed.tests.test_spans ‑ test_collections_metadata[1003-5000]
distributed.tests.test_spans ‑ test_collections_metadata[1004-5000]
distributed.tests.test_spans ‑ test_collections_metadata[1005-5000]
distributed.tests.test_spans ‑ test_collections_metadata[1006-5000]
…

♻️ This comment has been updated with latest results.

@crusaderky crusaderky force-pushed the test_collection_metadata branch from 8408ded to d964072 Compare July 2, 2026 14:25
@crusaderky crusaderky force-pushed the test_collection_metadata branch from d964072 to 354ec06 Compare July 2, 2026 14:49
crusaderky and others added 2 commits July 2, 2026 17:07
Span.add_metadata deduplicated incoming metadata via a class-level set of
id(metadata) values that was never pruned. Since CPython recycles the id of
garbage-collected objects, the metadata of a new graph submission could
collide with the id of a previous, dead metadata dict and be silently
dropped, leaving Span.metadata as None. The set also grew unboundedly over
the lifetime of the scheduler.

The only deduplication actually needed is within a single observe_tasks
call, where all tasks of one graph submission share the same metadata
object. Do that with a local set instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@crusaderky crusaderky mentioned this pull request Jul 3, 2026
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

flaky test Intermittent failures on CI. tests Unit tests and/or continuous integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant