Skip to content

PERF: Speed up import pandas with PEP 810 lazy imports#65885

Open
eendebakpt wants to merge 2 commits into
pandas-dev:mainfrom
eendebakpt:pep810
Open

PERF: Speed up import pandas with PEP 810 lazy imports#65885
eendebakpt wants to merge 2 commits into
pandas-dev:mainfrom
eendebakpt:pep810

Conversation

@eendebakpt

Copy link
Copy Markdown

PEP 810 (explicit lazy imports, CPython 3.15+) lets a module defer imports via a __lazy_modules__ declaration. On older Pythons the attribute is simply ignored, so adding it is backward compatible. A small set of pandas subsystems are imported eagerly today but are only needed once a specific entry point is called. Deferring them with __lazy_modules__ keeps them out of a plain import pandas:

  • the I/O backends — read_* / to_* / Excel / parquet / html / sql / json /
    stata / sas / … and the optional third-party engines they pull in;
  • pandas.testing (the assert_* helpers);
  • the eval / query, resample, interchange, and pandas.api re-export stacks.

Make more imports lazy would require reworking pandas.core as the many dependencies in modules there make it hard to use PEP 810 without additional changes.

Benchmarks

The import time depends a bit on the workload, so adding multiple benchmarks.

workload (after import pandas) baseline: modules / ms lazy: modules / ms faster
import pandas (nothing else) 290 / 826 221 / 678 18%
pd.DataFrame({'a':[1,2,3],'b':[4,5,6]}) 290 / 838 221 / 682 19%
df.groupby('a').sum() 290 / 834 221 / 685 18%
pd.to_datetime([...]) 290 / 836 221 / 704 16%
df.eval('a + b') 290 / 843 229 / 715 15%
df.to_csv() 291 / 832 222 / 706 15%
pd.read_csv(StringIO) 290 / 834 265 / 812 3%
pd.testing.assert_frame_equal(df, df) 290 / 840 229 / 710 15%
  • [N/A] closes #xxxx (Replace xxxx with the GitHub issue number)
  • Tests added and passed if fixing a bug or adding a new feature
  • All code checks passed.
  • [N/A] Added type annotations to new arguments/methods/functions.
  • [N/A] Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.
  • I have reviewed and followed all the contribution guidelines
  • If I used AI to develop this pull request, I prompted it to follow AGENTS.md.

eendebakpt and others added 2 commits June 15, 2026 11:55
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant