BUG: read_csv pyarrow engine dtype handling with index_col/defaultdict#65930
Open
jbrockmendel wants to merge 2 commits into
Open
BUG: read_csv pyarrow engine dtype handling with index_col/defaultdict#65930jbrockmendel wants to merge 2 commits into
jbrockmendel wants to merge 2 commits into
Conversation
jbrockmendel
added
Bug
IO CSV
Split the dtype fixes out of pandas-dev#65859: a non-dict ``dtype`` passed with ``index_col`` no longer raises AttributeError, and a ``defaultdict`` ``dtype`` now applies its default to columns not explicitly listed (GH#41574). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jbrockmendel
force-pushed
the
tst-pyarrow-dtypes
branch
from
941d3fd to
e430100
Compare
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splits the two dtype-handling fixes out of GH-65859 into a focused PR. With
engine="pyarrow", :func:read_csv:AttributeError: type object 'str' has no attribute 'get'when a non-dictdtypewas passed together withindex_col. The per-index-column dtype handling now only runs for a dictdtype; a scalardtypeis applied to the whole frame as before.defaultdictdtype's default factory for columns not explicitly listed — membership/iteration over adefaultdictdoesn't trigger the default — so those columns kept pyarrow's inferred dtype (ENH: support defaultdict in read_csv dtype parameter #41574). Thedefaultdictis now materialized over all columns before conversion.Un-xfails
test_dtypes_defaultdictandtest_dtypes_defaultdict_invalid, and addstest_dtype_scalar_with_index_col. The column-name/dedup fixes from GH-65859 (and the tests that depend on them, including the dup-columndefaultdictcase) stay in that PR.🤖 Generated with Claude Code