These are the changes in pandas 3.1.0. See :ref:`release` for a full changelog including other versions of pandas.
{{ header }}
- :class:`Period` now supports f-string formatting via
__format__, e.g.f"{period:%Y-%m}"(:issue:`48536`) - :meth:`.DataFrameGroupBy.agg` now allows for the provided
functo return a NumPy array (:issue:`63957`) - :meth:`DataFrameGroupBy.transform` now accepts list-like and dict arguments similar to :meth:`GroupBy.agg`, and supports :class:`NamedFunc` (:issue:`58318`)
- :meth:`Series.to_json` now supports serializing custom ExtensionArrays (by correctly using the
_values_for_jsonmethod of an ExtensionArray) (:issue:`65047`) - :meth:`Timestamp.round`, :meth:`Timestamp.floor`, and :meth:`Timestamp.ceil` now officially accept :class:`Timedelta` arguments (:issue:`63687`)
- Added :class:`NamedFunc`, an alias to :class:`NamedAgg` for a more semantically accurate name when used with non-aggregation functions; either can accept arbitrary functions (:issue:`65164`)
- :meth:`ExtensionArray.map` now calls :meth:`ExtensionArray._cast_pointwise_result` to retain the dtype backend, e.g. Arrow-backed arrays now preserve their Arrow dtype through
map(:issue:`57189`, :issue:`62164`) - :func:`read_csv` now supports
dtype="complex64"anddtype="complex128"with the C engine, enabling round-tripping of complex-number columns written by :meth:`DataFrame.to_csv` (:issue:`9379`) - Added :meth:`ExtensionArray.count` (:issue:`64450`)
- Added :meth:`ExtensionArray.sort` for in-place sorting of :class:`ExtensionArray` (:issue:`64977`)
- Added :meth:`Index.replace` method to support value replacement functionality similar to :meth:`Series.replace` (:issue:`19495`)
- Added reduction methods as public API on pandas-implemented extension arrays where applicable (:issue:`63512`)
- Display formatting for float sequences in DataFrame cells now respects the
display.precisionoption (:issue:`60503`). - Improved the precision of float parsing in :func:`read_csv` (:issue:`64395`)
- Improved the string
reprof :class:`pd.core.arrays.SparseArray` (:issue:`64547`) - Improved type inference of comparison and arithmetic operators on :class:`Series` and :class:`DataFrame` for static type checkers (e.g.
ser == "a"is now inferred as :class:`Series` instead ofAny) (:issue:`40762`) - MSVC is no longer required to build on Windows, and build errors when using the MinGW compiler have been fixed (:issue:`63160`)
These are bug fixes that might have notable behavior changes.
Some minimum supported versions of dependencies were updated. If installed, we now require:
| Package | Minimum Version | Required | Changed |
|---|---|---|---|
| X | X |
For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.
| Package | Minimum Version | Changed |
|---|---|---|
| X |
See :ref:`install.dependencies` and :ref:`install.optional_dependencies` for more.
- :meth:`.DataFrameGroupBy.sum` and :meth:`.DataFrameGroupBy.mean` on float dtypes with sorted grouping keys may differ from prior versions in the last few floating-point digits, due to a faster summation algorithm that does not use Kahan compensation (:issue:`65103`)
- :meth:`DataFrame.to_hdf` no longer writes a
pandas_versionattribute into the HDF5 file. The value was hardcoded to"0.15.2"and was only used internally to detect files written by pandas versions older than 0.10.1 (:issue:`62792`) - APIs that accept an
engine="numba"parameter withengine_kwargswill no longer pass through anopythonargument tonumba.jit. This argument has had no effect since numba 0.59.0 (:issue:`64483`). - Removed the
freqandfreqstrattributes from :class:`.DatetimeArray` and :class:`.TimedeltaArray`. Frequency is now stored only on :class:`DatetimeIndex` and :class:`TimedeltaIndex`; accessSeries.dt.freqor wrap the array in an Index to retrieve a frequency. Thecheck_freqkeyword on :func:`testing.assert_extension_array_equal` for these array types has also been removed (:issue:`24566`).
- Deprecated :attr:`Timestamp.dayofweek`, :attr:`Timestamp.dayofyear`, :attr:`Timestamp.daysinmonth` in favor of :attr:`Timestamp.day_of_week`, :attr:`Timestamp.day_of_year`, :attr:`Timestamp.days_in_month`, respectively. The same deprecation applies to the corresponding attributes on :class:`Period`, :class:`DatetimeIndex`, :class:`PeriodIndex`, and :attr:`Series.dt` (:issue:`46768`)
- Deprecated :class:`PeriodIndex` and :class:`PeriodArray` inferring the frequency from a :class:`Series` of datetime64 data when
freqis not provided. Passfreqexplicitly instead (:issue:`64241`) - Deprecated :func:`infer_freq`, :attr:`DatetimeIndex.inferred_freq`, and :attr:`TimedeltaIndex.inferred_freq` returning a string; in a future version these will return a :class:`BaseOffset` instead. Use
pd.set_option('future.infer_freq_returns_offset', True)to opt in to the future behavior (:issue:`55504`) - Deprecated :func:`set_eng_float_format`. Use
pd.set_option("display.precision", N)to control decimal precision, or pass a custom callable topd.set_option("display.float_format", func)(:issue:`64460`) - Deprecated :meth:`.DataFrameGroupBy.agg` and :meth:`.Resampler.agg` unpacking a scalar when the provided
funcreturns a Series or array of length 1; in the future this will result in the Series or array being in the result. Users should unpack the scalar infuncitself (:issue:`64014`) - Deprecated :meth:`ExcelFile.parse`, use :func:`read_excel` instead (:issue:`58247`)
- Deprecated
engine="fastparquet"andengine="auto"in :func:`read_parquet` and :meth:`DataFrame.to_parquet`. Thefastparquetlibrary has been retired; useengine="pyarrow"or do not passengineto use the default. (:issue:`64597`) - Deprecated arithmetic operations between pandas objects (:class:`DataFrame`, :class:`Series`, :class:`Index`, and pandas-implemented :class:`ExtensionArray` subclasses) and list-likes other than
list,np.ndarray, :class:`ExtensionArray`, :class:`Index`, :class:`Series`, :class:`DataFrame`. For e.g.tupleorrange, explicitly cast these to a supported object instead. In a future version, these will be treated as scalar-like for pointwise operation (:issue:`62423`) - Deprecated automatic dtype promotion when reindexing with a
fill_valuethat cannot be held by the original dtype. Explicitly cast to a common dtype instead (:issue:`53910`) - Deprecated grouping by an index level name when the name matches multiple levels of a :class:`MultiIndex`. Use the level number instead (:issue:`49434`)
- Deprecated implicit conversion of
datetime.dateobjects to :class:`Timestamp` when indexing or joining a :class:`DatetimeIndex`. Use :func:`to_datetime` to explicitly convert to :class:`DatetimeIndex` instead (:issue:`62158`) - Deprecated passing a
dictto :meth:`DataFrame.from_records`, use the :class:`DataFrame` constructor or :meth:`DataFrame.from_dict` instead (:issue:`22025`) - Deprecated passing a non-dict (e.g. a list of dicts) to :meth:`DataFrame.from_dict`. Use the :class:`DataFrame` constructor instead (:issue:`58862`)
- Deprecated passing unnecessary
*argsand**kwargsto :meth:`.GroupBy.cumsum`, :meth:`.GroupBy.cumprod`, :meth:`.GroupBy.cummin`, :meth:`.GroupBy.cummax`, :meth:`.SeriesGroupBy.skew`, :meth:`.DataFrameGroupBy.skew`, :meth:`.SeriesGroupBy.take`, and :meth:`.DataFrameGroupBy.take`. Theskipnaparameter for the cum* methods is now an explicit keyword argument (:issue:`50407`) - Deprecated setting values with :meth:`DataFrame.at` and :meth:`Series.at` when the key does not exist in the index, which previously expanded the object. Use
.locinstead (:issue:`48323`) - Deprecated the
%ndirective in :meth:`Period.strftime` for nanoseconds; use%Ninstead.%nis a newline directive in Cstrftime(and Python'stime.strftime/datetime.strftime) (:issue:`65432`) - Deprecated the
.nameproperty of offset objects (e.g., :class:`~pandas.tseries.offsets.Day`, :class:`~pandas.tseries.offsets.Hour`). Use.rule_codeinstead (:issue:`64207`) - Deprecated the
dropnakeyword in :meth:`DataFrame.to_hdf`, :meth:`HDFStore.put`, :meth:`HDFStore.append`, and :meth:`HDFStore.append_to_multiple`, and theio.hdf.dropna_tableoption. Use :meth:`DataFrame.dropna` before writing instead (:issue:`32038`) - Deprecated the
float_precisionargument in :func:`read_csv`, :func:`read_table`, and :func:`read_fwf`. All float precision modes now use the same converter (:issue:`64395`) - Deprecated the
includeandexcludearguments of :meth:`Series.describe`. They had no effect on a Series; filter dtypes upstream of the call instead (:issue:`54193`) - Deprecated the
weekdayproperty on :class:`DatetimeIndex`, :class:`.DatetimeArray`, :class:`PeriodIndex`, :class:`.PeriodArray`, and :class:`Period`. Useday_of_weekinstead.Timestamp.weekday()remains a method consistent with :meth:`datetime.datetime.weekday` (:issue:`12816`) - Deprecated the
xlrdandpyxlsbengines in :func:`read_excel`. Useengine="calamine"instead (:issue:`56542`) - Deprecated the default value of
exactin :func:`assert_index_equal`; in a future version this will default toTrueinstead of "equiv" (:issue:`57436`) - Deprecated the default value of
track_timesin :meth:`HDFStore.put`. In a future version, the default will change fromTruetoFalseso that HDF5 files are deterministic by default (:issue:`51456`) - Deprecated passing a non-boolean value for
numeric_onlyto :meth:`DataFrame.mean`, :meth:`DataFrame.min`, :meth:`DataFrame.max`, :meth:`DataFrame.median`, :meth:`DataFrame.skew`, :meth:`DataFrame.kurt`, :meth:`DataFrame.std`, :meth:`DataFrame.var`, :meth:`DataFrame.sem`, :meth:`DataFrame.sum`, :meth:`DataFrame.prod` (and their :class:`Series` equivalents); this will raise in a future version of pandas (:issue:`53098`) - Deprecated the inference of
datetime64dtype from data containingdatetime.dateobjects when used in comparisons or :meth:`Index.equals` with :class:`DatetimeIndex`. Use :func:`pandas.to_datetime` to explicitly convert todatetime64instead (:issue:`65056`) - Deprecated
inplacekeyword for :meth:`DataFrame.rename` and :meth:`DataFrame.drop`. This keyword will be removed in a future version (:issue:`63207`); see also PDEP-8
- Performance improvement in :class:`.DataFrameGroupBy` aggregations (
sum,mean,min,max,prod) when the grouping keys are already sorted (:issue:`65103`) - Performance improvement in casting integer and boolean dtypes to
string[pyarrow]by using PyArrow's native cast instead of element-wise conversion (:issue:`56505`) - Performance improvement in :meth:`DataFrame.__getitem__` when selecting a single column by label on a :class:`DataFrame` with duplicate column names. (:issue:`64126`).
- Performance improvement in :attr:`Series.is_monotonic_increasing` and :attr:`Series.is_monotonic_decreasing` for :class:`ArrowDtype` and masked dtypes by dispatching to the :class:`ExtensionArray` (:issue:`56619`)
- Performance improvement in :class:`DataFrame` repr by avoiding redundant formatting when columns exceed terminal width (:issue:`64863`)
- Performance improvement in :class:`GroupBy` reductions and transformations for :class:`SparseDtype` columns (:issue:`36123`)
- Performance improvement in :func:`bdate_range` and :func:`date_range` with
freq="B"orfreq="C"(business day frequencies) (:issue:`16463`) - Performance improvement in :func:`concat` and :meth:`DataFrame.astype` to extension dtypes (:issue:`65672`)
- Performance improvement in :func:`infer_freq` (:issue:`64463`)
- Performance improvement in :func:`merge` and :meth:`DataFrame.join` for many-to-many joins with
sort=False(:issue:`56564`) - Performance improvement in :func:`merge` for many-to-one joins with unique right keys (:issue:`38418`)
- Performance improvement in :func:`merge` with
how="cross"(:issue:`38082`) - Performance improvement in :func:`merge` with
how="left"(:issue:`64370`) - Performance improvement in :func:`merge` with
how="left"andsort=Falsewhen joining on a right index with unique keys (:issue:`65160`) - Performance improvement in :func:`merge` with
sort=Falsefor single-keyhow="left"/how="right"joins when the opposite join key is sorted, unique, and range-like (:issue:`64146`) - Performance improvement in :func:`read_csv` with
engine="c"when reading from binary file-like objects (e.g. PyArrow S3 file handles) by avoiding unnecessaryTextIOWrapperwrapping (:issue:`46823`) - Performance improvement in :func:`read_hdf` for the fixed (default) format, especially on large frames (:issue:`47726`)
- Performance improvement in :func:`read_html` and the Python CSV parser when
thousandsis set, fixing catastrophic regex backtracking on cells with many comma-separated digit groups followed by non-numeric text (:issue:`52619`) - Performance improvement in :func:`read_sas` by reading page header fields directly in Cython instead of falling back to Python (:issue:`47339`)
- Performance improvement in :func:`read_sas` for SAS7BDAT files by pre-computing date/datetime column classification once during metadata parsing instead of per chunk (:issue:`47339`)
- Performance improvement in :func:`read_sas` for SAS7BDAT files with full-precision (8-byte) numeric columns, with up to ~2x speedup on bulk reads (:issue:`47339`)
- Performance improvement in :func:`read_sas` for compressed SAS7BDAT files by reusing the decompression buffer instead of allocating per row (:issue:`47339`)
- Performance improvement in :func:`read_sas` when decoding strings (:issue:`47339`)
- Performance improvement in :func:`read_sql` with ADBC connections by requesting only table metadata when checking whether an input string names a table (:issue:`65652`)
- Performance improvement in :func:`to_datetime` with the default
cache=Truefor inputs that are already datetime-typed or use aunit(:issue:`65380`) - Performance improvement in :func:`tseries.frequencies.to_offset` parsing of frequency strings, especially for tick-resolution offsets (e.g.
"h","5min","3s") and compound expressions (e.g."1D1h") (:issue:`65395`) - Performance improvement in :func:`util.hash_pandas_object` for PyArrow-backed string and binary types by using PyArrow's
dictionary_encodeinstead of converting to NumPy for factorization (:issue:`48964`) - Performance improvement in :meth:`.DataFrameGroupBy.agg` and :meth:`.SeriesGroupBy.agg` with user-defined functions (:issue:`46505`)
- Performance improvement in :meth:`DataFrame.apply` with
axis=1when the :class:`DataFrame` has :class:`ExtensionDtype` columns (e.g. :class:`ArrowDtype`) (:issue:`61747`) - Performance improvement in :meth:`DataFrame.corr` and :meth:`DataFrame.cov` when data contains no NaN values (:issue:`64857`)
- Performance improvement in :meth:`DataFrame.equals`, :meth:`DataFrame.select_dtypes`, and other operations performing shallow column slicing on Arrow-backed columns (:issue:`58966`)
- Performance improvement in :meth:`DataFrame.fillna` and :meth:`Series.fillna` with scalar fill value for float, object, nullable, and datetime-like dtypes (:issue:`42147`)
- Performance improvement in :meth:`DataFrame.from_records` when passing a 2D :class:`numpy.ndarray` (:issue:`22025`)
- Performance improvement in :meth:`DataFrame.insert` when the number of blocks is small (:issue:`57641`)
- Performance improvement in :meth:`DataFrame.loc` with non-unique masked index (:issue:`56759`)
- Performance improvement in :meth:`DataFrame.query` and :meth:`DataFrame.eval` when the :class:`DataFrame` contains :class:`PeriodDtype` or :class:`IntervalDtype` columns (:issue:`35247`)
- Performance improvement in :meth:`DataFrame.rank` and :meth:`Series.rank` for non-nullable numeric dtypes (:issue:`65054`)
- Performance improvement in :meth:`DataFrame.sort_index` and :meth:`Series.sort_index` with the
levelparameter when the index is already sorted and not a :class:`MultiIndex` (:issue:`64883`) - Performance improvement in :meth:`DataFrame.sort_values` with multiple numeric columns by avoiding unnecessary :class:`Categorical` conversion (:issue:`15389`)
- Performance improvement in :meth:`DataFrame.sum`, :meth:`DataFrame.prod`, :meth:`DataFrame.min`, :meth:`DataFrame.max`, :meth:`DataFrame.mean`, :meth:`DataFrame.any`, and :meth:`DataFrame.all` with
axis=1for multi-block DataFrames by avoiding a transpose (:issue:`51474`) - Performance improvement in :meth:`DataFrame.take`, :meth:`Series.take`, :meth:`DataFrame.reindex`, :meth:`Series.reindex`, and boolean-array indexing for NumPy-backed dtypes (:issue:`65295`)
- Performance improvement in :meth:`DataFrame.to_excel` with the
openpyxlengine when usingengine_kwargs={"write_only": True}, reducing memory consumption (:issue:`41681`) - Performance improvement in :meth:`DataFrame.to_hdf` and :meth:`HDFStore.append` for table format when appending to an existing wide table with many
data_columns(:issue:`25839`) - Performance improvement in :meth:`DataFrame.to_stata` when writing object-dtype datetime columns with date formats that require year/month extraction (:issue:`64555`)
- Performance improvement in :meth:`DataFrame.unstack` and :meth:`Series.unstack` when the :class:`MultiIndex` is already sorted and the unstacked level is the last level (:issue:`65107`)
- Performance improvement in :meth:`DatetimeIndex.month_name` and :meth:`DatetimeIndex.day_name` when using the default string dtype by using PyArrow compute instead of going through an intermediate object array (:issue:`65104`)
- Performance improvement in :meth:`DatetimeIndex.strftime` and :meth:`Series.dt.strftime` for formats composed of common directives (
%Y,%m,%d,%H,%M,%S,%f) (:issue:`44764`) - Performance improvement in :meth:`GroupBy.any` and :meth:`GroupBy.all` for boolean-dtype columns (:issue:`37850`)
- Performance improvement in :meth:`GroupBy.first` and :meth:`GroupBy.last` for Extension Array dtypes, which no longer fall back to a slow
apply-based implementation (:issue:`57591`) - Performance improvement in :meth:`GroupBy.quantile` (:issue:`64330`)
- Performance improvement in :meth:`GroupBy.size` (:issue:`51750`)
- Performance improvement in :meth:`HDFStore.select_as_multiple` when no
whereclause is given, by avoiding a coordinate-based read (:issue:`26771`) - Performance improvement in :meth:`Index.get_indexer` for large monotonic indexes, which now uses binary search instead of building a hash table when the number of targets is small (:issue:`14273`)
- Performance improvement in :meth:`Index.join` and :meth:`Index.union` for :class:`RangeIndex` by avoiding unnecessary memory allocation in the libjoin fastpath (:issue:`54646`)
- Performance improvement in :meth:`IntervalIndex.get_indexer` for monotonic non-overlapping indexes, which now uses binary search instead of the interval tree (:issue:`47614`)
- Performance improvement in :meth:`NDFrame.__finalize__`, :meth:`Series.to_numpy`, :attr:`DataFrame.dtypes`, and :meth:`DataFrame.__getitem__` (:issue:`57431`)
- Performance improvement in :meth:`Timedelta.total_seconds` (:issue:`65388`)
- Performance improvement in :meth:`arrays.SparseArray.isna` by avoiding a dense-then-resparsify round-trip (:issue:`41023`)
- Performance improvement in datetime/timedelta unit conversion (e.g.
datetime64[s]todatetime64[ns]) (:issue:`35025`) - Performance improvement in indexing a :class:`DataFrame` with a :class:`CategoricalIndex` of :class:`Interval` categories (:issue:`61928`)
- Performance improvement in indexing a :class:`MultiIndex` with a list-like indexer (:issue:`55786`)
- Performance improvement in partial-string indexing on a monotonic decreasing :class:`DatetimeIndex` or :class:`PeriodIndex` (:issue:`64811`)
- Performance improvement in plotting :class:`DatetimeIndex` with multiplied frequencies (e.g.
"1000ms","100s") (:issue:`50355`) - Performance improvement in reading zip-compressed files (e.g. :func:`read_pickle`, :func:`read_csv`) on Python < 3.12 (:issue:`59279`)
- Performance improvement in reductions along
axis=1and other operations on DataFrames produced by :meth:`DataFrame.copy` (:issue:`60469`) - Performance improvement in repr of :class:`Series` and :class:`DataFrame` containing third-party array-like objects (e.g. xarray
DataArray) in object dtype columns (:issue:`61809`) - Performance improvement in :meth:`DataFrame.loc` and :meth:`DataFrame.iloc` setitem with a 2D list-of-lists value by avoiding a wasteful round-trip through an intermediate object array (:issue:`64229`).
- Performance improvement in :meth:`Series.reindex` and :meth:`DataFrame.reindex` for non-nanosecond
datetime64andtimedelta64dtypes (:issue:`24566`) - Performance improvement in :meth:`Series.iloc` and :meth:`DataFrame.iloc` when setting datetimelike values into object-dtype data with list-like indexers (:issue:`64250`).
- Performance improvement in the :attr:`Series.dt` duration component accessors (
days,seconds,microseconds,nanoseconds,components, etc.) for :class:`ArrowDtype` durations by using PyArrow compute instead of converting to :class:`TimedeltaArray` (:issue:`63470`)
- Fixed bug in :class:`Index` repr where attributes were not wrapped to respect
display.width(:issue:`11552`) - Fixed bug in :func:`to_timedelta` and :class:`Timedelta` not accepting Day offsets (:issue:`64240`)
- Bug in :meth:`Categorical.__repr__` where the values and categories lines could exceed
display.width(:issue:`12066`) - Bug in :meth:`Categorical.map` where mapping with a
defaultdictandna_action=Nonewould bypass the default factory by usingdict.get, causingNAvalues to be replaced withNaNinstead of the mapper's default value (:issue:`62710`) - Bug in :meth:`CategoricalIndex.union` and :meth:`CategoricalIndex.intersection` giving incorrect results when the two indexes have the same unordered categories in different orders (:issue:`55335`)
- Bug in :meth:`Index.fillna` raising
TypeErrorwhen filling with a tuple value (e.g. on object-dtype or :class:`CategoricalIndex` with tuple categories) (:issue:`37681`)
- Bug in :class:`ArrowExtensionArray` where adding a :class:`DateOffset` to a
date32[pyarrow]ordate64[pyarrow]Series raised anArrowTypeError(:issue:`57168`) - Bug in :class:`DatetimeIndex` constructor raising
ValueErrorwhen passing equivalent but not equal frequencies (e.g.QS-FEBvsQS-MAY) (:issue:`61086`) - Bug in :class:`DatetimeIndex` raising
AttributeErrorwhen comparing against Arrow date types (date32, date64) (:issue:`62051`) - Bug in :class:`Timestamp` constructor where passing
np.str_objects would fail in Cython string parsing (:issue:`48974`) - Bug in :class:`Timestamp` constructor where strings with a negative year of fewer than 4 digits (e.g.
"-111-01-01") silently dropped the leading"-"and were parsed as a positive year; BC dates with 1-4 digit years now parse correctly, matching :class:`numpy.datetime64` (:issue:`55954`) - Bug in :class:`Timestamp` constructor, :class:`Timedelta` constructor, :func:`to_datetime`, and :func:`to_timedelta` with non-round
floatinput andunitfailing to raise when the value is just outside the representable bounds (:issue:`57366`) - Bug in :func:`api.types.infer_dtype` returning
"date"or"mixed"instead of"datetime"/"timedelta"for lists of :class:`Timestamp`/:class:`Timedelta` values mixed withpd.NA(:issue:`53023`) - Bug in :func:`date_range` where
inclusive="left"andinclusive="right"returned a single-element result instead of empty whenstartequalsend(:issue:`55293`) - Bug in :func:`date_range` where
inclusiveparameter failed to filter endpoints when onlystartandperiodsorendandperiodswere specified (:issue:`46331`) - Bug in :func:`date_range` where
periods=1with offsets that disallown=0(e.g. :class:`offsets.LastWeekOfMonth`, :class:`offsets.FY5253`) raisedValueError(:issue:`41563`) - Bug in :func:`date_range` where calendar-based offsets (e.g.
MS,ME,QS,YS) could exclude the last offset boundary whenend's time-of-day was earlier thanstart's (:issue:`35342`) - Bug in :func:`to_datetime` and :func:`to_timedelta` on ARM platforms where round
floatvalues outside the int64 domain (e.g.float(2**63)) could silently produce incorrect results instead of raising (:issue:`64619`) - Bug in :func:`to_datetime` and :func:`to_timedelta` where
uint64values greater thanint64max silently overflowed instead of raising :class:`OutOfBoundsDatetime` or :class:`OutOfBoundsTimedelta` (:issue:`60677`) - Bug in :func:`to_datetime` when using a low time resolution
unit, higher resolution inoriginis now preserved instead of silently dropped (e.g.unit="D"with microsecond precision origin) (:issue:`63419`) - Bug in :meth:`DataFrame.replace` and :meth:`Series.replace` raising
AssertionErrorinstead of :class:`OutOfBoundsDatetime` when replacing with adatetimevalue outside thedatetime64[ns]range (:issue:`61671`) - Bug in :meth:`DataFrame.to_string` and :meth:`Series.to_string` where
na_repwas ignored for datetime and timedelta columns, always displayingNaT(:issue:`55426`) - Bug in :meth:`DatetimeArray.isin` and :meth:`TimedeltaArray.isin` where mismatched resolutions could silently truncate finer-resolution values, leading to false matches (:issue:`64545`)
- Bug in adding non-nano :class:`DatetimeIndex` with non-vectorized offsets (e.g. :class:`CustomBusinessDay`, :class:`CustomBusinessMonthEnd`) having a sub-unit
offsetparameter incorrectly truncating the result or raisingAttributeError(:issue:`56586`) - Bug in subtracting :class:`BusinessHour` (or :class:`CustomBusinessHour`) from a :class:`Timestamp` giving incorrect results when the subtraction would land exactly on the business-hour opening time (:issue:`33682`)
- Bug in :attr:`TimedeltaIndex.resolution` raising when the index has no frequency (:issue:`65186`)
- Bug in :class:`DateOffset` where
DateOffset(1)andDateOffset(days=1)returned different results near daylight saving time transitions (:issue:`61862`) - Bug in :class:`Timedelta` constructor where keyword arguments (e.g.
days=365000) that exceeded nanosecond int64 bounds raisedOutOfBoundsTimedeltainstead of falling back to a coarser resolution (:issue:`46587`) - Bug in :func:`to_timedelta` where passing
np.str_objects would fail in Cython string parsing (:issue:`48974`) - Bug in :meth:`Series.sum` on an overflowing
timedelta64series raising a plainValueErrorinstead of :class:`OutOfBoundsTimedelta` (:issue:`43178`) - Bug in
Series.dt.secondsandSeries.dt.microsecondswith :class:`ArrowDtype` durations returning theSeries.dt.componentsfield values (e.g. 0-59 for seconds, or 0 formicrosecondson a"ms"unit) instead of the totals within each day and second respectively, inconsistent with NumPy-backed timedeltas (:issue:`63470`)
- Bug in :class:`DatetimeIndex` addition with a :class:`DateOffset` that has only timedelta components (e.g.
DateOffset(hours=-2)) raisingValueErrornear DST transitions, while scalar :class:`Timestamp` addition worked correctly (:issue:`28610`) - Bug in :meth:`Timestamp.to_julian_date` and :meth:`DatetimeIndex.to_julian_date` returning the Julian date of the local wall clock for timezone-aware inputs instead of the underlying UTC instant (:issue:`54763`)
- Fixed bug in :func:`read_excel` where having a column with mixture of numeric and boolean values will typecast the values based on the first appearance data type since 1==True and 0==False (:issue:`60088`)
- Fixed bug in :meth:`Series.clip` where passing a scalar numpy array (e.g.
np.array(0)) would raise aTypeError(:issue:`59053`) - Fixed bug in :meth:`Series.mean` and :meth:`Series.sum` (and their :class:`DataFrame` counterparts) overflowing for
float16dtypes instead of upcasting tofloat64(:issue:`43929`) - Fixed bug in :meth:`Series.skew` and :meth:`Series.kurt` (and their :class:`DataFrame` counterparts) returning
0.0for degenerate distributions; these now returnNaN(:issue:`62864`) - Fixed bug in complex-dtype :meth:`Series.duplicated` and :meth:`Series.unique` (and related hashtable-backed methods) raising
TypeErrorwhen backed by a :class:`NumpyExtensionArray` (:issue:`54761`) - Fixed bug where :class:`DataFrame` arithmetic operations with :class:`Series` did not support the fill_value parameter(:issue:`61581`)
- Bug in :class:`DataFrame` constructor where
NaTin a :class:`TimedeltaIndex` row was incorrectly inferred asdatetime64instead oftimedelta64(:issue:`23985`) - Bug in :class:`DataFrame` constructor where constructing from a list of uniform-dtype arrays (e.g. pyarrow, :class:`CategoricalDtype`, nullable dtypes) lost the dtype (:issue:`49593`)
- Bug in :func:`pd.array` silently converting NaN to a nonsensical integer when given float data containing NaN and a NumPy integer dtype (:issue:`41724`)
- Fixed :func:`pandas.array` to preserve mask information when converting NumPy masked arrays, converting masked values to missing values (:issue:`63879`)
- Fixed bug in :meth:`DataFrame.from_records` where
excludewas ignored whendatawas an iterator andnrows=0(:issue:`63774`)
- Bug in :meth:`DataFrame.replace` with
regex=Truemutating the underlying :class:`StringArray` when the replacement value was not a string (:issue:`57733`)
- Bug in :class:`IntervalArray` and :class:`IntervalIndex` constructors unnecessarily upcasting sub-64-bit numeric dtypes (e.g.
float32,int32) to 64-bit (:issue:`45412`)
- Bug in :meth:`DataFrame.loc` and :meth:`Series.loc` replacing the index name with the key's name when indexing with an :class:`Index` (:issue:`17110`)
- Bug in :meth:`DataFrame.loc` raising
ValueErrorwhen setting a row on a :class:`DataFrame` with no columns and the label is not in the index (:issue:`17895`) - Bug in :meth:`DataFrame.loc` returning incorrect dtype when the column key is a
slice(:issue:`63071`) - Bug in :meth:`Index.get_indexer` where
method="pad","backfill", or"nearest"returned incorrect results when the target containedNaTorNaNinstead of-1(:issue:`32572`) - Bugs in setitem-with-expansion when adding new rows failing to keep the original dtype in some cases (:issue:`32346`, :issue:`15231`, :issue:`47503`, :issue:`6485`, :issue:`25383`, :issue:`52235`, :issue:`17026`, :issue:`56010`)
- Bug in :meth:`DataFrame.__getitem__` raising
InvalidIndexErrorwhen indexing with a tuple containing asliceon a :class:`DataFrame` with :class:`MultiIndex` columns (e.g.,df[:, "t1"]) (:issue:`26511`) - Bug in :meth:`DataFrame.at` raising
TypeErrorwhen accessing a :class:`MultiIndex` with a partial date string on a :class:`DatetimeIndex` level (:issue:`43395`) - Bug in :meth:`DataFrame.duplicated` returning an empty :class:`Series` without the DataFrame's index when the DataFrame had no columns (:issue:`61191`)
- Bug in :meth:`DataFrame.iloc` setitem raising
AttributeErrorwhen assigning a :class:`Series` or :class:`Index` with a nullable EA dtype (e.g.Int64,Float64,boolean) into a column with a NumPy dtype (:issue:`47776`) - Bug in :meth:`DataFrame.loc` raising
ValueErrorwhen assigning a list of tuples to an object-dtype column with a boolean mask on a mixed-dtype DataFrame (:issue:`37629`) - Bug in :meth:`DataFrame.loc` raising
ValueErrorwhen setting a row with a list-like value on a single-column :class:`DataFrame` with :class:`ExtensionArray` dtype (:issue:`44103`) - Bug in :meth:`DataFrame.loc` setitem-with-expansion writing the truncated string
"n"(or raisingTypeError) into rows outside the indexer when adding a new column from a list of strings or booleans (:issue:`42099`) - Bug in :meth:`DataFrame.loc` with a :class:`MultiIndex` returning wrong results instead of raising
KeyErrorwhen passing string keys for numeric index levels (:issue:`60104`) - Bug in :meth:`DataFrame.mask` with
inplace=Truewhere incorrect values were produced whenotherwas a :class:`Series` with :class:`ExtensionArray` values (:issue:`64635`) - Bug in :meth:`DataFrame.rename` and :meth:`Series.rename` not preserving nullable extension dtype (e.g.
Int64,Float64) when relabeling index or column labels (:issue:`65315`) - Bug in :meth:`DataFrame.where` and :meth:`DataFrame.mask` raising
TypeErrorwhencondis a :class:`Series` andaxis=1(:issue:`58190`) - Bug in :meth:`DataFrame.xs` where
drop_level=Falsewas ignored for fully specified :class:`MultiIndex` keys whenlevelwas not explicitly provided (:issue:`6507`) - Bug in :meth:`Index.get_indexer_non_unique` raising
ZeroDivisionErrorinstead of returning an all-missing result when called on an empty index with a non-empty target (:issue:`54746`) - Bug in :meth:`Index.get_level_values` mishandling boolean, NA-like (
np.nan,pd.NA,pd.NaT) and integer index names (:issue:`62169`) - Bug in :meth:`Index.get_loc` raising
KeyErrorwhen looking up a tuple in an object-dtype :class:`Index` with duplicates (:issue:`37800`) - Bug in :meth:`Index.insert` silently casting booleans to numeric when used with nullable numeric dtypes like
Float64orInt64(:issue:`61709`) - Bug in :meth:`Index.take` where
fill_valuewas silently ignored and integer-dtype indexes raisedValueErrorinstead of filling with the provided value. Passingfill_value=Nonenow fills-1entries with the Index's NA value (matching the :class:`~pandas.api.extensions.ExtensionArray` convention); omitfill_valueto retain the previous behavior where negative indices wrap (:issue:`65210`) - Bug in :meth:`Index.where` and :meth:`Index.putmask` preserving
numpy.datetime64/numpy.timedelta64NaTscalars in the object-dtype result for mismatched-dtype inputs, instead of normalizing to :attr:`pandas.NaT` as :meth:`Series.where` does (:issue:`55174`) - Bug in :meth:`MultiIndex.get_loc` returning a slice instead of an integer for a unique key when the :class:`MultiIndex` contained duplicates elsewhere, causing
.locto return a :class:`Series` instead of a scalar (:issue:`42102`) - Bug in :meth:`Series.where` and :meth:`Series.mask` raising
ValueErrorwhenotheris a tuple on object-dtype :class:`Series` (:issue:`37681`) - Fixed bug in :meth:`DataFrame.loc` where assigning an iterable to a single cell in an
objectdtype column incorrectly raised aValueError(:issue:`57962`) - Fixed bug in :meth:`DataFrame.loc` where assigning with duplicate column names and new columns corrupted unrelated columns (:issue:`58317`)
- Fixed segfault in :meth:`DataFrame.loc` when repeatedly adding new rows to an object-dtype-indexed :class:`DataFrame` (:issue:`21968`)
- Bug in :meth:`DataFrame.fillna` with a dict value raising
RecursionErrorwhen columns are a :class:`MultiIndex` with duplicate entries (:issue:`53498`) - Bug in :meth:`Series.combine_first` crashing when Series names are :class:`Timestamp` objects (:issue:`65333`)
- Bug in :meth:`DataFrame.loc` with a :class:`MultiIndex` where using a tuple indexer with a scalar and a list (e.g.,
(scalar, list)) did not drop the scalar-indexed level (:issue:`18631`) - Bug in :meth:`MultiIndex.get_loc` where looking up a tuple key containing a scalar inside an :class:`IntervalIndex` level with overlapping intervals raised
KeyErroror returned incorrect results (:issue:`27456`) - Bug in :meth:`MultiIndex.sortlevel` not raising
TypeErrorwhen sorting a level with incomparable types (e.g.,Timestampandstr) (:issue:`21136`)
- :func:`read_csv` with
memory_map=Trueand an in-memory buffer (e.g.BytesIO) now raises a clearValueErrorinstead of a crypticUnsupportedOperation: fileno(:issue:`45630`) - Fixed bug in :func:`read_csv` with the
cengine where an embedded\rfollowed by a space in an unquoted field could cause an infinite re-parsing loop, producing spurious rows or a buffer overflow (:issue:`51141`) - Fixed bug in :func:`read_excel` where usage of
skiprowscould lead to an infinite loop (:issue:`64027`) - Fixed bug where :func:`read_html` parsed nested tables incorrectly when using
html5liborbs4flavors (:issue:`64524`) - Fixed :func:`read_json` with
lines=Trueandchunksizeto respectnrowswhen the requested row count is not a multiple of the chunk size (:issue:`64025`) - :meth:`HDFStore.put` and :meth:`HDFStore.append` now support storing :class:`Series` and :class:`DataFrame` columns with :class:`PeriodDtype` in both
"fixed"and"table"formats (:issue:`41978`) - Bug in :meth:`DataFrame.__repr__` raising
TypeErrorfor a column with a NumPy structured dtype (e.g. produced by :meth:`DataFrame.from_records` from a structuredndarray) (:issue:`55011`) - Bug in :meth:`DataFrame.__repr__` where horizontally truncated output could exceed the terminal width by up to 4 characters (:issue:`32461`)
- Bug in :meth:`DataFrame.to_stata` raising
KeyErrorwhen column names require renaming andconvert_datesis specified for a different column (:issue:`60536`) - Bug in :meth:`DataFrame.to_string` where
formattersdict was applied to wrong columns when output was horizontally truncated viamax_cols(:issue:`35410`) - Fixed :func:`read_json` with
lines=Trueandnrows=0to return an empty DataFrame (:issue:`64025`) - :meth:`DataFrame.to_hdf` now raises a clear :class:`NotImplementedError` when writing a column or :class:`Index` of an unsupported extension dtype (such as :class:`IntervalDtype`, :class:`SparseDtype`, or the nullable integer/float/boolean dtypes), instead of a low-level
AttributeErroror PyTablesTypeError(:issue:`26144`, :issue:`38305`, :issue:`42070`) - :func:`read_hdf` can again read fixed-format files written by very old pandas versions (
<=0.15.x) that stored afreqattribute on non-datetimelike indexes, which previously failed with aTypeErrororValueError(:issue:`33186`) - :meth:`DataFrame.to_hdf` with
format="fixed"now compressesobjectdtype (e.g. string) columns whencomplib/complevelare given; previously the compression settings were silently ignored for these columns, producing much larger files (:issue:`45286`) - :meth:`HDFStore.put`, :meth:`HDFStore.append`, and :meth:`DataFrame.to_hdf` now emit a
UserWarninginstead of silently doing nothing when writing an empty :class:`DataFrame` or :class:`Series` withformat='table'orappend=True(:issue:`13016`) - :meth:`HDFStore.select` and :func:`read_hdf` now warn when a nested
whereof the form"(A & B) | (C & D)"over indexed columns may return incorrect results because of an upstream PyTables bug, suggesting writing withindex=Falseor running theORbranches as separate queries (:issue:`50598`) - :meth:`HDFStore.select` now raises a clear
ValueErrorwith a workaround, instead of an opaquetoo many inputserror, when awhereexpression has too many comparisons for a query against indexed columns (:issue:`39752`) - :meth:`HDFStore.select` now raises an informative
NotImplementedErrorinstead of a crypticKeyErrorwhen a column selection such aswhere="columns=['A']"is combined withiterator=Trueorchunksize; pass thecolumnsargument instead (:issue:`12953`) - :meth:`HDFStore.select` now raises an informative
NotImplementedErrorwhen awhereclause contains an arithmetic expression such as"(A % 3) == 0", instead of an opaque PyTablesTypeError; arithmetic inwherefilters is not supported (:issue:`41100`) - Bug in :meth:`HDFStore.select` where a
wherequery on a categorical data column for a value that is not one of the categories incorrectly matched rows with missing (NaN) values (:issue:`22977`) - Fixed
MemoryErrorin :meth:`HDFStore.select` when iterating large tables withchunksizeand nowherefilter (:issue:`15937`) - Fixed bug in :func:`read_hdf` raising on files written by older pandas versions whose
freqindex attribute could not be decoded; thefreqis now dropped with a warning instead of corrupting the index (:issue:`35917`) - Fixed bug in :func:`read_hdf` where a categorical column containing a category equal to the
nan_repstring (e.g. the default"nan") raisedValueError: operands could not be broadcast togetherinstead of reading that category back asNaN(:issue:`21741`) - Fixed bug in :func:`read_hdf` where the literal string
"nan"in a string :class:`Index` was incorrectly converted toNaNon read, even when a customnan_repwas supplied (:issue:`9604`) - Fixed bug in :meth:`DataFrame.to_hdf` and :meth:`HDFStore.put` where writing an object to a key silently deleted any nested keys stored beneath it (:issue:`17267`)
- Fixed bug in :meth:`DataFrame.to_hdf` raising
TypeErrorwhen the index had a non-tick :class:`DateOffset`freq(e.g.DateOffset(years=1)) (:issue:`45790`) - Fixed bug in :meth:`DataFrame.to_hdf` with
format="table"where a :class:`TimedeltaIndex` was reconstructed as a :class:`PeriodIndex` (whenfreqwas set) or an integer :class:`Index` (otherwise) on read-back (:issue:`21466`) - Fixed :meth:`DataFrame.to_hdf` and :meth:`Series.to_hdf` to round-trip a :class:`CategoricalIndex` in both
"fixed"and"table"formats; previously raisedAssertionError(:issue:`33909`, :issue:`16118`) - Fixed bug in :meth:`DataFrame.to_parquet` (
pyarrowengine) where a local file path was opened twice, once by pandas and again by pyarrow, wasting a syscall and silently truncating output to 0 bytes on filesystems that finalize a file's contents on close (:issue:`65810`) - Fixed bug in :meth:`HDFStore.get_storer` where
.shapereported a phantom row for a fixed-format :class:`Series` or :class:`DataFrame` stored with no rows (:issue:`37235`) - Fixed bug in :meth:`HDFStore.remove` where a
whereclause selecting on more than 31 values (e.g."index in [...]") deleted every row in the table instead of only the matching rows (:issue:`17567`) - Fixed bug in :meth:`HDFStore.select` where passing
whereas a list of conditions referencing caller-scope variables failed on Python 3.12+ due to PEP 709 inlining list comprehension stack frames (:issue:`64881`) - Fixed bug in :meth:`HDFStore.select` with
format="table"where reading a frame with a string :class:`Index` could crash with a bus error on strict-alignment platforms such as 32-bit ARM (:issue:`54396`) - Storing a :class:`DataFrame` or :class:`Series` with a :class:`MultiIndex` level named
'index'via :meth:`HDFStore.put` or :meth:`HDFStore.append` withformat='table'now raises a clearValueErrorinstead of an opaque reshape error (:issue:`6208`) - The :class:`~pandas.errors.PerformanceWarning` emitted by :meth:`DataFrame.to_hdf` for object columns now names only the columns that cannot be mapped to a c-type, instead of every object column sharing the same block (:issue:`28460`)
- Writing a :class:`DataFrame` with
format='table'and a column named'index'as adata_columnsentry (includingdata_columns=True) now raises a clearValueErrorinstead of an opaque reshape error (:issue:`41437`)
- Bug in :class:`Period` constructor where passing
np.str_objects would fail in Cython string parsing (:issue:`48974`) - Bug in :meth:`Period.strftime` where unknown format directives (e.g.
"%Q") silently produced platform-dependent output and crashed the Python process on Windows; anInvalid format stringValueErroris now raised on all platforms (:issue:`53562`)
- Bug in :meth:`DataFrame.plot.hexbin` ignoring
rcParams["image.cmap"]and always defaulting to"BuGn"when no colormap was specified (:issue:`31871`)
- Bug in :meth:`.DataFrameGroupBy.agg` when there are no groups, multiple keys, and
group_keys=False(:issue:`51445`) - Bug in :meth:`.DataFrameGroupBy.agg` would operate on the group as a whole when
argsorkwargsare supplied for the providedfunc; now this method only operates on each Series of the group (:issue:`39169`) - Bug in :meth:`.DataFrameGroupBy.apply` with
as_index=Falsewhere applying on an empty :class:`DataFrame` returned inconsistent index metadata compared to non-empty results (:issue:`48135`) - Bug in :meth:`.DataFrameGroupBy.cumprod`, :meth:`.DataFrameGroupBy.cummin`, and :meth:`.DataFrameGroupBy.cummax` (and Series variants) returning
Float64instead of preserving the nullable integer dtype (e.g.Int64) when the group key containsNA(:issue:`65550`) - Bug in :meth:`.GroupBy.any` and :meth:`.GroupBy.all` returning
NaNwithfloat64dtype for unobserved categorical groups on NumPybooldata instead of the boolean identity value withbooldtype (:issue:`65100`) - Bug in :meth:`.Resampler.agg` raising
ValueErrorwith a dict of aggregations when applied to a :meth:`DataFrame.groupby` withas_index=False(:issue:`52397`) - Bug in :meth:`.Rolling.corr` and :meth:`.Rolling.cov` computing incorrect results on degenerate windows (:issue:`24019`)
- Bug in :meth:`.Rolling.skew` and :meth:`.Rolling.kurt` (and their :class:`GroupBy` counterparts) returning
0.0and-3.0respectively for degenerate windows or groups; these now returnNaN(:issue:`62864`) - Bug in :meth:`.Rolling.skew` and :meth:`.Rolling.kurt` returning
NaNfor low-variance windows (:issue:`62946`) - Bug in :meth:`.Rolling.sum`, :meth:`.Rolling.mean`, :meth:`.Rolling.median`, :meth:`.Rolling.min`, and :meth:`.Rolling.max` with
method="table",engine="numba", andengine_kwargs={"parallel": True}could cause a segfault (:issue:`40454`) - Bug in :meth:`.SeriesGroupBy.ohlc` ignoring
as_index=False(:issue:`65140`) - Bug in :meth:`DataFrame.groupby` with a :class:`Grouper` with
freqraisingAttributeErrorwhen all grouping keys areNaT(:issue:`43486`) - Bug in :meth:`Series.resample` and :meth:`DataFrame.resample` where same-frequency resampling with monthly, quarterly, or annual frequencies bypassed aggregation, returning the original values instead of the aggregation result (:issue:`18553`)
- Bug in :func:`merge` where merging on a :class:`MultiIndex` containing
NaNvalues mappedNaNkeys to the last level value instead ofNaN(:issue:`64492`) - Bug in :meth:`DataFrame.pivot_table` with
margins=TrueraisingTypeErrorwhenvalueshas an :class:`ExtensionDtype` that cannot holdNA(e.g. :class:`IntervalDtype` with an integer subtype) and nocolumnswere specified (:issue:`55484`) - Bug in :meth:`Index.union` where the result could be unsorted when both inputs were monotonic increasing but disjoint, when
sortwas notFalse(:issue:`54646`) - Fixed bug in :meth:`Series.sort_values` where
ignore_index=Truehad no effect on an already-sorted Series (:issue:`65833`) - In :func:`pivot_table`, when
valuesis empty, the aggregation will be computed on a Series of all NA values (:issue:`46475`)
- Bug in :meth:`SparseArray.astype` where converting a datetime64 :class:`SparseArray` with
NaTfill value to"Sparse[int64]"silently replaced the fill value with0instead ofiNaT(:issue:`49631`) - Bug in indexing a :class:`SparseArray` with an out-of-bounds integer with the value of the length of the array returning the fill value instead of raising an
IndexError(:issue:`64183`).
- Bug in numpy ufuncs like :func:`numpy.isnan` raising
TypeErroron :class:`Series` or :class:`Index` backed by PyArrow dtypes whenfuture.distinguish_nan_and_naisTrue(:issue:`62506`) - Fixed bug in :meth:`Series.apply` and :meth:`Series.map` where nullable integer dtypes were converted to float, causing precision loss for large integers; now the nullable dtype will be preserved (:issue:`63903`).
- Fixed the :attr:`~Series.is_monotonic_increasing` and :attr:`~Series.is_monotonic_decreasing` properties to dispatch to the underlying :class:`ExtensionArray` implementation (:issue:`65585`)
- Bug in :meth:`DataFrame.eval` where a duplicate column name was resolved to a single column (yielding a :class:`Series`), inconsistent with :func:`pandas.eval` using
resolvers=(df,)and with :meth:`DataFrame.__getitem__`, which include every column with that label (:issue:`65588`) - Bug in :meth:`Series.transform` and :meth:`DataFrame.transform` where passing a list of duplicate function names did not raise :class:`errors.SpecificationError` (:issue:`54929`)