BUG: ewm(adjust=False) wrong result with com=1 and NaN (GH#31178)

pandas-devGH-59142 added a `com == 1` special case to the ewm loop to handle
irregular intervals when `times` is provided, but did not gate it on
`times` actually being given, so it leaked into the equally-spaced path.
With `adjust=False`, `ignore_na=False` and NaN present, com=1 (equivalently
span=3, alpha=0.5, halflife=1) then returned a value discontinuous in
alpha: `Series([1, NaN, 5]).ewm(com=1, adjust=False).mean()` gave 4.0
rather than 3.6667, while com=1±eps gave the correct result.

Gate the branch on whether `times` was provided. The cython loop already
derives this as `deltas is not None`; both numba generators always receive
`deltas`, so thread a `use_deltas` flag through from `ewm.py`.

The `com == 1` term is dropped: when `times` is provided with
`adjust=False`, `ExponentialMovingWindow.__init__` rejects com/span/alpha
and forces `_com = 1.0`, so the test was invariantly true under the new
gate and only served to encode the com-dependence that caused this bug.

Also document the `adjust=False` + `ignore_na` recursion in window.rst,
which was the original 2020 question in the same issue.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: jbrockmendel

doc/source/user_guide/window.rst

@@ -612,6 +612,30 @@ Whereas if ``ignore_na=True``, the weighted average would be calculated as
 
         \frac{(1-\alpha) \cdot 3 + 1 \cdot 5}{(1-\alpha) + 1}.
 
+``ignore_na`` makes the same distinction when ``adjust=False``, except that the
+decay is applied to the running average rather than to each observation. For
+equally spaced points (that is, when ``times`` is not supplied), letting
+:math:`u` be the index of the most recent non-null observation before :math:`t`,
+the recursion with ``ignore_na=False`` is
+
+.. math::
+
+    y_t = \frac{(1-\alpha)^{t-u} y_u + \alpha x_t}{(1-\alpha)^{t-u} + \alpha},
+
+so the weighted average of ``3, NaN, 5`` is
+
+.. math::
+
+        \frac{(1-\alpha)^2 \cdot 3 + \alpha \cdot 5}{(1-\alpha)^2 + \alpha}.
+
+With ``ignore_na=True`` the intermediate null does not count towards the
+exponent, so it is always :math:`1` and the recursion reduces to
+:math:`y_t = (1-\alpha) y_u + \alpha x_t`, giving
+
+.. math::
+
+        (1-\alpha) \cdot 3 + \alpha \cdot 5.
+
 The :meth:`~ExponentialMovingWindow.var`, :meth:`~ExponentialMovingWindow.std`, and :meth:`~ExponentialMovingWindow.cov` functions have a ``bias`` argument,
 specifying whether the result should contain biased or unbiased statistics.
 For example, if ``bias=True``, ``ewmvar(x)`` is calculated as

doc/source/whatsnew/v3.1.0.rst

@@ -580,6 +580,7 @@ Groupby/resample/rolling
 - Bug in :meth:`.Rolling.skew` and :meth:`.Rolling.kurt` returning ``NaN`` for low-variance windows (:issue:`62946`)
 - Bug in :meth:`.Rolling.sum`, :meth:`.Rolling.mean`, :meth:`.Rolling.median`, :meth:`.Rolling.min`, and :meth:`.Rolling.max` with ``method="table"``, ``engine="numba"``, and ``engine_kwargs={"parallel": True}`` could cause a segfault (:issue:`40454`)
 - Bug in :meth:`.SeriesGroupBy.ohlc` ignoring ``as_index=False`` (:issue:`65140`)
+- Bug in :meth:`DataFrame.ewm` and :meth:`Series.ewm` with ``adjust=False`` and ``ignore_na=False`` returning incorrect results for data containing ``NaN`` when the decay was given as ``com=1`` (equivalently ``span=3``, ``alpha=0.5``, or ``halflife=1``), a regression in 3.0.0; the result no longer differs from that of a decay an arbitrarily small amount away (:issue:`31178`)
 - Bug in :meth:`DataFrame.groupby` with a :class:`Grouper` with ``freq`` raising ``AttributeError`` when all grouping keys are ``NaT`` (:issue:`43486`)
 - Bug in :meth:`DataFrame.resample` and :meth:`Series.resample` with a timezone-naive index where using a ``Day`` frequency (e.g. ``"7D"``) produced different bin edges than the equivalent ``Hour`` frequency (e.g. ``"168h"``), and where ``origin`` and ``offset`` were ignored (:issue:`44996`, :issue:`62200`)
 - Bug in :meth:`DataFrame.resample` dropping the result index name when resampling ``on`` a column with a pyarrow-backed datetime or duration dtype (:issue:`59823`)

pandas/_libs/window/aggregations.pyx

@@ -2125,8 +2125,10 @@ def ewm(const float64_t[:] vals, const int64_t[:] start, const int64_t[:] end,
                             if normalize:
                                 # avoid numerical errors on constant series
                                 if weighted != cur:
-                                    if not adjust and com == 1:
-                                        # update in case of irregular-interval series
+                                    if use_deltas and not adjust:
+                                        # times were provided: weight by the
+                                        # elapsed interval so old_wt and new_wt
+                                        # sum to 1 (GH#31178)
                                         new_wt = 1. - old_wt
                                     weighted = old_wt * weighted + new_wt * cur
                                     weighted /= (old_wt + new_wt)

pandas/core/window/ewm.py

@@ -602,6 +602,7 @@ def mean(
                 adjust=self.adjust,
                 ignore_na=self.ignore_na,
                 deltas=tuple(self._deltas),
+                use_deltas=self.times is not None,
                 normalize=True,
             )
             return self._apply(ewm_func, name="mean")
@@ -693,6 +694,7 @@ def sum(
                 adjust=self.adjust,
                 ignore_na=self.ignore_na,
                 deltas=tuple(self._deltas),
+                use_deltas=self.times is not None,
                 normalize=False,
             )
             return self._apply(ewm_func, name="sum")

pandas/core/window/numba_.py

@@ -83,6 +83,7 @@ def generate_numba_ewm_func(
     adjust: bool,
     ignore_na: bool,
     deltas: tuple,
+    use_deltas: bool,
     normalize: bool,
 ):
     """
@@ -99,6 +100,9 @@ def generate_numba_ewm_func(
     adjust : bool
     ignore_na : bool
     deltas : tuple
+    use_deltas : bool
+        Whether `deltas` reflects user-provided `times`. False means the points
+        are equally spaced and `deltas` is all ones.
     normalize : bool
 
     Returns
@@ -143,8 +147,9 @@ def ewm(
                             # note that len(deltas) = len(vals) - 1 and deltas[i]
                             # is to be used in conjunction with vals[i+1]
                             old_wt *= old_wt_factor ** deltas[start + j - 1]
-                            if not adjust and com == 1:
-                                # update in case of irregular-interval time series
+                            if use_deltas and not adjust:
+                                # times were provided: weight by the elapsed interval
+                                # so old_wt and new_wt sum to 1 (GH#31178)
                                 new_wt = 1.0 - old_wt
                         else:
                             weighted = old_wt_factor * weighted
@@ -265,6 +270,7 @@ def generate_numba_ewm_table_func(
     adjust: bool,
     ignore_na: bool,
     deltas: tuple,
+    use_deltas: bool,
     normalize: bool,
 ):
     """
@@ -281,6 +287,9 @@ def generate_numba_ewm_table_func(
     adjust : bool
     ignore_na : bool
     deltas : tuple
+    use_deltas : bool
+        Whether `deltas` reflects user-provided `times`. False means the points
+        are equally spaced and `deltas` is all ones.
     normalize: bool
 
     Returns
@@ -319,8 +328,9 @@ def ewm_table(
                             # note that len(deltas) = len(vals) - 1 and deltas[i]
                             # is to be used in conjunction with vals[i+1]
                             old_wt[j] *= old_wt_factor ** deltas[i - 1]
-                            if not adjust and com == 1:
-                                # update in case of irregular-interval time series
+                            if use_deltas and not adjust:
+                                # times were provided: weight by the elapsed interval
+                                # so old_wt and new_wt sum to 1 (GH#31178)
                                 new_wt = 1.0 - old_wt[j]
                         else:
                             weighted[j] = old_wt_factor * weighted[j]

pandas/tests/window/test_ewm.py

@@ -93,6 +93,10 @@ def test_ewma_halflife_without_times(halflife_with_times):
 )
 @pytest.mark.parametrize("min_periods", [0, 2])
 def test_ewma_with_times_equal_spacing(halflife_with_times, times, min_periods):
+    # Deliberately not parametrized over adjust: with adjust=False and
+    # ignore_na=False the times path weights NaN gaps as a convex combination
+    # over the elapsed interval, which differs from the no-times recursion
+    # (GH#31178)
     halflife = halflife_with_times
     data = np.arange(10.0)
     data[::2] = np.nan
@@ -437,6 +441,29 @@ def test_ewma_nan_handling_cases(s, adjust, ignore_na, w):
         tm.assert_series_equal(result, expected)
 
 
+@pytest.mark.parametrize(
+    "kwargs", [{"com": 1}, {"span": 3}, {"alpha": 0.5}, {"halflife": 1}]
+)
+def test_ewma_nan_handling_adjust_false_com1(kwargs):
+    # GH#31178 com=1 was special-cased for the times= path, but the branch was
+    # not gated on times being provided, so it leaked into the equally-spaced
+    # path and returned 4.0 here.
+    ser = Series([1.0, np.nan, 5.0])
+    result = ser.ewm(adjust=False, ignore_na=False, **kwargs).mean()
+    # alpha = 0.5, so y[2] = (0.5**2 * 1 + 0.5 * 5) / (0.5**2 + 0.5)
+    expected = Series([1.0, 1.0, 11 / 3])
+    tm.assert_series_equal(result, expected)
+
+
+def test_ewma_nan_handling_adjust_false_com1_continuous():
+    # GH#31178 com=1 must not be discontinuous in alpha
+    ser = Series([1.0, np.nan, 5.0])
+    result = ser.ewm(com=1, adjust=False, ignore_na=False).mean()
+    for com in [1 - 1e-9, 1 + 1e-9]:
+        neighbor = ser.ewm(com=com, adjust=False, ignore_na=False).mean()
+        tm.assert_series_equal(result, neighbor, atol=1e-8, rtol=0)
+
+
 def test_ewm_alpha():
     # GH 10789
     arr = np.random.default_rng(2).standard_normal(100)

pandas/tests/window/test_numba.py

@@ -337,8 +337,23 @@ def test_cython_vs_numba(self, grouper, method, nogil, parallel, ignore_na, adju
 
         tm.assert_frame_equal(result, expected)
 
+    @pytest.mark.parametrize("method", ["single", "table"])
+    def test_ewm_nan_adjust_false_com1(self, method, nogil, parallel):
+        # GH#31178 com=1 was special-cased in both numba ewm generators without
+        # gating on times being provided. Checked against the expected value
+        # rather than the cython engine, which had the same bug.
+        df = DataFrame({"B": [1.0, np.nan, 5.0]})
+        ewm = df.ewm(com=1, adjust=False, ignore_na=False, method=method)
+
+        engine_kwargs = {"nogil": nogil, "parallel": parallel}
+        result = ewm.mean(engine="numba", engine_kwargs=engine_kwargs)
+        # alpha = 0.5, so y[2] = (0.5**2 * 1 + 0.5 * 5) / (0.5**2 + 0.5)
+        expected = DataFrame({"B": [1.0, 1.0, 11 / 3]})
+
+        tm.assert_frame_equal(result, expected)
+
     @pytest.mark.parametrize("grouper", ["None", "groupby"])
-    def test_cython_vs_numba_times(self, grouper, nogil, parallel, ignore_na):
+    def test_cython_vs_numba_times(self, grouper, nogil, parallel, ignore_na, adjust):
         # GH 40951
 
         df = DataFrame({"B": [0, 0, 1, 1, 2, 2]})
@@ -360,7 +375,7 @@ def test_cython_vs_numba_times(self, grouper, nogil, parallel, ignore_na):
             ]
         )
         ewm = grouper(df).ewm(
-            halflife=halflife, adjust=True, ignore_na=ignore_na, times=times
+            halflife=halflife, adjust=adjust, ignore_na=ignore_na, times=times
         )
 
         engine_kwargs = {"nogil": nogil, "parallel": parallel}