PERF: add lib.has_sentinel for integer-indexer sentinel scans#65923
Open
jbrockmendel wants to merge 2 commits into
Open
PERF: add lib.has_sentinel for integer-indexer sentinel scans#65923jbrockmendel wants to merge 2 commits into
jbrockmendel wants to merge 2 commits into
Conversation
Add a fused-type (int8/16/32/64) Cython helper has_sentinel(arr, sentinel), a short-circuiting, allocation-free equivalent of (arr == sentinel).any() with an 8x-unrolled inner loop, and wire up the clear (indexer == -1).any() call sites in merge (non-inner), single-level unstack, groupby key lifting, MultiIndex NaN-key indexing, and DataFrame.__setitem__ with non-unique columns. Split out from GH#65298. The unrolling of the pre-existing has_infs / is_range_indexer / is_sequence_range helpers is a separate, independent PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 tasks
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out from GH-65298. The other piece — unrolling the pre-existing
has_infs/is_range_indexer/is_sequence_rangehelpers — is independent and will be its own PR.Summary
Adds
lib.has_sentinel(arr, sentinel), a fused-type (int8/16/32/64) Cython helper that is a short-circuiting, allocation-free equivalent of(arr == sentinel).any()on integer indexers, with an 8x-unrolled inner loop. Wires up the clear(indexer == -1).any()/(ilocs < 0).any()call sites:_MergeOperation._maybe_add_join_keys(non-inner merges)_Unstacker.new_index(single-level unstack)sorting.get_group_index/decons_obs_group_ids(key lifting)MultiIndex._get_indexer_strictNaN-key pathDataFrame.__setitem__non-unique-columns pathBenchmarks
Micro,
has_sentinel(arr, -1)vs(arr == -1).any(), int64, best-of-9:(==-1).any()has_sentinelThe gains come from short-circuiting (large when a sentinel sits early) and from never allocating the N-element boolean array (dominant for small / overhead-bound arrays). On a large full scan with no early match the 8-wide scalar unroll trails numpy's SIMD comparison — most visible on narrow dtypes.
End-to-end this is roughly neutral on the operations that call it: each calls the helper O(1) times, so a single scan is a negligible fraction of the operation. No whatsnew entry for that reason. (Plain
GroupByaggregations don't reach the helper — they go throughget_group_index(..., xnull=True).)Test plan
pandas/tests/libs/test_lib.py: correctness across int8/16/32/64, every-position coverage around the unroll boundary, and empty inputreshape/merge,frame/indexing/test_setitem,indexes/multi/test_indexing,test_sorting,frame/test_stack_unstack,reshape/test_crosstab,groupby/) — all green