Support aggregate rolling expressions inside grouped over - #23627
Support aggregate rolling expressions inside grouped over#23627rjzamora wants to merge 8 commits into
Conversation
github-actions
Bot
added
Python
rjzamora
added
feature request
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
python/cudf_polars/tests/expressions/test_rolling.py (1)
208-208: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winTest restoration to the original input-row order.
These fixtures sort rows by group and rolling index. This does not exercise the required backward remap after grouped rolling evaluation. Use interleaved groups while keeping each group's index ordered, such as
A0, B0, A1, B2, A3, B3. Keepcheck_row_order=Truefor both execution paths.
python/cudf_polars/tests/expressions/test_rolling.py#L208-L208: use interleavedricrows and enablecheck_row_order=True.python/cudf_polars/tests/expressions/test_rolling.py#L241-L241: use interleavedgrows and enablecheck_row_order=True.python/cudf_polars/tests/streaming/test_rolling.py#L127-L127: use interleavedricrows instead of globally sorting byric.python/cudf_polars/tests/streaming/test_rolling.py#L163-L163: use interleavedgrows instead of globally sorting byg.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/cudf_polars/tests/expressions/test_rolling.py` at line 208, Restore interleaved input-row ordering to exercise backward remapping after grouped rolling evaluation: in python/cudf_polars/tests/expressions/test_rolling.py lines 208-208 and 241-241, use interleaved ric/g rows with each group’s index ordered and enable check_row_order=True for both execution paths; in python/cudf_polars/tests/streaming/test_rolling.py lines 127-127 and 163-163, use interleaved ric/g rows instead of globally sorting by the group column.python/cudf_polars/cudf_polars/dsl/expressions/rolling.py (1)
775-874: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider caching the resolved rolling index name.
_rolling_orderby_nameruns at Line 785 and again at Line 1488 for the samenamed_exprs. You can store the name onRollingWindowOpat dispatch time and read it here. This removes the duplicate validation pass and keeps one source of truth for the index column.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/cudf_polars/cudf_polars/dsl/expressions/rolling.py` around lines 775 - 874, Cache the resolved rolling index name on RollingWindowOp during dispatch, where _rolling_orderby_name is already evaluated for the same named_exprs. Update the _apply_unary_op handler for RollingWindowOp to read that cached name instead of calling _rolling_orderby_name again, keeping validation and index-column selection centralized.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@python/cudf_polars/cudf_polars/dsl/expressions/rolling.py`:
- Around line 775-874: Cache the resolved rolling index name on RollingWindowOp
during dispatch, where _rolling_orderby_name is already evaluated for the same
named_exprs. Update the _apply_unary_op handler for RollingWindowOp to read that
cached name instead of calling _rolling_orderby_name again, keeping validation
and index-column selection centralized.
In `@python/cudf_polars/tests/expressions/test_rolling.py`:
- Line 208: Restore interleaved input-row ordering to exercise backward
remapping after grouped rolling evaluation: in
python/cudf_polars/tests/expressions/test_rolling.py lines 208-208 and 241-241,
use interleaved ric/g rows with each group’s index ordered and enable
check_row_order=True for both execution paths; in
python/cudf_polars/tests/streaming/test_rolling.py lines 127-127 and 163-163,
use interleaved ric/g rows instead of globally sorting by the group column.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 73728fb8-69f4-482b-a0e8-722b8031ffc0
📒 Files selected for processing (5)
python/cudf_polars/cudf_polars/dsl/expressions/rolling.pypython/cudf_polars/cudf_polars/dsl/translate.pypython/cudf_polars/cudf_polars/dsl/utils/aggregations.pypython/cudf_polars/tests/expressions/test_rolling.pypython/cudf_polars/tests/streaming/test_rolling.py
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe change adds grouped range-based rolling aggregates in cuDF-Polars. It introduces ordered input gathering, shared unary-operation execution, dependency handling, aggregation decomposition, and expression and streaming test coverage. ChangesGrouped rolling support
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
python/cudf_polars/tests/expressions/test_rolling.py (1)
233-250: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd coverage for the new branches.
These two tests cover the happy path only. The new grouped range-rolling code adds branches that no test exercises:
- The null-index
RuntimeErrorin theRollingWindowOphandler.- The
NotImplementedErrorfrom_rolling_orderby_namewhen two rolling expressions use different index columns.- The
INT64cast for a non-Int64integral index. Thetscolumn here isInt64, so the cast is skipped.Add cases with a nullable index column, with two different index columns in one
select, and with anInt32/UInt32index.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/cudf_polars/tests/expressions/test_rolling.py` around lines 233 - 250, Extend test_rolling_common_aggs_over coverage to exercise the new RollingWindowOp error branches and index casting: add a nullable index case that expects the null-index RuntimeError, a select containing rolling expressions ordered by different index columns that expects _rolling_orderby_name’s NotImplementedError, and an equivalent case using Int32 or UInt32 index data to verify the INT64 cast path.python/cudf_polars/tests/streaming/test_rolling.py (1)
105-108: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the repeated skip condition.
The same
skipifcondition and reason appear twice.python/cudf_polars/tests/expressions/test_rolling.pyalready uses a named marker constant for the identical condition. Define one constant in this module and reuse it.♻️ Proposed refactor
+skip_rolling_expr_136_to_138 = pytest.mark.skipif( + not POLARS_VERSION_LT_136 and POLARS_VERSION_LT_139, + reason="Rolling window expressions are not accessible in polars 1.36-1.38", +) + + -@pytest.mark.skipif( - not POLARS_VERSION_LT_136 and POLARS_VERSION_LT_139, - reason="Rolling window expressions are not accessible in polars 1.36-1.38", -) +@skip_rolling_expr_136_to_138 def test_rolling_sum_over(engine):Apply the same replacement above
test_rolling_common_aggs_over.Also applies to: 152-155
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/cudf_polars/tests/streaming/test_rolling.py` around lines 105 - 108, Define a named skip marker constant in the rolling test module for the existing POLARS_VERSION condition and reason, matching the established constant used by the expressions rolling tests. Replace both repeated skipif decorators, including the one above test_rolling_common_aggs_over, with that shared constant.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/cudf_polars/cudf_polars/dsl/expressions/rolling.py`:
- Around line 1487-1500: Update the range-rolling branch in the surrounding
window-expression implementation to detect when a range-rolling expression is
combined with an over-clause order_by, using the existing order-direction and
null-ordering fields (over_ob_desc and over_ob_nulls_last). Raise
NotImplementedError for that combination unless _apply_ordered_unary_op is
updated to honor those ordering semantics; preserve the current rolling-index
ordering for cases without over(order_by=...).
In `@python/cudf_polars/cudf_polars/dsl/translate.py`:
- Around line 1219-1244: Update the dependency collection loop for named
aggregations to handle FixedSizeRollingWindow values explicitly. Add
v.children[0] to child_deps so FixedSizeRollingOp receives its input column
after projection, while preserving the existing RollingWindow and aggregation
handling.
In `@python/cudf_polars/cudf_polars/dsl/utils/aggregations.py`:
- Around line 166-177: Extend the nested-window guard in the RollingWindow
translation branch to also detect an expr.RollingWindow within agg.children[0],
alongside _contains_window_only_unary and _contains_fixed_size_rolling_window.
Raise NotImplementedError during translation so nested range rolling follows the
existing unsupported-operation fallback instead of reaching
RollingWindow.do_evaluate.
---
Nitpick comments:
In `@python/cudf_polars/tests/expressions/test_rolling.py`:
- Around line 233-250: Extend test_rolling_common_aggs_over coverage to exercise
the new RollingWindowOp error branches and index casting: add a nullable index
case that expects the null-index RuntimeError, a select containing rolling
expressions ordered by different index columns that expects
_rolling_orderby_name’s NotImplementedError, and an equivalent case using Int32
or UInt32 index data to verify the INT64 cast path.
In `@python/cudf_polars/tests/streaming/test_rolling.py`:
- Around line 105-108: Define a named skip marker constant in the rolling test
module for the existing POLARS_VERSION condition and reason, matching the
established constant used by the expressions rolling tests. Replace both
repeated skipif decorators, including the one above
test_rolling_common_aggs_over, with that shared constant.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 7a52a88e-8ead-4d01-8f1e-7d994923a59a
📒 Files selected for processing (5)
python/cudf_polars/cudf_polars/dsl/expressions/rolling.pypython/cudf_polars/cudf_polars/dsl/translate.pypython/cudf_polars/cudf_polars/dsl/utils/aggregations.pypython/cudf_polars/tests/expressions/test_rolling.pypython/cudf_polars/tests/streaming/test_rolling.py
| if rolling_named := unary_window_ops["range_rolling"]: | ||
| orderby_name = self._rolling_orderby_name(rolling_named) | ||
| rolling_order_by_col = df.column_map[orderby_name] | ||
| broadcasted_cols.extend( | ||
| self._reorder_to_input( | ||
| row_id, | ||
| self._apply_ordered_unary_op( | ||
| RollingWindowOp(named_exprs=rolling_named), | ||
| df, | ||
| grouper, | ||
| by_cols, | ||
| df.num_rows, | ||
| tables, | ||
| names, | ||
| dtypes, | ||
| order_index=order_index, | ||
| stream=df.stream, | ||
| row_id, | ||
| order_by_col=rolling_order_by_col, | ||
| require_sorted_groups=True, | ||
| ) | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
polars over order_by combined with rolling expression semantics
💡 Result:
In Polars, over and rolling represent two distinct paradigms for windowing operations, and they interact with order_by differently [1][2][3]. The over expression is used for window functions that operate over defined groups, where the order_by argument allows you to impose a specific sequence within those groups before the calculation is performed [1]. This is essential for operations that are sensitive to row order, such as cum_sum, cum_min, cum_max, or shift [1][4]. When using over(..., order_by=...), Polars conceptually sorts the data within each partition according to the specified columns, applies the expression, and maps the result back to the original row positions [1]. In contrast, the rolling expression is specialized for time-series or index-based interval calculations [3]. It is explicitly designed for window operations where the window size is defined by a period (e.g., "3d", "5h") or an integer range based on an index_column [3]. Key distinctions and interaction points include: 1. Order Sensitivity: over relies on order_by to handle sequence-dependent operations [1]. rolling requires the provided index_column to be pre-sorted in ascending order [3]. 2. Semantics: over is typically used for cumulative, ranking, or shift operations across fixed groups [1][5]. rolling is used to aggregate over a moving temporal or integer-based window [3]. 3. Combining Logic: Because rolling and over serve different structural purposes, they are not typically nested as "a rolling expression inside an over". If you need to perform a rolling aggregation within groups, you would generally use group_by in combination with rolling or standard windowing logic, or rely on over for functions that support cumulative definitions [1][3]. 4. SQL Compatibility: Recent updates have improved Polars' over clause to better align with SQL-style windowing semantics (e.g., handling of NULLs in cumulative operations) when order_by is specified, making over behave more like standard SQL OVER (ORDER BY ...) clauses [6][4]. In summary, use over with order_by when you need to define an explicit row sequence for group-local operations like cumulative sums [1], and use rolling when you need to compute aggregations based on temporal or continuous index intervals [3].
Citations:
- 1: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.over.html
- 2: https://docs.pola.rs/api/python/stable/reference/expressions/window.html
- 3: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling.html
- 4: pola-rs/polars@5ad3d2e
- 5: https://docs.pola.rs/user-guide/expressions/window-functions/
- 6: fix: Respect SQL semantics for cumulative functions mapped via
OVERclause pola-rs/polars#26570
🏁 Script executed:
#!/bin/bash
set -eu
file="python/cudf_polars/cudf_polars/dsl/expressions/rolling.py"
printf '%s\n' '--- target implementation ---'
sed -n '1420,1525p' "$file"
printf '%s\n' '--- order_by and range_rolling references ---'
rg -n -C 5 '_order_by_expr|over_ob_desc|over_ob_nulls_last|range_rolling|_rolling_orderby_name|require_sorted_groups' \
python/cudf_polars/cudf_polars python/cudf_polars/tests tests 2>/dev/null || true
printf '%s\n' '--- rolling-related tests and fixtures ---'
git ls-files | rg '(^|/)(test|tests|python/cudf_polars).*rolling|rolling.*test|window' | head -200Repository: NVIDIA/cudf
Length of output: 31207
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- Python rolling tests involving over/order_by ---'
rg -n -C 8 'rolling|over\(' python/cudf_polars/tests/expressions/test_rolling.py \
python/cudf_polars/tests/test_rolling.py \
python/cudf_polars/tests/test_window_functions.py \
python/cudf_polars/tests/streaming/test_rolling.py
printf '%s\n' '--- RollingWindow and decomposition definitions ---'
sed -n '180,255p' python/cudf_polars/cudf_polars/dsl/expressions/rolling.py
sed -n '330,430p' python/cudf_polars/cudf_polars/dsl/expressions/rolling.py
sed -n '920,980p' python/cudf_polars/cudf_polars/dsl/expressions/rolling.py
sed -n '1210,1430p' python/cudf_polars/cudf_polars/dsl/expressions/rolling.py
printf '%s\n' '--- Polars version and local compatibility references ---'
rg -n 'polars|required.*polars|POLARS' pyproject.toml python/cudf_polars -g '*.toml' -g '*.py' | head -200Repository: NVIDIA/cudf
Length of output: 50368
🏁 Script executed:
#!/bin/bash
set -eu
file="python/cudf_polars/cudf_polars/dsl/expressions/rolling.py"
printf '%s\n' '--- RollingWindow implementation ---'
sed -n '180,255p' "$file"
printf '%s\n' '--- Window expression constructor ---'
sed -n '330,430p' "$file"
printf '%s\n' '--- Rolling decomposition and order_by handling ---'
sed -n '920,980p' "$file"
sed -n '1210,1345p' "$file"
sed -n '1385,1420p' "$file"
printf '%s\n' '--- Exact rolling-over tests after the first test block ---'
sed -n '185,255p' python/cudf_polars/tests/expressions/test_rolling.py
rg -n -C 4 'order_by|order_by=' python/cudf_polars/tests/expressions/test_rolling.py \
python/cudf_polars/tests/test_window_functions.pyRepository: NVIDIA/cudf
Length of output: 36867
🌐 Web query:
Polars Expr rolling over order_by "order_by" "rolling" over window expression
💡 Result:
In Polars, expressions for rolling windows and window operations like over() handle ordering and partitioning differently. As of August 2026, there is no direct way to pass an order_by argument to Expr.rolling() because rolling() is intended for specific index-based or temporal windowing rather than general window aggregation [1][2]. 1. Understanding Expr.rolling() The Expr.rolling() method is used for temporal or index-based rolling windows (e.g., sum, mean) [1]. It requires an index_column that must be pre-sorted [1][3]. You cannot provide an order_by argument to Expr.rolling() to change the internal sort order; the data must be sorted correctly before or during the rolling operation [1][3]. 2. Understanding Expr.over() The Expr.over() method is a true window function used for partitioning and ordering [4]. It fully supports partition_by and order_by arguments, which are used to control the sorting of rows within each partition before the expression is evaluated [4]. This is the correct tool for operations that require sorting, such as cum_sum, diff, or shift [4][5]. 3. Combining Operations If you need to perform rolling-style aggregations on grouped, ordered data, you have two primary approaches: - Pre-sort the DataFrame: Sort your DataFrame by the desired ordering columns using df.sort() before applying rolling() [1][3]. If you have groups, you can use group_by_dynamic() or DataFrame.rolling() (with the group_by parameter) to handle partitions [3]. - Use rolling_*_by expressions: Polars provides specific expressions like rolling_mean_by, rolling_sum_by, etc., which accept an by argument (a column to associate with values for the window) [2][6]. These can be combined with over() for grouped operations [2]. In summary, use Expr.over(partition_by=..., order_by=...) for general ordered window operations [4], and use DataFrame.rolling() or Expr.rolling_*_by expressions for time/index-based rolling aggregations where sorting is managed via the index or the by parameter [2][3][6].
Citations:
- 1: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling.html
- 2: window expression not allowed in aggregation: allow chained
.over()and.rolling()aggregations pola-rs/polars#12051 - 3: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.rolling.html
- 4: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.over.html
- 5: https://stackoverflow.com/questions/79387448/how-to-sort-inside-a-window-in-descending-order-in-polars
- 6: https://docs.pola.rs/docs/rust/dev/polars/prelude/enum.Expr.html
🌐 Web query:
site:github.com/pola-rs/polars rolling over order_by RollingGroupOptions order_by
💡 Result:
In the Polars library, RollingGroupOptions is a configuration struct used to define the behavior of rolling window operations, typically within the context of a rolling group-by (e.g., groupby_rolling) [1][2][3]. It does not have an order_by field or property [2][3]. Rolling window operations in Polars are generally expected to be performed on data that is already sorted by the time index [4][5]. The structure typically includes the following configuration options [2][3]: - index_column: The name of the column representing the time or index [1][2]. - period: The duration or length of the rolling window [1][2][3]. - offset: The offset used to shift the window relative to the current index [1][5][2][3]. - closed_window: Determines whether the window is closed on the left, right, both, or neither [4][5][2][3]. - check_sorted: A boolean flag to verify if the index column is sorted [5][3]. If you need to order your data before performing a rolling operation, you should apply a sorting operation (such as sort) to your DataFrame or LazyFrame before initiating the rolling group-by [5]. The rolling operation itself relies on this pre-sorted order to function correctly [4][5].
Citations:
- 1: https://github.com/pola-rs/polars/blob/main/crates/polars-lazy/src/frame/mod.rs
- 2: https://github.com/pola-rs/polars/blob/3e39d238/crates/polars-python/src/lazyframe/visitor/expr_nodes.rs
- 3: pola-rs/polars@81f4ac2
- 4: Depending on data distribution rolling kernels are not used leading to performance degradation pola-rs/polars#25083
- 5: Wrong subgroup labels using
groupby_rollingwithbyset and positiveoffsetpola-rs/polars#9973
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- Current rolling-over tests and nearby unsupported cases ---'
sed -n '185,255p' python/cudf_polars/tests/expressions/test_rolling.py
sed -n '540,640p' python/cudf_polars/tests/expressions/test_rolling.py
printf '%s\n' '--- All test uses combining rolling and over(order_by) ---'
rg -n -U -P '(?s)rolling\\([^\\n]*\\).*?over\\([^\\n]*order_by|over\\([^\\n]*order_by[^\\n]*\\).*?rolling\\(' \
python/cudf_polars/tests python/cudf_polars/cudf_polars || trueRepository: NVIDIA/cudf
Length of output: 5071
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
import importlib.util
spec = importlib.util.find_spec("polars")
print("polars_available:", spec is not None)
if spec is None:
raise SystemExit(0)
import polars as pl
print("polars_version:", pl.__version__)
df = pl.DataFrame(
{
"g": ["a", "a", "a", "a"],
"idx": [1, 2, 3, 4],
"ob": [40, 10, 30, 20],
"x": [1, 2, 4, 8],
}
).lazy()
expr = pl.col("x").sum().rolling("idx", period="2i").over(
"g", order_by="ob"
)
try:
result = df.select(expr.alias("out")).collect()
print("result:")
print(result)
except Exception as exc:
print("exception_type:", type(exc).__name__)
print("exception:", exc)
PYRepository: NVIDIA/cudf
Length of output: 173
🌐 Web query:
Polars source GroupedWindow rolling expression over order_by Rolling
💡 Result:
In Polars, the concepts of "GroupedWindow" (often referred to as window functions via the.over expression) and "Rolling" operations are distinct but complementary mechanisms [1][2][3]. 1. Rolling Operations and Order By Rolling operations in Polars are generally index-based or time-based rather than explicitly ordered by a generic column within an.over context [4][5]. - DataFrame.rolling: This method creates rolling groups based on a temporal or integer index column [4]. When using rolling windows, the index column must be sorted in ascending order (or sorted within each group if a 'group_by' is provided) [4][6][5]. - Rolling Expression Methods: Methods like.rolling_sum_by or.rolling_mean_by allow you to compute rolling statistics over a window defined by an index column [3][7]. These are designed to handle non-uniform intervals or specific index-based logic [4]. 2. Window Functions (over) and Order By If you are looking to perform order-sensitive operations (like cumulative sums or window-based offsets) within groups, you use the.over expression, which explicitly supports the 'order_by' parameter [8][9]. - The.over method allows you to partition data (partition_by) and order rows within those partitions (order_by) before applying an expression [8]. - This is the standard way to handle order-sensitive window functions in Polars [8]. Key Distinction - If your goal is a rolling window (a sliding window over time or index), use.rolling or the.rolling__by expression family [4][3]. Ensure your index column is sorted [4][5]. - If your goal is an order-dependent calculation within a group (e.g., cumulative sum, shifted values relative to a sorted column), use.over(partition_by=..., order_by=...) [8][9]. You cannot combine an explicit "rolling" window definition inside an.over clause in the same way you define an SQL window frame; instead, choose the mechanism that matches your requirement: either a time/index-based sliding window (Rolling) or an order-sensitive partition-based calculation (Over) [8][4].
Citations:
- 1: https://docs.pola.rs/api/python/stable/reference/expressions/window.html
- 2: https://pola-rs.github.io/polars-book/user-guide/expressions/window/
- 3: https://docs.pola.rs/api/python/stable/reference/expressions/index.html
- 4: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.rolling.html
- 5: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling.html
- 6: https://docs.pola.rs/api/python/version/0.20/reference/dataframe/api/polars.DataFrame.group_by_rolling.html
- 7: https://docs.pola.rs/docs/python/dev/reference/expressions/computation.html
- 8: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.over.html
- 9: https://stackoverflow.com/questions/79387448/how-to-sort-inside-a-window-in-descending-order-in-polars
🌐 Web query:
site:github.com/pola-rs/polars "rolling" "order_by" "over" "InvalidOperationError"
💡 Result:
The InvalidOperationError encountered when using rolling operations (e.g., rolling_mean, rolling_sum, or the.rolling method) in Polars typically occurs because the library requires the index column (or the 'by' column) to be explicitly sorted in ascending order [1][2][3]. When Polars performs a rolling calculation, it relies on the data being sorted to efficiently compute the windowed results [3]. If the index column is not marked as sorted, Polars raises an InvalidOperationError [2][3]. To resolve this issue, you have a few options: 1. Sort the data: Ensure your DataFrame is sorted by the relevant index or 'by' column before performing the rolling operation [1][2]. 2. Set the sorted flag: If your data is already sorted but Polars has lost the sortedness metadata (which can sometimes happen during intermediate operations), you can explicitly inform Polars by using the.set_sorted method on the relevant column [2]. 3. Use group_by_rolling: If you are performing rolling operations within groups, ensure you are using group_by_rolling or the appropriate.rolling method on the DataFrame, which is designed to handle these operations correctly when configured with the correct index and grouping columns [2][3]. Note that in some cases, particularly in older versions or specific contexts like streaming, this check might behave inconsistently (sometimes failing to raise an error even when data is unsorted, or conversely, raising an error even when data is sorted due to metadata tracking issues) [1][4]. Always ensure you are using the latest version of Polars, as many of these issues are actively tracked and fixed by the maintainers [4].
Citations:
- 1: streaming engine does not raise for
rollingwhen index_column is unsorted pola-rs/polars#27231 - 2:
rolling_meandoesn't respect a sorted date column when applied over a group pola-rs/polars#11225 - 3: rolling_* functions don't check sortedness pola-rs/polars#10991
- 4: df.rolling with group_by raises “data not sorted” even when data is sorted pola-rs/polars#21476
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re
path = Path("python/cudf_polars/tests/expressions/test_rolling.py")
text = path.read_text()
rolling_over = []
for match in re.finditer(r"rolling\(", text):
start = max(0, match.start() - 180)
end = min(len(text), match.end() + 300)
snippet = text[start:end]
if ".over(" in snippet:
rolling_over.append(" ".join(snippet.split()))
print("rolling_over_occurrences:", len(rolling_over))
for snippet in rolling_over:
print(snippet)
PYRepository: NVIDIA/cudf
Length of output: 3822
🌐 Web query:
"rolling(\"ts\"" ".over(\"g\", order_by" Polars
💡 Result:
In Polars, there is no direct expression syntax exactly like rolling("ts").over("g", order_by=...); rather, rolling window operations and grouping operations are handled through distinct API patterns [1]. Depending on your goal, you should choose one of the following approaches: 1. Rolling Aggregations over Groups (DataFrame level) If you need to perform multiple aggregations over a rolling window within groups, use the DataFrame.rolling method with the group_by parameter [2][3]. The index_column must be sorted within each group [2][3]. df.rolling(index_column="ts", period="1d", group_by="g").agg( pl.col("value").sum.alias("sum_val"), pl.col("value").mean.alias("mean_val")) 2. Rolling Expression (Expression level) If you are performing a single rolling aggregation (like rolling_mean, rolling_sum, etc.) on an expression, you can use the built-in rolling expressions directly [4][5][6]. To apply these within groups, chain them with.over [7]. Note that rolling_sum and rolling_mean do not take an explicit order_by argument because they operate on the existing order of the expression [4][6]. df.with_columns( pl.col("value").rolling_sum(window_size=5).over("g")) 3. Rolling by Another Column If your rolling operation depends on a time column (and not just row counts), use the specific rolling_..._by methods (e.g., rolling_sum_by) [5][6]. df.with_columns( pl.col("value").rolling_sum_by(by="ts", window_size="1d").over("g")) Key Implementation Details: - Sorting: Methods like DataFrame.rolling require the index_column to be sorted [2][3]. - Expression over: The.over method supports an order_by parameter to control row order within partitions for operations like cum_sum or diff, but it does not define the window size for rolling operations [8][9]. - Performance: DataFrame.rolling is generally more performant than chaining multiple rolling expressions with.over because it can cache the window calculations [4][5][6].
Citations:
- 1: https://docs.pola.rs/api/python/stable/reference/expressions/window.html
- 2: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.rolling.html
- 3: https://docs.pola.rs/py-polars/html/reference/dataframe/api/polars.DataFrame.rolling.html
- 4: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling_mean.html
- 5: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling_mean_by.html
- 6: https://docs.pola.rs/py-polars/html/reference/expressions/api/polars.Expr.rolling_sum.html
- 7: https://stackoverflow.com/questions/76164821/how-to-group-by-and-rolling-sum-in-polars
- 8: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.over.html
- 9: order_by in over is incorrect when grouping column is marked as sorted pola-rs/polars#18943
🌐 Web query:
"Expr.rolling" "over" "order_by" Polars issue
💡 Result:
In Polars, the primary requirement when using rolling operations (both Expr.rolling and DataFrame.rolling) is that the index_column must be sorted in ascending order [1][2][3][4]. If the data is not sorted, Polars will typically raise an InvalidOperationError because the rolling window logic relies on the assumption of sorted temporal or integer index values [5][3]. Key points regarding common "order_by" and rolling issues include: 1. Requirement for Sorted Data: The index_column used in rolling must be sorted [1][3]. You should ensure your data is sorted before performing the operation, for example by using .set_sorted() if you know the data is already sorted, or by performing a full .sort() if necessary [1][2]. 2. Difference Between Expression and DataFrame Rolling: - Expr.rolling is used for rolling aggregations based on a time or integer window. It does not accept a by argument [6][7]. - DataFrame.rolling is often more flexible and supports a group_by parameter, which allows you to perform rolling operations independently within each group (where each group must be sorted by the index column) [3][4]. 3. Common Misconceptions: - Users sometimes attempt to pass order_by or by arguments into Expr.rolling based on similarities with other Polars functions (like over), which causes a TypeError or InvalidOperationError because these arguments are not implemented for Expr.rolling [6][7]. - If you need to perform rolling operations within groups (partitioned data), use DataFrame.rolling(..., group_by="your_column") rather than trying to force it into a single expression [3][4]. 4. Streaming Engine Behavior: Note that there have been reports where the streaming engine might not raise an error for unsorted index columns in rolling operations, even when the in-memory engine does [5]. It is best practice to always ensure your data is explicitly sorted regardless of the engine being used [1][3]. If you need to perform operations that involve specific row ordering within windows that are not time-based, consider using Expr.over with order_by, which is designed for order-sensitive operations such as cum_sum or diff, but note that over is generally for window functions, not the dynamic window-based rolling aggregations provided by rolling [8][9].
Citations:
- 1: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling.html
- 2: https://docs.pola.rs/docs/python/dev/reference/expressions/api/polars.Expr.rolling.html
- 3: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.rolling.html
- 4: https://docs.pola.rs/py-polars/html/reference/dataframe/api/polars.DataFrame.rolling.html
- 5: streaming engine does not raise for
rollingwhen index_column is unsorted pola-rs/polars#27231 - 6: https://stackoverflow.com/questions/78203785/polars-rolling-by-option-not-allowed
- 7:
rolling: add grouping by rows (no index) pola-rs/polars#12014 - 8: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.over.html
- 9: https://stackoverflow.com/questions/79387448/how-to-sort-inside-a-window-in-descending-order-in-polars
🏁 Script executed:
#!/bin/bash
set -eu
# Query the public Polars repository without cloning it.
for url in \
"https://api.github.com/search/code?q=rolling+over+order_by+repo%3Apola-rs%2Fpolars" \
"https://api.github.com/repos/pola-rs/polars/contents/crates/polars-plan/src/dsl" \
"https://api.github.com/repos/pola-rs/polars/contents/crates/polars-plan/src/logical_plan"
do
printf '\n--- %s ---\n' "$url"
curl -L --fail --silent --show-error "$url" |
python3 -c 'import json,sys; x=json.load(sys.stdin); print(json.dumps(x, indent=2)[:30000])' ||
true
doneRepository: NVIDIA/cudf
Length of output: 2393
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
urls=(
"https://raw.githubusercontent.com/pola-rs/polars/main/crates/polars-plan/src/dsl/expr.rs"
"https://raw.githubusercontent.com/pola-rs/polars/main/crates/polars-plan/src/dsl/window.rs"
"https://raw.githubusercontent.com/pola-rs/polars/main/crates/polars-plan/src/logical_plan/expr.rs"
"https://raw.githubusercontent.com/pola-rs/polars/main/crates/polars-lazy/src/dsl/mod.rs"
)
for i in "${!urls[@]}"; do
file="$tmp/$i"
if curl -L --fail --silent --show-error "${urls[$i]}" -o "$file"; then
printf '\n--- %s ---\n' "${urls[$i]}"
rg -n -C 4 'Rolling|rolling|Window|order_by' "$file" | head -250 || true
fi
doneRepository: NVIDIA/cudf
Length of output: 393
🌐 Web query:
github polars "RollingGroupOptions" "order_by" "Window"
💡 Result:
In the Polars library, RollingGroupOptions is a configuration structure used within the rolling window operations, primarily in the Rust API [1]. There is no direct "order_by" parameter within the RollingGroupOptions struct itself [1]; instead, Polars handles ordering requirements for rolling operations primarily through the mandatory sorting of the index column [2][3][4]. Key details regarding ordering and rolling windows in Polars include: 1. Rolling Operations and Index Columns: When performing a rolling operation (e.g., using.rolling in Python or.rolling on an Expr in Rust), you must specify an index_column [2][3][5]. This column must be sorted in ascending order [3][4]. If a "group_by" (or "by") column is specified, the index_column must be sorted within each group [2][3][4]. 2. RollingGroupOptions Struct: This structure is used in the Rust API to define the rolling configuration (index_column, period, offset, and closed_window) [1]. It does not contain an "order_by" field [1]. 3. Window Functions and Ordering: While rolling operations rely on a pre-sorted index column, there have been long-standing feature requests to support explicit "order_by" clauses within window functions (e.g., inside.over) to avoid the need for pre-sorting the entire DataFrame [6]. Currently, to achieve specific ordering within an.over window, users generally must ensure the DataFrame is sorted by the desired keys before applying the window function [6]. 4. Distinction from Window: The Window object in Polars (seen in the Rust/internal representation) is distinct from rolling options [7]. It manages partitioning (partition_by) and contains fields related to ordering (order_by, order_by_descending, etc.) that are used for window functions like shift, rank, or cumulative aggregations, rather than rolling temporal/index windows [7]. In summary, there is no RollingGroupOptions.order_by field because rolling windows in Polars are fundamentally defined by the sorted state of the index column provided at execution time [2][3][4]. If you require custom ordering for non-rolling window operations, you should use the.over method with a pre-sorted DataFrame [6].
Citations:
- 1: https://docs.pola.rs/api/rust/dev/polars/prelude/struct.RollingGroupOptions.html
- 2: https://docs.pola.rs/api/python/version/0.18/reference/dataframe/api/polars.DataFrame.groupby_rolling.html
- 3: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.rolling.html
- 4: https://docs.pola.rs/api/python/version/0.20/reference/dataframe/api/polars.DataFrame.group_by_rolling.html
- 5: https://docs.pola.rs/docs/rust/dev/polars/prelude/enum.Expr.html
- 6: Support
ORDER BYin conjunction withovermethod pola-rs/polars#8662 - 7: https://github.com/pola-rs/polars/blob/3e39d238/crates/polars-python/src/lazyframe/visitor/expr_nodes.rs
🌐 Web query:
github polars "rolling_expr" "Window" "order_by"
💡 Result:
In Polars, the functionality for ordering rows within windows is handled primarily through the order_by parameter in the Expr.over method, rather than within Expr.rolling [1][2]. Expr.over and order_by The Expr.over method is used to compute expressions over partitions (groups) of data [1][3]. The order_by parameter allows you to specify a column to sort rows within each partition before the expression is evaluated [1][4]. This is essential for operations that are sensitive to row order, such as cum_sum, diff, or shift [1][5]. Example: df.with_columns( cumulative_sales=pl.col("sales").cum_sum.over("store_id", order_by="date")) Rolling Operations Expr.rolling and Expr.rolling_mean_by are designed for time-series or index-based window calculations [6][7]. Unlike over, which sorts within groups, rolling methods generally require the input index column to be pre-sorted [6][8]. - Expr.rolling(index_column=...): Requires the index_column to be sorted in ascending order [6]. - DataFrame.rolling(index_column=..., group_by=...): If a group_by is specified, the index_column must be sorted within each group [8]. In summary, if you need to control the order of evaluation within a window, use over(..., order_by=...) [1]. If you are performing a rolling window calculation, ensure your data is already sorted by the index column before applying the rolling operation [6][8].
Citations:
- 1: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.over.html
- 2: https://docs.pola.rs/api/python/version/1/reference/expressions/window.html
- 3: https://docs.pola.rs/user-guide/expressions/window-functions/
- 4: https://docs.pola.rs/api/python/version/1/reference/expressions/api/polars.Expr.over.html
- 5: https://pola-rs.github.io/polars-book/user-guide/expressions/window/
- 6: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling.html
- 7: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling_mean_by.html
- 8: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.rolling.html
🌐 Web query:
Polars rolling expression over partition order_by semantics source code
💡 Result:
In Polars, the concepts of partitioning and ordering are handled differently depending on whether you are using window functions via.over or rolling aggregation methods..over Semantics The.over expression is designed for window functions (like cum_sum, rank, or aggregates) and explicitly supports both partition_by and order_by [1][2]. - partition_by: Defines the groups over which the expression is calculated [1]. - order_by: Defines the sorting order within each partition before the expression is evaluated [1]. This is required for operations that are sensitive to row order, such as cumulative sums [1][2]. - Source Behavior: Internally, Polars uses these parameters to perform efficient window computations. Recent updates have focused on ensuring that these operations align with standard SQL window function semantics, particularly regarding how null values are handled in cumulative operations [3]..rolling Semantics Rolling operations in Polars are handled either via DataFrame.rolling (for time-based or index-based windows) or via expression-level methods like Expr.rolling_sum [4][5][6]. - Indexing: Unlike.over, rolling operations require a specific index_column (e.g., date, datetime, or integer) that must be sorted [4][6]. - Grouping: If grouping is required, the rolling method accepts a group_by parameter [6]. The requirement is that the data must be sorted by the index_column within each of those groups [4][6]. - Source Logic: The DataFrame.rolling implementation creates a RollingGroupBy object [6]. Its internal logic (found in polars/dataframe/group_by.py) performs a lazy operation using the specified index_column and group_by columns, ensuring data is grouped and aggregated correctly while respecting the temporal or positional order [7][8]. Key Distinction While.over allows you to specify arbitrary expressions for ordering within partitions [1],.rolling is strictly bound to a temporal or index column that defines the window's boundaries [4][6]. If you need to perform rolling operations over partitioned groups, you must use the.rolling method on the DataFrame (or a LazyFrame) and specify the group_by parameter, ensuring the data is pre-sorted by the index column within those groups [6].
Citations:
- 1: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.over.html
- 2: https://docs.pola.rs/api/python/version/1/reference/expressions/api/polars.Expr.over.html
- 3: fix: Respect SQL semantics for cumulative functions mapped via
OVERclause pola-rs/polars#26570 - 4: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling.html
- 5: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.rolling_sum.html
- 6: https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.rolling.html
- 7: https://github.com/pola-rs/polars/blob/py-1.43.0/py-polars/src/polars/dataframe/group_by.py
- 8: https://github.com/pola-rs/polars/blob/py-1.42.1/py-polars/src/polars/dataframe/group_by.py
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
base="https://raw.githubusercontent.com/pola-rs/polars/py-1.43.0"
paths=(
"crates/polars-plan/src/dsl/mod.rs"
"crates/polars-plan/src/dsl/window.rs"
"crates/polars-plan/src/logical_plan/expr.rs"
"crates/polars-lazy/src/dsl/mod.rs"
"crates/polars-lazy/src/frame/mod.rs"
"crates/polars-python/src/lazyframe/visitor/expr_nodes.rs"
)
for path in "${paths[@]}"; do
out="$tmp/$(basename "$path")"
if curl -L --fail --silent --show-error "$base/$path" -o "$out"; then
printf '\n--- %s ---\n' "$path"
rg -n -C 5 'Rolling|rolling|order_by|Window' "$out" | head -300 || true
else
printf 'unavailable: %s\n' "$path"
fi
doneRepository: NVIDIA/cudf
Length of output: 27099
🌐 Web query:
site:github.com/pola-rs/polars "AExpr::Over" "AExpr::Rolling"
💡 Result:
AExpr::Over and AExpr::Rolling are internal variants of the AExpr (Arena Expression) enum within the Polars library, specifically found in the polars-plan crate [1]. These variants represent analytical expressions used in the library's internal expression planning and logical query optimization pipeline [1][2]. AExpr::Over This variant represents window function expressions that operate over a set of rows [1]. It encompasses the logic for windowing operations that include partition-by keys, optional ordering, and window mapping strategies [1][3]. It is used to transform standard aggregate or window functions into a form that the execution engine can process as a window operation [2]. AExpr::Rolling This variant represents rolling window operations, which are typically time-series or index-based window calculations [1]. It is used when a calculation needs to be performed over a dynamic window defined by a period and offset relative to an index column [1][2]. This variant is conditional on the dynamic_group_by feature flag in Polars [1][4]. In the context of the Polars execution planner, both variants are considered non-scalar expressions because they implicitly zip with the context (i.e., they are not simple element-wise operations) [1]. During the planning phase, these AExpr nodes are converted into physical expressions that the execution engine uses to perform the actual data calculations [2].
Citations:
- 1: https://github.com/pola-rs/polars/blob/b6ae1153/crates/polars-plan/src/plans/aexpr/mod.rs
- 2: https://github.com/pola-rs/polars/blob/b6ae1153/crates/polars-expr/src/planner.rs
- 3: https://github.com/pola-rs/polars/blob/b6ae1153/crates/polars-plan/src/plans/conversion/ir_to_dsl.rs
- 4: https://github.com/pola-rs/polars/blob/b6ae1153/crates/polars-plan/src/plans/visitor/expr.rs
🌐 Web query:
site:github.com/pola-rs/polars "IRNode::Window" "Rolling"
💡 Result:
In the Polars library, IRNode::Window (and related rolling window structures) refers to the internal representation (IR) nodes used to handle rolling window operations within the lazy execution engine [1][2]. Key aspects of these nodes include: Internal Representation (IR) Variants: Polars uses specific IR variants to distinguish different types of rolling operations [1][2]. For example: - RollingExpr: Handles fixed-size window operations where the window size is defined by a number of slots [1][2]. - RollingExprBy: Handles operations that rely on a dynamic window or specific index-based logic [1][2]. Technical Implementation: - These nodes are defined within the Polars Rust codebase (primarily in the polars-plan crate) [1]. - They encapsulate the rolling function (e.g., Min, Max, Mean, Sum, Quantile, Var, Std, Rank, Skew, Kurtosis) along with specific options such as window size, minimum periods, and window weights [1][3][2]. - Recent developments have focused on exposing these internal expressions to the Python API, enabling external engines (such as cudf-polars) to intercept and optimize rolling operations, for example by offloading them to GPU execution [4][3]. Context: The IRNode structures allow the Polars query planner to perform optimizations—such as fusion and predicate pushdown—before the plan is executed [2][5][6]. The system separates the definition of the rolling operation from the execution strategy, which is critical for supporting both standard CPU-based rolling windows and specialized implementations for dynamic or time-based windows [7][6].
Citations:
- 1: https://github.com/pola-rs/polars/blob/3e39d238/crates/polars-plan/src/plans/aexpr/function_expr/mod.rs
- 2: https://github.com/pola-rs/polars/blob/3e39d238/crates/polars-plan/src/plans/conversion/dsl_to_ir/functions.rs
- 3: feat(rust): Expose fixed-size rolling window expressions in Python visitor pola-rs/polars#27108
- 4: Expose rolling expression node in Python API pola-rs/polars#25001
- 5: https://github.com/pola-rs/polars/blob/3e39d238/crates/polars-plan/src/plans/conversion/dsl_to_ir/mod.rs
- 6: pola-rs/polars@5a202a0
- 7: feat: postfix
rollingexpression as a special case of window functions. pola-rs/polars#11445
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
base="https://raw.githubusercontent.com/pola-rs/polars/py-1.43.0"
paths=(
"crates/polars-lazy/src/physical_plan/executors/window.rs"
"crates/polars-lazy/src/physical_plan/executors/mod.rs"
"crates/polars-stream/src/nodes/compute/window.rs"
"crates/polars-stream/src/ir/mod.rs"
"crates/polars-plan/src/plans/aexpr/mod.rs"
"crates/polars-plan/src/plans/ir/mod.rs"
)
for path in "${paths[@]}"; do
out="$tmp/$(echo "$path" | tr / _)"
if curl -L --fail --silent --show-error "$base/$path" -o "$out"; then
printf '\n--- %s ---\n' "$path"
rg -n -C 5 'Rolling|rolling|Window|order_by|AExpr::Over' "$out" | head -400 || true
else
printf 'unavailable: %s\n' "$path"
fi
doneRepository: NVIDIA/cudf
Length of output: 1246
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
base="https://raw.githubusercontent.com/pola-rs/polars/b6ae1153"
paths=(
"crates/polars-expr/src/planner.rs"
"crates/polars-plan/src/plans/conversion/ir_to_dsl.rs"
"crates/polars-plan/src/plans/conversion/dsl_to_ir/mod.rs"
"crates/polars-plan/src/plans/conversion/dsl_to_ir/functions.rs"
"crates/polars-plan/src/plans/aexpr/function_expr/mod.rs"
)
for path in "${paths[@]}"; do
out="$tmp/$(echo "$path" | tr / _)"
if curl -L --fail --silent --show-error "$base/$path" -o "$out"; then
printf '\n--- %s ---\n' "$path"
rg -n -C 8 'AExpr::Over|AExpr::Rolling|Expr::Over|Expr::Rolling|order_by|rolling' "$out" | head -500 || true
else
printf 'unavailable: %s\n' "$path"
fi
doneRepository: NVIDIA/cudf
Length of output: 13631
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
base="https://raw.githubusercontent.com/pola-rs/polars/b6ae1153"
paths=(
"crates/polars-expr/src/expressions/window.rs"
"crates/polars-expr/src/expressions/window/mod.rs"
"crates/polars-expr/src/window.rs"
"crates/polars-expr/src/window/mod.rs"
"crates/polars-expr/src/expressions/mod.rs"
)
for path in "${paths[@]}"; do
out="$tmp/$(echo "$path" | tr / _)"
if curl -L --fail --silent --show-error "$base/$path" -o "$out"; then
printf '\n--- %s ---\n' "$path"
rg -n -C 8 'order_by|phys_function|evaluate|Rolling|rolling|sort' "$out" | head -600 || true
else
printf 'unavailable: %s\n' "$path"
fi
doneRepository: NVIDIA/cudf
Length of output: 35372
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
base="https://raw.githubusercontent.com/pola-rs/polars/b6ae1153"
path="crates/polars-expr/src/expressions/rolling.rs"
out="$tmp/rolling.rs"
curl -L --fail --silent --show-error "$base/$path" -o "$out"
printf '%s\n' '--- RollingExpr implementation ---'
rg -n -C 12 'struct RollingExpr|impl PhysicalExpr for RollingExpr|evaluate_on_groups|evaluate_impl|order|sorted|Rolling' "$out" | head -800Repository: NVIDIA/cudf
Length of output: 6839
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
url="https://raw.githubusercontent.com/pola-rs/polars/b6ae1153/crates/polars-expr/src/expressions/rolling.rs"
out="$tmp/rolling.rs"
curl -L --fail --silent --show-error "$url" -o "$out"
printf '%s\n' '--- RollingExpr group ordering and window construction ---'
sed -n '77,245p' "$out"
printf '%s\n' '--- RollingWindower sortedness checks in Polars time crate ---'
for path in \
"crates/polars-time/src/group_by/rolling.rs" \
"crates/polars-time/src/chunkedarray/rolling_window.rs" \
"crates/polars-time/src/group_by/mod.rs"
do
candidate="$tmp/$(echo "$path" | tr / _)"
if curl -L --fail --silent --show-error \
"https://raw.githubusercontent.com/pola-rs/polars/b6ae1153/$path" \
-o "$candidate"; then
printf '\n--- %s ---\n' "$path"
rg -n -C 8 'RollingWindower|sorted|monotonic|window|out of order' "$candidate" | head -500 || true
fi
doneRepository: NVIDIA/cudf
Length of output: 7434
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
import ast
from pathlib import Path
path = Path("python/cudf_polars/cudf_polars/dsl/expressions/rolling.py")
tree = ast.parse(path.read_text())
matches = []
for node in ast.walk(tree):
if not isinstance(node, ast.If):
continue
test = ast.unparse(node.test)
if 'unary_window_ops["range_rolling"]' not in test:
continue
calls = [
n for n in ast.walk(node)
if isinstance(n, ast.Call)
and isinstance(n.func, ast.Attribute)
and n.func.attr == "_apply_ordered_unary_op"
]
for call in calls:
keywords = {kw.arg: ast.unparse(kw.value) for kw in call.keywords}
matches.append(
{
"guard": test,
"order_by_col": keywords.get("order_by_col"),
"ob_desc": keywords.get("ob_desc"),
"ob_nulls_last": keywords.get("ob_nulls_last"),
"require_sorted_groups": keywords.get("require_sorted_groups"),
}
)
print(matches)
print("range_rolling_guard_checks_self_order_by:",
any(
"self._order_by_expr" in ast.unparse(n.test)
for n in ast.walk(tree)
if isinstance(n, ast.If)
and 'unary_window_ops["range_rolling"]' in ast.unparse(n.test)
))
PYRepository: NVIDIA/cudf
Length of output: 200
Reject over(order_by=...) with range-rolling expressions. The CPU window implementation sorts each group by order_by before evaluating the rolling expression. This branch always orders by the rolling index and omits over_ob_desc and over_ob_nulls_last, which can produce different results. Raise NotImplementedError when both conditions apply, or implement the same ordering semantics.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/cudf_polars/cudf_polars/dsl/expressions/rolling.py` around lines 1487
- 1500, Update the range-rolling branch in the surrounding window-expression
implementation to detect when a range-rolling expression is combined with an
over-clause order_by, using the existing order-direction and null-ordering
fields (over_ob_desc and over_ob_nulls_last). Raise NotImplementedError for that
combination unless _apply_ordered_unary_op is updated to honor those ordering
semantics; preserve the current rolling-index ordering for cases without
over(order_by=...).
| child_deps: list[expr.Expr] = [] | ||
| for ne in named_aggs: | ||
| v = ne.value | ||
| if ( | ||
| isinstance(v, expr.UnaryFunction) | ||
| and v.name == "fill_null_with_strategy" | ||
| and isinstance(v.children[0], expr.UnaryFunction) | ||
| and v.children[0].name == "cum_sum" | ||
| ) | ||
| else v.children[0] | ||
| for ne in named_aggs | ||
| for v in (ne.value,) | ||
| if isinstance(v, expr.Agg) | ||
| or ( | ||
| ): | ||
| child_deps.append(v.children[0].children[0]) | ||
| elif isinstance(v, expr.RollingWindow): | ||
| child_deps.append(v.children[0]) | ||
| child_deps.append(expr.Col(schema[v.orderby], v.orderby)) | ||
| elif isinstance(v, expr.Agg) or ( | ||
| isinstance(v, expr.UnaryFunction) | ||
| and v.name | ||
| in { | ||
| "rank", | ||
| "fill_null_with_strategy", | ||
| "cum_sum", | ||
| "diff", | ||
| "shift", | ||
| "shift_and_fill", | ||
| } | ||
| ) | ||
| ] | ||
| ): | ||
| child_deps.append(v.children[0]) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Inspect the previous child-dependency construction for GroupedWindow and locate FixedSizeRollingWindow handling.
set -euo pipefail
# Show the current window translation block.
rg -n -C 6 'child_deps' python/cudf_polars/cudf_polars/dsl/translate.py
# Show the previous revision of the same region.
git log -1 --format=%H -- python/cudf_polars/cudf_polars/dsl/translate.py
git show HEAD~1:python/cudf_polars/cudf_polars/dsl/translate.py 2>/dev/null | rg -n -C 8 'by_exprs|FixedSizeRollingWindow|children = \(' | head -80
# Find where FixedSizeRollingWindow children are evaluated at runtime.
rg -n -C 4 'FixedSizeRollingWindow' python/cudf_polars/cudf_polars/dsl/Repository: NVIDIA/cudf
Length of output: 14000
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- parent implementation ---'
git show HEAD~1:python/cudf_polars/cudf_polars/dsl/translate.py | sed -n '1180,1260p'
echo '--- current implementation ---'
sed -n '1180,1260p' python/cudf_polars/cudf_polars/dsl/translate.py
echo '--- aggregation decomposition ---'
sed -n '110,180p' python/cudf_polars/cudf_polars/dsl/utils/aggregations.py
echo '--- GroupedWindow dependency consumers ---'
rg -n -C 8 'class GroupedWindow|by_cols_for_scan|op\.children|children' \
python/cudf_polars/cudf_polars/dsl/expressions/rolling.pyRepository: NVIDIA/cudf
Length of output: 34386
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- expression dependency traversal ---'
rg -n -C 10 'def .*gather.*column|_gather_columns|\.children' \
python/cudf_polars/cudf_polars/dsl python/cudf_polars
echo '--- projection dependency handling ---'
rg -n -C 10 'GroupedWindow|named_aggs|child_deps|children.*evaluate|evaluate.*children' \
python/cudf_polars/cudf_polars
echo '--- parent/current diff for this file ---'
git diff --no-ext-diff --unified=20 HEAD~1 HEAD -- \
python/cudf_polars/cudf_polars/dsl/translate.py | sed -n '1,220p'Repository: NVIDIA/cudf
Length of output: 50368
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- dependency helper locations ---'
rg -n --glob '*.py' 'def _gather_columns|def gather_columns|_gather_columns\(' \
python/cudf_polars/cudf_polars/dsl python/cudf_polars/cudf_polars
echo '--- base expression implementation ---'
rg -l --glob '*.py' 'class Expr' python/cudf_polars/cudf_polars/dsl \
| head -20
echo '--- current commit diff only ---'
git diff --no-ext-diff --unified=8 HEAD~1 HEAD -- \
python/cudf_polars/cudf_polars/dsl/translate.py \
| rg -n -C 12 'child_deps|FixedSizeRollingWindow|^[-+]' \
| head -240Repository: NVIDIA/cudf
Length of output: 1087
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- expression base class ---'
sed -n '1,240p' python/cudf_polars/cudf_polars/dsl/expressions/base.py
echo '--- GroupedWindow dependency setup ---'
sed -n '1000,1080p' python/cudf_polars/cudf_polars/dsl/expressions/rolling.py
echo '--- projection and expression-column dependency code ---'
rg -n --glob '*.py' -C 5 \
'projection|column_names|columns_needed|required_columns|collect.*column|traversal\(' \
python/cudf_polars/cudf_polars | head -300Repository: NVIDIA/cudf
Length of output: 39428
Register fixed-size rolling dependencies.
FixedSizeRollingWindow is accepted by GroupedWindow, but its data child is not added to child_deps. Add a branch that appends v.children[0] so FixedSizeRollingOp can evaluate the required input column after projection.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/cudf_polars/cudf_polars/dsl/translate.py` around lines 1219 - 1244,
Update the dependency collection loop for named aggregations to handle
FixedSizeRollingWindow values explicitly. Add v.children[0] to child_deps so
FixedSizeRollingOp receives its input column after projection, while preserving
the existing RollingWindow and aggregation handling.
| if isinstance(agg, expr.RollingWindow): | ||
| if context != ExecutionContext.WINDOW: | ||
| raise NotImplementedError( | ||
| "Range rolling is not supported in groupby or rolling context" | ||
| ) | ||
| if _contains_window_only_unary(agg.children[0]) or ( | ||
| _contains_fixed_size_rolling_window(agg.children[0]) | ||
| ): | ||
| raise NotImplementedError( | ||
| "Range rolling over a window does not support nested window expressions" | ||
| ) | ||
| return [(named_expr, True)], named_expr.reconstruct(expr.Col(agg.dtype, name)) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Also reject a nested RollingWindow.
The guard checks _contains_window_only_unary and _contains_fixed_size_rolling_window on agg.children[0]. It does not check for a nested expr.RollingWindow. A nested range rolling therefore passes translation and reaches RollingWindow.do_evaluate, which raises RuntimeError for a non-FRAME context. A RuntimeError does not trigger the CPU fallback, so the user gets a hard failure instead of an unsupported-operation fallback.
Add a nested range-rolling check so translation raises NotImplementedError first.
🛡️ Proposed guard
+def _contains_range_rolling_window(value: expr.Expr) -> bool:
+ return any(isinstance(node, expr.RollingWindow) for node in traversal([value]))
+
+
def decompose_single_agg( if isinstance(agg, expr.RollingWindow):
if context != ExecutionContext.WINDOW:
raise NotImplementedError(
"Range rolling is not supported in groupby or rolling context"
)
if _contains_window_only_unary(agg.children[0]) or (
_contains_fixed_size_rolling_window(agg.children[0])
+ ) or _contains_range_rolling_window(agg.children[0]):
- ):
raise NotImplementedError(
"Range rolling over a window does not support nested window expressions"
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if isinstance(agg, expr.RollingWindow): | |
| if context != ExecutionContext.WINDOW: | |
| raise NotImplementedError( | |
| "Range rolling is not supported in groupby or rolling context" | |
| ) | |
| if _contains_window_only_unary(agg.children[0]) or ( | |
| _contains_fixed_size_rolling_window(agg.children[0]) | |
| ): | |
| raise NotImplementedError( | |
| "Range rolling over a window does not support nested window expressions" | |
| ) | |
| return [(named_expr, True)], named_expr.reconstruct(expr.Col(agg.dtype, name)) | |
| def _contains_range_rolling_window(value: expr.Expr) -> bool: | |
| return any(isinstance(node, expr.RollingWindow) for node in traversal([value])) | |
| if isinstance(agg, expr.RollingWindow): | |
| if context != ExecutionContext.WINDOW: | |
| raise NotImplementedError( | |
| "Range rolling is not supported in groupby or rolling context" | |
| ) | |
| if _contains_window_only_unary(agg.children[0]) or ( | |
| _contains_fixed_size_rolling_window(agg.children[0]) | |
| ) or _contains_range_rolling_window(agg.children[0]): | |
| raise NotImplementedError( | |
| "Range rolling over a window does not support nested window expressions" | |
| ) | |
| return [(named_expr, True)], named_expr.reconstruct(expr.Col(agg.dtype, name)) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/cudf_polars/cudf_polars/dsl/utils/aggregations.py` around lines 166 -
177, Extend the nested-window guard in the RollingWindow translation branch to
also detect an expr.RollingWindow within agg.children[0], alongside
_contains_window_only_unary and _contains_fixed_size_rolling_window. Raise
NotImplementedError during translation so nested range rolling follows the
existing unsupported-operation fallback instead of reaching
RollingWindow.do_evaluate.
1 participant
Adds cudf-polars support for aggregate range-rolling expressions inside grouped
over(...), including common forms likesum,min,max,mean,count, andlen.Closes #23623.
Partially addresses #23606.