Shift and Rolling Windows
Learn how to shift time-series observations, compare current values with previous values, and calculate rolling statistics using Pandas.
1. Introduction
Shift and rolling window operations are useful for analyzing sequential data. Shifting moves values relative to their index, while rolling windows calculate statistics over a moving group of observations.
Learning Objectives
- Understand the Pandas
shift()method. - Compare current values with previous observations.
- Calculate differences and percentage changes.
- Use rolling means, sums, minimums, and maximums.
- Control rolling calculations with
min_periods. - Apply these methods to practical time-series data.
Shift repositions observations; rolling operations calculate statistics over consecutive observations.
2. Creating Sample Time-Series Data
First, create a Series containing daily sales. The datetime index allows the values to be analyzed as a time series.
import pandas as pd
dates = pd.date_range(
start="2025-01-01",
periods=6,
freq="D"
)
sales = pd.Series(
[100, 120, 115, 140, 160, 155],
index=dates,
name="Sales"
)
print(sales)
Output
2025-01-01 100
2025-01-02 120
2025-01-03 115
2025-01-04 140
2025-01-05 160
2025-01-06 155
Freq: D, Name: Sales, dtype: int64
3. The shift() Method
The shift() method moves data by a specified number
of positions while keeping the original index. By default,
values are shifted without changing the index labels.
previous_sales = sales.shift(1)
print(previous_sales)
Output
2025-01-01 NaN
2025-01-02 100.0
2025-01-03 120.0
2025-01-04 115.0
2025-01-05 140.0
2025-01-06 160.0
Freq: D, Name: Sales, dtype: float64
Each value moves down by one row. The first position becomes
NaN because no previous observation exists.
shift(1) is commonly used
to retrieve the previous observation. shift(-1)
retrieves the next observation relative to each index position.
4. Shifting by Multiple Periods
Use the periods argument to shift observations by
more than one position.
shift_one = sales.shift(periods=1)
shift_two = sales.shift(periods=2)
print("Shift by one:")
print(shift_one)
print("\\nShift by two:")
print(shift_two)
Output
Shift by one:
2025-01-01 NaN
2025-01-02 100.0
2025-01-03 120.0
2025-01-04 115.0
2025-01-05 140.0
2025-01-06 160.0
Freq: D, Name: Sales, dtype: float64
Shift by two:
2025-01-01 NaN
2025-01-02 NaN
2025-01-03 100.0
2025-01-04 120.0
2025-01-05 115.0
2025-01-06 140.0
Freq: D, Name: Sales, dtype: float64
5. Shifting in the Opposite Direction
A negative shift moves values in the opposite direction. For
example, shift(-1) places the next observation at
the current index position.
next_sales = sales.shift(-1)
print(next_sales)
Output
2025-01-01 120.0
2025-01-02 115.0
2025-01-03 140.0
2025-01-04 160.0
2025-01-05 155.0
2025-01-06 NaN
Freq: D, Name: Sales, dtype: float64
6. Comparing Current and Previous Values
Shifted data can be used to compare each observation with the previous one. Subtracting the shifted Series calculates the absolute change.
previous_sales = sales.shift(1)
sales_change = sales - previous_sales
print(sales_change)
Output
2025-01-01 NaN
2025-01-02 20.0
2025-01-03 -5.0
2025-01-04 25.0
2025-01-05 20.0
2025-01-06 -5.0
Freq: D, Name: Sales, dtype: float64
Positive values indicate an increase compared with the previous observation, while negative values indicate a decrease.
7. Calculating Percentage Change with shift()
Percentage change compares the difference with the previous value. The formula is:
Percentage Change
Percentage change = ((Current Value - Previous Value) / Previous Value) × 100
percentage_change = (
(sales - sales.shift(1)) / sales.shift(1)
) * 100
print(percentage_change.round(2))
Output
2025-01-01 NaN
2025-01-02 20.00
2025-01-03 -4.17
2025-01-04 21.74
2025-01-05 14.29
2025-01-06 -3.12
Freq: D, Name: Sales, dtype: float64
8. The diff() Method
The diff() method calculates the difference between
the current observation and an earlier observation. It is a
convenient alternative to subtracting a shifted Series.
print(sales.diff())
Output
2025-01-01 NaN
2025-01-02 20.0
2025-01-03 -5.0
2025-01-04 25.0
2025-01-05 20.0
2025-01-06 -5.0
Freq: D, Name: Sales, dtype: float64
To compare with the observation two positions earlier, specify
periods=2.
print(sales.diff(periods=2))
Output
2025-01-01 NaN
2025-01-02 NaN
2025-01-03 15.0
2025-01-04 20.0
2025-01-05 45.0
2025-01-06 15.0
Freq: D, Name: Sales, dtype: float64
9. The freq Parameter
When a datetime index is used, the freq parameter
shifts the index by a time offset instead of moving the data
values between rows. This differs from the default positional
shift.
daily_sales = pd.Series(
[100, 120, 115],
index=pd.date_range(
"2025-01-01",
periods=3,
freq="D"
)
)
shifted_index = daily_sales.shift(freq="D")
print(shifted_index)
Output
2025-01-02 100
2025-01-03 120
2025-01-04 115
Freq: D, dtype: int64
The original values remain associated with their observations, but the datetime index moves forward by one day.
shift(periods=1) for
positional shifting. Use shift(freq="D") when you
specifically need to move datetime index labels by one day.
10. Introduction to Rolling Windows
A rolling window calculates a statistic over a group of consecutive observations. The window moves forward as Pandas processes the Series.
For example, a three-observation rolling mean calculates the average of each group of three consecutive values.
rolling_mean = sales.rolling(window=3).mean()
print(rolling_mean.round(2))
Output
2025-01-01 NaN
2025-01-02 NaN
2025-01-03 111.67
2025-01-04 125.00
2025-01-05 138.33
2025-01-06 151.67
Freq: D, Name: Sales, dtype: float64
The first two results are missing because the default setting requires three valid observations for a three-observation window.
11. Rolling Sum
The rolling sum adds all values in each window. It is useful for calculating totals across consecutive observations.
rolling_sum = sales.rolling(window=3).sum()
print(rolling_sum)
Output
2025-01-01 NaN
2025-01-02 NaN
2025-01-03 335.0
2025-01-04 375.0
2025-01-05 415.0
2025-01-06 455.0
Freq: D, Name: Sales, dtype: float64
12. Rolling Minimum and Maximum
Use min() and max() to find the smallest
and largest values within each rolling window.
rolling_min = sales.rolling(window=3).min()
rolling_max = sales.rolling(window=3).max()
print("Rolling minimum:")
print(rolling_min)
print("\\nRolling maximum:")
print(rolling_max)
Output
Rolling minimum:
2025-01-01 NaN
2025-01-02 NaN
2025-01-03 100.0
2025-01-04 115.0
2025-01-05 115.0
2025-01-06 140.0
Freq: D, Name: Sales, dtype: float64
Rolling maximum:
2025-01-01 NaN
2025-01-02 NaN
2025-01-03 120.0
2025-01-04 140.0
2025-01-05 160.0
2025-01-06 160.0
Freq: D, Name: Sales, dtype: float64
13. Using min_periods
The min_periods parameter controls the minimum number
of valid observations needed to produce a rolling result.
rolling_mean = sales.rolling(
window=3,
min_periods=1
).mean()
print(rolling_mean.round(2))
Output
2025-01-01 100.00
2025-01-02 110.00
2025-01-03 111.67
2025-01-04 125.00
2025-01-05 138.33
2025-01-06 151.67
Freq: D, Name: Sales, dtype: float64
The first result uses one observation, the second uses two, and subsequent results use the full three-observation window.
14. Centered Rolling Windows
By default, rolling results are aligned to the right edge of the
window. Setting center=True aligns results to the
center of the window.
centered_mean = sales.rolling(
window=3,
center=True
).mean()
print(centered_mean.round(2))
Output
2025-01-01 NaN
2025-01-02 111.67
2025-01-03 125.00
2025-01-04 138.33
2025-01-05 151.67
2025-01-06 NaN
Freq: D, Name: Sales, dtype: float64
15. Combining Shift and Rolling Operations
Shift and rolling operations can be combined to compare current values with previous values and calculate short-term trends.
analysis = pd.DataFrame({
"Sales": sales,
"Previous Sales": sales.shift(1),
"Change": sales.diff(),
"Three-Day Average": sales.rolling(
window=3
).mean()
})
print(analysis.round(2))
Output
Sales Previous Sales Change Three-Day Average
2025-01-01 100 NaN NaN NaN
2025-01-02 120 100.0 20.0 NaN
2025-01-03 115 120.0 -5.0 111.67
2025-01-04 140 115.0 25.0 125.00
2025-01-05 160 140.0 20.0 138.33
2025-01-06 155 160.0 -5.0 151.67
This DataFrame combines the original sales, previous day's sales, day-to-day change, and three-observation moving average in one table.
16. Common Mistakes and Best Practices
Common Mistakes
- Forgetting that shifting creates missing values at the edges.
- Confusing positional shifting with datetime index shifting.
- Ignoring missing values in rolling calculations.
- Assuming a three-observation window always means three calendar days.
- Using future observations in a predictive feature.
Best Practices
- Sort observations chronologically when time order matters.
- Choose a window size appropriate for the analysis.
- Set
min_periodsintentionally. - Use
shift()anddiff()for lag comparisons. - Check whether a calculation uses future observations.
17. Practice Exercises
Exercise 1: Previous Observation
Create a Series of daily temperatures and use
shift(1) to display the previous observation.
temperature = pd.Series([22, 24, 23, 27, 26])
# Write your solution below
Exercise 2: Daily Difference
Calculate the difference between each observation and the observation immediately before it.
prices = pd.Series([100, 105, 103, 110, 108])
# Write your solution below
Exercise 3: Moving Average
Calculate a three-observation rolling average and allow results when at least one valid observation is available.
visitors = pd.Series([200, 220, 210, 250, 270])
# Write your solution below
Lesson Summary
The Pandas shift() method moves values relative to
their index and is useful for creating lagged observations.
The diff() method calculates differences between
observations, while shifted values can be used to calculate
percentage changes.
Rolling windows calculate statistics over consecutive groups of
observations. Common methods include mean(),
sum(), min(), and max().
The min_periods and center parameters
control how rolling results are calculated and aligned.
Combining shift and rolling operations helps analyze changes, smooth time-series data, and create useful features. Always consider missing values, index alignment, and future-data leakage when using these operations.