Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Shift and Rolling Windows

Learn how to shift time-series observations, compare current values with previous values, and calculate rolling statistics using Pandas.

1. Introduction

Shift and rolling window operations are useful for analyzing sequential data. Shifting moves values relative to their index, while rolling windows calculate statistics over a moving group of observations.

Learning Objectives

  • Understand the Pandas shift() method.
  • Compare current values with previous observations.
  • Calculate differences and percentage changes.
  • Use rolling means, sums, minimums, and maximums.
  • Control rolling calculations with min_periods.
  • Apply these methods to practical time-series data.
Shift Rolling Window 10 20 30 40 Move values relative to the index 10 20 30 40 Calculate a statistic over a window

Shift repositions observations; rolling operations calculate statistics over consecutive observations.

2. Creating Sample Time-Series Data

First, create a Series containing daily sales. The datetime index allows the values to be analyzed as a time series.

import pandas as pd

dates = pd.date_range(
    start="2025-01-01",
    periods=6,
    freq="D"
)

sales = pd.Series(
    [100, 120, 115, 140, 160, 155],
    index=dates,
    name="Sales"
)

print(sales)

Output

2025-01-01    100
2025-01-02    120
2025-01-03    115
2025-01-04    140
2025-01-05    160
2025-01-06    155
Freq: D, Name: Sales, dtype: int64

3. The shift() Method

The shift() method moves data by a specified number of positions while keeping the original index. By default, values are shifted without changing the index labels.

previous_sales = sales.shift(1)

print(previous_sales)

Output

2025-01-01      NaN
2025-01-02    100.0
2025-01-03    120.0
2025-01-04    115.0
2025-01-05    140.0
2025-01-06    160.0
Freq: D, Name: Sales, dtype: float64

Each value moves down by one row. The first position becomes NaN because no previous observation exists.

Remember: shift(1) is commonly used to retrieve the previous observation. shift(-1) retrieves the next observation relative to each index position.

4. Shifting by Multiple Periods

Use the periods argument to shift observations by more than one position.

shift_one = sales.shift(periods=1)
shift_two = sales.shift(periods=2)

print("Shift by one:")
print(shift_one)

print("\\nShift by two:")
print(shift_two)

Output

Shift by one:
2025-01-01      NaN
2025-01-02    100.0
2025-01-03    120.0
2025-01-04    115.0
2025-01-05    140.0
2025-01-06    160.0
Freq: D, Name: Sales, dtype: float64

Shift by two:
2025-01-01      NaN
2025-01-02      NaN
2025-01-03    100.0
2025-01-04    120.0
2025-01-05    115.0
2025-01-06    140.0
Freq: D, Name: Sales, dtype: float64

5. Shifting in the Opposite Direction

A negative shift moves values in the opposite direction. For example, shift(-1) places the next observation at the current index position.

next_sales = sales.shift(-1)

print(next_sales)

Output

2025-01-01    120.0
2025-01-02    115.0
2025-01-03    140.0
2025-01-04    160.0
2025-01-05    155.0
2025-01-06      NaN
Freq: D, Name: Sales, dtype: float64
Important: Negative shifts can expose future observations. They may be useful for retrospective comparisons, but should not be used as available inputs for a real-time forecasting model.

6. Comparing Current and Previous Values

Shifted data can be used to compare each observation with the previous one. Subtracting the shifted Series calculates the absolute change.

previous_sales = sales.shift(1)

sales_change = sales - previous_sales

print(sales_change)

Output

2025-01-01     NaN
2025-01-02    20.0
2025-01-03    -5.0
2025-01-04    25.0
2025-01-05    20.0
2025-01-06    -5.0
Freq: D, Name: Sales, dtype: float64

Positive values indicate an increase compared with the previous observation, while negative values indicate a decrease.

7. Calculating Percentage Change with shift()

Percentage change compares the difference with the previous value. The formula is:

Percentage Change

Percentage change = ((Current Value - Previous Value) / Previous Value) × 100

percentage_change = (
    (sales - sales.shift(1)) / sales.shift(1)
) * 100

print(percentage_change.round(2))

Output

2025-01-01          NaN
2025-01-02    20.00
2025-01-03    -4.17
2025-01-04    21.74
2025-01-05    14.29
2025-01-06    -3.12
Freq: D, Name: Sales, dtype: float64

8. The diff() Method

The diff() method calculates the difference between the current observation and an earlier observation. It is a convenient alternative to subtracting a shifted Series.

print(sales.diff())

Output

2025-01-01     NaN
2025-01-02    20.0
2025-01-03    -5.0
2025-01-04    25.0
2025-01-05    20.0
2025-01-06    -5.0
Freq: D, Name: Sales, dtype: float64

To compare with the observation two positions earlier, specify periods=2.

print(sales.diff(periods=2))

Output

2025-01-01     NaN
2025-01-02     NaN
2025-01-03    15.0
2025-01-04    20.0
2025-01-05    45.0
2025-01-06    15.0
Freq: D, Name: Sales, dtype: float64

9. The freq Parameter

When a datetime index is used, the freq parameter shifts the index by a time offset instead of moving the data values between rows. This differs from the default positional shift.

daily_sales = pd.Series(
    [100, 120, 115],
    index=pd.date_range(
        "2025-01-01",
        periods=3,
        freq="D"
    )
)

shifted_index = daily_sales.shift(freq="D")

print(shifted_index)

Output

2025-01-02    100
2025-01-03    120
2025-01-04    115
Freq: D, dtype: int64

The original values remain associated with their observations, but the datetime index moves forward by one day.

Note: Use shift(periods=1) for positional shifting. Use shift(freq="D") when you specifically need to move datetime index labels by one day.

10. Introduction to Rolling Windows

A rolling window calculates a statistic over a group of consecutive observations. The window moves forward as Pandas processes the Series.

For example, a three-observation rolling mean calculates the average of each group of three consecutive values.

rolling_mean = sales.rolling(window=3).mean()

print(rolling_mean.round(2))

Output

2025-01-01       NaN
2025-01-02       NaN
2025-01-03    111.67
2025-01-04    125.00
2025-01-05    138.33
2025-01-06    151.67
Freq: D, Name: Sales, dtype: float64

The first two results are missing because the default setting requires three valid observations for a three-observation window.

11. Rolling Sum

The rolling sum adds all values in each window. It is useful for calculating totals across consecutive observations.

rolling_sum = sales.rolling(window=3).sum()

print(rolling_sum)

Output

2025-01-01      NaN
2025-01-02      NaN
2025-01-03    335.0
2025-01-04    375.0
2025-01-05    415.0
2025-01-06    455.0
Freq: D, Name: Sales, dtype: float64

12. Rolling Minimum and Maximum

Use min() and max() to find the smallest and largest values within each rolling window.

rolling_min = sales.rolling(window=3).min()
rolling_max = sales.rolling(window=3).max()

print("Rolling minimum:")
print(rolling_min)

print("\\nRolling maximum:")
print(rolling_max)

Output

Rolling minimum:
2025-01-01      NaN
2025-01-02      NaN
2025-01-03    100.0
2025-01-04    115.0
2025-01-05    115.0
2025-01-06    140.0
Freq: D, Name: Sales, dtype: float64

Rolling maximum:
2025-01-01      NaN
2025-01-02      NaN
2025-01-03    120.0
2025-01-04    140.0
2025-01-05    160.0
2025-01-06    160.0
Freq: D, Name: Sales, dtype: float64

13. Using min_periods

The min_periods parameter controls the minimum number of valid observations needed to produce a rolling result.

rolling_mean = sales.rolling(
    window=3,
    min_periods=1
).mean()

print(rolling_mean.round(2))

Output

2025-01-01    100.00
2025-01-02    110.00
2025-01-03    111.67
2025-01-04    125.00
2025-01-05    138.33
2025-01-06    151.67
Freq: D, Name: Sales, dtype: float64

The first result uses one observation, the second uses two, and subsequent results use the full three-observation window.

14. Centered Rolling Windows

By default, rolling results are aligned to the right edge of the window. Setting center=True aligns results to the center of the window.

centered_mean = sales.rolling(
    window=3,
    center=True
).mean()

print(centered_mean.round(2))

Output

2025-01-01       NaN
2025-01-02    111.67
2025-01-03    125.00
2025-01-04    138.33
2025-01-05    151.67
2025-01-06       NaN
Freq: D, Name: Sales, dtype: float64
Forecasting caution: Centered windows can use observations occurring after the timestamp being analyzed. Avoid them when constructing features that must use only past and current information.

15. Combining Shift and Rolling Operations

Shift and rolling operations can be combined to compare current values with previous values and calculate short-term trends.

analysis = pd.DataFrame({
    "Sales": sales,
    "Previous Sales": sales.shift(1),
    "Change": sales.diff(),
    "Three-Day Average": sales.rolling(
        window=3
    ).mean()
})

print(analysis.round(2))

Output

            Sales  Previous Sales  Change  Three-Day Average
2025-01-01    100             NaN     NaN                 NaN
2025-01-02    120           100.0    20.0                 NaN
2025-01-03    115           120.0    -5.0              111.67
2025-01-04    140           115.0    25.0              125.00
2025-01-05    160           140.0    20.0              138.33
2025-01-06    155           160.0    -5.0              151.67

This DataFrame combines the original sales, previous day's sales, day-to-day change, and three-observation moving average in one table.

16. Common Mistakes and Best Practices

Common Mistakes

  • Forgetting that shifting creates missing values at the edges.
  • Confusing positional shifting with datetime index shifting.
  • Ignoring missing values in rolling calculations.
  • Assuming a three-observation window always means three calendar days.
  • Using future observations in a predictive feature.

Best Practices

  • Sort observations chronologically when time order matters.
  • Choose a window size appropriate for the analysis.
  • Set min_periods intentionally.
  • Use shift() and diff() for lag comparisons.
  • Check whether a calculation uses future observations.

17. Practice Exercises

Exercise 1: Previous Observation

Create a Series of daily temperatures and use shift(1) to display the previous observation.

temperature = pd.Series([22, 24, 23, 27, 26])

# Write your solution below

Exercise 2: Daily Difference

Calculate the difference between each observation and the observation immediately before it.

prices = pd.Series([100, 105, 103, 110, 108])

# Write your solution below

Exercise 3: Moving Average

Calculate a three-observation rolling average and allow results when at least one valid observation is available.

visitors = pd.Series([200, 220, 210, 250, 270])

# Write your solution below

Lesson Summary

The Pandas shift() method moves values relative to their index and is useful for creating lagged observations. The diff() method calculates differences between observations, while shifted values can be used to calculate percentage changes.

Rolling windows calculate statistics over consecutive groups of observations. Common methods include mean(), sum(), min(), and max(). The min_periods and center parameters control how rolling results are calculated and aligned.

Combining shift and rolling operations helps analyze changes, smooth time-series data, and create useful features. Always consider missing values, index alignment, and future-data leakage when using these operations.