Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Resampling Data in Pandas

Resampling is a powerful Pandas technique for changing the frequency of time series data. It allows you to aggregate high-frequency observations into daily, weekly, or monthly summaries and convert data into different time intervals.

1. Learning Objectives

After completing this lesson, you will be able to:

  • Understand the meaning and purpose of resampling.
  • Use resample() with a DatetimeIndex.
  • Aggregate data into daily, weekly, and monthly intervals.
  • Apply sum, mean, minimum, maximum, and count operations.
  • Resample multiple DataFrame columns.
  • Understand upsampling and missing values.
  • Use resampling to analyze real-world time series data.

2. What Is Resampling?

Resampling changes the frequency of time series observations. For example, hourly temperature measurements can be converted into daily averages, or daily sales can be grouped into monthly totals.

Common Applications

  • Convert daily sales into monthly sales totals.
  • Calculate daily average temperatures from hourly readings.
  • Summarize minute-level sensor readings into hourly values.
  • Aggregate financial data into weekly or monthly reports.
  • Prepare time series data for visualization and analysis.
How Resampling Works Daily Sales Many observations sum() Monthly Total Fewer observations Resampling groups time-based observations into new intervals.

Resampling can reduce many detailed observations into a smaller summary dataset.

3. Understanding the resample() Method

The resample() method groups observations into time-based intervals. You then apply an aggregation method such as sum() or mean() to calculate a result for each interval.

General syntax:

df.resample("frequency").aggregation_method()
  • df is the DataFrame or Series.
  • frequency defines the time interval.
  • aggregation_method() calculates the summary value.
Important: For the standard use of resample(), the index must be a DatetimeIndex, TimedeltaIndex, or PeriodIndex. Alternatively, specify a datetime column using the on parameter.

4. Creating Sample Time Series Data

Before resampling, create a DataFrame containing dates as the index and sales values as the data.

import pandas as pd

dates = pd.date_range(
    start="2026-01-01",
    periods=10,
    freq="D"
)

df = pd.DataFrame({
    "Sales": [100, 150, 120, 180, 200,
              170, 220, 190, 250, 230]
}, index=dates)

print(df)

Expected output:

            Sales
2026-01-01    100
2026-01-02    150
2026-01-03    120
2026-01-04    180
2026-01-05    200
2026-01-06    170
2026-01-07    220
2026-01-08    190
2026-01-09    250
2026-01-10    230

This DataFrame contains daily sales observations and can be resampled into larger time intervals.

5. Resampling Daily Data into Monthly Totals

Use the month-end frequency ME and sum() to calculate the total sales for each month.

monthly_sales = df["Sales"].resample("ME").sum()

print(monthly_sales)

Expected output:

2026-01-31    1810
Freq: ME, Name: Sales, dtype: int64

All ten observations fall in January 2026. Their sum is 1,810, so the monthly result contains one observation labeled with the month-end date.

Tip: In modern Pandas, use ME for month-end frequency. Some older examples use the legacy M alias.

6. Resampling into Weekly Data

Weekly resampling groups observations into weekly intervals. By default, the W frequency uses weeks ending on Sunday.

weekly_sales = df["Sales"].resample("W").sum()

print(weekly_sales)

Expected output:

2026-01-04    550
2026-01-11   1260
Freq: W-SUN, Name: Sales, dtype: int64

The first interval contains January 1 through January 4. The second contains January 5 through January 11, including only the available observations through January 10.

7. Resampling into Daily Averages

Resampling can also calculate statistics other than totals. Use mean() to calculate the average value in each interval.

daily_average = df["Sales"].resample("D").mean()

print(daily_average.head())

Expected output:

2026-01-01    100.0
2026-01-02    150.0
2026-01-03    120.0
2026-01-04    180.0
2026-01-05    200.0
Freq: D, Name: Sales, dtype: float64

Because each daily interval contains one observation in this dataset, the daily average equals the original daily sales value.

8. Common Aggregation Methods

Pandas provides several aggregation methods that can be applied after resampling.

Method Purpose
sum() Calculates the total for each interval.
mean() Calculates the average value.
min() Returns the smallest value.
max() Returns the largest value.
count() Counts non-missing observations.
first() Returns the first valid observation in each interval.
last() Returns the last valid observation in each interval.
median() Calculates the median value.
std() Calculates the sample standard deviation.

Example: Apply Multiple Aggregations

result = df["Sales"].resample("W").agg(
    ["sum", "mean", "min", "max", "count"]
)

print(result)

Expected output:

             sum        mean  min  max  count
2026-01-04   550  137.500000  100  180      4
2026-01-11  1260  210.000000  170  250      6

9. Resampling Multiple Columns

A DataFrame may contain several numeric columns. You can resample multiple columns together and calculate summaries for each one.

import pandas as pd

dates = pd.date_range(
    "2026-01-01",
    periods=6,
    freq="D"
)

df = pd.DataFrame({
    "Sales": [100, 150, 200, 120, 180, 250],
    "Orders": [10, 15, 20, 12, 18, 25]
}, index=dates)

monthly = df.resample("ME").sum()

print(monthly)

Expected output:

            Sales  Orders
2026-01-31   1000     100

Pandas sums each numeric column independently for the monthly interval.

10. Resampling with a Datetime Column

A DatetimeIndex is not always required. If your dates are stored in a DataFrame column, use the on parameter to identify the datetime column.

import pandas as pd

df = pd.DataFrame({
    "Date": pd.to_datetime([
        "2026-01-01",
        "2026-01-15",
        "2026-02-05",
        "2026-02-20"
    ]),
    "Sales": [100, 200, 150, 250]
})

monthly = df.resample("ME", on="Date")["Sales"].sum()

print(monthly)

Expected output:

Date
2026-01-31    300
2026-02-28    400
Freq: ME, Name: Sales, dtype: int64

The on="Date" argument tells Pandas to use the Date column for grouping observations into monthly intervals.

11. Understanding Upsampling

Upsampling increases the frequency of a time series. For example, daily data can be converted into hourly-indexed data. New time intervals may contain missing values because the original dataset has no observations for those timestamps.

Example: Convert Daily Data to 12-Hour Intervals

import pandas as pd

dates = pd.date_range(
    "2026-01-01",
    periods=3,
    freq="D"
)

series = pd.Series(
    [100, 200, 300],
    index=dates
)

result = series.resample("12h").asfreq()

print(result)

Expected output:

2026-01-01 00:00:00    100.0
2026-01-01 12:00:00      NaN
2026-01-02 00:00:00    200.0
2026-01-02 12:00:00      NaN
2026-01-03 00:00:00    300.0
Freq: 12h, dtype: float64

The new noon timestamps have no original values, so Pandas represents them as NaN.

12. Filling Missing Values After Upsampling

After upsampling, you can fill missing values using forward filling or backward filling. The correct strategy depends on the meaning of the data.

Example: Forward Fill

result = series.resample("12h").ffill()

print(result)

Expected output:

2026-01-01 00:00:00    100
2026-01-01 12:00:00    100
2026-01-02 00:00:00    200
2026-01-02 12:00:00    200
2026-01-03 00:00:00    300
Freq: 12h, dtype: int64

Forward filling propagates the most recent available value into newly created intervals until another observation appears.

Important: Filling missing observations changes the resulting data. Do not use forward fill or interpolation unless the chosen approach is suitable for the dataset.

13. Resampling with First and Last Values

Some time series applications require the first or last observation in each interval rather than a sum or average.

import pandas as pd

dates = pd.date_range(
    "2026-01-01",
    periods=6,
    freq="D"
)

prices = pd.Series(
    [100, 110, 105, 120, 125, 130],
    index=dates
)

weekly_first = prices.resample("W").first()
weekly_last = prices.resample("W").last()

print("First:")
print(weekly_first)

print("Last:")
print(weekly_last)

Expected output:

First:
2026-01-04    100
2026-01-11    125
Freq: W-SUN, dtype: int64

Last:
2026-01-04    120
2026-01-11    130
Freq: W-SUN, dtype: int64

The first weekly interval ends on January 4. The second ends on January 11 and contains the remaining available observations.

14. Common Resampling Frequencies

Frequency Description Example Application
hHourly intervalsHourly sensor readings
12h12-hour intervalsHalf-day summaries
DDaily intervalsDaily sales reports
WWeekly intervals ending Sunday by defaultWeekly sales reports
MSMonth-start intervalsMonthly periods beginning at month start
MEMonth-end intervalsMonth-end summaries
QEQuarter-end intervalsQuarterly financial summaries
YEYear-end intervalsAnnual totals

The correct frequency depends on the reporting period. Check the supported aliases for your installed Pandas version, especially when using calendar-end frequencies.

15. Real-World Example: Monthly Revenue Report

Consider a business that records transactions on different days. Resampling can calculate total monthly revenue from the daily records.

import pandas as pd

df = pd.DataFrame({
    "Date": pd.to_datetime([
        "2026-01-05",
        "2026-01-18",
        "2026-02-03",
        "2026-02-22",
        "2026-03-10"
    ]),
    "Revenue": [5000, 7000, 6000, 9000, 8000]
})

monthly_revenue = (
    df.resample("ME", on="Date")["Revenue"]
      .sum()
)

print(monthly_revenue)

Expected output:

Date
2026-01-31    12000
2026-02-28    15000
2026-03-31     8000
Freq: ME, Name: Revenue, dtype: int64

The result shows total revenue for January, February, and March 2026. This type of summary is useful for monthly reporting and trend analysis.

16. Common Mistakes to Avoid

Using a Non-Datetime Index

Convert the date column with pd.to_datetime() and set it as the index, or pass the datetime column through the on parameter.

Choosing the Wrong Frequency

Daily, weekly, and month-end intervals produce different groups. Choose the frequency that matches the reporting rule.

Ignoring Missing Values

Upsampling may create missing observations. Inspect them before deciding whether to fill, interpolate, or leave them missing.

Assuming Every Interval Has Data

Some periods may have no observations. Depending on the operation, the resulting interval can contain missing values or an aggregation identity such as zero for a sum.

17. Practice Exercises

Exercise 1: Monthly Sales

Create daily sales data for January and February 2026 and calculate monthly sales totals.

Exercise 2: Weekly Average

Generate 20 daily observations and calculate their weekly average.

Exercise 3: Multiple Aggregations

Calculate the sum, mean, minimum, and maximum of weekly sales.

Exercise 4: Upsampling

Convert daily observations into 12-hour intervals and identify the newly created missing values.

Exercise 5: Revenue Report

Create a DataFrame with a Date column and Revenue column, then calculate monthly revenue using the on parameter.

18. Knowledge Check

Question 1

Which method is used to resample time series data?

Answer: resample()

Question 2

Which operation calculates monthly sales totals?

Answer: resample("ME").sum()

Question 3

What is upsampling?

Answer: Increasing the frequency of a time series, potentially creating new intervals without original observations.

Question 4

How can you resample using a datetime column without setting it as the index?

Answer: Use the on parameter, for example df.resample("ME", on="Date").

Summary

Resampling is used to change the frequency of time series data. The resample() method groups observations into daily, weekly, monthly, or other time intervals, after which aggregation methods calculate totals, averages, and other statistics.

Resampling works with datetime-based indexes or a datetime column specified through on. Downsampling creates summaries over larger intervals, while upsampling creates finer intervals and may introduce missing values. Choosing appropriate frequencies and missing-value strategies is essential for accurate analysis.