Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Time Series Data in Pandas

Time series analysis is the process of working with data collected, recorded, or measured over time. Pandas provides powerful tools for handling dates, timestamps, time-based indexing, resampling, and rolling calculations.

1. Learning Objectives

After completing this lesson, you will be able to:

  • Understand time series data and its applications.
  • Convert strings into Pandas datetime values.
  • Create date ranges using pd.date_range().
  • Use datetime values as a DataFrame index.
  • Select records using dates and time periods.
  • Resample data into daily, weekly, or monthly intervals.
  • Use shifting and rolling calculations for time-based analysis.

2. What Is Time Series Data?

Time series data consists of observations associated with specific points or periods in time. The observations may be collected every second, minute, hour, day, month, or year.

Examples of Time Series Data

  • Daily stock prices.
  • Hourly temperature readings.
  • Monthly sales revenue.
  • Website traffic recorded every minute.
  • Daily electricity consumption.
Daily Sales Over Time Mon Tue Wed Thu Fri Sat Sales Time →

A time series chart displays how a value changes over time.

3. Import Pandas

Start by importing the Pandas library.

import pandas as pd

Expected output:

# No output is displayed.
# Pandas is now available through the pd alias.

4. Understanding Datetime Values

Dates are sometimes stored as strings. Pandas can convert these strings into datetime values, making date-based operations easier.

Example: Convert Strings to Datetime

import pandas as pd

dates = pd.Series([
    "2026-01-01",
    "2026-01-02",
    "2026-01-03"
])

dates = pd.to_datetime(dates)

print(dates)
print(dates.dtype)

Expected output:

0   2026-01-01
1   2026-01-02
2   2026-01-03
dtype: datetime64[ns]
datetime64[ns]
Tip: Use pd.to_datetime() when you need to convert date strings into datetime values.

5. Creating a Date Range

The pd.date_range() function generates a sequence of dates or timestamps. It is useful when creating time series indexes.

Example: Generate Daily Dates

import pandas as pd

dates = pd.date_range(
    start="2026-01-01",
    periods=5,
    freq="D"
)

print(dates)

Expected output:

DatetimeIndex([
    '2026-01-01',
    '2026-01-02',
    '2026-01-03',
    '2026-01-04',
    '2026-01-05'
], dtype='datetime64[ns]', freq='D')

Common Frequency Aliases

Frequency Meaning Example
D Daily One interval per calendar day
h Hourly One interval per hour
min Every minute One interval per minute
W Weekly Weekly date intervals
MS Month start First day of each month
ME Month end Last day of each month
Important: Frequency aliases can differ across Pandas versions. In modern Pandas, use ME for month-end frequency; older examples may use M.

6. Using Datetime as a DataFrame Index

A datetime index allows Pandas to identify records by time and supports convenient date-based selection and resampling.

Example: Create a Time Series DataFrame

import pandas as pd

dates = pd.date_range(
    start="2026-01-01",
    periods=4,
    freq="D"
)

df = pd.DataFrame({
    "Sales": [1200, 1500, 1350, 1800]
}, index=dates)

print(df)

Expected output:

            Sales
2026-01-01   1200
2026-01-02   1500
2026-01-03   1350
2026-01-04   1800

Check the Index Type

print(type(df.index))

Expected output:

<class 'pandas.core.indexes.datetimes.DatetimeIndex'>

7. Selecting Data by Date

Once the index contains datetime values, you can select individual dates or date ranges using familiar DataFrame indexing techniques.

Select One Date

print(df.loc["2026-01-02"])

Expected output:

Sales    1500
Name: 2026-01-02 00:00:00, dtype: int64

Select a Date Range

print(df.loc["2026-01-02":"2026-01-03"])

Expected output:

            Sales
2026-01-02   1500
2026-01-03   1350
Tip: Label-based date slicing with .loc includes both endpoints when those labels exist in the index.

8. Resampling Time Series Data

Resampling changes the frequency of time series observations. It can aggregate detailed data into larger time intervals or convert data into a finer frequency.

For example, daily sales can be grouped into monthly totals. The index must be datetime-like for standard datetime resampling.

Example: Calculate Monthly Sales

import pandas as pd

dates = pd.date_range(
    start="2026-01-01",
    periods=6,
    freq="D"
)

sales = [100, 200, 150, 300, 250, 400]

df = pd.DataFrame({
    "Sales": sales
}, index=dates)

monthly_sales = df["Sales"].resample("ME").sum()

print(monthly_sales)

Expected output:

2026-01-31    1400
Freq: ME, Name: Sales, dtype: int64

Common Resampling Operations

Operation Purpose
.sum() Calculate the total for each interval.
.mean() Calculate the average for each interval.
.min() Find the minimum value for each interval.
.max() Find the maximum value for each interval.
.count() Count observations in each interval.

9. Shifting Time Series Data

The shift() method moves values by a specified number of rows. It is commonly used to compare current values with earlier or later observations.

Example: Compare Current and Previous Sales

import pandas as pd

sales = pd.Series(
    [100, 120, 90, 150],
    index=pd.date_range(
        "2026-01-01",
        periods=4,
        freq="D"
    )
)

previous_sales = sales.shift(1)

print(previous_sales)

Expected output:

2026-01-01      NaN
2026-01-02    100.0
2026-01-03    120.0
2026-01-04     90.0
Freq: D, dtype: float64

The first observation becomes NaN because no previous value exists. The other values move down by one position.

10. Rolling Window Calculations

A rolling window calculates a statistic over a moving group of observations. Rolling averages can help smooth short-term fluctuations and reveal patterns.

Example: Three-Observation Moving Average

import pandas as pd

sales = pd.Series([100, 120, 90, 150, 180])

rolling_average = sales.rolling(window=3).mean()

print(rolling_average)

Expected output:

0           NaN
1           NaN
2    103.333333
3    120.000000
4    140.000000
dtype: float64

With a window size of three, the first two positions are NaN because they do not yet contain three observations. By default, a complete window is required.

Tip: Use rolling(window=3).mean() for a three-row moving average. A row-based window is different from a time-based window such as rolling("3D").

11. Extracting Date and Time Components

The datetime accessor .dt provides properties for extracting information such as year, month, day, and weekday.

Example

import pandas as pd

dates = pd.Series(
    pd.to_datetime([
        "2026-01-05",
        "2026-02-10",
        "2026-03-15"
    ])
)

print(dates.dt.year)
print(dates.dt.month)
print(dates.dt.day)

Expected output:

0    2026
1    2026
2    2026
dtype: int32

0    1
1    2
2    3
dtype: int32

0     5
1    10
2    15
dtype: int32

The exact integer dtype displayed can vary by Pandas version. The extracted values remain the same.

12. Handling Missing Time Series Values

Time series data may contain missing observations because of sensor failures, incomplete records, or irregular data collection. Pandas provides methods to identify and handle missing values.

Example: Fill Missing Values

import pandas as pd

series = pd.Series(
    [10.0, None, 30.0, None, 50.0]
)

print(series.ffill())

Expected output:

0    10.0
1    10.0
2    30.0
3    30.0
4    50.0
dtype: float64

The ffill() method propagates the most recent non-missing value forward. Choose a filling strategy based on the meaning of your data; carrying values forward is not always appropriate.

13. Important Time Series Methods

Method Purpose
pd.to_datetime() Convert values into datetime objects.
pd.date_range() Generate a sequence of dates.
.resample() Group observations into time intervals.
.shift() Move values across rows.
.rolling() Calculate statistics over moving windows.
.dt.year Extract the year from datetime values.
.dt.month Extract the month from datetime values.
.ffill() Fill missing values using previous values.

14. Practice Exercises

Exercise 1: Create Dates

Create a date range containing seven daily dates starting from February 1, 2026.

Exercise 2: Build a Time Series

Create a DataFrame with daily dates as the index and a column containing seven sales values.

Exercise 3: Select a Date Range

Select observations from February 3 through February 5, 2026.

Exercise 4: Resample Data

Calculate monthly total sales from a DataFrame containing daily sales.

Exercise 5: Calculate a Moving Average

Calculate a three-observation rolling average for a Series of daily temperatures.

15. Knowledge Check

Question 1

Which function converts a column of date strings into datetime values?

Answer: pd.to_datetime()

Question 2

Which function creates a sequence of dates?

Answer: pd.date_range()

Question 3

Which method groups time series observations into monthly intervals?

Answer: .resample(), using an appropriate monthly frequency.

Question 4

Why can a rolling average begin with missing values?

Answer: The window may not contain enough observations to calculate a complete result.

Summary

Time series data represents observations over time. Pandas makes it easier to convert strings to datetime values, generate date ranges, create datetime indexes, select date-based records, and aggregate observations with resampling.

Methods such as shift(), rolling(), .dt, and ffill() support comparisons, moving calculations, date-component extraction, and missing-value handling. These tools are useful for analyzing sales, finance, weather, sensor readings, and other time-dependent datasets.