Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Time-Based Indexing in Pandas

Time-based indexing allows you to select, filter, and analyze records using dates and timestamps. Pandas provides powerful datetime indexes and label-based selection tools for working with time-dependent datasets.

1. Learning Objectives

After completing this lesson, you will be able to:

  • Understand time-based indexing in Pandas.
  • Create a DataFrame with a DatetimeIndex.
  • Select records for a particular date.
  • Filter data using date ranges.
  • Use partial date strings for selection.
  • Apply Boolean conditions to datetime columns.
  • Sort and slice time-based data correctly.

2. What Is Time-Based Indexing?

Time-based indexing is a technique for accessing data using dates or timestamps instead of ordinary integer positions. It is particularly useful when a dataset contains daily sales, hourly temperatures, stock prices, or event records.

Example Applications

  • Retrieve sales recorded on a specific day.
  • Analyze temperatures recorded during a week.
  • Extract all transactions from a particular month.
  • Compare performance across different time periods.
  • Inspect sensor readings recorded at specific timestamps.
Time-Based Indexing Date Sales Action 2026-04-01 120 Select by date 2026-04-02 150 Select by date 2026-04-03 180 Select by date Use a DatetimeIndex to select observations by time.

A DatetimeIndex makes date-based selection convenient and readable.

3. Import Pandas

Import Pandas before creating a time series.

import pandas as pd

Expected output:

# No output is displayed.

4. Creating a DatetimeIndex

A DatetimeIndex stores dates and timestamps as index labels. It enables date-based selection through methods such as .loc[].

Example: Create a Time-Based DataFrame

import pandas as pd

dates = pd.date_range(
    start="2026-04-01",
    periods=5,
    freq="D"
)

df = pd.DataFrame({
    "Sales": [120, 150, 180, 140, 200],
    "Customers": [12, 15, 18, 14, 20]
}, index=dates)

print(df)

Expected output:

            Sales  Customers
2026-04-01    120         12
2026-04-02    150         15
2026-04-03    180         18
2026-04-04    140         14
2026-04-05    200         20

Check the Index Type

print(type(df.index))

Expected output:

<class 'pandas.core.indexes.datetimes.DatetimeIndex'>

5. Selecting Data for a Specific Date

Use .loc[] to select a row using its datetime index label.

print(df.loc["2026-04-03"])

Expected output:

Sales        180
Customers     18
Name: 2026-04-03 00:00:00, dtype: int64

The result is a Series because one row containing multiple columns has been selected.

Tip: Use .loc[] when selecting by index labels. Use .iloc[] when selecting by integer position.

6. Selecting a Date Range

You can select a continuous range of dates by passing a start and end date to .loc[].

Example

print(df.loc["2026-04-02":"2026-04-04"])

Expected output:

            Sales  Customers
2026-04-02    150         15
2026-04-03    180         18
2026-04-04    140         14

For label-based slicing with .loc[], both endpoints are included when matching index labels exist.

7. Partial Date Indexing

Pandas supports partial date strings when the DataFrame has a DatetimeIndex. This can make it easier to select all observations from a year or month.

Example: Select a Month

import pandas as pd

dates = pd.date_range(
    "2026-04-01",
    "2026-06-30",
    freq="D"
)

df = pd.DataFrame({
    "Sales": range(100, 191)
}, index=dates)

april_data = df.loc["2026-04"]

print(april_data.head())
print(len(april_data))

Expected output:

            Sales
2026-04-01    100
2026-04-02    101
2026-04-03    102
2026-04-04    103
2026-04-05    104
30

The expression df.loc["2026-04"] selects observations from April 2026. The head() method displays the first five rows, while len() returns the number of selected rows.

8. Selecting Data by Year

A year string can be used to select all observations belonging to that year.

import pandas as pd

dates = pd.date_range(
    "2025-12-29",
    periods=7,
    freq="D"
)

df = pd.DataFrame({
    "Visitors": [80, 90, 100, 110, 120, 130, 140]
}, index=dates)

year_data = df.loc["2026"]

print(year_data)

Expected output:

            Visitors
2026-01-01       100
2026-01-02       110
2026-01-03       120
2026-01-04       130
2026-01-05       140

Only the rows whose index belongs to 2026 are returned.

9. Filtering with a Datetime Column

A datetime column can be filtered using Boolean conditions even when it is not the index.

import pandas as pd

df = pd.DataFrame({
    "Date": pd.to_datetime([
        "2026-04-01",
        "2026-04-05",
        "2026-04-10",
        "2026-04-15"
    ]),
    "Sales": [100, 200, 300, 400]
})

filtered = df[
    (df["Date"] >= "2026-04-05") &
    (df["Date"] <= "2026-04-10")
]

print(filtered)

Expected output:

        Date  Sales
1 2026-04-05    200
2 2026-04-10    300
Important: Use parentheses around each condition when combining Pandas comparisons with & or |.

10. Filtering by Month and Year

If the date is stored in a regular DataFrame column, use the .dt accessor to filter by components such as year and month.

import pandas as pd

df = pd.DataFrame({
    "Date": pd.to_datetime([
        "2026-04-05",
        "2026-04-12",
        "2026-05-02",
        "2026-05-10"
    ]),
    "Sales": [100, 150, 200, 250]
})

may_data = df[
    (df["Date"].dt.year == 2026) &
    (df["Date"].dt.month == 5)
]

print(may_data)

Expected output:

        Date  Sales
2 2026-05-02    200
3 2026-05-10    250

11. Sorting Time-Based Data

Time-based data is often sorted chronologically before slicing, plotting, or calculating time-dependent statistics.

import pandas as pd

df = pd.DataFrame({
    "Sales": [300, 100, 200]
}, index=pd.to_datetime([
    "2026-04-03",
    "2026-04-01",
    "2026-04-02"
]))

df = df.sort_index()

print(df)

Expected output:

            Sales
2026-04-01    100
2026-04-02    200
2026-04-03    300
Best practice: Use sort_index() when chronological order is required. Do not assume incoming data is already sorted.

12. Working with Timestamps

When an index contains date and time values, you can select a specific timestamp rather than an entire day.

import pandas as pd

dates = pd.to_datetime([
    "2026-04-01 09:00:00",
    "2026-04-01 12:00:00",
    "2026-04-01 15:00:00"
])

df = pd.DataFrame({
    "Temperature": [24, 28, 26]
}, index=dates)

print(df.loc["2026-04-01 12:00:00"])

Expected output:

Temperature    28
Name: 2026-04-01 12:00:00, dtype: int64

13. Selecting Data by Time of Day

When the index contains timestamps, the .between_time() method can select records whose times fall within a specified daily time interval.

import pandas as pd

dates = pd.date_range(
    "2026-04-01 09:00",
    periods=5,
    freq="h"
)

df = pd.DataFrame({
    "Temperature": [22, 24, 27, 26, 23]
}, index=dates)

result = df.between_time("10:00", "12:00")

print(result)

Expected output:

                     Temperature
2026-04-01 10:00:00           24
2026-04-01 11:00:00           27
2026-04-01 12:00:00           26

By default, the start and end times are included. The method operates on the time component of a DatetimeIndex, making it useful for selecting daily business hours across multiple dates.

14. Common Time-Based Indexing Methods

Method Purpose
pd.date_range() Generate a sequence of dates or timestamps.
pd.to_datetime() Convert values into datetime data.
.loc[] Select data by date or index label.
.iloc[] Select data by integer position.
.sort_index() Sort rows by index labels.
.between_time() Select rows by time of day.
.dt.year Extract the year from a datetime column.
.dt.month Extract the month from a datetime column.

15. Practice Exercises

Exercise 1: Create a DatetimeIndex

Create seven daily dates starting from June 1, 2026, and use them as the index of a DataFrame containing sales values.

Exercise 2: Select a Specific Date

Retrieve the row for June 4, 2026, using .loc[].

Exercise 3: Select a Month

Create daily observations for April and May 2026, then select all April observations using partial date indexing.

Exercise 4: Filter by Date

Filter a DataFrame to include only dates between June 5 and June 15, 2026.

Exercise 5: Select Business Hours

Create hourly observations and select records between 09:00 and 12:00 using .between_time().

16. Knowledge Check

Question 1

Which index type is commonly used for time-based indexing?

Answer: DatetimeIndex

Question 2

Which method selects rows using datetime index labels?

Answer: .loc[]

Question 3

What does df.loc["2026-04"] select when the DataFrame has a suitable DatetimeIndex?

Answer: Observations belonging to April 2026.

Question 4

Which method selects rows based on the time of day?

Answer: .between_time()

Question 5

Why is sorting a DatetimeIndex useful?

Answer: It keeps observations in chronological order and supports predictable time-based workflows.

Summary

Time-based indexing enables convenient selection and filtering using dates and timestamps. A DatetimeIndex supports exact-date selection, date-range slicing, and partial string indexing by month or year.

Datetime columns can also be filtered using Boolean conditions and the .dt accessor. Additional tools such as sort_index() and between_time() help organize observations and select records by time of day.