Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

DateTime Operations in Pandas

Pandas provides powerful datetime tools for converting date strings, extracting date components, performing date arithmetic, filtering records by time, and working with time zones.

1. Learning Objectives

After completing this lesson, you will be able to:

  • Convert strings and columns into datetime values.
  • Understand the difference between timestamps and timedeltas.
  • Extract year, month, day, and weekday information.
  • Perform date arithmetic and calculate time differences.
  • Format datetime values as strings.
  • Filter DataFrame rows using date conditions.
  • Understand basic timezone localization and conversion.

2. Converting Values to Datetime

Date values are often imported from CSV files, spreadsheets, or databases as strings. The pd.to_datetime() function converts supported date representations into Pandas datetime values.

Example

import pandas as pd

dates = pd.Series([
    "2026-04-10",
    "2026-04-11",
    "2026-04-12"
])

dates = pd.to_datetime(dates)

print(dates)
print(dates.dtype)

Expected output:

0   2026-04-10
1   2026-04-11
2   2026-04-12
dtype: datetime64[ns]
datetime64[ns]
Tip: Convert date columns to datetime before performing reliable date comparisons, sorting, or time-based calculations.

3. Handling Different Date Formats

Dates may be represented in different formats, such as 2026-04-10, 10/04/2026, or April 10, 2026. Pandas can parse many common formats. When the input format is known, specifying it makes the conversion more explicit.

Example: Specify the Format

import pandas as pd

dates = pd.Series([
    "10/04/2026",
    "11/04/2026",
    "12/04/2026"
])

converted = pd.to_datetime(
    dates,
    format="%d/%m/%Y"
)

print(converted)

Expected output:

0   2026-04-10
1   2026-04-11
2   2026-04-12
dtype: datetime64[ns]

Common Date Format Directives

Directive Meaning Example
%YFour-digit year2026
%yTwo-digit year26
%mMonth number04
%dDay of month10
%HHour in 24-hour format14
%MMinute30
%SSecond45
Important: The format must match the input. For example, use %d/%m/%Y for day/month/year. Ambiguous dates can otherwise be interpreted incorrectly.

4. Handling Invalid Dates

If a column contains invalid date strings, conversion can fail. Use errors="coerce" to convert invalid values into NaT, which represents a missing datetime value.

import pandas as pd

dates = pd.Series([
    "2026-04-10",
    "not-a-date",
    "2026-04-12"
])

converted = pd.to_datetime(
    dates,
    errors="coerce"
)

print(converted)

Expected output:

0   2026-04-10
1          NaT
2   2026-04-12
dtype: datetime64[ns]

You can identify invalid or missing datetime values using converted.isna().

5. Extracting Date Components

The .dt accessor provides convenient properties for extracting components from a datetime Series.

import pandas as pd

dates = pd.Series(
    pd.to_datetime([
        "2026-04-10",
        "2026-07-15",
        "2026-12-25"
    ])
)

print(dates.dt.year)
print(dates.dt.month)
print(dates.dt.day)

Expected output:

0    2026
1    2026
2    2026
dtype: int32

0     4
1     7
2    12
dtype: int32

0    10
1    15
2    25
dtype: int32

The exact integer dtype may vary by Pandas version. The extracted values are what matter.

Other Useful Components

Property Description
.dt.hourExtracts the hour.
.dt.minuteExtracts the minute.
.dt.secondExtracts the second.
.dt.dayofweekReturns weekday number, Monday = 0 and Sunday = 6.
.dt.day_name()Returns the weekday name.
.dt.month_name()Returns the month name.
.dt.quarterReturns the quarter from 1 to 4.
.dt.is_month_endChecks whether each date is the final day of its month.

6. Working with Weekday Names

Weekday names can be useful for analyzing business activity, scheduling, and comparing weekday and weekend patterns.

import pandas as pd

dates = pd.Series(
    pd.to_datetime([
        "2026-04-06",
        "2026-04-07",
        "2026-04-08"
    ])
)

print(dates.dt.day_name())

Expected output:

0     Monday
1    Tuesday
2  Wednesday
dtype: object

7. Date Arithmetic with Timedelta

A Timedelta represents a duration. You can add or subtract a duration from a timestamp to calculate another date or time.

Example: Add Days to a Date

import pandas as pd

date = pd.Timestamp("2026-04-10")

future_date = date + pd.Timedelta(days=5)
previous_date = date - pd.Timedelta(days=3)

print(future_date)
print(previous_date)

Expected output:

2026-04-15 00:00:00
2026-04-07 00:00:00

Common Timedelta Units

pd.Timedelta(days=2)
pd.Timedelta(hours=5)
pd.Timedelta(minutes=30)
pd.Timedelta(seconds=45)

These expressions create durations that can be used in date-time arithmetic.

8. Calculating Time Differences

Subtracting one datetime from another produces a Timedelta. This is useful for calculating durations between events.

import pandas as pd

start = pd.Timestamp("2026-04-10 09:00:00")
end = pd.Timestamp("2026-04-10 17:30:00")

duration = end - start

print(duration)
print(duration.total_seconds() / 3600)

Expected output:

0 days 08:30:00
8.5

The duration is eight hours and thirty minutes, or 8.5 hours. For durations, total_seconds() is useful when you need a numeric total rather than individual time components.

9. Formatting Datetime Values

The .dt.strftime() method converts datetime values into formatted strings. It is useful for reports and display labels.

import pandas as pd

dates = pd.Series(
    pd.to_datetime([
        "2026-04-10",
        "2026-04-11"
    ])
)

formatted = dates.dt.strftime("%d-%m-%Y")

print(formatted)

Expected output:

0    10-04-2026
1    11-04-2026
dtype: object
Remember: strftime() returns strings. If you need date calculations later, keep the original datetime values or convert the formatted values back to datetime.

10. Filtering Data by Date

Datetime values can be used in conditions to select records from a DataFrame.

import pandas as pd

df = pd.DataFrame({
    "Date": pd.to_datetime([
        "2026-04-01",
        "2026-04-05",
        "2026-04-10",
        "2026-04-15"
    ]),
    "Sales": [100, 200, 300, 400]
})

filtered = df[
    (df["Date"] >= "2026-04-05") &
    (df["Date"] <= "2026-04-10")
]

print(filtered)

Expected output:

        Date  Sales
1 2026-04-05    200
2 2026-04-10    300

The & operator combines the two conditions. Each condition must be enclosed in parentheses when combining Pandas Series comparisons.

11. Using Datetime Index Slicing

If dates are stored in a sorted DatetimeIndex, you can use .loc to select a date or a range of dates.

import pandas as pd

dates = pd.date_range(
    "2026-04-01",
    periods=5,
    freq="D"
)

df = pd.DataFrame({
    "Temperature": [25, 27, 26, 29, 28]
}, index=dates)

print(df.loc["2026-04-02":"2026-04-04"])

Expected output:

            Temperature
2026-04-02           27
2026-04-03           26
2026-04-04           29
Tip: A sorted datetime index makes date-range selection easier and more predictable.

12. Timezone Localization

A timezone-naive timestamp does not contain timezone information. Localization assigns a timezone to a naive timestamp without changing its displayed wall-clock time.

import pandas as pd

date = pd.Timestamp("2026-04-10 09:00:00")

localized = date.tz_localize("UTC")

print(localized)

Expected output:

2026-04-10 09:00:00+00:00

The timestamp now includes UTC timezone information.

13. Timezone Conversion

The tz_convert() method converts a timezone-aware timestamp to another timezone while preserving the same instant in time.

import pandas as pd

utc_date = pd.Timestamp(
    "2026-04-10 09:00:00",
    tz="UTC"
)

india_date = utc_date.tz_convert("Asia/Kolkata")

print(utc_date)
print(india_date)

Expected output:

2026-04-10 09:00:00+00:00
2026-04-10 14:30:00+05:30
Important: Use tz_localize() to assign a timezone to naive datetime values, and tz_convert() to convert values that already have timezone information.

14. Common DateTime Methods

Method or Property Purpose
pd.to_datetime()Convert values to datetime.
pd.Timestamp()Create a timestamp.
pd.Timedelta()Represent a duration.
.dt.yearExtract the year.
.dt.monthExtract the month.
.dt.dayExtract the day of the month.
.dt.day_name()Get the weekday name.
.dt.strftime()Format datetime values as strings.
.tz_localize()Assign timezone information to naive datetime values.
.tz_convert()Convert timezone-aware values to another timezone.

15. Practice Exercises

Exercise 1: Convert Dates

Convert the strings 2026-05-01, 2026-05-02, and 2026-05-03 into datetime values.

Exercise 2: Extract Components

Create a datetime Series and extract its year, month, and day using .dt.

Exercise 3: Calculate a Duration

Calculate the difference between 08:15 and 16:45 on the same day.

Exercise 4: Filter Records

Filter a DataFrame to include only records dated between May 5 and May 15, 2026.

Exercise 5: Convert Timezones

Create a UTC timestamp and convert it to the Asia/Kolkata timezone.

16. Knowledge Check

Question 1

Which function converts a Series of date strings into datetime values?

Answer: pd.to_datetime()

Question 2

Which accessor is used to extract the month from a datetime Series?

Answer: .dt.month

Question 3

What does subtracting one timestamp from another produce?

Answer: A Timedelta representing the duration between the timestamps.

Question 4

What is the difference between tz_localize() and tz_convert()?

Answer: Localization assigns a timezone to naive datetime values; conversion changes the timezone representation of timezone-aware values.

Summary

Pandas datetime operations make it easier to prepare, inspect, and analyze date and time data. You can convert strings to datetime values, extract date components, calculate durations, format dates, and filter rows by date conditions.

Timezone localization and conversion help manage timestamps across regions. These operations are important when analyzing events, transactions, schedules, and time-dependent datasets.