Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Performance and Large Datasets

Learn how to improve Pandas performance, reduce memory consumption, process large datasets efficiently and select the right tools for scalable data analysis.

1. Introduction to Pandas Performance

Pandas is a powerful Python library for data analysis. However, processing millions of rows or working with memory-intensive DataFrames can require careful optimization.

Performance optimization means reducing unnecessary computation, using memory efficiently and choosing appropriate methods for the size and structure of your data.

Learning Objectives

  • Understand common performance bottlenecks.
  • Measure execution time using Python tools.
  • Inspect DataFrame memory consumption.
  • Choose efficient data types.
  • Process large CSV files in chunks.
  • Understand PyArrow, Dask and Polars.
  • Apply practical optimization techniques.

2. Why Performance Optimization Matters

Large datasets can increase processing time and memory requirements. Inefficient operations may slow down a program or cause it to run out of available memory.

Problem Possible Cause Recommended Approach
High memory usage Unnecessarily large data types or unused columns. Optimize data types and select required columns.
Slow calculations Repeated Python-level loops. Prefer vectorized operations when appropriate.
Slow file loading Reading unnecessary columns or rows. Use usecols and suitable data types.
Memory exhaustion Loading a dataset that is too large at once. Read and process data in chunks.
Repeated calculations Performing the same expensive operation multiple times. Reuse results where appropriate.

3. Measuring Execution Time

Before optimizing code, measure its execution time. Python provides the time module for basic timing and timeit for repeated measurements.

Example: Measure an Operation

import time
import pandas as pd

numbers = pd.Series(range(1_000_000))

start = time.perf_counter()

result = numbers * 2

end = time.perf_counter()

print("Execution time:", round(end - start, 4), "seconds")

The program records the elapsed time for the multiplication operation. The exact result depends on your hardware, Python environment and system workload.

Best practice: Measure a representative workload several times before comparing optimization strategies. Do not assume a change is faster without testing.

4. Inspecting DataFrame Memory Usage

Use memory_usage() and info(memory_usage="deep") to understand how much memory a DataFrame consumes.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Amit", "Neha", "Raj", "Priya"],
    "Age": [25, 30, 22, 28],
    "Salary": [35000, 52000, 28000, 45000]
})

print(df.info(memory_usage="deep"))

print(df.memory_usage(deep=True))

total_bytes = df.memory_usage(deep=True).sum()

print("Total memory:", total_bytes, "bytes")

The output reports the data types, non-null counts and memory usage of the DataFrame. The precise byte counts depend on the stored values and Pandas version.

Note: deep=True provides a more comprehensive estimate for object-backed data, including the memory occupied by contained objects.

5. Avoid Loading Unnecessary Columns

If a CSV file contains many columns, load only the columns required for your analysis. This can reduce both memory usage and file-reading overhead.

import pandas as pd

df = pd.read_csv(
    "sales.csv",
    usecols=["Product", "Quantity", "Price"]
)

print(df.head())

In this example, Pandas loads only the three specified columns from sales.csv.

Reading a Small Sample

sample = pd.read_csv(
    "sales.csv",
    nrows=1000
)

print(sample.shape)

The nrows parameter is useful when exploring a file or testing an analysis on a small sample before processing the complete dataset.

6. Prefer Vectorized Operations

Vectorized operations apply calculations across a Series or array without requiring an explicit Python loop for every value. They are generally preferable for simple numerical transformations.

Vectorized Approach

import pandas as pd

df = pd.DataFrame({
    "Price": [100, 200, 300, 400]
})

df["PriceWithTax"] = df["Price"] * 1.18

print(df)

Output

   Price  PriceWithTax
0    100         118.0
1    200         236.0
2    300         354.0
3    400         472.0

The calculation is applied to the complete column. For straightforward arithmetic, this is usually clearer than iterating through every row.

7. Choose Efficient Data Types

Data types influence how much memory a DataFrame uses. For example, a small integer range may not require a 64-bit integer representation.

import pandas as pd

df = pd.DataFrame({
    "Age": [20, 25, 30, 35],
    "Quantity": [1, 2, 3, 4]
})

print("Before optimization:")
print(df.dtypes)

df["Age"] = df["Age"].astype("int8")
df["Quantity"] = df["Quantity"].astype("int8")

print("After optimization:")
print(df.dtypes)

Both columns contain values that fit within the int8 range. Choosing a smaller integer type can reduce memory usage for suitable datasets.

Important: Do not downcast blindly. Confirm the minimum and maximum values first. A value outside the target type's range can cause incorrect results or conversion problems.

8. Optimize Repeated Text Values with category

Columns containing a limited number of repeated strings may benefit from Pandas' category data type. This is especially useful for columns such as department, status or product category.

import pandas as pd

df = pd.DataFrame({
    "Department": [
        "IT", "HR", "IT", "Finance",
        "HR", "IT", "Finance"
    ]
})

print("Before:")
print(df.memory_usage(deep=True))

df["Department"] = df["Department"].astype("category")

print("After:")
print(df.memory_usage(deep=True))

print(df.dtypes)

The category data type stores category information separately from the underlying category codes. The memory benefit depends on the number of distinct values and the size of the dataset.

Use category when: a column has many repeated values relative to the number of unique values. Test memory consumption on your actual dataset.

9. Processing Large Datasets in Chunks

When a CSV file is too large to fit comfortably into memory, Pandas can read it in smaller batches using chunksize.

import pandas as pd

total_sales = 0

for chunk in pd.read_csv(
    "sales.csv",
    chunksize=10000
):
    total_sales += chunk["Sales"].sum()

print("Total sales:", total_sales)

Instead of loading the entire file into one DataFrame, this example processes up to 10,000 rows per chunk and accumulates the total.

Practical use: Chunking is useful for totals, counts, aggregations and transformations that can be processed incrementally. Some analyses require additional logic to combine results correctly across chunks.

10. Using query() for Filtering

The query() method lets you express suitable filtering conditions using a readable string expression.

import pandas as pd

df = pd.DataFrame({
    "Product": ["Laptop", "Mouse", "Keyboard", "Monitor"],
    "Price": [60000, 800, 1500, 12000],
    "Stock": [5, 50, 25, 10]
})

result = df.query("Price > 1000 and Stock >= 10")

print(result)

Output

    Product  Price  Stock
2  Keyboard   1500     25
3   Monitor  12000     10

The example selects rows whose price is above 1,000 and whose stock is at least 10.

Performance note: query() is not guaranteed to be faster than Boolean indexing for every dataset. Benchmark both approaches for performance-critical workloads.

11. Avoid Unnecessary Copies and Intermediate DataFrames

Creating many temporary DataFrames can increase memory usage. Select required columns and combine compatible operations where doing so keeps the code readable.

Select Only Required Columns

import pandas as pd

df = pd.read_csv("sales.csv")

analysis_df = df[["Product", "Sales"]]

print(analysis_df.head())

This selects only the columns required for the analysis. Depending on the operation and Pandas version, selecting data may produce a copy or a view-like object. Do not assume that every selection avoids copying memory.

Combine Compatible Calculations

df["Revenue"] = df["Quantity"] * df["Price"]

df["DiscountedRevenue"] = (
    df["Revenue"] * (1 - df["DiscountRate"])
)

Here, the calculations are explicit and each derived column can be reused by subsequent operations.

12. Choosing the Right Tool for Large Data

Pandas is suitable for many data analysis tasks, but other tools may be useful when datasets, execution requirements or system resources change.

Tool Main Strength When to Consider It
Pandas Flexible DataFrame-based analysis. Everyday analysis and datasets that fit available resources.
PyArrow Columnar data formats and efficient data interchange. Parquet, Arrow-backed data and supported I/O workflows.
Dask Parallel and partitioned DataFrame computations. Workloads that benefit from partitioning or distributed execution.
Polars Expression-based DataFrame processing and query optimization. Workloads suited to its execution model and supported operations.
Remember: A different library is not automatically faster for every workload. Consider data size, available memory, I/O patterns, required operations and the cost of changing your existing workflow.

13. A Practical Performance Checklist

Before Processing

  • Inspect the shape and data types of your dataset.
  • Identify which columns are actually required.
  • Measure current execution time and memory usage.
  • Check for unnecessary repeated calculations.

During Optimization

  • Prefer vectorized operations where appropriate.
  • Evaluate smaller numeric data types safely.
  • Consider categorical types for repeated strings.
  • Use chunking when processing large CSV files.
  • Benchmark changes using representative data.

After Optimization

  • Verify that the output remains correct.
  • Compare execution time before and after.
  • Check the final memory footprint.
  • Document any assumptions about data types and missing values.

14. Practice Exercises

Exercise 1: Inspect Memory

Create a DataFrame containing names, ages and departments. Display its data types and memory usage.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Amit", "Neha", "Raj", "Priya"],
    "Age": [22, 25, 30, 28],
    "Department": ["IT", "HR", "IT", "Finance"]
})

# Write your solution below

Exercise 2: Optimize Data Types

Convert a suitable integer column to a smaller numeric type after checking its range. Compare memory usage before and after.

quantities = pd.Series([1, 2, 3, 4, 5])

# Write your solution below

Exercise 3: Process a CSV in Chunks

Read a large sales CSV in batches and calculate the total value of its Sales column.

# Use pd.read_csv() with chunksize
# Accumulate the Sales total from each chunk

Summary

Pandas performance optimization focuses on reducing unnecessary computation, controlling memory consumption and choosing efficient operations for the dataset.

Inspect memory usage, select only required columns, prefer vectorized calculations and use appropriate data types. For large CSV files, chunked processing can avoid loading the complete dataset into memory.

For larger or more demanding workloads, evaluate PyArrow, Dask or Polars based on your requirements. Always benchmark your changes and verify that the optimized output remains correct.