Performance and Large Datasets
Learn how to improve Pandas performance, reduce memory consumption, process large datasets efficiently and select the right tools for scalable data analysis.
1. Introduction to Pandas Performance
Pandas is a powerful Python library for data analysis. However, processing millions of rows or working with memory-intensive DataFrames can require careful optimization.
Performance optimization means reducing unnecessary computation, using memory efficiently and choosing appropriate methods for the size and structure of your data.
Learning Objectives
- Understand common performance bottlenecks.
- Measure execution time using Python tools.
- Inspect DataFrame memory consumption.
- Choose efficient data types.
- Process large CSV files in chunks.
- Understand PyArrow, Dask and Polars.
- Apply practical optimization techniques.
2. Why Performance Optimization Matters
Large datasets can increase processing time and memory requirements. Inefficient operations may slow down a program or cause it to run out of available memory.
| Problem | Possible Cause | Recommended Approach |
|---|---|---|
| High memory usage | Unnecessarily large data types or unused columns. | Optimize data types and select required columns. |
| Slow calculations | Repeated Python-level loops. | Prefer vectorized operations when appropriate. |
| Slow file loading | Reading unnecessary columns or rows. | Use usecols and suitable data types. |
| Memory exhaustion | Loading a dataset that is too large at once. | Read and process data in chunks. |
| Repeated calculations | Performing the same expensive operation multiple times. | Reuse results where appropriate. |
3. Measuring Execution Time
Before optimizing code, measure its execution time.
Python provides the time module for basic
timing and timeit for repeated measurements.
Example: Measure an Operation
import time
import pandas as pd
numbers = pd.Series(range(1_000_000))
start = time.perf_counter()
result = numbers * 2
end = time.perf_counter()
print("Execution time:", round(end - start, 4), "seconds")
The program records the elapsed time for the multiplication operation. The exact result depends on your hardware, Python environment and system workload.
4. Inspecting DataFrame Memory Usage
Use memory_usage() and
info(memory_usage="deep") to understand
how much memory a DataFrame consumes.
import pandas as pd
df = pd.DataFrame({
"Name": ["Amit", "Neha", "Raj", "Priya"],
"Age": [25, 30, 22, 28],
"Salary": [35000, 52000, 28000, 45000]
})
print(df.info(memory_usage="deep"))
print(df.memory_usage(deep=True))
total_bytes = df.memory_usage(deep=True).sum()
print("Total memory:", total_bytes, "bytes")
The output reports the data types, non-null counts and memory usage of the DataFrame. The precise byte counts depend on the stored values and Pandas version.
deep=True provides
a more comprehensive estimate for object-backed data,
including the memory occupied by contained objects.
5. Avoid Loading Unnecessary Columns
If a CSV file contains many columns, load only the columns required for your analysis. This can reduce both memory usage and file-reading overhead.
import pandas as pd
df = pd.read_csv(
"sales.csv",
usecols=["Product", "Quantity", "Price"]
)
print(df.head())
In this example, Pandas loads only the three specified
columns from sales.csv.
Reading a Small Sample
sample = pd.read_csv(
"sales.csv",
nrows=1000
)
print(sample.shape)
The nrows parameter is useful when exploring
a file or testing an analysis on a small sample before
processing the complete dataset.
6. Prefer Vectorized Operations
Vectorized operations apply calculations across a Series or array without requiring an explicit Python loop for every value. They are generally preferable for simple numerical transformations.
Vectorized Approach
import pandas as pd
df = pd.DataFrame({
"Price": [100, 200, 300, 400]
})
df["PriceWithTax"] = df["Price"] * 1.18
print(df)
Output
Price PriceWithTax
0 100 118.0
1 200 236.0
2 300 354.0
3 400 472.0
The calculation is applied to the complete column. For straightforward arithmetic, this is usually clearer than iterating through every row.
7. Choose Efficient Data Types
Data types influence how much memory a DataFrame uses. For example, a small integer range may not require a 64-bit integer representation.
import pandas as pd
df = pd.DataFrame({
"Age": [20, 25, 30, 35],
"Quantity": [1, 2, 3, 4]
})
print("Before optimization:")
print(df.dtypes)
df["Age"] = df["Age"].astype("int8")
df["Quantity"] = df["Quantity"].astype("int8")
print("After optimization:")
print(df.dtypes)
Both columns contain values that fit within the
int8 range. Choosing a smaller integer
type can reduce memory usage for suitable datasets.
8. Optimize Repeated Text Values with category
Columns containing a limited number of repeated strings
may benefit from Pandas' category data type.
This is especially useful for columns such as department,
status or product category.
import pandas as pd
df = pd.DataFrame({
"Department": [
"IT", "HR", "IT", "Finance",
"HR", "IT", "Finance"
]
})
print("Before:")
print(df.memory_usage(deep=True))
df["Department"] = df["Department"].astype("category")
print("After:")
print(df.memory_usage(deep=True))
print(df.dtypes)
The category data type stores category information separately from the underlying category codes. The memory benefit depends on the number of distinct values and the size of the dataset.
9. Processing Large Datasets in Chunks
When a CSV file is too large to fit comfortably into
memory, Pandas can read it in smaller batches using
chunksize.
import pandas as pd
total_sales = 0
for chunk in pd.read_csv(
"sales.csv",
chunksize=10000
):
total_sales += chunk["Sales"].sum()
print("Total sales:", total_sales)
Instead of loading the entire file into one DataFrame, this example processes up to 10,000 rows per chunk and accumulates the total.
10. Using query() for Filtering
The query() method lets you express suitable
filtering conditions using a readable string expression.
import pandas as pd
df = pd.DataFrame({
"Product": ["Laptop", "Mouse", "Keyboard", "Monitor"],
"Price": [60000, 800, 1500, 12000],
"Stock": [5, 50, 25, 10]
})
result = df.query("Price > 1000 and Stock >= 10")
print(result)
Output
Product Price Stock
2 Keyboard 1500 25
3 Monitor 12000 10
The example selects rows whose price is above 1,000 and whose stock is at least 10.
query()
is not guaranteed to be faster than Boolean indexing
for every dataset. Benchmark both approaches for
performance-critical workloads.
11. Avoid Unnecessary Copies and Intermediate DataFrames
Creating many temporary DataFrames can increase memory usage. Select required columns and combine compatible operations where doing so keeps the code readable.
Select Only Required Columns
import pandas as pd
df = pd.read_csv("sales.csv")
analysis_df = df[["Product", "Sales"]]
print(analysis_df.head())
This selects only the columns required for the analysis. Depending on the operation and Pandas version, selecting data may produce a copy or a view-like object. Do not assume that every selection avoids copying memory.
Combine Compatible Calculations
df["Revenue"] = df["Quantity"] * df["Price"]
df["DiscountedRevenue"] = (
df["Revenue"] * (1 - df["DiscountRate"])
)
Here, the calculations are explicit and each derived column can be reused by subsequent operations.
12. Choosing the Right Tool for Large Data
Pandas is suitable for many data analysis tasks, but other tools may be useful when datasets, execution requirements or system resources change.
| Tool | Main Strength | When to Consider It |
|---|---|---|
| Pandas | Flexible DataFrame-based analysis. | Everyday analysis and datasets that fit available resources. |
| PyArrow | Columnar data formats and efficient data interchange. | Parquet, Arrow-backed data and supported I/O workflows. |
| Dask | Parallel and partitioned DataFrame computations. | Workloads that benefit from partitioning or distributed execution. |
| Polars | Expression-based DataFrame processing and query optimization. | Workloads suited to its execution model and supported operations. |
13. A Practical Performance Checklist
Before Processing
- Inspect the shape and data types of your dataset.
- Identify which columns are actually required.
- Measure current execution time and memory usage.
- Check for unnecessary repeated calculations.
During Optimization
- Prefer vectorized operations where appropriate.
- Evaluate smaller numeric data types safely.
- Consider categorical types for repeated strings.
- Use chunking when processing large CSV files.
- Benchmark changes using representative data.
After Optimization
- Verify that the output remains correct.
- Compare execution time before and after.
- Check the final memory footprint.
- Document any assumptions about data types and missing values.
14. Practice Exercises
Exercise 1: Inspect Memory
Create a DataFrame containing names, ages and departments. Display its data types and memory usage.
import pandas as pd
df = pd.DataFrame({
"Name": ["Amit", "Neha", "Raj", "Priya"],
"Age": [22, 25, 30, 28],
"Department": ["IT", "HR", "IT", "Finance"]
})
# Write your solution below
Exercise 2: Optimize Data Types
Convert a suitable integer column to a smaller numeric type after checking its range. Compare memory usage before and after.
quantities = pd.Series([1, 2, 3, 4, 5])
# Write your solution below
Exercise 3: Process a CSV in Chunks
Read a large sales CSV in batches and calculate the total value of its Sales column.
# Use pd.read_csv() with chunksize
# Accumulate the Sales total from each chunk
Summary
Pandas performance optimization focuses on reducing unnecessary computation, controlling memory consumption and choosing efficient operations for the dataset.
Inspect memory usage, select only required columns, prefer vectorized calculations and use appropriate data types. For large CSV files, chunked processing can avoid loading the complete dataset into memory.
For larger or more demanding workloads, evaluate PyArrow, Dask or Polars based on your requirements. Always benchmark your changes and verify that the optimized output remains correct.