Processing Large CSV Files with chunksize
Learn how to process large CSV files efficiently using Pandas chunksize, read data in smaller batches, perform aggregations, filter records and save processed results without loading the entire dataset into memory.
1. Introduction to Processing Large CSV Files
CSV files are commonly used to store and exchange structured data. However, when a CSV file contains millions of records, loading the entire file into a single Pandas DataFrame may require a large amount of memory.
Pandas provides the chunksize parameter
in read_csv() to process a file in
smaller batches instead of reading all records
at once.
Learning Objectives
- Understand the challenges of large CSV files.
- Learn how the chunksize parameter works.
- Read CSV files in smaller batches.
- Perform calculations on individual chunks.
- Calculate totals, counts and averages.
- Filter records from large datasets.
- Combine results from multiple chunks.
- Export processed data into a new CSV file.
2. Why Process CSV Files in Chunks?
Consider a CSV file containing several million customer or transaction records. Loading every record into memory may consume significant system resources.
Chunk processing divides the file into smaller DataFrames. Each DataFrame is processed separately, allowing the application to work with a portion of the data at a time.
| Normal CSV Reading | Chunk-Based Reading |
|---|---|
| Loads the complete dataset. | Loads a limited number of rows at a time. |
| May require significant memory. | Can reduce peak DataFrame memory usage. |
| Suitable for datasets that fit in memory. | Useful for datasets larger than available memory. |
| Processes one complete DataFrame. | Processes multiple smaller DataFrames. |
3. Understanding the chunksize Parameter
The chunksize parameter specifies how
many rows Pandas should read into each DataFrame
batch.
import pandas as pd
chunks = pd.read_csv(
"sales.csv",
chunksize=1000
)
for chunk in chunks:
print(chunk.head())
In this example, Pandas reads up to 1,000 rows into each chunk. The loop processes one chunk before requesting the next one.
| chunksize | Maximum Rows per Batch |
|---|---|
| 100 | 100 rows |
| 1,000 | 1,000 rows |
| 5,000 | 5,000 rows |
| 10,000 | 10,000 rows |
| 50,000 | 50,000 rows |
The final chunk may contain fewer rows if the total number of records is not evenly divisible by the selected chunksize.
4. Creating a Sample CSV File
Before processing a large CSV file, create a small sample dataset to understand the structure.
import pandas as pd
data = {
"OrderID": [101, 102, 103, 104, 105],
"Product": [
"Laptop",
"Mouse",
"Keyboard",
"Monitor",
"Laptop"
],
"Quantity": [2, 5, 3, 1, 4],
"Price": [50000, 500, 1500, 12000, 50000]
}
df = pd.DataFrame(data)
df.to_csv("sales.csv", index=False)
print("CSV file created successfully!")
Output
CSV file created successfully!
The to_csv() method saves the DataFrame
into a CSV file. Setting index=False
prevents Pandas from writing the DataFrame index
as an additional column.
5. Reading CSV Files in Chunks
Use pd.read_csv() with chunksize to
create an iterator that returns one DataFrame
at a time.
import pandas as pd
chunk_iterator = pd.read_csv(
"sales.csv",
chunksize=2
)
for chunk_number, chunk in enumerate(
chunk_iterator,
start=1
):
print("Chunk:", chunk_number)
print(chunk)
print("----------------")
Output
Chunk: 1
OrderID Product Quantity Price
0 101 Laptop 2 50000
1 102 Mouse 5 500
----------------
Chunk: 2
OrderID Product Quantity Price
2 103 Keyboard 3 1500
3 104 Monitor 1 12000
----------------
Chunk: 3
OrderID Product Quantity Price
4 105 Laptop 4 50000
----------------
6. Processing Each Chunk Individually
Each chunk returned by Pandas is an ordinary DataFrame. You can perform filtering, calculations, transformations and aggregations on it.
import pandas as pd
for chunk in pd.read_csv(
"sales.csv",
chunksize=2
):
chunk["Total"] = (
chunk["Quantity"] * chunk["Price"]
)
print(chunk)
In this example, each batch receives a new
Total column calculated by multiplying
Quantity by Price.
Example Output
OrderID Product Quantity Price Total
0 101 Laptop 2 50000 100000
1 102 Mouse 5 500 2500
OrderID Product Quantity Price Total
2 103 Keyboard 3 1500 4500
3 104 Monitor 1 12000 12000
OrderID Product Quantity Price Total
4 105 Laptop 4 50000 200000
7. Calculating Total Sales from Large CSV Files
One common use of chunk processing is calculating the total sales amount without loading the complete dataset into memory.
import pandas as pd
total_sales = 0
for chunk in pd.read_csv(
"sales.csv",
chunksize=1000
):
chunk_total = (
chunk["Quantity"] * chunk["Price"]
).sum()
total_sales += chunk_total
print("Total Sales:", total_sales)
Output for the sample CSV
Total Sales: 319000
Each chunk calculates its own sales total. The program then adds the partial results to calculate the final total.
8. Counting Records in Large CSV Files
You can count records incrementally by calculating the number of rows in every chunk.
import pandas as pd
total_records = 0
for chunk in pd.read_csv(
"sales.csv",
chunksize=1000
):
total_records += len(chunk)
print("Total Records:", total_records)
Output
Total Records: 5
This approach is useful when you need the total number of records without storing the complete dataset in a single DataFrame.
9. Calculating Average from Multiple Chunks
To calculate an accurate overall average, maintain the total sum and count across all chunks. Do not simply average the individual chunk averages when chunk sizes may differ.
import pandas as pd
total_quantity = 0
total_records = 0
for chunk in pd.read_csv(
"sales.csv",
chunksize=2
):
total_quantity += chunk["Quantity"].sum()
total_records += chunk["Quantity"].count()
if total_records > 0:
average_quantity = (
total_quantity / total_records
)
else:
average_quantity = 0
print("Average Quantity:", average_quantity)
Output
Average Quantity: 3.0
10. Filtering Records from Large CSV Files
You can filter records inside every chunk and process only the rows that satisfy a particular condition.
import pandas as pd
for chunk in pd.read_csv(
"sales.csv",
chunksize=2
):
high_value_orders = chunk[
chunk["Price"] > 10000
]
print(high_value_orders)
This example selects products whose price is greater than 10,000 from each batch.
Example Output
OrderID Product Quantity Price
0 101 Laptop 2 50000
OrderID Product Quantity Price
4 105 Laptop 4 50000
11. Saving Processed Chunks into a New CSV File
After processing each chunk, you can append its results to an output CSV file. This avoids combining every processed batch into one large DataFrame in memory.
import pandas as pd
import os
input_file = "sales.csv"
output_file = "processed_sales.csv"
# Remove previous output if it exists
if os.path.exists(output_file):
os.remove(output_file)
first_chunk = True
for chunk in pd.read_csv(
input_file,
chunksize=2
):
chunk["Total"] = (
chunk["Quantity"] * chunk["Price"]
)
chunk.to_csv(
output_file,
mode="w" if first_chunk else "a",
header=first_chunk,
index=False
)
first_chunk = False
print("Processed CSV saved successfully!")
Output
Processed CSV saved successfully!
The first batch creates the output file and writes the header. Subsequent batches append records without writing the header again.
12. Selecting Specific Columns While Reading Chunks
If your CSV contains many columns, use
usecols to load only the fields
required for your processing task.
import pandas as pd
for chunk in pd.read_csv(
"sales.csv",
usecols=["OrderID", "Product", "Price"],
chunksize=1000
):
print(chunk.head())
This approach avoids loading unnecessary columns into each chunk.
usecols, suitable data types and
chunksize when processing large CSV files.
13. Handling Missing Values in Chunks
Missing values can appear in any batch. Handle them inside the chunk-processing loop according to the requirements of your application.
import pandas as pd
for chunk in pd.read_csv(
"sales.csv",
chunksize=1000
):
chunk["Price"] = chunk["Price"].fillna(0)
chunk["Quantity"] = (
chunk["Quantity"].fillna(0)
)
chunk["Total"] = (
chunk["Price"] * chunk["Quantity"]
)
print(chunk.head())
In this example, missing Price and Quantity values are replaced with zero before calculating Total.
14. Handling Errors During Chunk Processing
Large file-processing tasks should handle missing files, invalid numeric values and unexpected input formats gracefully.
import pandas as pd
try:
total_sales = 0
for chunk in pd.read_csv(
"sales.csv",
chunksize=1000
):
chunk["Price"] = pd.to_numeric(
chunk["Price"],
errors="coerce"
)
chunk["Quantity"] = pd.to_numeric(
chunk["Quantity"],
errors="coerce"
)
chunk = chunk.dropna(
subset=["Price", "Quantity"]
)
total_sales += (
chunk["Price"] * chunk["Quantity"]
).sum()
print("Total Sales:", total_sales)
except FileNotFoundError:
print("CSV file not found.")
except pd.errors.EmptyDataError:
print("CSV file is empty.")
except pd.errors.ParserError:
print("Unable to parse CSV file.")
This example converts invalid numeric entries into missing values and removes records that cannot be used in the calculation.
15. Choosing an Appropriate Chunk Size
The best chunk size depends on available memory, row width, data types and the complexity of operations performed on each batch.
| Chunk Size | Suitable Scenario | Consideration |
|---|---|---|
| 100 | Small testing and demonstrations. | More iterations for large files. |
| 1,000 | Small batch processing. | Easy to inspect and debug. |
| 10,000 | General data-processing workloads. | Balance between batch size and memory. |
| 50,000 | Larger processing batches. | Requires more memory per chunk. |
These are example values, not universal performance recommendations. Test different chunk sizes using your actual CSV files.
16. Common Mistakes and Best Practices
Common Mistakes
- Loading the entire CSV instead of using chunksize.
- Converting the iterator into a list of all chunks.
- Combining all chunks into one large DataFrame unnecessarily.
- Writing the CSV header for every chunk.
- Calculating averages using unweighted chunk averages.
- Ignoring missing values and invalid numeric data.
- Choosing an excessively large chunk size.
Best Practices
- Use an appropriate chunksize based on available memory.
- Select required columns using usecols.
- Process each chunk independently whenever possible.
- Accumulate sums and counts for overall statistics.
- Write processed batches directly to an output file.
- Validate input records and handle errors.
- Monitor memory and execution time.
17. Practice Exercises
Exercise 1: Read CSV in Chunks
Read a CSV file in batches of 500 rows and display the first five records of each chunk.
import pandas as pd
# Write your solution below
Exercise 2: Calculate Total Sales
Process a sales CSV in chunks and calculate the overall sales amount using Price and Quantity.
# Read the file using chunksize
# Calculate Price * Quantity
# Accumulate the total sales
Exercise 3: Filter High-Value Orders
Select records where Price is greater than 10,000 and save the matching records into a new CSV file.
# Read CSV in chunks
# Apply the Price condition
# Append matching records to an output CSV
Exercise 4: Calculate Average Quantity
Calculate the average Quantity across all chunks using the combined sum and count.
# Initialize total_quantity and total_count
# Process every chunk
# Calculate the final average
Summary
Pandas provides the chunksize parameter
in read_csv() to process large CSV files
in smaller DataFrame batches. This approach helps
reduce the memory required for each individual batch.
Each chunk can be used independently for filtering, calculations, aggregations and data transformations. Partial results can be combined to calculate totals, counts and averages without loading the entire dataset into memory.
For efficient processing, choose a suitable chunk size, select only the required columns, handle missing values, and write processed results incrementally to output files. Always validate the results and monitor memory usage.