Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Processing Large CSV Files with chunksize

Learn how to process large CSV files efficiently using Pandas chunksize, read data in smaller batches, perform aggregations, filter records and save processed results without loading the entire dataset into memory.

1. Introduction to Processing Large CSV Files

CSV files are commonly used to store and exchange structured data. However, when a CSV file contains millions of records, loading the entire file into a single Pandas DataFrame may require a large amount of memory.

Pandas provides the chunksize parameter in read_csv() to process a file in smaller batches instead of reading all records at once.

Learning Objectives

  • Understand the challenges of large CSV files.
  • Learn how the chunksize parameter works.
  • Read CSV files in smaller batches.
  • Perform calculations on individual chunks.
  • Calculate totals, counts and averages.
  • Filter records from large datasets.
  • Combine results from multiple chunks.
  • Export processed data into a new CSV file.

2. Why Process CSV Files in Chunks?

Consider a CSV file containing several million customer or transaction records. Loading every record into memory may consume significant system resources.

Chunk processing divides the file into smaller DataFrames. Each DataFrame is processed separately, allowing the application to work with a portion of the data at a time.

Normal CSV Reading Chunk-Based Reading
Loads the complete dataset. Loads a limited number of rows at a time.
May require significant memory. Can reduce peak DataFrame memory usage.
Suitable for datasets that fit in memory. Useful for datasets larger than available memory.
Processes one complete DataFrame. Processes multiple smaller DataFrames.
Remember: Chunking reduces the memory needed for each individual batch. It does not automatically make every operation faster or guarantee that total application memory remains constant.

3. Understanding the chunksize Parameter

The chunksize parameter specifies how many rows Pandas should read into each DataFrame batch.

import pandas as pd

chunks = pd.read_csv(
    "sales.csv",
    chunksize=1000
)

for chunk in chunks:
    print(chunk.head())

In this example, Pandas reads up to 1,000 rows into each chunk. The loop processes one chunk before requesting the next one.

chunksize Maximum Rows per Batch
100 100 rows
1,000 1,000 rows
5,000 5,000 rows
10,000 10,000 rows
50,000 50,000 rows

The final chunk may contain fewer rows if the total number of records is not evenly divisible by the selected chunksize.

4. Creating a Sample CSV File

Before processing a large CSV file, create a small sample dataset to understand the structure.

import pandas as pd

data = {
    "OrderID": [101, 102, 103, 104, 105],
    "Product": [
        "Laptop",
        "Mouse",
        "Keyboard",
        "Monitor",
        "Laptop"
    ],
    "Quantity": [2, 5, 3, 1, 4],
    "Price": [50000, 500, 1500, 12000, 50000]
}

df = pd.DataFrame(data)

df.to_csv("sales.csv", index=False)

print("CSV file created successfully!")

Output

CSV file created successfully!

The to_csv() method saves the DataFrame into a CSV file. Setting index=False prevents Pandas from writing the DataFrame index as an additional column.

5. Reading CSV Files in Chunks

Use pd.read_csv() with chunksize to create an iterator that returns one DataFrame at a time.

import pandas as pd

chunk_iterator = pd.read_csv(
    "sales.csv",
    chunksize=2
)

for chunk_number, chunk in enumerate(
    chunk_iterator,
    start=1
):
    print("Chunk:", chunk_number)
    print(chunk)
    print("----------------")

Output

Chunk: 1

   OrderID Product  Quantity  Price
0      101  Laptop         2  50000
1      102   Mouse         5    500

----------------

Chunk: 2

   OrderID   Product  Quantity  Price
2      103  Keyboard         3   1500
3      104   Monitor         1  12000

----------------

Chunk: 3

   OrderID Product  Quantity  Price
4      105  Laptop         4  50000

----------------
Important: The iterator reads additional data as you loop through it. You do not need to load all the chunks into a list.

6. Processing Each Chunk Individually

Each chunk returned by Pandas is an ordinary DataFrame. You can perform filtering, calculations, transformations and aggregations on it.

import pandas as pd

for chunk in pd.read_csv(
    "sales.csv",
    chunksize=2
):
    chunk["Total"] = (
        chunk["Quantity"] * chunk["Price"]
    )

    print(chunk)

In this example, each batch receives a new Total column calculated by multiplying Quantity by Price.

Example Output

   OrderID Product  Quantity  Price   Total
0      101  Laptop         2  50000  100000
1      102   Mouse         5    500    2500

   OrderID   Product  Quantity  Price   Total
2      103  Keyboard         3   1500    4500
3      104   Monitor         1  12000   12000

   OrderID Product  Quantity  Price   Total
4      105  Laptop         4  50000  200000

7. Calculating Total Sales from Large CSV Files

One common use of chunk processing is calculating the total sales amount without loading the complete dataset into memory.

import pandas as pd

total_sales = 0

for chunk in pd.read_csv(
    "sales.csv",
    chunksize=1000
):
    chunk_total = (
        chunk["Quantity"] * chunk["Price"]
    ).sum()

    total_sales += chunk_total

print("Total Sales:", total_sales)

Output for the sample CSV

Total Sales: 319000

Each chunk calculates its own sales total. The program then adds the partial results to calculate the final total.

8. Counting Records in Large CSV Files

You can count records incrementally by calculating the number of rows in every chunk.

import pandas as pd

total_records = 0

for chunk in pd.read_csv(
    "sales.csv",
    chunksize=1000
):
    total_records += len(chunk)

print("Total Records:", total_records)

Output

Total Records: 5

This approach is useful when you need the total number of records without storing the complete dataset in a single DataFrame.

9. Calculating Average from Multiple Chunks

To calculate an accurate overall average, maintain the total sum and count across all chunks. Do not simply average the individual chunk averages when chunk sizes may differ.

import pandas as pd

total_quantity = 0
total_records = 0

for chunk in pd.read_csv(
    "sales.csv",
    chunksize=2
):
    total_quantity += chunk["Quantity"].sum()

    total_records += chunk["Quantity"].count()

if total_records > 0:
    average_quantity = (
        total_quantity / total_records
    )
else:
    average_quantity = 0

print("Average Quantity:", average_quantity)

Output

Average Quantity: 3.0
Remember: Calculate the average using the combined sum divided by the combined count. This produces the correct weighted result even when the final chunk is smaller.

10. Filtering Records from Large CSV Files

You can filter records inside every chunk and process only the rows that satisfy a particular condition.

import pandas as pd

for chunk in pd.read_csv(
    "sales.csv",
    chunksize=2
):
    high_value_orders = chunk[
        chunk["Price"] > 10000
    ]

    print(high_value_orders)

This example selects products whose price is greater than 10,000 from each batch.

Example Output

   OrderID Product  Quantity  Price
0      101  Laptop         2  50000

   OrderID Product  Quantity  Price
4      105  Laptop         4  50000

11. Saving Processed Chunks into a New CSV File

After processing each chunk, you can append its results to an output CSV file. This avoids combining every processed batch into one large DataFrame in memory.

import pandas as pd
import os

input_file = "sales.csv"
output_file = "processed_sales.csv"

# Remove previous output if it exists
if os.path.exists(output_file):
    os.remove(output_file)

first_chunk = True

for chunk in pd.read_csv(
    input_file,
    chunksize=2
):
    chunk["Total"] = (
        chunk["Quantity"] * chunk["Price"]
    )

    chunk.to_csv(
        output_file,
        mode="w" if first_chunk else "a",
        header=first_chunk,
        index=False
    )

    first_chunk = False

print("Processed CSV saved successfully!")

Output

Processed CSV saved successfully!

The first batch creates the output file and writes the header. Subsequent batches append records without writing the header again.

Important: In production applications, validate input and output paths and handle file permissions. Do not delete an existing output file unless overwriting it is intended.

12. Selecting Specific Columns While Reading Chunks

If your CSV contains many columns, use usecols to load only the fields required for your processing task.

import pandas as pd

for chunk in pd.read_csv(
    "sales.csv",
    usecols=["OrderID", "Product", "Price"],
    chunksize=1000
):
    print(chunk.head())

This approach avoids loading unnecessary columns into each chunk.

Best practice: Combine usecols, suitable data types and chunksize when processing large CSV files.

13. Handling Missing Values in Chunks

Missing values can appear in any batch. Handle them inside the chunk-processing loop according to the requirements of your application.

import pandas as pd

for chunk in pd.read_csv(
    "sales.csv",
    chunksize=1000
):
    chunk["Price"] = chunk["Price"].fillna(0)

    chunk["Quantity"] = (
        chunk["Quantity"].fillna(0)
    )

    chunk["Total"] = (
        chunk["Price"] * chunk["Quantity"]
    )

    print(chunk.head())

In this example, missing Price and Quantity values are replaced with zero before calculating Total.

Note: Replacing missing values with zero is not appropriate for every dataset. Choose a missing-value strategy based on the meaning of the data.

14. Handling Errors During Chunk Processing

Large file-processing tasks should handle missing files, invalid numeric values and unexpected input formats gracefully.

import pandas as pd

try:
    total_sales = 0

    for chunk in pd.read_csv(
        "sales.csv",
        chunksize=1000
    ):
        chunk["Price"] = pd.to_numeric(
            chunk["Price"],
            errors="coerce"
        )

        chunk["Quantity"] = pd.to_numeric(
            chunk["Quantity"],
            errors="coerce"
        )

        chunk = chunk.dropna(
            subset=["Price", "Quantity"]
        )

        total_sales += (
            chunk["Price"] * chunk["Quantity"]
        ).sum()

    print("Total Sales:", total_sales)

except FileNotFoundError:
    print("CSV file not found.")

except pd.errors.EmptyDataError:
    print("CSV file is empty.")

except pd.errors.ParserError:
    print("Unable to parse CSV file.")

This example converts invalid numeric entries into missing values and removes records that cannot be used in the calculation.

Tip: Add logging and validation when building production data-processing applications so that skipped records and processing failures can be investigated.

15. Choosing an Appropriate Chunk Size

The best chunk size depends on available memory, row width, data types and the complexity of operations performed on each batch.

Chunk Size Suitable Scenario Consideration
100 Small testing and demonstrations. More iterations for large files.
1,000 Small batch processing. Easy to inspect and debug.
10,000 General data-processing workloads. Balance between batch size and memory.
50,000 Larger processing batches. Requires more memory per chunk.

These are example values, not universal performance recommendations. Test different chunk sizes using your actual CSV files.

16. Common Mistakes and Best Practices

Common Mistakes

  • Loading the entire CSV instead of using chunksize.
  • Converting the iterator into a list of all chunks.
  • Combining all chunks into one large DataFrame unnecessarily.
  • Writing the CSV header for every chunk.
  • Calculating averages using unweighted chunk averages.
  • Ignoring missing values and invalid numeric data.
  • Choosing an excessively large chunk size.

Best Practices

  • Use an appropriate chunksize based on available memory.
  • Select required columns using usecols.
  • Process each chunk independently whenever possible.
  • Accumulate sums and counts for overall statistics.
  • Write processed batches directly to an output file.
  • Validate input records and handle errors.
  • Monitor memory and execution time.

17. Practice Exercises

Exercise 1: Read CSV in Chunks

Read a CSV file in batches of 500 rows and display the first five records of each chunk.

import pandas as pd

# Write your solution below

Exercise 2: Calculate Total Sales

Process a sales CSV in chunks and calculate the overall sales amount using Price and Quantity.

# Read the file using chunksize
# Calculate Price * Quantity
# Accumulate the total sales

Exercise 3: Filter High-Value Orders

Select records where Price is greater than 10,000 and save the matching records into a new CSV file.

# Read CSV in chunks
# Apply the Price condition
# Append matching records to an output CSV

Exercise 4: Calculate Average Quantity

Calculate the average Quantity across all chunks using the combined sum and count.

# Initialize total_quantity and total_count
# Process every chunk
# Calculate the final average

Summary

Pandas provides the chunksize parameter in read_csv() to process large CSV files in smaller DataFrame batches. This approach helps reduce the memory required for each individual batch.

Each chunk can be used independently for filtering, calculations, aggregations and data transformations. Partial results can be combined to calculate totals, counts and averages without loading the entire dataset into memory.

For efficient processing, choose a suitable chunk size, select only the required columns, handle missing values, and write processed results incrementally to output files. Always validate the results and monitor memory usage.