Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Memory Optimization and Numerical Downcasting

Learn how to reduce DataFrame memory consumption by choosing efficient data types, converting columns, optimizing numeric values and managing large datasets effectively.

1. Introduction to Memory Optimization

Memory optimization is the process of reducing the amount of RAM required to store and process data. Pandas DataFrames can consume significant memory when they contain millions of rows or unnecessary large data types.

By selecting appropriate data types and removing unnecessary data, you can make your data analysis workflow more efficient.

Learning Objectives

  • Understand DataFrame memory consumption.
  • Inspect memory usage of individual columns.
  • Understand integer and floating-point data types.
  • Perform numerical downcasting.
  • Convert object columns into category types.
  • Handle missing values with nullable data types.
  • Compare memory usage before and after optimization.

2. Understanding Pandas Data Types

Every DataFrame column has a data type that determines how its values are stored and processed.

Data Type Description Example
int64 64-bit integer values. 100, 200, 300
int32 32-bit integer values. 1000, 2000
int16 16-bit integer values. 100, 200
int8 8-bit integer values. 10, 20, 30
float64 64-bit floating-point values. 10.50, 20.75
float32 32-bit floating-point values. 12.5, 25.5
object General Python objects, often strings. "Amit", "Neha"
category Stores values using categorical codes and categories. "IT", "HR"

3. Checking DataFrame Memory Usage

Pandas provides memory_usage() to calculate the memory consumed by individual DataFrame columns.

import pandas as pd

df = pd.DataFrame({
    "Age": [22, 25, 30, 28],
    "Salary": [25000, 35000, 45000, 50000],
    "Experience": [1, 2, 5, 3]
})

print(df.dtypes)

print(df.memory_usage(deep=True))

total_memory = df.memory_usage(deep=True).sum()

print("Total Memory:", total_memory, "bytes")

Output

Age           int64
Salary        int64
Experience    int64
dtype: object

Index         132
Age            32
Salary         32
Experience     32
dtype: int64

Total Memory: 228 bytes
Tip: Use deep=True to get a more complete memory estimate, especially for columns containing Python objects such as strings.

4. Understanding Numerical Downcasting

Downcasting means converting a numeric column from a larger data type into a smaller compatible data type.

For example, an int64 column may be converted into int8 or int16 if all its values fit within the supported range.

Data Type Memory per Value Value Range
int8 1 byte -128 to 127
int16 2 bytes -32,768 to 32,767
int32 4 bytes -2,147,483,648 to 2,147,483,647
int64 8 bytes 64-bit signed integer range

Smaller numeric types can reduce memory consumption, particularly when a DataFrame contains many rows.

5. Integer Downcasting Using astype()

The astype() method converts a column into a specified data type.

import pandas as pd

df = pd.DataFrame({
    "Age": [20, 25, 30, 35],
    "Quantity": [1, 2, 3, 4]
})

print("Before Optimization:")
print(df.dtypes)

df["Age"] = df["Age"].astype("int8")
df["Quantity"] = df["Quantity"].astype("int8")

print("\nAfter Optimization:")
print(df.dtypes)

Output

Before Optimization:

Age         int64
Quantity    int64
dtype: object

After Optimization:

Age         int8
Quantity    int8
dtype: object
Important: Always check the minimum and maximum values before converting numeric columns. Converting values outside the target type's range can result in incorrect data.

6. Automatic Integer Downcasting

Pandas provides pd.to_numeric() with downcast options to choose a smaller compatible numeric type.

import pandas as pd

df = pd.DataFrame({
    "Quantity": [10, 20, 30, 40],
    "Sales": [1000, 2000, 3000, 4000]
})

df["Quantity"] = pd.to_numeric(
    df["Quantity"],
    downcast="integer"
)

df["Sales"] = pd.to_numeric(
    df["Sales"],
    downcast="integer"
)

print(df.dtypes)

Output

Quantity    int8
Sales       int16
dtype: object

Pandas chooses a smaller integer type that can represent the values in each column. The selected type depends on the actual data range.

7. Floating-Point Downcasting

Floating-point columns can also be converted into smaller numeric types when the required precision allows it.

import pandas as pd

df = pd.DataFrame({
    "Price": [100.50, 200.75, 300.25, 400.80],
    "Discount": [5.5, 10.5, 15.5, 20.5]
})

print("Before:")
print(df.dtypes)

df["Price"] = pd.to_numeric(
    df["Price"],
    downcast="float"
)

df["Discount"] = pd.to_numeric(
    df["Discount"],
    downcast="float"
)

print("\nAfter:")
print(df.dtypes)

Output

Before:

Price       float64
Discount    float64
dtype: object

After:

Price       float32
Discount    float32
dtype: object
Precision warning: float32 provides less precision than float64. For financial calculations and high-precision scientific work, verify that the conversion is acceptable.

8. Optimizing Object Columns with Category

A Pandas object column containing repeated string values can consume significant memory. The category data type can reduce storage requirements when the number of unique values is relatively small.

import pandas as pd

df = pd.DataFrame({
    "Department": [
        "IT", "HR", "IT", "Finance",
        "HR", "IT", "Finance"
    ]
})

print("Before:")
print(df.dtypes)

print(df.memory_usage(deep=True))

df["Department"] = df["Department"].astype("category")

print("\nAfter:")
print(df.dtypes)

print(df.memory_usage(deep=True))

Output

Before:
Department    object
dtype: object

After:
Department    category
dtype: object

The category type stores unique category labels and represents individual values using integer codes.

Practical example: Department names, employee status, product categories and city names are common candidates for categorical storage when values repeat frequently.

9. Converting String Columns

Pandas supports dedicated string data types through astype("string"). This can make text handling more consistent than using a general object column.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Amit", "Neha", "Raj", "Priya"]
})

df["Name"] = df["Name"].astype("string")

print(df.dtypes)

print(df["Name"].str.upper())

Output

Name    str
dtype: object

0     AMIT
1     NEHA
2      RAJ
3    PRIYA
Name: Name, dtype: string

The string extension type provides consistent string operations and supports missing values using Pandas' nullable data model.

10. Nullable Integer Data Types

Standard NumPy integer types cannot represent missing values directly. Pandas provides nullable integer types such as Int8, Int16, Int32 and Int64.

import pandas as pd

df = pd.DataFrame({
    "Age": [20, 25, None, 35]
})

df["Age"] = df["Age"].astype("Int8")

print(df)

print(df.dtypes)

Output

    Age
0    20
1    25
2  <NA>
3    35

Age    Int8
dtype: object

The capitalized Int8 type is a Pandas nullable integer type. It can store integer values together with missing values represented by pd.NA.

11. Optimizing Multiple DataFrame Columns

You can optimize several columns by applying conversions to each column based on its data type.

import pandas as pd

df = pd.DataFrame({
    "Age": [20, 25, 30, 35],
    "Salary": [25000, 35000, 45000, 50000],
    "Department": ["IT", "HR", "IT", "HR"],
    "Rating": [4.5, 3.8, 4.2, 4.7]
})

# Optimize integer columns
for column in ["Age", "Salary"]:
    df[column] = pd.to_numeric(
        df[column],
        downcast="integer"
    )

# Optimize floating-point columns
df["Rating"] = pd.to_numeric(
    df["Rating"],
    downcast="float"
)

# Convert repeated text values
df["Department"] = df["Department"].astype("category")

print(df.dtypes)

Each column is converted according to its data type and actual values.

12. Comparing Memory Before and After Optimization

Memory comparison helps determine whether an optimization actually reduces storage requirements.

import pandas as pd

df = pd.DataFrame({
    "Age": [20, 25, 30, 35, 40],
    "Department": [
        "IT", "HR", "IT", "HR", "IT"
    ]
})

# Memory before optimization
before = df.memory_usage(deep=True).sum()

# Optimize columns
df["Age"] = pd.to_numeric(
    df["Age"],
    downcast="integer"
)

df["Department"] = df["Department"].astype("category")

# Memory after optimization
after = df.memory_usage(deep=True).sum()

print("Before:", before, "bytes")
print("After:", after, "bytes")

if before > 0:
    reduction = ((before - after) / before) * 100
    print("Memory reduction:", round(reduction, 2), "%")

The program calculates the memory difference and percentage reduction. Actual results depend on the number of rows, unique values, data types and Pandas version.

Remember: For very small DataFrames, categorical metadata can sometimes make the optimized version use more memory. Test with representative data.

13. Best Practices for Memory Optimization

Recommended Practices

  • Inspect memory usage before optimization.
  • Select only the columns required for analysis.
  • Use smaller integer types when their ranges are sufficient.
  • Downcast floating-point values only when precision allows.
  • Convert repeated strings to categorical types when beneficial.
  • Use nullable types when integer columns contain missing values.
  • Process large CSV files in chunks when needed.
  • Compare memory usage after every optimization.
  • Validate the accuracy of transformed values.

Common Mistakes

  • Converting all numeric columns to int8 without checking ranges.
  • Using float32 for calculations that require greater precision.
  • Converting every object column to category without checking cardinality.
  • Ignoring missing values during data type conversion.
  • Assuming that every optimization will reduce memory usage.

14. Practice Exercises

Exercise 1: Inspect Memory Usage

Create a DataFrame with Age, Salary and Quantity columns. Display each column's memory usage.

import pandas as pd

df = pd.DataFrame({
    "Age": [20, 25, 30, 35],
    "Salary": [20000, 30000, 40000, 50000],
    "Quantity": [1, 2, 3, 4]
})

# Write your solution below

Exercise 2: Integer Downcasting

Convert the Quantity column into the smallest suitable integer type using pd.to_numeric().

quantity = pd.Series([10, 20, 30, 40, 50])

# Write your solution below

Exercise 3: Float Optimization

Convert a floating-point column into a smaller compatible floating-point type and compare its data type.

prices = pd.Series([
    100.50, 200.75, 300.25, 400.80
])

# Write your solution below

Exercise 4: Category Conversion

Convert the Department column into category type and compare its memory usage before and after.

department = pd.Series([
    "IT", "HR", "IT", "Finance", "HR"
])

# Write your solution below

Summary

Memory optimization helps reduce the RAM required by Pandas DataFrames. Choosing suitable data types, inspecting memory usage and removing unnecessary columns can improve the efficiency of data processing.

Numerical downcasting converts compatible integer and floating-point columns into smaller data types. Categorical types can reduce memory consumption for columns containing repeated values, while nullable integer types support missing values.

Always compare memory usage before and after optimization, validate numeric ranges and preserve the required precision. The best optimization depends on the structure and size of the dataset.