Memory Optimization and Numerical Downcasting
Learn how to reduce DataFrame memory consumption by choosing efficient data types, converting columns, optimizing numeric values and managing large datasets effectively.
1. Introduction to Memory Optimization
Memory optimization is the process of reducing the amount of RAM required to store and process data. Pandas DataFrames can consume significant memory when they contain millions of rows or unnecessary large data types.
By selecting appropriate data types and removing unnecessary data, you can make your data analysis workflow more efficient.
Learning Objectives
- Understand DataFrame memory consumption.
- Inspect memory usage of individual columns.
- Understand integer and floating-point data types.
- Perform numerical downcasting.
- Convert object columns into category types.
- Handle missing values with nullable data types.
- Compare memory usage before and after optimization.
2. Understanding Pandas Data Types
Every DataFrame column has a data type that determines how its values are stored and processed.
| Data Type | Description | Example |
|---|---|---|
int64 |
64-bit integer values. | 100, 200, 300 |
int32 |
32-bit integer values. | 1000, 2000 |
int16 |
16-bit integer values. | 100, 200 |
int8 |
8-bit integer values. | 10, 20, 30 |
float64 |
64-bit floating-point values. | 10.50, 20.75 |
float32 |
32-bit floating-point values. | 12.5, 25.5 |
object |
General Python objects, often strings. | "Amit", "Neha" |
category |
Stores values using categorical codes and categories. | "IT", "HR" |
3. Checking DataFrame Memory Usage
Pandas provides memory_usage() to calculate
the memory consumed by individual DataFrame columns.
import pandas as pd
df = pd.DataFrame({
"Age": [22, 25, 30, 28],
"Salary": [25000, 35000, 45000, 50000],
"Experience": [1, 2, 5, 3]
})
print(df.dtypes)
print(df.memory_usage(deep=True))
total_memory = df.memory_usage(deep=True).sum()
print("Total Memory:", total_memory, "bytes")
Output
Age int64
Salary int64
Experience int64
dtype: object
Index 132
Age 32
Salary 32
Experience 32
dtype: int64
Total Memory: 228 bytes
deep=True to get
a more complete memory estimate, especially for columns
containing Python objects such as strings.
4. Understanding Numerical Downcasting
Downcasting means converting a numeric column from a larger data type into a smaller compatible data type.
For example, an int64 column may be converted
into int8 or int16 if all its
values fit within the supported range.
| Data Type | Memory per Value | Value Range |
|---|---|---|
int8 |
1 byte | -128 to 127 |
int16 |
2 bytes | -32,768 to 32,767 |
int32 |
4 bytes | -2,147,483,648 to 2,147,483,647 |
int64 |
8 bytes | 64-bit signed integer range |
Smaller numeric types can reduce memory consumption, particularly when a DataFrame contains many rows.
5. Integer Downcasting Using astype()
The astype() method converts a column into
a specified data type.
import pandas as pd
df = pd.DataFrame({
"Age": [20, 25, 30, 35],
"Quantity": [1, 2, 3, 4]
})
print("Before Optimization:")
print(df.dtypes)
df["Age"] = df["Age"].astype("int8")
df["Quantity"] = df["Quantity"].astype("int8")
print("\nAfter Optimization:")
print(df.dtypes)
Output
Before Optimization:
Age int64
Quantity int64
dtype: object
After Optimization:
Age int8
Quantity int8
dtype: object
6. Automatic Integer Downcasting
Pandas provides pd.to_numeric() with
downcast options to choose a smaller
compatible numeric type.
import pandas as pd
df = pd.DataFrame({
"Quantity": [10, 20, 30, 40],
"Sales": [1000, 2000, 3000, 4000]
})
df["Quantity"] = pd.to_numeric(
df["Quantity"],
downcast="integer"
)
df["Sales"] = pd.to_numeric(
df["Sales"],
downcast="integer"
)
print(df.dtypes)
Output
Quantity int8
Sales int16
dtype: object
Pandas chooses a smaller integer type that can represent the values in each column. The selected type depends on the actual data range.
7. Floating-Point Downcasting
Floating-point columns can also be converted into smaller numeric types when the required precision allows it.
import pandas as pd
df = pd.DataFrame({
"Price": [100.50, 200.75, 300.25, 400.80],
"Discount": [5.5, 10.5, 15.5, 20.5]
})
print("Before:")
print(df.dtypes)
df["Price"] = pd.to_numeric(
df["Price"],
downcast="float"
)
df["Discount"] = pd.to_numeric(
df["Discount"],
downcast="float"
)
print("\nAfter:")
print(df.dtypes)
Output
Before:
Price float64
Discount float64
dtype: object
After:
Price float32
Discount float32
dtype: object
float32
provides less precision than float64.
For financial calculations and high-precision scientific
work, verify that the conversion is acceptable.
8. Optimizing Object Columns with Category
A Pandas object column containing repeated string values
can consume significant memory. The category
data type can reduce storage requirements when the
number of unique values is relatively small.
import pandas as pd
df = pd.DataFrame({
"Department": [
"IT", "HR", "IT", "Finance",
"HR", "IT", "Finance"
]
})
print("Before:")
print(df.dtypes)
print(df.memory_usage(deep=True))
df["Department"] = df["Department"].astype("category")
print("\nAfter:")
print(df.dtypes)
print(df.memory_usage(deep=True))
Output
Before:
Department object
dtype: object
After:
Department category
dtype: object
The category type stores unique category labels and represents individual values using integer codes.
9. Converting String Columns
Pandas supports dedicated string data types through
astype("string"). This can make text
handling more consistent than using a general
object column.
import pandas as pd
df = pd.DataFrame({
"Name": ["Amit", "Neha", "Raj", "Priya"]
})
df["Name"] = df["Name"].astype("string")
print(df.dtypes)
print(df["Name"].str.upper())
Output
Name str
dtype: object
0 AMIT
1 NEHA
2 RAJ
3 PRIYA
Name: Name, dtype: string
The string extension type provides consistent string operations and supports missing values using Pandas' nullable data model.
10. Nullable Integer Data Types
Standard NumPy integer types cannot represent missing
values directly. Pandas provides nullable integer
types such as Int8, Int16,
Int32 and Int64.
import pandas as pd
df = pd.DataFrame({
"Age": [20, 25, None, 35]
})
df["Age"] = df["Age"].astype("Int8")
print(df)
print(df.dtypes)
Output
Age
0 20
1 25
2 <NA>
3 35
Age Int8
dtype: object
The capitalized Int8 type is a Pandas
nullable integer type. It can store integer values
together with missing values represented by
pd.NA.
11. Optimizing Multiple DataFrame Columns
You can optimize several columns by applying conversions to each column based on its data type.
import pandas as pd
df = pd.DataFrame({
"Age": [20, 25, 30, 35],
"Salary": [25000, 35000, 45000, 50000],
"Department": ["IT", "HR", "IT", "HR"],
"Rating": [4.5, 3.8, 4.2, 4.7]
})
# Optimize integer columns
for column in ["Age", "Salary"]:
df[column] = pd.to_numeric(
df[column],
downcast="integer"
)
# Optimize floating-point columns
df["Rating"] = pd.to_numeric(
df["Rating"],
downcast="float"
)
# Convert repeated text values
df["Department"] = df["Department"].astype("category")
print(df.dtypes)
Each column is converted according to its data type and actual values.
12. Comparing Memory Before and After Optimization
Memory comparison helps determine whether an optimization actually reduces storage requirements.
import pandas as pd
df = pd.DataFrame({
"Age": [20, 25, 30, 35, 40],
"Department": [
"IT", "HR", "IT", "HR", "IT"
]
})
# Memory before optimization
before = df.memory_usage(deep=True).sum()
# Optimize columns
df["Age"] = pd.to_numeric(
df["Age"],
downcast="integer"
)
df["Department"] = df["Department"].astype("category")
# Memory after optimization
after = df.memory_usage(deep=True).sum()
print("Before:", before, "bytes")
print("After:", after, "bytes")
if before > 0:
reduction = ((before - after) / before) * 100
print("Memory reduction:", round(reduction, 2), "%")
The program calculates the memory difference and percentage reduction. Actual results depend on the number of rows, unique values, data types and Pandas version.
13. Best Practices for Memory Optimization
Recommended Practices
- Inspect memory usage before optimization.
- Select only the columns required for analysis.
- Use smaller integer types when their ranges are sufficient.
- Downcast floating-point values only when precision allows.
- Convert repeated strings to categorical types when beneficial.
- Use nullable types when integer columns contain missing values.
- Process large CSV files in chunks when needed.
- Compare memory usage after every optimization.
- Validate the accuracy of transformed values.
Common Mistakes
- Converting all numeric columns to
int8without checking ranges. - Using
float32for calculations that require greater precision. - Converting every object column to category without checking cardinality.
- Ignoring missing values during data type conversion.
- Assuming that every optimization will reduce memory usage.
14. Practice Exercises
Exercise 1: Inspect Memory Usage
Create a DataFrame with Age, Salary and Quantity columns. Display each column's memory usage.
import pandas as pd
df = pd.DataFrame({
"Age": [20, 25, 30, 35],
"Salary": [20000, 30000, 40000, 50000],
"Quantity": [1, 2, 3, 4]
})
# Write your solution below
Exercise 2: Integer Downcasting
Convert the Quantity column into the smallest
suitable integer type using pd.to_numeric().
quantity = pd.Series([10, 20, 30, 40, 50])
# Write your solution below
Exercise 3: Float Optimization
Convert a floating-point column into a smaller compatible floating-point type and compare its data type.
prices = pd.Series([
100.50, 200.75, 300.25, 400.80
])
# Write your solution below
Exercise 4: Category Conversion
Convert the Department column into category type and compare its memory usage before and after.
department = pd.Series([
"IT", "HR", "IT", "Finance", "HR"
])
# Write your solution below
Summary
Memory optimization helps reduce the RAM required by Pandas DataFrames. Choosing suitable data types, inspecting memory usage and removing unnecessary columns can improve the efficiency of data processing.
Numerical downcasting converts compatible integer and floating-point columns into smaller data types. Categorical types can reduce memory consumption for columns containing repeated values, while nullable integer types support missing values.
Always compare memory usage before and after optimization, validate numeric ranges and preserve the required precision. The best optimization depends on the structure and size of the dataset.