Handling Missing Values in Pandas
Learn how to detect, analyze, remove, and replace missing values in Pandas DataFrames using practical data cleaning techniques.
1. Introduction to Missing Values
Missing values occur when information is unavailable, was not recorded, or cannot be determined. They are common in real-world datasets and can affect calculations, visualizations, and machine learning workflows.
Pandas provides several methods for detecting and handling missing data. The appropriate strategy depends on why the values are missing and how the data will be used.
Common Representations of Missing Data
| Representation | Description |
|---|---|
NaN |
A common representation of missing numeric data. |
None |
The Python null object, often treated as missing by Pandas. |
pd.NA |
Pandas' scalar missing value for nullable data types. |
NaT |
Represents missing datetime or timedelta values. |
Missing values can be removed, replaced, or retained, depending on the requirements of the analysis.
2. Create a Sample DataFrame
The following dataset contains missing values in the Age, Salary, and City columns.
import pandas as pd
import numpy as np
df = pd.DataFrame({
"Name": [
"Samadhan",
"Pradnya",
"Samiksha",
"Rahul",
"Neha",
"Amit"
],
"Age": [22, np.nan, 24, 21, np.nan, 25],
"Salary": [45000, 52000, np.nan, 48000, 39000, np.nan],
"City": [
"Pune",
"Mumbai",
None,
"Nashik",
"Mumbai",
"Pune"
]
})
print(df)
Expected Output:
Name Age Salary City 0 Samadhan 22.0 45000.0 Pune 1 Pradnya NaN 52000.0 Mumbai 2 Samiksha 24.0 NaN None 3 Rahul 21.0 48000.0 Nashik 4 Neha NaN 39000.0 Mumbai 5 Amit 25.0 NaN Pune
NaN. The exact displayed representation can depend
on the column's data type.
3. Detect Missing Values with isna()
The isna() method identifies missing values.
It returns True for missing entries and False for non-missing
entries.
Example 1: Check Every Cell
print(df.isna())
Expected Output:
Name Age Salary City 0 False False False False 1 False True False False 2 False False True True 3 False False False False 4 False True False False 5 False False True False
Example 2: Count Missing Values in Each Column
print(df.isna().sum())
Expected Output:
Name 0 Age 2 Salary 2 City 1 dtype: int64
Example 3: Count All Missing Values
total_missing = df.isna().sum().sum() print(total_missing)
Expected Output:
5
Example 4: Calculate the Percentage of Missing Values
missing_percentage = (
df.isna().mean() * 100
).round(2)
print(missing_percentage)
Expected Output:
Name 0.00 Age 33.33 Salary 33.33 City 16.67 dtype: float64
The percentage helps determine whether a column has only a few missing values or a substantial amount of incomplete data.
4. Find Rows with Missing Values
You can filter rows based on whether a particular column, or any column in the row, contains missing data.
Example 1: Rows with Missing Age
result = df[
df["Age"].isna()
]
print(result)
Expected Output:
Name Age Salary City 1 Pradnya NaN 52000.0 Mumbai 4 Neha NaN 39000.0 Mumbai
Example 2: Rows with Any Missing Value
result = df[
df.isna().any(axis=1)
]
print(result)
The expression any(axis=1) checks whether each
row contains at least one True value in the missing-value mask.
Example 3: Rows with Missing Values in Every Column
result = df[
df.isna().all(axis=1)
]
print(result)
This selects rows in which every column is missing. For the current dataset, the result is an empty DataFrame because no row has all four values missing.
any(axis=1) to find rows with at least one
missing value and all(axis=1) to find rows
where every selected column is missing.
5. Remove Missing Values with dropna()
The dropna() method removes rows or columns
containing missing values. By default, it removes rows that
contain at least one missing value.
Example 1: Remove Rows with Any Missing Value
result = df.dropna() print(result)
Expected Output:
Name Age Salary City 0 Samadhan 22.0 45000.0 Pune 3 Rahul 21.0 48000.0 Nashik
Example 2: Remove Rows Only When All Values Are Missing
result = df.dropna(how="all") print(result)
Since no row in this dataset is entirely missing, every row is retained.
Example 3: Remove Rows with Missing Values in Specific Columns
result = df.dropna(
subset=["Age", "Salary"]
)
print(result[["Name", "Age", "Salary"]])
Expected Output:
Name Age Salary 0 Samadhan 22.0 45000.0 3 Rahul 21.0 48000.0
Example 4: Remove Columns with Missing Values
result = df.dropna(axis=1) print(result.columns.tolist())
Expected Output:
['Name']
In this example, Age, Salary, and City all contain at least one missing value, so they are removed. Use this option carefully because it can discard useful columns.
dropna().
6. Replace Missing Values with fillna()
The fillna() method replaces missing values with
a specified value or with values calculated from the dataset.
Example 1: Fill Missing Values with a Constant
result = df.copy()
result["City"] = result["City"].fillna(
"Unknown"
)
print(result[["Name", "City"]])
Expected Output:
Name City 0 Samadhan Pune 1 Pradnya Mumbai 2 Samiksha Unknown 3 Rahul Nashik 4 Neha Mumbai 5 Amit Pune
Example 2: Fill Missing Ages with the Mean
result = df.copy()
mean_age = result["Age"].mean()
result["Age"] = result["Age"].fillna(
mean_age
)
print(result[["Name", "Age"]])
Expected Output:
Name Age 0 Samadhan 22.0 1 Pradnya 23.0 2 Samiksha 24.0 3 Rahul 21.0 4 Neha 23.0 5 Amit 25.0
The mean of the available ages is 23.0. Missing age values are replaced with this calculated value.
Example 3: Fill Missing Salaries with the Median
result = df.copy()
median_salary = result["Salary"].median()
result["Salary"] = result["Salary"].fillna(
median_salary
)
print(result[["Name", "Salary"]])
Expected Output:
Name Salary 0 Samadhan 45000.0 1 Pradnya 52000.0 2 Samiksha 46500.0 3 Rahul 48000.0 4 Neha 39000.0 5 Amit 46500.0
The median of the available salaries is 46,500. The median is often useful for skewed numeric data because it is less affected by extreme values than the mean.
Example 4: Fill Different Columns with Different Values
result = df.copy()
result = result.fillna({
"Age": result["Age"].median(),
"Salary": result["Salary"].median(),
"City": "Unknown"
})
print(result.isna().sum())
Expected Output:
Name 0 Age 0 Salary 0 City 0 dtype: int64
7. Forward Fill and Backward Fill
Forward fill replaces a missing value with the previous available value. Backward fill replaces it with the next available value.
Example Dataset
data = pd.DataFrame({
"Day": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"],
"Sales": [100, np.nan, np.nan, 160, np.nan]
})
print(data)
Expected Output:
Day Sales 0 Monday 100.0 1 Tuesday NaN 2 Wednesday NaN 3 Thursday 160.0 4 Friday NaN
Example 1: Forward Fill
data["ForwardFill"] = data["Sales"].ffill() print(data[["Day", "ForwardFill"]])
Expected Output:
Day ForwardFill 0 Monday 100.0 1 Tuesday 100.0 2 Wednesday 100.0 3 Thursday 160.0 4 Friday 160.0
Example 2: Backward Fill
data["BackwardFill"] = data["Sales"].bfill() print(data[["Day", "BackwardFill"]])
Expected Output:
Day BackwardFill 0 Monday 100.0 1 Tuesday 160.0 2 Wednesday 160.0 3 Thursday 160.0 4 Friday NaN
8. Limit the Number of Filled Values
The limit argument controls how many consecutive
missing values are filled during forward or backward filling.
data = pd.Series([
100,
np.nan,
np.nan,
np.nan,
200
])
result = data.ffill(limit=2)
print(result)
Expected Output:
0 100.0 1 100.0 2 100.0 3 NaN 4 200.0 dtype: float64
Only two consecutive missing values are filled from the preceding value. The third missing value remains unchanged.
9. Interpolate Missing Numeric Values
Interpolation estimates missing values using neighboring observations. Linear interpolation is useful when values represent a continuous sequence and a straight-line estimate is reasonable.
data = pd.Series([
10,
np.nan,
30,
np.nan,
50
])
result = data.interpolate()
print(result)
Expected Output:
0 10.0 1 20.0 2 30.0 3 40.0 4 50.0 dtype: float64
The missing values are estimated using the neighboring numeric values. The method and assumptions should match the meaning and ordering of the data.
Linear interpolation estimates missing values along a straight line between known numeric observations.
10. Replace Placeholder Values with Missing Values
Some datasets use placeholders such as "Unknown", "N/A", or -999 to represent missing information. These values may not be automatically recognized as missing by Pandas.
Example 1: Replace Text Placeholders
data = pd.DataFrame({
"Name": ["Samadhan", "Pradnya", "Samiksha"],
"City": ["Pune", "N/A", "Unknown"]
})
data["City"] = data["City"].replace(
["N/A", "Unknown"],
np.nan
)
print(data)
Expected Output:
Name City 0 Samadhan Pune 1 Pradnya NaN 2 Samiksha NaN
Example 2: Replace a Numeric Sentinel Value
data = pd.DataFrame({
"Name": ["Samadhan", "Pradnya", "Samiksha"],
"Score": [85, -999, 92]
})
data["Score"] = data["Score"].replace(
-999,
np.nan
)
print(data)
Expected Output:
Name Score 0 Samadhan 85.0 1 Pradnya NaN 2 Samiksha 92.0
11. Choose an Appropriate Missing-Value Strategy
| Strategy | Suitable Use Case | Potential Concern |
|---|---|---|
| Remove rows | Only a small number of records are incomplete. | Can discard useful observations. |
| Mean imputation | Numeric data where the mean is a reasonable estimate. | Can be affected by outliers and reduce variability. |
| Median imputation | Numeric data with skewed distributions or outliers. | Can hide meaningful differences in the data. |
| Mode imputation | Categorical values when the most frequent category is appropriate. | Can overrepresent the most common category. |
| Forward or backward fill | Ordered observations where carrying adjacent values is justified. | Can propagate stale or future information. |
| Interpolation | Ordered numeric observations with a meaningful interpolation model. | Estimates may not reflect the real process. |
12. Validate the Cleaned DataFrame
After handling missing values, check the dataset again to confirm that the intended changes have been applied.
print("Missing values per column:")
print(df.isna().sum())
print("\nTotal missing values:")
print(df.isna().sum().sum())
print("\nData types:")
print(df.dtypes)
print("\nDataFrame shape:")
print(df.shape)
These checks show the remaining missing values, column data types, and number of rows and columns. The expected results depend on which cleaning operations were applied.
13. Practical Exercises
Exercise 1: Count Missing Values
Create a DataFrame with missing values in at least two columns. Count missing values in each column.
Hint: Use isna().sum().
Exercise 2: Remove Incomplete Records
Remove rows that contain missing values in the Age column.
Hint: Use dropna(subset=["Age"]).
Exercise 3: Fill Numeric Missing Values
Replace missing values in a Salary column with the median of its available values.
Hint: Combine median() and fillna().
Exercise 4: Forward Fill
Fill missing values in an ordered sales series using the preceding available value.
Hint: Use ffill().
Exercise 5: Replace Placeholder Values
Replace the text values "N/A" and "Unknown" with missing values before calculating the number of missing entries.
Hint: Use replace() and isna().sum().
14. Quick Revision Questions
- What are missing values, and why should they be inspected?
- What is the difference between
isna()andnotna()? - How can you count missing values in every DataFrame column?
- What is the difference between
dropna()andfillna()? - How does forward fill differ from backward fill?
- When can interpolation be useful?
- Why should placeholder values be verified before replacing them?
- Why should a cleaned DataFrame be validated again?
Summary
Pandas provides several methods for handling missing values.
Use isna() to detect missing entries,
dropna() to remove incomplete rows or columns,
and fillna() to replace missing values with
appropriate constants or calculated statistics.
Forward fill, backward fill, and interpolation are additional techniques for ordered or numeric data when their assumptions are appropriate. Placeholder values can be converted to recognized missing values after checking the dataset's documentation.
Always inspect and validate your data before and after cleaning. The best strategy preserves useful information while avoiding unsupported assumptions about missing values.