Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Handling Missing Values in Pandas

Learn how to detect, analyze, remove, and replace missing values in Pandas DataFrames using practical data cleaning techniques.

1. Introduction to Missing Values

Missing values occur when information is unavailable, was not recorded, or cannot be determined. They are common in real-world datasets and can affect calculations, visualizations, and machine learning workflows.

Pandas provides several methods for detecting and handling missing data. The appropriate strategy depends on why the values are missing and how the data will be used.

Common Representations of Missing Data

Representation Description
NaN A common representation of missing numeric data.
None The Python null object, often treated as missing by Pandas.
pd.NA Pandas' scalar missing value for nullable data types.
NaT Represents missing datetime or timedelta values.
Dataset with Missing Values Detect Missing Data Remove dropna() Replace fillna() Preserve Keep as missing Validate the Result

Missing values can be removed, replaced, or retained, depending on the requirements of the analysis.

2. Create a Sample DataFrame

The following dataset contains missing values in the Age, Salary, and City columns.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "Name": [
        "Samadhan",
        "Pradnya",
        "Samiksha",
        "Rahul",
        "Neha",
        "Amit"
    ],
    "Age": [22, np.nan, 24, 21, np.nan, 25],
    "Salary": [45000, 52000, np.nan, 48000, 39000, np.nan],
    "City": [
        "Pune",
        "Mumbai",
        None,
        "Nashik",
        "Mumbai",
        "Pune"
    ]
})

print(df)

Expected Output:

        Name   Age   Salary    City
0   Samadhan  22.0  45000.0    Pune
1    Pradnya   NaN  52000.0  Mumbai
2   Samiksha  24.0      NaN    None
3      Rahul  21.0  48000.0  Nashik
4       Neha   NaN  39000.0  Mumbai
5       Amit  25.0      NaN    Pune
Important: Pandas commonly displays missing numeric values as NaN. The exact displayed representation can depend on the column's data type.

3. Detect Missing Values with isna()

The isna() method identifies missing values. It returns True for missing entries and False for non-missing entries.

Example 1: Check Every Cell

print(df.isna())

Expected Output:

    Name    Age  Salary   City
0  False  False   False  False
1  False   True   False  False
2  False  False    True   True
3  False  False   False  False
4  False   True   False  False
5  False  False    True  False

Example 2: Count Missing Values in Each Column

print(df.isna().sum())

Expected Output:

Name      0
Age       2
Salary    2
City      1
dtype: int64

Example 3: Count All Missing Values

total_missing = df.isna().sum().sum()

print(total_missing)

Expected Output:

5

Example 4: Calculate the Percentage of Missing Values

missing_percentage = (
    df.isna().mean() * 100
).round(2)

print(missing_percentage)

Expected Output:

Name       0.00
Age       33.33
Salary    33.33
City      16.67
dtype: float64

The percentage helps determine whether a column has only a few missing values or a substantial amount of incomplete data.

4. Find Rows with Missing Values

You can filter rows based on whether a particular column, or any column in the row, contains missing data.

Example 1: Rows with Missing Age

result = df[
    df["Age"].isna()
]

print(result)

Expected Output:

      Name  Age   Salary    City
1  Pradnya  NaN  52000.0  Mumbai
4     Neha  NaN  39000.0  Mumbai

Example 2: Rows with Any Missing Value

result = df[
    df.isna().any(axis=1)
]

print(result)

The expression any(axis=1) checks whether each row contains at least one True value in the missing-value mask.

Example 3: Rows with Missing Values in Every Column

result = df[
    df.isna().all(axis=1)
]

print(result)

This selects rows in which every column is missing. For the current dataset, the result is an empty DataFrame because no row has all four values missing.

Remember: Use any(axis=1) to find rows with at least one missing value and all(axis=1) to find rows where every selected column is missing.

5. Remove Missing Values with dropna()

The dropna() method removes rows or columns containing missing values. By default, it removes rows that contain at least one missing value.

Example 1: Remove Rows with Any Missing Value

result = df.dropna()

print(result)

Expected Output:

    Name   Age   Salary    City
0 Samadhan 22.0  45000.0    Pune
3    Rahul 21.0  48000.0  Nashik

Example 2: Remove Rows Only When All Values Are Missing

result = df.dropna(how="all")

print(result)

Since no row in this dataset is entirely missing, every row is retained.

Example 3: Remove Rows with Missing Values in Specific Columns

result = df.dropna(
    subset=["Age", "Salary"]
)

print(result[["Name", "Age", "Salary"]])

Expected Output:

        Name   Age   Salary
0   Samadhan  22.0  45000.0
3      Rahul  21.0  48000.0

Example 4: Remove Columns with Missing Values

result = df.dropna(axis=1)

print(result.columns.tolist())

Expected Output:

['Name']

In this example, Age, Salary, and City all contain at least one missing value, so they are removed. Use this option carefully because it can discard useful columns.

Important: Removing rows can reduce the amount of available data. Check how many records will be lost before choosing dropna().

6. Replace Missing Values with fillna()

The fillna() method replaces missing values with a specified value or with values calculated from the dataset.

Example 1: Fill Missing Values with a Constant

result = df.copy()

result["City"] = result["City"].fillna(
    "Unknown"
)

print(result[["Name", "City"]])

Expected Output:

        Name     City
0   Samadhan     Pune
1    Pradnya   Mumbai
2   Samiksha  Unknown
3      Rahul   Nashik
4       Neha   Mumbai
5       Amit     Pune

Example 2: Fill Missing Ages with the Mean

result = df.copy()

mean_age = result["Age"].mean()

result["Age"] = result["Age"].fillna(
    mean_age
)

print(result[["Name", "Age"]])

Expected Output:

        Name   Age
0   Samadhan  22.0
1    Pradnya  23.0
2   Samiksha  24.0
3      Rahul  21.0
4       Neha  23.0
5       Amit  25.0

The mean of the available ages is 23.0. Missing age values are replaced with this calculated value.

Example 3: Fill Missing Salaries with the Median

result = df.copy()

median_salary = result["Salary"].median()

result["Salary"] = result["Salary"].fillna(
    median_salary
)

print(result[["Name", "Salary"]])

Expected Output:

        Name   Salary
0   Samadhan  45000.0
1    Pradnya  52000.0
2   Samiksha  46500.0
3      Rahul  48000.0
4       Neha  39000.0
5        Amit  46500.0

The median of the available salaries is 46,500. The median is often useful for skewed numeric data because it is less affected by extreme values than the mean.

Example 4: Fill Different Columns with Different Values

result = df.copy()

result = result.fillna({
    "Age": result["Age"].median(),
    "Salary": result["Salary"].median(),
    "City": "Unknown"
})

print(result.isna().sum())

Expected Output:

Name      0
Age       0
Salary    0
City      0
dtype: int64

7. Forward Fill and Backward Fill

Forward fill replaces a missing value with the previous available value. Backward fill replaces it with the next available value.

Example Dataset

data = pd.DataFrame({
    "Day": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"],
    "Sales": [100, np.nan, np.nan, 160, np.nan]
})

print(data)

Expected Output:

         Day  Sales
0     Monday  100.0
1    Tuesday    NaN
2  Wednesday    NaN
3   Thursday  160.0
4     Friday    NaN

Example 1: Forward Fill

data["ForwardFill"] = data["Sales"].ffill()

print(data[["Day", "ForwardFill"]])

Expected Output:

         Day  ForwardFill
0     Monday        100.0
1    Tuesday        100.0
2  Wednesday        100.0
3   Thursday        160.0
4     Friday        160.0

Example 2: Backward Fill

data["BackwardFill"] = data["Sales"].bfill()

print(data[["Day", "BackwardFill"]])

Expected Output:

         Day  BackwardFill
0     Monday         100.0
1    Tuesday         160.0
2  Wednesday         160.0
3   Thursday         160.0
4     Friday           NaN
Important: Forward fill and backward fill should only be used when carrying a previous or subsequent value forward or backward makes sense for the data. Backward fill can introduce future information into earlier records, which may cause data leakage in predictive time-series workflows.

8. Limit the Number of Filled Values

The limit argument controls how many consecutive missing values are filled during forward or backward filling.

data = pd.Series([
    100,
    np.nan,
    np.nan,
    np.nan,
    200
])

result = data.ffill(limit=2)

print(result)

Expected Output:

0    100.0
1    100.0
2    100.0
3      NaN
4    200.0
dtype: float64

Only two consecutive missing values are filled from the preceding value. The third missing value remains unchanged.

9. Interpolate Missing Numeric Values

Interpolation estimates missing values using neighboring observations. Linear interpolation is useful when values represent a continuous sequence and a straight-line estimate is reasonable.

data = pd.Series([
    10,
    np.nan,
    30,
    np.nan,
    50
])

result = data.interpolate()

print(result)

Expected Output:

0    10.0
1    20.0
2    30.0
3    40.0
4    50.0
dtype: float64

The missing values are estimated using the neighboring numeric values. The method and assumptions should match the meaning and ordering of the data.

10 20 30 40 50 Ordered Observations Numeric Value

Linear interpolation estimates missing values along a straight line between known numeric observations.

10. Replace Placeholder Values with Missing Values

Some datasets use placeholders such as "Unknown", "N/A", or -999 to represent missing information. These values may not be automatically recognized as missing by Pandas.

Example 1: Replace Text Placeholders

data = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "City": ["Pune", "N/A", "Unknown"]
})

data["City"] = data["City"].replace(
    ["N/A", "Unknown"],
    np.nan
)

print(data)

Expected Output:

       Name  City
0  Samadhan  Pune
1   Pradnya   NaN
2  Samiksha   NaN

Example 2: Replace a Numeric Sentinel Value

data = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Score": [85, -999, 92]
})

data["Score"] = data["Score"].replace(
    -999,
    np.nan
)

print(data)

Expected Output:

       Name  Score
0  Samadhan   85.0
1   Pradnya    NaN
2  Samiksha   92.0
Best Practice: Replace placeholders only when the dataset's documentation confirms that they represent missing values. A value such as -999 could be meaningful in some specialized datasets.

11. Choose an Appropriate Missing-Value Strategy

Strategy Suitable Use Case Potential Concern
Remove rows Only a small number of records are incomplete. Can discard useful observations.
Mean imputation Numeric data where the mean is a reasonable estimate. Can be affected by outliers and reduce variability.
Median imputation Numeric data with skewed distributions or outliers. Can hide meaningful differences in the data.
Mode imputation Categorical values when the most frequent category is appropriate. Can overrepresent the most common category.
Forward or backward fill Ordered observations where carrying adjacent values is justified. Can propagate stale or future information.
Interpolation Ordered numeric observations with a meaningful interpolation model. Estimates may not reflect the real process.

12. Validate the Cleaned DataFrame

After handling missing values, check the dataset again to confirm that the intended changes have been applied.

print("Missing values per column:")
print(df.isna().sum())

print("\nTotal missing values:")
print(df.isna().sum().sum())

print("\nData types:")
print(df.dtypes)

print("\nDataFrame shape:")
print(df.shape)

These checks show the remaining missing values, column data types, and number of rows and columns. The expected results depend on which cleaning operations were applied.

Important: A dataset does not always need to contain zero missing values. Retaining missing values may be more appropriate than introducing unsupported assumptions through imputation.

13. Practical Exercises

Exercise 1: Count Missing Values

Create a DataFrame with missing values in at least two columns. Count missing values in each column.

Hint: Use isna().sum().

Exercise 2: Remove Incomplete Records

Remove rows that contain missing values in the Age column.

Hint: Use dropna(subset=["Age"]).

Exercise 3: Fill Numeric Missing Values

Replace missing values in a Salary column with the median of its available values.

Hint: Combine median() and fillna().

Exercise 4: Forward Fill

Fill missing values in an ordered sales series using the preceding available value.

Hint: Use ffill().

Exercise 5: Replace Placeholder Values

Replace the text values "N/A" and "Unknown" with missing values before calculating the number of missing entries.

Hint: Use replace() and isna().sum().

14. Quick Revision Questions

  1. What are missing values, and why should they be inspected?
  2. What is the difference between isna() and notna()?
  3. How can you count missing values in every DataFrame column?
  4. What is the difference between dropna() and fillna()?
  5. How does forward fill differ from backward fill?
  6. When can interpolation be useful?
  7. Why should placeholder values be verified before replacing them?
  8. Why should a cleaned DataFrame be validated again?

Summary

Pandas provides several methods for handling missing values. Use isna() to detect missing entries, dropna() to remove incomplete rows or columns, and fillna() to replace missing values with appropriate constants or calculated statistics.

Forward fill, backward fill, and interpolation are additional techniques for ordered or numeric data when their assumptions are appropriate. Placeholder values can be converted to recognized missing values after checking the dataset's documentation.

Always inspect and validate your data before and after cleaning. The best strategy preserves useful information while avoiding unsupported assumptions about missing values.