Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Handling Duplicate Data in Pandas

Learn how to identify duplicate rows, inspect repeated records, remove duplicates safely, and validate a cleaned DataFrame.

1. What Is Duplicate Data?

Duplicate data occurs when the same record appears more than once in a DataFrame. Duplicate records can inflate totals, distort statistical results, and create incorrect reports.

Example: Student Records

Samadhan is analyzing student marks. The same record for Pradnya appears twice. If both records are included in a report, the student may be counted twice.

Before cleaning Student Subject Marks Pradnya Python 92 Pradnya Python 92 Repeated record Clean After cleaning Student Subject Marks Pradnya Python 92 One record retained Always verify whether repeated records are truly duplicates.

Duplicate removal reduces repeated records without changing unique records.

2. Create a Sample DataFrame

The following DataFrame contains duplicate student records. We will use it throughout this lesson.

import pandas as pd

df = pd.DataFrame({
    "Student": [
        "Samadhan",
        "Pradnya",
        "Samiksha",
        "Pradnya",
        "Samadhan",
        "Aarav"
    ],
    "Subject": [
        "Python",
        "Python",
        "SQL",
        "Python",
        "Python",
        "SQL"
    ],
    "Marks": [85, 92, 88, 92, 85, 79]
})

print(df)

Expected Output

    Student Subject  Marks
0  Samadhan  Python     85
1   Pradnya  Python     92
2  Samiksha     SQL     88
3   Pradnya  Python     92
4  Samadhan  Python     85
5     Aarav     SQL     79

Rows at indexes 0 and 4 contain the same values. Rows at indexes 1 and 3 also contain the same values.

3. Detect Duplicate Rows with duplicated()

The duplicated() method identifies rows that have already appeared earlier in the DataFrame. By default, the first occurrence is marked as False, while subsequent occurrences are marked as True.

print(df.duplicated())

Expected Output

0    False
1    False
2    False
3     True
4     True
5    False
dtype: bool

How to Read the Result

  • False means the row is not a repeated occurrence.
  • True means the row duplicates an earlier row.
  • Indexes 3 and 4 are identified as duplicates.

Count Duplicate Rows

duplicate_count = df.duplicated().sum()

print(duplicate_count)

Expected Output:

2

Because Boolean values are treated numerically in this calculation, sum() counts the number of True values.

4. Keep All Duplicate Occurrences Visible

By default, the first occurrence is not marked as a duplicate. Use keep=False to mark every occurrence of a repeated row as True.

print(df.duplicated(keep=False))

Expected Output

0     True
1     True
2    False
3     True
4     True
5    False
dtype: bool

This result marks both occurrences of the Samadhan record and both occurrences of the Pradnya record.

Practical tip: Use keep=False when you need to inspect every row participating in a duplicate group.

5. Inspect Duplicate Records

Detecting duplicates is only the first step. Inspect the matching records before deciding whether to remove them.

duplicate_rows = df[df.duplicated(keep=False)]

print(duplicate_rows)

Expected Output

    Student Subject  Marks
0  Samadhan  Python     85
1   Pradnya  Python     92
3   Pradnya  Python     92
4  Samadhan  Python     85

The filtered DataFrame contains every row that belongs to a duplicate group. Rows for Samiksha and Aarav do not appear because they have no exact duplicate in the sample.

6. Remove Duplicate Rows with drop_duplicates()

The drop_duplicates() method removes repeated records. By default, it keeps the first occurrence and removes subsequent matching rows.

clean_df = df.drop_duplicates()

print(clean_df)

Expected Output

    Student Subject  Marks
0  Samadhan  Python     85
1   Pradnya  Python     92
2  Samiksha     SQL     88
5     Aarav     SQL     79

The cleaned DataFrame contains four rows instead of six. The original DataFrame remains unchanged because drop_duplicates() returns a new DataFrame by default.

Check the Number of Rows

print("Original rows:", len(df))
print("Cleaned rows:", len(clean_df))

Expected Output:

Original rows: 6
Cleaned rows: 4

7. Control Which Duplicate to Keep

The keep parameter controls which occurrence is retained.

Option Behavior Typical Use
keep="first" Keeps the first occurrence. Default behavior when the earliest row should remain.
keep="last" Keeps the last occurrence. When the latest occurrence should remain based on row order.
keep=False Removes every occurrence of a duplicated record. When only records appearing exactly once should remain.

Keep the Last Occurrence

last_df = df.drop_duplicates(keep="last")

print(last_df)

Expected Output

    Student Subject  Marks
2  Samiksha     SQL     88
3   Pradnya  Python     92
4  Samadhan  Python     85
5     Aarav     SQL     79

The last occurrence of each repeated record is retained. The remaining rows keep their original index labels.

Remove Every Repeated Occurrence

unique_only_df = df.drop_duplicates(keep=False)

print(unique_only_df)

Expected Output

    Student Subject  Marks
2  Samiksha     SQL     88
5     Aarav     SQL     79

8. Find Duplicates Using Selected Columns

Sometimes two records should be considered duplicates based on only a few columns. Use the subset parameter to specify those columns.

Example: Same Student and Subject

Suppose the same student can appear more than once for different assessments. You may want to check repeated student-subject combinations rather than matching every column in the row.

print(df.duplicated(subset=["Student", "Subject"]))

Expected Output

0    False
1    False
2    False
3     True
4     True
5    False
dtype: bool

The comparison uses only Student and Subject. The value in Marks is ignored when determining whether the selected combination has appeared before.

Remove Repeated Student-Subject Combinations

student_subject_df = df.drop_duplicates(
    subset=["Student", "Subject"]
)

print(student_subject_df)

Expected Output:

    Student Subject  Marks
0  Samadhan  Python     85
1   Pradnya  Python     92
2  Samiksha     SQL     88
5     Aarav     SQL     79
Important: Choosing a subset changes the meaning of a duplicate. If a student has multiple legitimate assessments in the same subject, removing rows based only on student and subject could discard valid data.

9. Keep the Record with the Highest Marks

The keep="last" option keeps the last row in the current order; it does not automatically identify the row with the highest marks. Sort the data first when a particular record should be retained.

best_record_df = (
    df.sort_values("Marks", ascending=False)
      .drop_duplicates(subset=["Student", "Subject"])
)

print(best_record_df)

Expected Output

    Student Subject  Marks
1   Pradnya  Python     92
2  Samiksha     SQL     88
0  Samadhan  Python     85
5     Aarav     SQL     79

Sorting places higher marks first. Duplicate removal then retains the first record for each student-subject combination.

Remember: The sort order determines which duplicate is kept. Check the sorting rule before removing data.

10. Reset the Index After Removing Duplicates

Duplicate removal does not automatically renumber the index. Use reset_index(drop=True) when a fresh sequential index is useful.

clean_df = (
    df.drop_duplicates()
      .reset_index(drop=True)
)

print(clean_df)

Expected Output

    Student Subject  Marks
0  Samadhan  Python     85
1   Pradnya  Python     92
2  Samiksha     SQL     88
3     Aarav     SQL     79

The original index labels are replaced with 0, 1, 2, and 3. Setting drop=True prevents the old index from being added as a new column.

11. Understand the Duplicate-Removal Workflow

1. Load the DataFrame Inspect columns and row counts 2. Detect Duplicates Use duplicated() and subset 3. Review the Matches Confirm that rows are truly duplicates 4. Remove Duplicates Use drop_duplicates() 5. Validate the Result Recheck duplicates and row counts

A reliable duplicate-cleaning workflow includes inspection and validation.

Validate the Cleaned Data

clean_df = df.drop_duplicates()

print("Rows before:", len(df))
print("Rows after:", len(clean_df))
print("Remaining duplicates:", clean_df.duplicated().sum())

Expected Output:

Rows before: 6
Rows after: 4
Remaining duplicates: 0

12. Practical Example: Clean Student Results

Samiksha receives a DataFrame containing repeated exam records. The goal is to retain one record for each student and subject, keeping the highest mark when multiple records exist.

results = pd.DataFrame({
    "Student": [
        "Samadhan",
        "Pradnya",
        "Samadhan",
        "Samiksha",
        "Pradnya"
    ],
    "Subject": [
        "Python",
        "SQL",
        "Python",
        "SQL",
        "SQL"
    ],
    "Marks": [85, 90, 85, 88, 94]
})

clean_results = (
    results.sort_values("Marks", ascending=False)
           .drop_duplicates(subset=["Student", "Subject"])
           .reset_index(drop=True)
)

print(clean_results)

Expected Output

    Student Subject  Marks
0   Pradnya     SQL     94
1   Pradnya     SQL     90
2  Samiksha     SQL     88
3  Samadhan  Python     85
Check the result carefully: The two Pradnya records have different marks, but both have the same student and subject. The code above keeps the first occurrence after sorting only if the selected subset matches. Because both records have the same student and subject, the expected result contains only the 94-mark record.

The correct output is therefore:

    Student Subject  Marks
0   Pradnya     SQL     94
1  Samiksha     SQL     88
2  Samadhan  Python     85

This example shows why the duplicate rule must match the business requirement. If separate assessments are legitimate, include an assessment identifier in the subset instead of removing records based only on student and subject.

13. Common Mistakes to Avoid

  • Deleting without inspection: Check whether the repeated rows represent the same real-world record.
  • Using the wrong subset: Selecting too few columns can remove valid records.
  • Assuming keep="last" means highest value: It means the last row in the current order.
  • Expecting the original DataFrame to change: By default, drop_duplicates() returns a new DataFrame.
  • Ignoring the index: Use reset_index(drop=True) when sequential indexing is required.
  • Assuming identical names always mean duplicate records: A person may legitimately have multiple transactions, assessments, or events.

14. Quick Revision

Method or Parameter Purpose
df.duplicated() Identifies repeated row occurrences.
df.duplicated(keep=False) Marks all occurrences of duplicated rows.
df.duplicated().sum() Counts duplicate occurrences after the first.
df.drop_duplicates() Removes repeated rows, keeping the first occurrence.
keep="last" Keeps the last occurrence.
keep=False Removes all rows that belong to duplicate groups.
subset=["Student"] Checks duplicates using only the selected column or columns.
reset_index(drop=True) Creates a fresh sequential index without retaining the old index.

15. Practice Questions

Concept Questions

  1. What is the difference between duplicated() and drop_duplicates()?
  2. What does keep=False do?
  3. Why would you use the subset parameter?
  4. Does drop_duplicates() modify the original DataFrame by default?
  5. Why might two rows with the same student name not be duplicates?

Hands-On Exercises

  1. Create a DataFrame with at least seven rows and two duplicate records.
  2. Display the Boolean duplicate mask.
  3. Count duplicate occurrences using sum().
  4. Display every row belonging to a duplicate group.
  5. Remove duplicates while keeping the last occurrence.
  6. Remove duplicates using two selected columns.
  7. Reset the index and verify that no exact duplicate rows remain.
Learning goal: Before moving on, make sure you can explain why a duplicate is being removed, not just which Pandas method removes it.

Summary

Duplicate records can affect counts, totals, and analytical results. Pandas provides duplicated() to detect repeated rows and drop_duplicates() to remove them. Use keep to control which occurrence remains and subset to define the columns used for comparison. Inspect duplicate groups before deleting records, and validate the cleaned DataFrame afterward. Use sorting when the requirement is to retain a record with a particular value, such as the highest mark.