Handling Duplicate Data in Pandas
Learn how to identify duplicate rows, inspect repeated records, remove duplicates safely, and validate a cleaned DataFrame.
1. What Is Duplicate Data?
Duplicate data occurs when the same record appears more than once in a DataFrame. Duplicate records can inflate totals, distort statistical results, and create incorrect reports.
Example: Student Records
Samadhan is analyzing student marks. The same record for Pradnya appears twice. If both records are included in a report, the student may be counted twice.
Duplicate removal reduces repeated records without changing unique records.
2. Create a Sample DataFrame
The following DataFrame contains duplicate student records. We will use it throughout this lesson.
import pandas as pd
df = pd.DataFrame({
"Student": [
"Samadhan",
"Pradnya",
"Samiksha",
"Pradnya",
"Samadhan",
"Aarav"
],
"Subject": [
"Python",
"Python",
"SQL",
"Python",
"Python",
"SQL"
],
"Marks": [85, 92, 88, 92, 85, 79]
})
print(df)
Expected Output
Student Subject Marks 0 Samadhan Python 85 1 Pradnya Python 92 2 Samiksha SQL 88 3 Pradnya Python 92 4 Samadhan Python 85 5 Aarav SQL 79
Rows at indexes 0 and 4 contain the same values. Rows at indexes 1 and 3 also contain the same values.
3. Detect Duplicate Rows with duplicated()
The duplicated() method identifies rows that have
already appeared earlier in the DataFrame. By default, the first
occurrence is marked as False, while subsequent
occurrences are marked as True.
print(df.duplicated())
Expected Output
0 False 1 False 2 False 3 True 4 True 5 False dtype: bool
How to Read the Result
Falsemeans the row is not a repeated occurrence.Truemeans the row duplicates an earlier row.- Indexes 3 and 4 are identified as duplicates.
Count Duplicate Rows
duplicate_count = df.duplicated().sum() print(duplicate_count)
Expected Output:
2
Because Boolean values are treated numerically in this calculation,
sum() counts the number of True values.
4. Keep All Duplicate Occurrences Visible
By default, the first occurrence is not marked as a duplicate.
Use keep=False to mark every occurrence of a repeated
row as True.
print(df.duplicated(keep=False))
Expected Output
0 True 1 True 2 False 3 True 4 True 5 False dtype: bool
This result marks both occurrences of the Samadhan record and both occurrences of the Pradnya record.
keep=False when
you need to inspect every row participating in a duplicate group.
5. Inspect Duplicate Records
Detecting duplicates is only the first step. Inspect the matching records before deciding whether to remove them.
duplicate_rows = df[df.duplicated(keep=False)] print(duplicate_rows)
Expected Output
Student Subject Marks 0 Samadhan Python 85 1 Pradnya Python 92 3 Pradnya Python 92 4 Samadhan Python 85
The filtered DataFrame contains every row that belongs to a duplicate group. Rows for Samiksha and Aarav do not appear because they have no exact duplicate in the sample.
6. Remove Duplicate Rows with drop_duplicates()
The drop_duplicates() method removes repeated records.
By default, it keeps the first occurrence and removes subsequent
matching rows.
clean_df = df.drop_duplicates() print(clean_df)
Expected Output
Student Subject Marks 0 Samadhan Python 85 1 Pradnya Python 92 2 Samiksha SQL 88 5 Aarav SQL 79
The cleaned DataFrame contains four rows instead of six. The
original DataFrame remains unchanged because
drop_duplicates() returns a new DataFrame by default.
Check the Number of Rows
print("Original rows:", len(df))
print("Cleaned rows:", len(clean_df))
Expected Output:
Original rows: 6 Cleaned rows: 4
7. Control Which Duplicate to Keep
The keep parameter controls which occurrence is
retained.
| Option | Behavior | Typical Use |
|---|---|---|
keep="first" |
Keeps the first occurrence. | Default behavior when the earliest row should remain. |
keep="last" |
Keeps the last occurrence. | When the latest occurrence should remain based on row order. |
keep=False |
Removes every occurrence of a duplicated record. | When only records appearing exactly once should remain. |
Keep the Last Occurrence
last_df = df.drop_duplicates(keep="last") print(last_df)
Expected Output
Student Subject Marks 2 Samiksha SQL 88 3 Pradnya Python 92 4 Samadhan Python 85 5 Aarav SQL 79
The last occurrence of each repeated record is retained. The remaining rows keep their original index labels.
Remove Every Repeated Occurrence
unique_only_df = df.drop_duplicates(keep=False) print(unique_only_df)
Expected Output
Student Subject Marks 2 Samiksha SQL 88 5 Aarav SQL 79
8. Find Duplicates Using Selected Columns
Sometimes two records should be considered duplicates based on
only a few columns. Use the subset parameter to
specify those columns.
Example: Same Student and Subject
Suppose the same student can appear more than once for different assessments. You may want to check repeated student-subject combinations rather than matching every column in the row.
print(df.duplicated(subset=["Student", "Subject"]))
Expected Output
0 False 1 False 2 False 3 True 4 True 5 False dtype: bool
The comparison uses only Student and
Subject. The value in Marks is ignored
when determining whether the selected combination has appeared
before.
Remove Repeated Student-Subject Combinations
student_subject_df = df.drop_duplicates(
subset=["Student", "Subject"]
)
print(student_subject_df)
Expected Output:
Student Subject Marks 0 Samadhan Python 85 1 Pradnya Python 92 2 Samiksha SQL 88 5 Aarav SQL 79
9. Keep the Record with the Highest Marks
The keep="last" option keeps the last row in the
current order; it does not automatically identify the row with
the highest marks. Sort the data first when a particular record
should be retained.
best_record_df = (
df.sort_values("Marks", ascending=False)
.drop_duplicates(subset=["Student", "Subject"])
)
print(best_record_df)
Expected Output
Student Subject Marks 1 Pradnya Python 92 2 Samiksha SQL 88 0 Samadhan Python 85 5 Aarav SQL 79
Sorting places higher marks first. Duplicate removal then retains the first record for each student-subject combination.
10. Reset the Index After Removing Duplicates
Duplicate removal does not automatically renumber the index.
Use reset_index(drop=True) when a fresh sequential
index is useful.
clean_df = (
df.drop_duplicates()
.reset_index(drop=True)
)
print(clean_df)
Expected Output
Student Subject Marks 0 Samadhan Python 85 1 Pradnya Python 92 2 Samiksha SQL 88 3 Aarav SQL 79
The original index labels are replaced with 0, 1, 2, and 3.
Setting drop=True prevents the old index from being
added as a new column.
11. Understand the Duplicate-Removal Workflow
A reliable duplicate-cleaning workflow includes inspection and validation.
Validate the Cleaned Data
clean_df = df.drop_duplicates()
print("Rows before:", len(df))
print("Rows after:", len(clean_df))
print("Remaining duplicates:", clean_df.duplicated().sum())
Expected Output:
Rows before: 6 Rows after: 4 Remaining duplicates: 0
12. Practical Example: Clean Student Results
Samiksha receives a DataFrame containing repeated exam records. The goal is to retain one record for each student and subject, keeping the highest mark when multiple records exist.
results = pd.DataFrame({
"Student": [
"Samadhan",
"Pradnya",
"Samadhan",
"Samiksha",
"Pradnya"
],
"Subject": [
"Python",
"SQL",
"Python",
"SQL",
"SQL"
],
"Marks": [85, 90, 85, 88, 94]
})
clean_results = (
results.sort_values("Marks", ascending=False)
.drop_duplicates(subset=["Student", "Subject"])
.reset_index(drop=True)
)
print(clean_results)
Expected Output
Student Subject Marks 0 Pradnya SQL 94 1 Pradnya SQL 90 2 Samiksha SQL 88 3 Samadhan Python 85
The correct output is therefore:
Student Subject Marks 0 Pradnya SQL 94 1 Samiksha SQL 88 2 Samadhan Python 85
This example shows why the duplicate rule must match the business requirement. If separate assessments are legitimate, include an assessment identifier in the subset instead of removing records based only on student and subject.
13. Common Mistakes to Avoid
- Deleting without inspection: Check whether the repeated rows represent the same real-world record.
- Using the wrong subset: Selecting too few columns can remove valid records.
- Assuming keep="last" means highest value: It means the last row in the current order.
-
Expecting the original DataFrame to change:
By default,
drop_duplicates()returns a new DataFrame. -
Ignoring the index:
Use
reset_index(drop=True)when sequential indexing is required. - Assuming identical names always mean duplicate records: A person may legitimately have multiple transactions, assessments, or events.
14. Quick Revision
| Method or Parameter | Purpose |
|---|---|
df.duplicated() |
Identifies repeated row occurrences. |
df.duplicated(keep=False) |
Marks all occurrences of duplicated rows. |
df.duplicated().sum() |
Counts duplicate occurrences after the first. |
df.drop_duplicates() |
Removes repeated rows, keeping the first occurrence. |
keep="last" |
Keeps the last occurrence. |
keep=False |
Removes all rows that belong to duplicate groups. |
subset=["Student"] |
Checks duplicates using only the selected column or columns. |
reset_index(drop=True) |
Creates a fresh sequential index without retaining the old index. |
15. Practice Questions
Concept Questions
- What is the difference between
duplicated()anddrop_duplicates()? - What does
keep=Falsedo? - Why would you use the
subsetparameter? - Does
drop_duplicates()modify the original DataFrame by default? - Why might two rows with the same student name not be duplicates?
Hands-On Exercises
- Create a DataFrame with at least seven rows and two duplicate records.
- Display the Boolean duplicate mask.
- Count duplicate occurrences using
sum(). - Display every row belonging to a duplicate group.
- Remove duplicates while keeping the last occurrence.
- Remove duplicates using two selected columns.
- Reset the index and verify that no exact duplicate rows remain.
Summary
Duplicate records can affect counts, totals, and analytical
results. Pandas provides duplicated() to detect
repeated rows and drop_duplicates() to remove them.
Use keep to control which occurrence remains and
subset to define the columns used for comparison.
Inspect duplicate groups before deleting records, and validate
the cleaned DataFrame afterward. Use sorting when the requirement
is to retain a record with a particular value, such as the highest
mark.