Filtering Groups in Pandas
Learn how to keep or remove entire groups based on group size, average values, totals, and custom conditions.
1. What Is Group Filtering?
Group filtering means selecting complete groups that satisfy a condition. Instead of checking each row independently, Pandas evaluates a condition for each group and keeps or removes the entire group.
Example: Student Performance
Suppose a school has students in three classes. You want to keep only classes whose average score is at least 75. Every student in a qualifying class remains in the result, even if an individual student scored below 75.
If the group average is at least 75, keep every row belonging to that group.
2. Create a Sample DataFrame
We will use student scores grouped by class throughout this lesson.
import pandas as pd
df = pd.DataFrame({
"Student": [
"Samadhan", "Pradnya", "Samiksha",
"Aarav", "Diya", "Rohan",
"Meera", "Kabir", "Anaya"
],
"Class": [
"A", "A", "A",
"B", "B", "B",
"C", "C", "C"
],
"Score": [80, 90, 85, 60, 70, 65, 75, 80, 85],
"Attendance": [95, 98, 92, 70, 80, 75, 88, 90, 94]
})
print(df)
Expected output
Student Class Score Attendance 0 Samadhan A 80 95 1 Pradnya A 90 98 2 Samiksha A 85 92 3 Aarav B 60 70 4 Diya B 70 80 5 Rohan B 65 75 6 Meera C 75 88 7 Kabir C 80 90 8 Anaya C 85 94
3. Row Filtering vs Group Filtering
These techniques answer different questions. Row filtering checks each record; group filtering checks a condition for the group as a whole.
| Technique | Condition applies to | Typical question |
|---|---|---|
| Boolean row filtering | Each row | Which students scored at least 80? |
GroupBy.filter() |
Each complete group | Which classes have an average score of at least 75? |
Example A: Keep individual students with scores of at least 80
high_scorers = df[df["Score"] >= 80] print(high_scorers[["Student", "Class", "Score"]])
Expected output
Student Class Score 0 Samadhan A 80 1 Pradnya A 90 2 Samiksha A 85 7 Kabir C 80 8 Anaya C 85
Example B: Keep entire classes whose average score is at least 75
qualified_classes = df.groupby("Class").filter(
lambda group: group["Score"].mean() >= 75
)
print(qualified_classes[["Student", "Class", "Score"]])
Expected output
Student Class Score 0 Samadhan A 80 1 Pradnya A 90 2 Samiksha A 85 6 Meera C 75 7 Kabir C 80 8 Anaya C 85
4. How GroupBy.filter() Works
The filter() method evaluates a function for each group. If the function returns True, the group is retained; if it returns False, the group is excluded.
The condition is evaluated separately for each group, and the result contains rows from qualifying groups.
result = df.groupby("Class").filter(
lambda group: group["Score"].mean() >= 75
)
In this example:
groupby("Class")divides the DataFrame into classes A, B, and C.lambda grouprepresents the current class's DataFrame.group["Score"].mean()calculates the current class's average score.>= 75determines whether the group qualifies.
5. Filter Groups by Number of Rows
You can retain groups based on how many records they contain. This is useful when a department, class, store, or category needs a minimum number of observations.
Example: Keep classes with at least three students
large_classes = df.groupby("Class").filter(
lambda group: len(group) >= 3
)
print(large_classes.groupby("Class").size())
Expected output
Class A 3 B 3 C 3 dtype: int64
Every class qualifies because each contains three student records.
Example: Keep groups with more than three rows
large_classes = df.groupby("Class").filter(
lambda group: len(group) > 3
)
print(large_classes)
Expected output
Empty DataFrame Columns: [Student, Class, Score, Attendance] Index: []
6. Filter Groups by Average Score
Average-based filtering is useful when you want to keep groups whose typical performance meets a target.
high_average_classes = df.groupby("Class").filter(
lambda group: group["Score"].mean() >= 80
)
print(high_average_classes[["Student", "Class", "Score"]])
Expected output
Student Class Score 0 Samadhan A 80 1 Pradnya A 90 2 Samiksha A 85 6 Meera C 75 7 Kabir C 80 8 Anaya C 85
Class A has an average of 85, Class B has an average of 65, and Class C has an average of 80. Therefore, classes A and C are retained.
7. Filter Groups by Total Score
Instead of checking the average, you can evaluate the sum of a group's values.
high_total_classes = df.groupby("Class").filter(
lambda group: group["Score"].sum() >= 240
)
print(high_total_classes[["Student", "Class", "Score"]])
Expected output
Student Class Score 0 Samadhan A 80 1 Pradnya A 90 2 Samiksha A 85 6 Meera C 75 7 Kabir C 80 8 Anaya C 85
Class A totals 255, Class B totals 195, and Class C totals 240. Only classes A and C meet the threshold.
8. Filter Groups Using Multiple Conditions
A group can qualify only when multiple conditions are satisfied. Use and when both conditions must be true.
Example: Average score of at least 75 and average attendance of at least 90
qualified_groups = df.groupby("Class").filter(
lambda group:
group["Score"].mean() >= 75
and group["Attendance"].mean() >= 90
)
print(qualified_groups[["Student", "Class", "Score", "Attendance"]])
Expected output
Student Class Score Attendance 0 Samadhan A 80 95 1 Pradnya A 90 98 2 Samiksha A 85 92
Class A meets both requirements. Class C meets the score requirement but has an average attendance below 90, so it is excluded.
and and or between complete scalar conditions in the filter function. Do not use Python's and to combine two Pandas Series.
9. Filter Groups by Maximum or Minimum
You can evaluate the highest or lowest value within each group.
Example: Keep classes where every student scored at least 65
consistent_classes = df.groupby("Class").filter(
lambda group: group["Score"].min() >= 65
)
print(consistent_classes[["Student", "Class", "Score"]])
Expected output
Student Class Score 0 Samadhan A 80 1 Pradnya A 90 2 Samiksha A 85 3 Aarav B 60 4 Diya B 70 5 Rohan B 65 6 Meera C 75 7 Kabir C 80 8 Anaya C 85
All classes qualify because their minimum scores are 80, 60, and 75 respectively? Only classes A and C meet the condition of a minimum score of at least 65.
10. Filter Groups by a Custom Rule
The filter function can contain any valid condition that returns a single Boolean value for the current group.
Example: Keep classes with at least one score of 90 or higher
classes_with_top_scores = df.groupby("Class").filter(
lambda group: (group["Score"] >= 90).any()
)
print(classes_with_top_scores[["Student", "Class", "Score"]])
Expected output
Student Class Score 0 Samadhan A 80 1 Pradnya A 90 2 Samiksha A 85
The expression (group["Score"] >= 90).any() checks whether at least one student in the current class has a score of 90 or above. Class A qualifies; classes B and C do not.
Using all()
classes_all_scores_high = df.groupby("Class").filter(
lambda group: (group["Score"] >= 70).all()
)
The all() method returns True only when every value satisfies the condition. Here, Class B is excluded because it contains a score of 60 and another score of 65.
11. Filtering Groups Without Losing the Original Data
By default, groupby().filter() returns the rows of qualifying groups as a DataFrame. Assigning that result to a new variable lets you keep the original DataFrame unchanged.
original_df = df.copy()
filtered_df = df.groupby("Class").filter(
lambda group: group["Score"].mean() >= 75
)
print("Original rows:", len(original_df))
print("Filtered rows:", len(filtered_df))
Expected output
Original rows: 9 Filtered rows: 6
dropna=True, groups that fail the filter are excluded. The optional dropna=False behavior instead marks rows from failing groups with missing values in the returned result. For most beginner workflows, the default is the simplest choice.
12. Common Mistakes
Mistake 1: Confusing row and group conditions
Incorrect for selecting complete classes: df[df["Score"] >= 75]
This selects individual students. Use groupby().filter() when the condition applies to an entire group.
Mistake 2: Returning a Series instead of a Boolean
A filter function should return one Boolean value per group. Expressions such as group["Score"] >= 75 return a Series, not one Boolean.
Use an aggregation or reduction such as group["Score"].mean() >= 75, (group["Score"] >= 75).any(), or (group["Score"] >= 75).all().
Mistake 3: Using the wrong threshold
Check whether the question asks for a minimum average, maximum value, total, or number of rows. Each condition produces a different result.
Mistake 4: Assuming the filtered rows are renumbered
The result usually retains the original row index. If you need a new sequential index, use reset_index(drop=True).
filtered_df = filtered_df.reset_index(drop=True)
13. Practice Exercises
Use the sample DataFrame from this lesson to solve the following tasks.
Exercise 1: Minimum group size
Keep classes that contain at least three student records.
Hint: Use len(group) >= 3.
Exercise 2: Average score
Keep complete classes whose average score is greater than 70.
Hint: Use group["Score"].mean().
Exercise 3: Attendance requirement
Keep classes whose average attendance is at least 90.
Hint: Use group["Attendance"].mean().
Exercise 4: All students must pass
Keep classes where every student has a score of at least 70.
Hint: Use (group["Score"] >= 70).all().
Exercise 5: At least one top scorer
Keep classes that contain at least one student with a score of 85 or higher.
Hint: Use (group["Score"] >= 85).any().
14. Revision Questions
- What is group filtering in Pandas?
- How does
GroupBy.filter()differ from Boolean row filtering? - What does the filter function need to return for each group?
- How can you retain groups with a minimum number of rows?
- How do
any()andall()differ? - What happens when no group satisfies the filter condition?
- How can you filter groups by both average score and attendance?
Summary
Group filtering selects or excludes complete groups based on conditions evaluated at the group level. The Pandas groupby().filter() method is useful for checking group size, average, total, minimum, maximum, and custom Boolean rules. Use ordinary Boolean indexing when you want to filter individual rows, and use group filtering when the entire group must qualify.
Key methods: groupby(), filter(), mean(), sum(), min(), max(), any(), all(), and reset_index().