Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Filtering Groups in Pandas

Learn how to keep or remove entire groups based on group size, average values, totals, and custom conditions.

1. What Is Group Filtering?

Group filtering means selecting complete groups that satisfy a condition. Instead of checking each row independently, Pandas evaluates a condition for each group and keeps or removes the entire group.

Example: Student Performance

Suppose a school has students in three classes. You want to keep only classes whose average score is at least 75. Every student in a qualifying class remains in the result, even if an individual student scored below 75.

Group-Level Filtering Class A Scores: 80, 90, 85 Average: 85 Keep group Class B Scores: 60, 70, 65 Average: 65 Remove group Class C Scores: 75, 80, 85 Average: 80 Keep group Result: All rows from Classes A and C

If the group average is at least 75, keep every row belonging to that group.

2. Create a Sample DataFrame

We will use student scores grouped by class throughout this lesson.

import pandas as pd

df = pd.DataFrame({
    "Student": [
        "Samadhan", "Pradnya", "Samiksha",
        "Aarav", "Diya", "Rohan",
        "Meera", "Kabir", "Anaya"
    ],
    "Class": [
        "A", "A", "A",
        "B", "B", "B",
        "C", "C", "C"
    ],
    "Score": [80, 90, 85, 60, 70, 65, 75, 80, 85],
    "Attendance": [95, 98, 92, 70, 80, 75, 88, 90, 94]
})

print(df)

Expected output

    Student Class  Score  Attendance
0  Samadhan     A     80          95
1    Pradnya     A     90          98
2   Samiksha     A     85          92
3      Aarav     B     60          70
4       Diya     B     70          80
5      Rohan     B     65          75
6      Meera     C     75          88
7      Kabir     C     80          90
8      Anaya     C     85          94

3. Row Filtering vs Group Filtering

These techniques answer different questions. Row filtering checks each record; group filtering checks a condition for the group as a whole.

Technique Condition applies to Typical question
Boolean row filtering Each row Which students scored at least 80?
GroupBy.filter() Each complete group Which classes have an average score of at least 75?

Example A: Keep individual students with scores of at least 80

high_scorers = df[df["Score"] >= 80]

print(high_scorers[["Student", "Class", "Score"]])

Expected output

    Student Class  Score
0  Samadhan     A     80
1    Pradnya     A     90
2   Samiksha     A     85
7      Kabir     C     80
8      Anaya     C     85

Example B: Keep entire classes whose average score is at least 75

qualified_classes = df.groupby("Class").filter(
    lambda group: group["Score"].mean() >= 75
)

print(qualified_classes[["Student", "Class", "Score"]])

Expected output

    Student Class  Score
0  Samadhan     A     80
1    Pradnya     A     90
2   Samiksha     A     85
6      Meera     C     75
7      Kabir     C     80
8      Anaya     C     85
Remember: Class C is retained completely, including Meera, whose score is 75. Group filtering does not remove individual members simply because their scores are below the group threshold.

4. How GroupBy.filter() Works

The filter() method evaluates a function for each group. If the function returns True, the group is retained; if it returns False, the group is excluded.

DataFrame Student records groupby() Split into classes filter() Evaluate condition Filtered DataFrame Only qualifying groups True: keep group False: remove group

The condition is evaluated separately for each group, and the result contains rows from qualifying groups.

result = df.groupby("Class").filter(
    lambda group: group["Score"].mean() >= 75
)

In this example:

  • groupby("Class") divides the DataFrame into classes A, B, and C.
  • lambda group represents the current class's DataFrame.
  • group["Score"].mean() calculates the current class's average score.
  • >= 75 determines whether the group qualifies.

5. Filter Groups by Number of Rows

You can retain groups based on how many records they contain. This is useful when a department, class, store, or category needs a minimum number of observations.

Example: Keep classes with at least three students

large_classes = df.groupby("Class").filter(
    lambda group: len(group) >= 3
)

print(large_classes.groupby("Class").size())

Expected output

Class
A    3
B    3
C    3
dtype: int64

Every class qualifies because each contains three student records.

Example: Keep groups with more than three rows

large_classes = df.groupby("Class").filter(
    lambda group: len(group) > 3
)

print(large_classes)

Expected output

Empty DataFrame
Columns: [Student, Class, Score, Attendance]
Index: []
Important: An empty result is valid when no group satisfies the condition. Check whether the threshold matches the data you have.

6. Filter Groups by Average Score

Average-based filtering is useful when you want to keep groups whose typical performance meets a target.

high_average_classes = df.groupby("Class").filter(
    lambda group: group["Score"].mean() >= 80
)

print(high_average_classes[["Student", "Class", "Score"]])

Expected output

    Student Class  Score
0  Samadhan     A     80
1    Pradnya     A     90
2   Samiksha     A     85
6      Meera     C     75
7      Kabir     C     80
8      Anaya     C     85

Class A has an average of 85, Class B has an average of 65, and Class C has an average of 80. Therefore, classes A and C are retained.

7. Filter Groups by Total Score

Instead of checking the average, you can evaluate the sum of a group's values.

high_total_classes = df.groupby("Class").filter(
    lambda group: group["Score"].sum() >= 240
)

print(high_total_classes[["Student", "Class", "Score"]])

Expected output

    Student Class  Score
0  Samadhan     A     80
1    Pradnya     A     90
2   Samiksha     A     85
6      Meera     C     75
7      Kabir     C     80
8      Anaya     C     85

Class A totals 255, Class B totals 195, and Class C totals 240. Only classes A and C meet the threshold.

8. Filter Groups Using Multiple Conditions

A group can qualify only when multiple conditions are satisfied. Use and when both conditions must be true.

Example: Average score of at least 75 and average attendance of at least 90

qualified_groups = df.groupby("Class").filter(
    lambda group:
        group["Score"].mean() >= 75
        and group["Attendance"].mean() >= 90
)

print(qualified_groups[["Student", "Class", "Score", "Attendance"]])

Expected output

    Student Class  Score  Attendance
0  Samadhan     A     80          95
1    Pradnya     A     90          98
2   Samiksha     A     85          92

Class A meets both requirements. Class C meets the score requirement but has an average attendance below 90, so it is excluded.

Tip: Use and and or between complete scalar conditions in the filter function. Do not use Python's and to combine two Pandas Series.

9. Filter Groups by Maximum or Minimum

You can evaluate the highest or lowest value within each group.

Example: Keep classes where every student scored at least 65

consistent_classes = df.groupby("Class").filter(
    lambda group: group["Score"].min() >= 65
)

print(consistent_classes[["Student", "Class", "Score"]])

Expected output

    Student Class  Score
0  Samadhan     A     80
1    Pradnya     A     90
2   Samiksha     A     85
3      Aarav     B     60
4       Diya     B     70
5      Rohan     B     65
6      Meera     C     75
7      Kabir     C     80
8      Anaya     C     85

All classes qualify because their minimum scores are 80, 60, and 75 respectively? Only classes A and C meet the condition of a minimum score of at least 65.

10. Filter Groups by a Custom Rule

The filter function can contain any valid condition that returns a single Boolean value for the current group.

Example: Keep classes with at least one score of 90 or higher

classes_with_top_scores = df.groupby("Class").filter(
    lambda group: (group["Score"] >= 90).any()
)

print(classes_with_top_scores[["Student", "Class", "Score"]])

Expected output

    Student Class  Score
0  Samadhan     A     80
1    Pradnya     A     90
2   Samiksha     A     85

The expression (group["Score"] >= 90).any() checks whether at least one student in the current class has a score of 90 or above. Class A qualifies; classes B and C do not.

Using all()

classes_all_scores_high = df.groupby("Class").filter(
    lambda group: (group["Score"] >= 70).all()
)

The all() method returns True only when every value satisfies the condition. Here, Class B is excluded because it contains a score of 60 and another score of 65.

11. Filtering Groups Without Losing the Original Data

By default, groupby().filter() returns the rows of qualifying groups as a DataFrame. Assigning that result to a new variable lets you keep the original DataFrame unchanged.

original_df = df.copy()

filtered_df = df.groupby("Class").filter(
    lambda group: group["Score"].mean() >= 75
)

print("Original rows:", len(original_df))
print("Filtered rows:", len(filtered_df))

Expected output

Original rows: 9
Filtered rows: 6
Note: With the default dropna=True, groups that fail the filter are excluded. The optional dropna=False behavior instead marks rows from failing groups with missing values in the returned result. For most beginner workflows, the default is the simplest choice.

12. Common Mistakes

Mistake 1: Confusing row and group conditions

Incorrect for selecting complete classes: df[df["Score"] >= 75]

This selects individual students. Use groupby().filter() when the condition applies to an entire group.

Mistake 2: Returning a Series instead of a Boolean

A filter function should return one Boolean value per group. Expressions such as group["Score"] >= 75 return a Series, not one Boolean.

Use an aggregation or reduction such as group["Score"].mean() >= 75, (group["Score"] >= 75).any(), or (group["Score"] >= 75).all().

Mistake 3: Using the wrong threshold

Check whether the question asks for a minimum average, maximum value, total, or number of rows. Each condition produces a different result.

Mistake 4: Assuming the filtered rows are renumbered

The result usually retains the original row index. If you need a new sequential index, use reset_index(drop=True).

filtered_df = filtered_df.reset_index(drop=True)

13. Practice Exercises

Use the sample DataFrame from this lesson to solve the following tasks.

Exercise 1: Minimum group size

Keep classes that contain at least three student records.

Hint: Use len(group) >= 3.

Exercise 2: Average score

Keep complete classes whose average score is greater than 70.

Hint: Use group["Score"].mean().

Exercise 3: Attendance requirement

Keep classes whose average attendance is at least 90.

Hint: Use group["Attendance"].mean().

Exercise 4: All students must pass

Keep classes where every student has a score of at least 70.

Hint: Use (group["Score"] >= 70).all().

Exercise 5: At least one top scorer

Keep classes that contain at least one student with a score of 85 or higher.

Hint: Use (group["Score"] >= 85).any().

14. Revision Questions

  1. What is group filtering in Pandas?
  2. How does GroupBy.filter() differ from Boolean row filtering?
  3. What does the filter function need to return for each group?
  4. How can you retain groups with a minimum number of rows?
  5. How do any() and all() differ?
  6. What happens when no group satisfies the filter condition?
  7. How can you filter groups by both average score and attendance?

Summary

Group filtering selects or excludes complete groups based on conditions evaluated at the group level. The Pandas groupby().filter() method is useful for checking group size, average, total, minimum, maximum, and custom Boolean rules. Use ordinary Boolean indexing when you want to filter individual rows, and use group filtering when the entire group must qualify.

Key methods: groupby(), filter(), mean(), sum(), min(), max(), any(), all(), and reset_index().