Filtering Helper Functions in Pandas
Learn how to use Pandas helper methods such as isin(), between(), isna(), notna(), and query() to filter DataFrame rows efficiently and clearly.
1. Introduction to Filtering Helper Functions
Filtering helper functions simplify common DataFrame filtering tasks. Instead of writing several repetitive comparison expressions, you can use built-in Pandas methods designed for specific filtering requirements.
These methods are useful when checking whether values belong to a list, selecting values within a range, locating missing data, or writing readable filtering expressions.
Common Filtering Methods
| Method | Purpose |
|---|---|
isin() |
Checks whether values belong to a collection. |
between() |
Selects values within a specified range. |
isna() |
Identifies missing values. |
notna() |
Identifies values that are not missing. |
query() |
Filters rows using a string expression. |
Different helper methods address different filtering requirements while returning Boolean masks or filtered rows.
2. Create a Sample DataFrame
The following examples use a student DataFrame containing names, marks, attendance, cities, and optional assignment scores.
import pandas as pd
df = pd.DataFrame({
"Name": [
"Samadhan",
"Pradnya",
"Samiksha",
"Rahul",
"Neha",
"Amit"
],
"Marks": [85, 68, 92, 76, 55, 70],
"Attendance": [90, 85, 75, 88, 60, 80],
"City": [
"Pune",
"Mumbai",
"Pune",
"Nashik",
"Mumbai",
"Pune"
],
"Assignment": [18, None, 20, 15, None, 17]
})
print(df)
Expected Output:
Name Marks Attendance City Assignment 0 Samadhan 85 90 Pune 18.0 1 Pradnya 68 85 Mumbai NaN 2 Samiksha 92 75 Pune 20.0 3 Rahul 76 88 Nashik 15.0 4 Neha 55 60 Mumbai NaN 5 Amit 70 80 Pune 17.0
NaN in
this example. These missing values will be useful when learning
isna() and notna().
3. Using isin() to Match Multiple Values
The isin() method checks whether each value in a
Series belongs to a specified collection. It returns a Boolean
Series that can be used directly to filter a DataFrame.
Example 1: Filter by City
Select students who live in Pune or Mumbai.
result = df[
df["City"].isin(["Pune", "Mumbai"])
]
print(result[["Name", "City"]])
Expected Output:
Name City 0 Samadhan Pune 1 Pradnya Mumbai 2 Samiksha Pune 4 Neha Mumbai 5 Amit Pune
Example 2: Filter by Several Marks Values
result = df[
df["Marks"].isin([70, 85, 92])
]
print(result[["Name", "Marks"]])
Expected Output:
Name Marks 0 Samadhan 85 2 Samiksha 92 5 Amit 70
Example 3: Combine isin() with Another Condition
result = df[
(df["City"].isin(["Pune", "Mumbai"])) &
(df["Marks"] >= 70)
]
print(result[["Name", "Marks", "City"]])
Expected Output:
Name Marks City 0 Samadhan 85 Pune 2 Samiksha 92 Pune 5 Amit 70 Pune
isin() when checking a column against multiple
acceptable values instead of repeatedly writing equality checks.
4. Using between() to Filter a Range
The between() method identifies values within a
specified interval. By default, the lower and upper endpoints
are included.
Example 1: Marks Between 70 and 90
result = df[
df["Marks"].between(70, 90)
]
print(result[["Name", "Marks"]])
Expected Output:
Name Marks 0 Samadhan 85 3 Rahul 76 5 Amit 70
Example 2: Exclude the Endpoints
Set inclusive="neither" when values equal to either
endpoint should not be included.
result = df[
df["Marks"].between(
70,
90,
inclusive="neither"
)
]
print(result[["Name", "Marks"]])
Expected Output:
Name Marks 0 Samadhan 85 3 Rahul 76
Example 3: Attendance Between 80 and 90
result = df[
df["Attendance"].between(80, 90)
]
print(result[["Name", "Attendance"]])
Expected Output:
Name Attendance 0 Samadhan 90 1 Pradnya 85 3 Rahul 88 5 Amit 80
inclusive argument supports
"both", "neither",
"left", and "right".
Use the option that matches your boundary requirements.
5. Using isna() to Find Missing Values
Real-world datasets often contain incomplete information.
The isna() method identifies missing values and
returns True for missing entries and False for non-missing entries.
Example 1: Find Rows with Missing Assignments
result = df[
df["Assignment"].isna()
]
print(result[["Name", "Assignment"]])
Expected Output:
Name Assignment 1 Pradnya NaN 4 Neha NaN
Example 2: Find Rows with Any Missing Value
result = df[
df.isna().any(axis=1)
]
print(result[["Name", "Assignment"]])
Expected Output:
Name Assignment 1 Pradnya NaN 4 Neha NaN
Here, isna() checks every cell,
any(axis=1) identifies rows containing at least one
missing value, and the resulting Boolean mask filters the rows.
Example 3: Count Missing Values
print(df["Assignment"].isna().sum())
Expected Output:
2
6. Using notna() to Find Available Values
The notna() method is the opposite of
isna(). It identifies values that are not missing.
Example 1: Select Students with Assignment Scores
result = df[
df["Assignment"].notna()
]
print(result[["Name", "Assignment"]])
Expected Output:
Name Assignment 0 Samadhan 18.0 2 Samiksha 20.0 3 Rahul 15.0 5 Amit 17.0
Example 2: Keep Rows with Available Assignments and Marks Above 70
result = df[
df["Assignment"].notna() &
(df["Marks"] > 70)
]
print(result[["Name", "Marks", "Assignment"]])
Expected Output:
Name Marks Assignment 0 Samadhan 85 18.0 2 Samiksha 92 20.0 3 Rahul 76 15.0
isna() to find missing entries and
notna() to select available entries.
7. Filtering with query()
The query() method filters rows using an expression
written as a string. It can make many filtering statements more
compact and readable.
Example 1: Filter by Marks
result = df.query("Marks > 80")
print(result[["Name", "Marks"]])
Expected Output:
Name Marks 0 Samadhan 85 2 Samiksha 92
Example 2: Combine Multiple Conditions
result = df.query(
"Marks >= 70 and Attendance >= 80"
)
print(result[["Name", "Marks", "Attendance"]])
Expected Output:
Name Marks Attendance 0 Samadhan 85 90 3 Rahul 76 88 5 Amit 70 80
Example 3: Filter String Values
result = df.query("City == 'Pune'")
print(result[["Name", "City"]])
Expected Output:
Name City 0 Samadhan Pune 2 Samiksha Pune 5 Amit Pune
Example 4: Use an External Variable
Prefix an external Python variable with @ inside
the query expression.
minimum_marks = 75
result = df.query("Marks >= @minimum_marks")
print(result[["Name", "Marks"]])
Expected Output:
Name Marks 0 Samadhan 85 2 Samiksha 92 3 Rahul 76
8. Combining Helper Methods
Helper methods become especially useful when combined with Boolean operators. The following example selects students who live in Pune or Mumbai, have marks between 70 and 90, and have a recorded assignment score.
result = df[
df["City"].isin(["Pune", "Mumbai"]) &
df["Marks"].between(70, 90) &
df["Assignment"].notna()
]
print(
result[
["Name", "Marks", "City", "Assignment"]
]
)
Expected Output:
Name Marks City Assignment 0 Samadhan 85 Pune 18.0 5 Amit 70 Pune 17.0
Each method handles one requirement: isin() checks
the city, between() checks the marks range, and
notna() verifies that the assignment score exists.
The AND operator requires every requirement to be satisfied.
9. Filtering Workflow
Choose a helper method according to the type of condition required, then apply it to the DataFrame.
10. Common Mistakes and Best Practices
| Common Mistake | Recommended Approach |
|---|---|
| Repeating multiple equality comparisons for a long list of accepted values. |
Use isin() to check membership in a list.
|
| Writing separate greater-than and less-than checks for every range. |
Use between() when checking an interval.
|
Comparing missing values directly with
== NaN.
|
Use isna() or notna().
|
Combining Pandas Boolean Series with Python's
and or or operators.
|
Use & or | with
parentheses around comparisons.
|
| Building query strings directly from untrusted input. | Validate values and use trusted query expressions. |
11. Practical Exercises
Exercise 1: Filter Multiple Cities
Select students who live in Pune or Nashik using
isin().
Expected Names: Samadhan, Samiksha, Rahul, and Amit.
Exercise 2: Filter a Marks Range
Select students whose marks are between 60 and 80, including both endpoints.
Expected Names: Pradnya, Rahul, and Amit.
Exercise 3: Find Missing Values
Select rows where the Assignment column contains a missing value.
Expected Names: Pradnya and Neha.
Exercise 4: Find Available Assignment Scores
Select students whose Assignment values are not missing.
Expected Names: Samadhan, Samiksha, Rahul, and Amit.
Exercise 5: Use query()
Select students whose marks are at least 70 and whose
attendance is at least 80 using query().
Expected Names: Samadhan, Rahul, and Amit.
12. Quick Revision Questions
- What is the purpose of the
isin()method? - How does
between()handle range endpoints by default? - What is the difference between
isna()andnotna()? - How can you identify DataFrame rows containing missing values?
- What is the purpose of the
query()method? - How can an external Python variable be referenced inside
query()? - Why should parentheses be used when combining Boolean Series?
- How can several helper methods be combined in one filter?
Summary
Pandas filtering helper methods simplify common data-selection
tasks. Use isin() to match multiple allowed values,
between() to select values within a range,
isna() to identify missing values, and
notna() to select non-missing values.
The query() method provides an alternative way to
express many filtering conditions. Helper methods can also be
combined with Boolean operators to build more precise filters.
Selecting the appropriate helper method improves code readability, reduces repetitive comparisons, and makes data analysis easier to maintain.