Filtering and Conditional Selection in Pandas
Learn how to filter DataFrame rows using comparison operators, Boolean conditions, multiple conditions, and useful helper methods.
1. What Is Filtering in Pandas?
Filtering means selecting only the rows that satisfy a condition. Instead of displaying the entire dataset, you can retrieve records matching your requirements.
For example, Samadhan may want students who scored more than 80 marks. Pradnya may want to find students from Pune. Pandas allows both tasks using Boolean conditions.
Create a Sample DataFrame
import pandas as pd
df = pd.DataFrame({
"Name": ["Samadhan", "Pradnya", "Samiksha", "Amit", "Neha"],
"Age": [21, 22, 20, 23, 21],
"City": ["Pune", "Mumbai", "Pune", "Nashik", "Mumbai"],
"Marks": [85, 92, 78, 88, 67]
})
print(df)
Output
Name Age City Marks 0 Samadhan 21 Pune 85 1 Pradnya 22 Mumbai 92 2 Samiksha 20 Pune 78 3 Amit 23 Nashik 88 4 Neha 21 Mumbai 67
2. Basic Filtering with Comparison Operators
Comparison operators compare values and return Boolean results:
True or False.
| Operator | Meaning | Example |
|---|---|---|
== |
Equal to | df["City"] == "Pune" |
!= |
Not equal to | df["City"] != "Pune" |
> |
Greater than | df["Marks"] > 80 |
< |
Less than | df["Age"] < 22 |
>= |
Greater than or equal to | df["Marks"] >= 85 |
<= |
Less than or equal to | df["Age"] <= 21 |
Example: Students scoring more than 80
high_marks = df[df["Marks"] > 80] print(high_marks)
Output
Name Age City Marks 0 Samadhan 21 Pune 85 1 Pradnya 22 Mumbai 92 3 Amit 23 Nashik 88
Only rows where Marks is greater than 80 are included. Students scoring exactly 80 would not match this condition.
3. Visual Guide: How Boolean Filtering Works
Pandas evaluates the condition for each row and uses the resulting Boolean mask to select matching records.
Figure 1: A Boolean condition creates a mask used to filter rows.
4. Filter Text Columns
You can filter text columns by comparing their values with a string.
Example: Students from Pune
pune_students = df[df["City"] == "Pune"] print(pune_students)
Output
Name Age City Marks 0 Samadhan 21 Pune 85 2 Samiksha 20 Pune 78
Example: Students not from Pune
other_cities = df[df["City"] != "Pune"] print(other_cities)
This returns the rows whose City value is not Pune. Missing values require special consideration because comparisons with missing values do not behave like ordinary strings.
5. Filter Using Multiple Conditions
You can combine conditions to select rows matching more than one requirement.
AND condition using &
Use & when both conditions must be true.
Put each condition inside parentheses.
result = df[
(df["Marks"] >= 80) &
(df["City"] == "Pune")
]
print(result)
Output
Name Age City Marks 0 Samadhan 21 Pune 85
OR condition using |
Use | when at least one condition must be true.
result = df[
(df["City"] == "Pune") |
(df["Marks"] >= 90)
]
print(result)
Output
Name Age City Marks 0 Samadhan 21 Pune 85 1 Pradnya 22 Mumbai 92 2 Samiksha 20 Pune 78
& and
| for element-wise Pandas conditions, not Python's
and and or. Parentheses around
individual comparisons help ensure the intended evaluation.
6. Filter with isin()
Use isin() when you want to match any value from
a list of allowed values.
Example: Students from Pune or Mumbai
cities = ["Pune", "Mumbai"] result = df[df["City"].isin(cities)] print(result)
Output
Name Age City Marks 0 Samadhan 21 Pune 85 1 Pradnya 22 Mumbai 92 2 Samiksha 20 Pune 78 4 Neha 21 Mumbai 67
This is often clearer than writing several OR conditions for the same column.
7. Filter Values Between a Range
Use between() to select numerical values within
a range. Both boundaries are included by default.
result = df[df["Marks"].between(80, 90)] print(result)
Output
Name Age City Marks 0 Samadhan 21 Pune 85 3 Amit 23 Nashik 88
A score of 80 or 90 would also match this condition because the default setting includes both endpoints.
8. Filter Text Using str.contains()
The string accessor str.contains() helps select
rows whose text contains a particular pattern.
Example: Names containing "a"
result = df[
df["Name"].str.contains("a", case=False, na=False)
]
print(result)
Output
Name Age City Marks 0 Samadhan 21 Pune 85 1 Pradnya 22 Mumbai 92 2 Samiksha 20 Pune 78 3 Amit 23 Nashik 88 4 Neha 21 Mumbai 67
The case=False argument makes the search
case-insensitive. The na=False argument treats
missing text values as non-matches.
9. Visual Guide: AND vs OR Conditions
Figure 2: AND narrows the result to rows meeting both conditions; OR includes rows meeting either condition.
10. Select Columns After Filtering
You can filter rows and then select only the columns required for a report.
result = df.loc[
df["Marks"] >= 80,
["Name", "Marks"]
]
print(result)
Output
Name Marks 0 Samadhan 85 1 Pradnya 92 3 Amit 88
Using loc[] is a convenient way to combine a
Boolean row condition with a selection of columns.
11. Common Filtering Mistakes
Mistake 1: Using and instead of &
For element-wise Pandas conditions, use
& and | with parentheses
around each comparison.
Mistake 2: Forgetting parentheses
# Correct df[(df["Marks"] >= 80) & (df["Age"] <= 22)]
Mistake 3: Using the wrong comparison operator
> 80 excludes 80, while
>= 80 includes it. Choose the operator
that matches the requirement.
Mistake 4: Ignoring missing text values
When using str.contains(), consider setting
na=False if missing values should not match.
12. Practice Exercises
Exercise 1: Basic Conditions
- Find students with marks greater than 80.
- Find students who belong to Mumbai.
- Find students aged 21 or younger.
Exercise 2: Multiple Conditions
- Find Pune students who scored at least 80.
- Find students from Pune or Nashik.
- Find students whose marks are between 70 and 90.
Exercise 3: String Filtering
- Find names containing the letter "a".
- Make the search case-insensitive.
- Display only Name and Marks for matching rows.
Suggested solution
# Marks greater than 80
print(df[df["Marks"] > 80])
# Students from Mumbai
print(df[df["City"] == "Mumbai"])
# Age 21 or younger
print(df[df["Age"] <= 21])
# Pune students with at least 80 marks
print(df[(df["City"] == "Pune") & (df["Marks"] >= 80)])
# Students from Pune or Nashik
print(df[df["City"].isin(["Pune", "Nashik"])])
# Marks between 70 and 90
print(df[df["Marks"].between(70, 90)])
# Case-insensitive text search
mask = df["Name"].str.contains("a", case=False, na=False)
print(df.loc[mask, ["Name", "Marks"]])
13. Revision Questions
- What is Boolean filtering in Pandas?
- What does a Boolean mask contain?
- What is the difference between
&and|? - Why should each condition be enclosed in parentheses?
- How do you filter a DataFrame using
isin()? - What does
between()do? - How does
str.contains()help filter text? - How do you select particular columns after filtering rows?
Summary
Filtering allows you to retrieve only the DataFrame rows
that meet specific conditions. Comparison operators create
Boolean masks, while & and |
combine multiple conditions.
Methods such as isin(), between(),
and str.contains() make common filtering tasks
easier. Use loc[] when you want to combine row
filtering with column selection.