Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Filtering and Conditional Selection in Pandas

Learn how to filter DataFrame rows using comparison operators, Boolean conditions, multiple conditions, and useful helper methods.

1. What Is Filtering in Pandas?

Filtering means selecting only the rows that satisfy a condition. Instead of displaying the entire dataset, you can retrieve records matching your requirements.

For example, Samadhan may want students who scored more than 80 marks. Pradnya may want to find students from Pune. Pandas allows both tasks using Boolean conditions.

Create a Sample DataFrame

import pandas as pd

df = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Amit", "Neha"],
    "Age": [21, 22, 20, 23, 21],
    "City": ["Pune", "Mumbai", "Pune", "Nashik", "Mumbai"],
    "Marks": [85, 92, 78, 88, 67]
})

print(df)

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
1   Pradnya   22  Mumbai     92
2  Samiksha   20    Pune     78
3      Amit   23  Nashik     88
4      Neha   21  Mumbai     67

2. Basic Filtering with Comparison Operators

Comparison operators compare values and return Boolean results: True or False.

Operator Meaning Example
== Equal to df["City"] == "Pune"
!= Not equal to df["City"] != "Pune"
> Greater than df["Marks"] > 80
< Less than df["Age"] < 22
>= Greater than or equal to df["Marks"] >= 85
<= Less than or equal to df["Age"] <= 21

Example: Students scoring more than 80

high_marks = df[df["Marks"] > 80]

print(high_marks)

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
1   Pradnya   22  Mumbai     92
3      Amit   23  Nashik     88

Only rows where Marks is greater than 80 are included. Students scoring exactly 80 would not match this condition.

3. Visual Guide: How Boolean Filtering Works

Pandas evaluates the condition for each row and uses the resulting Boolean mask to select matching records.

Filter: Marks greater than 80 Original Marks Samadhan: 85 → True Pradnya: 92 → True Samiksha: 78 → False Amit: 88 → True Neha: 67 → False Filtered Results Samadhan — 85 Pradnya — 92 Amit — 88 df[df["Marks"] > 80] True rows are retained; False rows are excluded.

Figure 1: A Boolean condition creates a mask used to filter rows.

4. Filter Text Columns

You can filter text columns by comparing their values with a string.

Example: Students from Pune

pune_students = df[df["City"] == "Pune"]

print(pune_students)

Output

       Name  Age  City  Marks
0  Samadhan   21  Pune     85
2  Samiksha   20  Pune     78

Example: Students not from Pune

other_cities = df[df["City"] != "Pune"]

print(other_cities)

This returns the rows whose City value is not Pune. Missing values require special consideration because comparisons with missing values do not behave like ordinary strings.

5. Filter Using Multiple Conditions

You can combine conditions to select rows matching more than one requirement.

AND condition using &

Use & when both conditions must be true. Put each condition inside parentheses.

result = df[
    (df["Marks"] >= 80) &
    (df["City"] == "Pune")
]

print(result)

Output

       Name  Age  City  Marks
0  Samadhan   21  Pune     85

OR condition using |

Use | when at least one condition must be true.

result = df[
    (df["City"] == "Pune") |
    (df["Marks"] >= 90)
]

print(result)

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
1   Pradnya   22  Mumbai     92
2  Samiksha   20    Pune     78
Important: Use & and | for element-wise Pandas conditions, not Python's and and or. Parentheses around individual comparisons help ensure the intended evaluation.

6. Filter with isin()

Use isin() when you want to match any value from a list of allowed values.

Example: Students from Pune or Mumbai

cities = ["Pune", "Mumbai"]

result = df[df["City"].isin(cities)]

print(result)

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
1   Pradnya   22  Mumbai     92
2  Samiksha   20    Pune     78
4      Neha   21  Mumbai     67

This is often clearer than writing several OR conditions for the same column.

7. Filter Values Between a Range

Use between() to select numerical values within a range. Both boundaries are included by default.

result = df[df["Marks"].between(80, 90)]

print(result)

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
3      Amit   23  Nashik     88

A score of 80 or 90 would also match this condition because the default setting includes both endpoints.

8. Filter Text Using str.contains()

The string accessor str.contains() helps select rows whose text contains a particular pattern.

Example: Names containing "a"

result = df[
    df["Name"].str.contains("a", case=False, na=False)
]

print(result)

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
1   Pradnya   22  Mumbai     92
2  Samiksha   20    Pune     78
3      Amit   23  Nashik     88
4      Neha   21  Mumbai     67

The case=False argument makes the search case-insensitive. The na=False argument treats missing text values as non-matches.

9. Visual Guide: AND vs OR Conditions

AND (&) OR (|) Both conditions must be true At least one condition must be true Marks ≥ 80 AND City = Pune City = Pune OR Marks ≥ 90 Samadhan Pune and 85 marks Samadhan, Pradnya, Samiksha Each matches at least one condition Combine conditions to control which rows are returned.

Figure 2: AND narrows the result to rows meeting both conditions; OR includes rows meeting either condition.

10. Select Columns After Filtering

You can filter rows and then select only the columns required for a report.

result = df.loc[
    df["Marks"] >= 80,
    ["Name", "Marks"]
]

print(result)

Output

       Name  Marks
0  Samadhan     85
1   Pradnya     92
3      Amit     88

Using loc[] is a convenient way to combine a Boolean row condition with a selection of columns.

11. Common Filtering Mistakes

Mistake 1: Using and instead of &

For element-wise Pandas conditions, use & and | with parentheses around each comparison.

Mistake 2: Forgetting parentheses

# Correct
df[(df["Marks"] >= 80) & (df["Age"] <= 22)]

Mistake 3: Using the wrong comparison operator

> 80 excludes 80, while >= 80 includes it. Choose the operator that matches the requirement.

Mistake 4: Ignoring missing text values

When using str.contains(), consider setting na=False if missing values should not match.

12. Practice Exercises

Exercise 1: Basic Conditions

  1. Find students with marks greater than 80.
  2. Find students who belong to Mumbai.
  3. Find students aged 21 or younger.

Exercise 2: Multiple Conditions

  1. Find Pune students who scored at least 80.
  2. Find students from Pune or Nashik.
  3. Find students whose marks are between 70 and 90.

Exercise 3: String Filtering

  1. Find names containing the letter "a".
  2. Make the search case-insensitive.
  3. Display only Name and Marks for matching rows.

Suggested solution

# Marks greater than 80
print(df[df["Marks"] > 80])

# Students from Mumbai
print(df[df["City"] == "Mumbai"])

# Age 21 or younger
print(df[df["Age"] <= 21])

# Pune students with at least 80 marks
print(df[(df["City"] == "Pune") & (df["Marks"] >= 80)])

# Students from Pune or Nashik
print(df[df["City"].isin(["Pune", "Nashik"])])

# Marks between 70 and 90
print(df[df["Marks"].between(70, 90)])

# Case-insensitive text search
mask = df["Name"].str.contains("a", case=False, na=False)
print(df.loc[mask, ["Name", "Marks"]])

13. Revision Questions

  1. What is Boolean filtering in Pandas?
  2. What does a Boolean mask contain?
  3. What is the difference between & and |?
  4. Why should each condition be enclosed in parentheses?
  5. How do you filter a DataFrame using isin()?
  6. What does between() do?
  7. How does str.contains() help filter text?
  8. How do you select particular columns after filtering rows?

Summary

Filtering allows you to retrieve only the DataFrame rows that meet specific conditions. Comparison operators create Boolean masks, while & and | combine multiple conditions.

Methods such as isin(), between(), and str.contains() make common filtering tasks easier. Use loc[] when you want to combine row filtering with column selection.