Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Boolean Filtering in Pandas

Learn how Boolean conditions produce True and False values, how Boolean masks select DataFrame rows, and how to combine logical conditions for precise data analysis.

1. What Is Boolean Filtering?

Boolean filtering is a technique that selects rows using conditions that evaluate to True or False. Pandas keeps the rows whose corresponding condition is true.

For example, if Samadhan wants to find students with marks of at least 80, Pandas can check each student's Marks value and return only the matching rows.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Amit", "Neha"],
    "Age": [21, 22, 20, 23, 21],
    "City": ["Pune", "Mumbai", "Pune", "Nashik", "Mumbai"],
    "Marks": [85, 92, 78, 88, 67]
})

print(df)

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
1   Pradnya   22  Mumbai     92
2  Samiksha   20    Pune     78
3      Amit   23  Nashik     88
4      Neha   21  Mumbai     67

2. Create a Boolean Mask

A Boolean mask is a Series of Boolean values generated by applying a condition to a column.

mask = df["Marks"] >= 80

print(mask)

Output

0     True
1     True
2    False
3     True
4    False
Name: Marks, dtype: bool

The mask is aligned with the DataFrame's row index. Each Boolean value indicates whether that row satisfies the condition.

Remember: Creating a mask only evaluates the condition. To retrieve the matching records, use the mask inside the DataFrame selection.
result = df[mask]

print(result)

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
1   Pradnya   22  Mumbai     92
3      Amit   23  Nashik     88

3. Boolean Operators

Comparison operators are used to build Boolean conditions.

Operator Meaning Example
== Equal to df["City"] == "Pune"
!= Not equal to df["City"] != "Pune"
> Greater than df["Marks"] > 80
< Less than df["Age"] < 22
>= Greater than or equal to df["Marks"] >= 80
<= Less than or equal to df["Age"] <= 21

Example: Equal to

print(df["City"] == "Pune")

Output

0     True
1    False
2     True
3    False
4    False
Name: City, dtype: bool

Example: Greater than

print(df["Marks"] > 85)

Output

0    False
1     True
2    False
3     True
4    False
Name: Marks, dtype: bool

4. Visual Guide: Boolean Mask to Filtered Rows

Condition: Marks >= 80 Boolean Mask Samadhan — True Pradnya — True Samiksha — False Amit — True Neha — False Selected Rows Samadhan — 85 Pradnya — 92 Amit — 88 True rows are selected; False rows are excluded.

Figure 1: Boolean indexing uses a mask to select matching rows.

5. Use the NOT Operator (~)

The tilde operator ~ performs element-wise logical negation on a Boolean mask. It changes True to False and False to True.

mask = df["City"] == "Pune"

print(~mask)

Output

0    False
1     True
2    False
3     True
4     True
Name: City, dtype: bool

Filter students who are not from Pune

not_pune = df[~(df["City"] == "Pune")]

print(not_pune)

Output

     Name  Age    City  Marks
1  Pradnya   22  Mumbai     92
3     Amit   23  Nashik     88
4     Neha   21  Mumbai     67
Tip: Negating an entire Boolean condition can be easier to read when the mask is stored in a variable.

6. Combine Boolean Conditions with AND (&)

The & operator keeps rows where both conditions are true.

mask = (
    (df["Marks"] >= 80) &
    (df["Age"] <= 22)
)

print(df[mask])

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
1   Pradnya   22  Mumbai     92

Both students meet the marks requirement and the age requirement.

7. Combine Boolean Conditions with OR (|)

The | operator keeps rows where at least one condition is true.

mask = (
    (df["City"] == "Pune") |
    (df["Marks"] >= 90)
)

print(df[mask])

Output

       Name  Age    City  Marks
0  Samadhan   21    Pune     85
1   Pradnya   22  Mumbai     92
2  Samiksha   20    Pune     78

Samadhan and Samiksha match the city condition. Pradnya matches the marks condition.

8. Check Whether Any or All Values Match

Boolean masks can be summarized with any() and all().

any(): At least one value is True

mask = df["Marks"] >= 90

print(mask.any())

Output

True

all(): Every value is True

mask = df["Marks"] >= 60

print(mask.all())

Output

True

Use any() to check whether at least one record matches and all() to check whether every record matches.

9. Filter and Select Specific Columns

Boolean filtering can be combined with loc[] to return only the columns needed in a report.

result = df.loc[
    df["Marks"] >= 80,
    ["Name", "Marks"]
]

print(result)

Output

       Name  Marks
0  Samadhan     85
1   Pradnya     92
3      Amit     88

This avoids displaying unnecessary columns while keeping the original row labels.

10. Visual Guide: AND, OR, and NOT

AND (&) OR (|) NOT (~) Both conditions true Either condition true Reverse Boolean values Marks ≥ 80 AND Age ≤ 22 City = Pune OR Marks ≥ 90 Original: True Negated: False Narrower results Must satisfy both Broader results Either can match Inverted mask True becomes False Use parentheses around individual comparisons.

Figure 2: AND combines requirements, OR allows alternatives, and NOT reverses a Boolean mask.

11. Common Mistakes and Best Practices

Mistake 1: Using Python's and/or

For element-wise Pandas conditions, use & and | rather than and and or.

Mistake 2: Forgetting parentheses

# Correct
mask = (df["Marks"] >= 80) & (df["Age"] <= 22)

Mistake 3: Misunderstanding the mask

A mask contains Boolean values aligned to the DataFrame index. It is not the filtered DataFrame itself.

Mistake 4: Ignoring missing values

Missing values can affect comparisons and text operations. Decide explicitly how missing values should be handled.

Best practice: Store complex conditions in clearly named variables such as high_marks or pune_students. This makes filtering code easier to read and debug.

12. Practice Exercises

Exercise 1: Boolean Masks

  1. Create a mask for marks greater than or equal to 80.
  2. Print the Boolean mask.
  3. Use the mask to display matching students.

Exercise 2: Logical Operators

  1. Find students aged 21 or younger who scored at least 80.
  2. Find students from Pune or Mumbai.
  3. Find students who are not from Pune.

Exercise 3: Mask Summary

  1. Check whether any student scored at least 90.
  2. Check whether all students scored at least 60.
  3. Display only the Name and Marks columns for scores of 80 or more.

Suggested solution

# Boolean mask
mask = df["Marks"] >= 80
print(mask)
print(df[mask])

# AND condition
print(df[(df["Age"] <= 21) & (df["Marks"] >= 80)])

# OR condition
print(df[df["City"].isin(["Pune", "Mumbai"])])

# NOT condition
print(df[~(df["City"] == "Pune")])

# Mask summaries
print((df["Marks"] >= 90).any())
print((df["Marks"] >= 60).all())

# Filter rows and select columns
print(df.loc[df["Marks"] >= 80, ["Name", "Marks"]])

13. Revision Questions

  1. What is Boolean filtering?
  2. What is a Boolean mask?
  3. How do comparison operators generate Boolean values?
  4. What is the purpose of the tilde operator ~?
  5. What is the difference between & and |?
  6. Why are parentheses important in multiple conditions?
  7. What do any() and all() return?
  8. How can you filter rows and select specific columns together?

Summary

Boolean filtering uses conditions that evaluate to True or False. A Boolean mask can be applied to a DataFrame to keep only rows that meet the required conditions.

Use & for AND, | for OR, and ~ for NOT. Methods such as any() and all() help summarize Boolean results. Combine masks with loc[] when you need to select specific columns as well as rows.