Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Filtering Helper Functions in Pandas

Learn how to use Pandas helper methods such as isin(), between(), isna(), notna(), and query() to filter DataFrame rows efficiently and clearly.

1. Introduction to Filtering Helper Functions

Filtering helper functions simplify common DataFrame filtering tasks. Instead of writing several repetitive comparison expressions, you can use built-in Pandas methods designed for specific filtering requirements.

These methods are useful when checking whether values belong to a list, selecting values within a range, locating missing data, or writing readable filtering expressions.

Common Filtering Methods

Method Purpose
isin() Checks whether values belong to a collection.
between() Selects values within a specified range.
isna() Identifies missing values.
notna() Identifies values that are not missing.
query() Filters rows using a string expression.
Original DataFrame isin() Match allowed values between() Match a range isna() / notna() Check missing data Filtered DataFrame Only matching rows remain

Different helper methods address different filtering requirements while returning Boolean masks or filtered rows.

2. Create a Sample DataFrame

The following examples use a student DataFrame containing names, marks, attendance, cities, and optional assignment scores.

import pandas as pd

df = pd.DataFrame({
    "Name": [
        "Samadhan",
        "Pradnya",
        "Samiksha",
        "Rahul",
        "Neha",
        "Amit"
    ],
    "Marks": [85, 68, 92, 76, 55, 70],
    "Attendance": [90, 85, 75, 88, 60, 80],
    "City": [
        "Pune",
        "Mumbai",
        "Pune",
        "Nashik",
        "Mumbai",
        "Pune"
    ],
    "Assignment": [18, None, 20, 15, None, 17]
})

print(df)

Expected Output:

        Name  Marks  Attendance    City  Assignment
0   Samadhan     85          90    Pune        18.0
1    Pradnya     68          85  Mumbai         NaN
2   Samiksha     92          75    Pune        20.0
3      Rahul     76          88  Nashik        15.0
4       Neha     55          60  Mumbai         NaN
5       Amit     70          80    Pune        17.0
Note: Pandas displays missing numeric values as NaN in this example. These missing values will be useful when learning isna() and notna().

3. Using isin() to Match Multiple Values

The isin() method checks whether each value in a Series belongs to a specified collection. It returns a Boolean Series that can be used directly to filter a DataFrame.

Example 1: Filter by City

Select students who live in Pune or Mumbai.

result = df[
    df["City"].isin(["Pune", "Mumbai"])
]

print(result[["Name", "City"]])

Expected Output:

        Name    City
0   Samadhan    Pune
1    Pradnya  Mumbai
2   Samiksha    Pune
4       Neha  Mumbai
5       Amit    Pune

Example 2: Filter by Several Marks Values

result = df[
    df["Marks"].isin([70, 85, 92])
]

print(result[["Name", "Marks"]])

Expected Output:

        Name  Marks
0   Samadhan     85
2   Samiksha     92
5       Amit     70

Example 3: Combine isin() with Another Condition

result = df[
    (df["City"].isin(["Pune", "Mumbai"])) &
    (df["Marks"] >= 70)
]

print(result[["Name", "Marks", "City"]])

Expected Output:

        Name  Marks    City
0   Samadhan     85    Pune
2   Samiksha     92    Pune
5       Amit     70    Pune
Best Practice: Use isin() when checking a column against multiple acceptable values instead of repeatedly writing equality checks.

4. Using between() to Filter a Range

The between() method identifies values within a specified interval. By default, the lower and upper endpoints are included.

Example 1: Marks Between 70 and 90

result = df[
    df["Marks"].between(70, 90)
]

print(result[["Name", "Marks"]])

Expected Output:

        Name  Marks
0   Samadhan     85
3      Rahul     76
5       Amit     70

Example 2: Exclude the Endpoints

Set inclusive="neither" when values equal to either endpoint should not be included.

result = df[
    df["Marks"].between(
        70,
        90,
        inclusive="neither"
    )
]

print(result[["Name", "Marks"]])

Expected Output:

        Name  Marks
0   Samadhan     85
3      Rahul     76

Example 3: Attendance Between 80 and 90

result = df[
    df["Attendance"].between(80, 90)
]

print(result[["Name", "Attendance"]])

Expected Output:

        Name  Attendance
0   Samadhan          90
1    Pradnya          85
3      Rahul          88
5       Amit          80
Important: The inclusive argument supports "both", "neither", "left", and "right". Use the option that matches your boundary requirements.

5. Using isna() to Find Missing Values

Real-world datasets often contain incomplete information. The isna() method identifies missing values and returns True for missing entries and False for non-missing entries.

Example 1: Find Rows with Missing Assignments

result = df[
    df["Assignment"].isna()
]

print(result[["Name", "Assignment"]])

Expected Output:

      Name  Assignment
1  Pradnya         NaN
4     Neha         NaN

Example 2: Find Rows with Any Missing Value

result = df[
    df.isna().any(axis=1)
]

print(result[["Name", "Assignment"]])

Expected Output:

      Name  Assignment
1  Pradnya         NaN
4     Neha         NaN

Here, isna() checks every cell, any(axis=1) identifies rows containing at least one missing value, and the resulting Boolean mask filters the rows.

Example 3: Count Missing Values

print(df["Assignment"].isna().sum())

Expected Output:

2

6. Using notna() to Find Available Values

The notna() method is the opposite of isna(). It identifies values that are not missing.

Example 1: Select Students with Assignment Scores

result = df[
    df["Assignment"].notna()
]

print(result[["Name", "Assignment"]])

Expected Output:

        Name  Assignment
0   Samadhan        18.0
2   Samiksha        20.0
3      Rahul        15.0
5       Amit        17.0

Example 2: Keep Rows with Available Assignments and Marks Above 70

result = df[
    df["Assignment"].notna() &
    (df["Marks"] > 70)
]

print(result[["Name", "Marks", "Assignment"]])

Expected Output:

        Name  Marks  Assignment
0   Samadhan     85        18.0
2   Samiksha     92        20.0
3      Rahul     76        15.0
Remember: Use isna() to find missing entries and notna() to select available entries.

7. Filtering with query()

The query() method filters rows using an expression written as a string. It can make many filtering statements more compact and readable.

Example 1: Filter by Marks

result = df.query("Marks > 80")

print(result[["Name", "Marks"]])

Expected Output:

        Name  Marks
0   Samadhan     85
2   Samiksha     92

Example 2: Combine Multiple Conditions

result = df.query(
    "Marks >= 70 and Attendance >= 80"
)

print(result[["Name", "Marks", "Attendance"]])

Expected Output:

        Name  Marks  Attendance
0   Samadhan     85          90
3      Rahul     76          88
5       Amit     70          80

Example 3: Filter String Values

result = df.query("City == 'Pune'")

print(result[["Name", "City"]])

Expected Output:

        Name  City
0   Samadhan  Pune
2   Samiksha  Pune
5       Amit  Pune

Example 4: Use an External Variable

Prefix an external Python variable with @ inside the query expression.

minimum_marks = 75

result = df.query("Marks >= @minimum_marks")

print(result[["Name", "Marks"]])

Expected Output:

        Name  Marks
0   Samadhan     85
2   Samiksha     92
3      Rahul     76
Security Note: Avoid constructing query expressions directly from untrusted input. Use trusted expressions and validate external values before processing them.

8. Combining Helper Methods

Helper methods become especially useful when combined with Boolean operators. The following example selects students who live in Pune or Mumbai, have marks between 70 and 90, and have a recorded assignment score.

result = df[
    df["City"].isin(["Pune", "Mumbai"]) &
    df["Marks"].between(70, 90) &
    df["Assignment"].notna()
]

print(
    result[
        ["Name", "Marks", "City", "Assignment"]
    ]
)

Expected Output:

        Name  Marks  City  Assignment
0   Samadhan     85  Pune        18.0
5       Amit     70  Pune        17.0

Each method handles one requirement: isin() checks the city, between() checks the marks range, and notna() verifies that the assignment score exists. The AND operator requires every requirement to be satisfied.

9. Filtering Workflow

Identify the Filtering Requirement Choose the Appropriate Helper isin() Match allowed values between() Check a numeric range isna() / notna() Check missing values Apply the Boolean Filter Inspect the resulting rows

Choose a helper method according to the type of condition required, then apply it to the DataFrame.

10. Common Mistakes and Best Practices

Common Mistake Recommended Approach
Repeating multiple equality comparisons for a long list of accepted values. Use isin() to check membership in a list.
Writing separate greater-than and less-than checks for every range. Use between() when checking an interval.
Comparing missing values directly with == NaN. Use isna() or notna().
Combining Pandas Boolean Series with Python's and or or operators. Use & or | with parentheses around comparisons.
Building query strings directly from untrusted input. Validate values and use trusted query expressions.

11. Practical Exercises

Exercise 1: Filter Multiple Cities

Select students who live in Pune or Nashik using isin().

Expected Names: Samadhan, Samiksha, Rahul, and Amit.

Exercise 2: Filter a Marks Range

Select students whose marks are between 60 and 80, including both endpoints.

Expected Names: Pradnya, Rahul, and Amit.

Exercise 3: Find Missing Values

Select rows where the Assignment column contains a missing value.

Expected Names: Pradnya and Neha.

Exercise 4: Find Available Assignment Scores

Select students whose Assignment values are not missing.

Expected Names: Samadhan, Samiksha, Rahul, and Amit.

Exercise 5: Use query()

Select students whose marks are at least 70 and whose attendance is at least 80 using query().

Expected Names: Samadhan, Rahul, and Amit.

12. Quick Revision Questions

  1. What is the purpose of the isin() method?
  2. How does between() handle range endpoints by default?
  3. What is the difference between isna() and notna()?
  4. How can you identify DataFrame rows containing missing values?
  5. What is the purpose of the query() method?
  6. How can an external Python variable be referenced inside query()?
  7. Why should parentheses be used when combining Boolean Series?
  8. How can several helper methods be combined in one filter?

Summary

Pandas filtering helper methods simplify common data-selection tasks. Use isin() to match multiple allowed values, between() to select values within a range, isna() to identify missing values, and notna() to select non-missing values.

The query() method provides an alternative way to express many filtering conditions. Helper methods can also be combined with Boolean operators to build more precise filters.

Selecting the appropriate helper method improves code readability, reduces repetitive comparisons, and makes data analysis easier to maintain.