Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Multiple Conditions in Pandas

Learn how to combine multiple filtering conditions using AND, OR, and NOT operators to retrieve specific rows from a Pandas DataFrame.

1. Introduction to Multiple Conditions

In Pandas, multiple conditions allow us to filter DataFrame rows based on two or more requirements. This technique is useful when a single condition is not sufficient to identify the required data.

For example, a student management system may need to find students who scored more than 70 marks and have attendance of at least 80%. Both requirements must be checked together.

Sample Student DataFrame

Name Marks Attendance City
Samadhan 85 90 Pune
Pradnya 68 85 Mumbai
Samiksha 92 75 Pune
Rahul 76 88 Nashik
Neha 55 60 Mumbai
Student DataFrame Condition 1 Marks > 70 Condition 2 Attendance >= 80 Combine Conditions Select Matching Rows

Multiple conditions are combined to determine which DataFrame rows satisfy the filtering requirements.

2. Create a Sample DataFrame

First, import Pandas and create a DataFrame containing student names, marks, attendance percentages, and cities.

import pandas as pd

df = pd.DataFrame({
    "Name": [
        "Samadhan",
        "Pradnya",
        "Samiksha",
        "Rahul",
        "Neha"
    ],
    "Marks": [85, 68, 92, 76, 55],
    "Attendance": [90, 85, 75, 88, 60],
    "City": [
        "Pune",
        "Mumbai",
        "Pune",
        "Nashik",
        "Mumbai"
    ]
})

print(df)

Expected Output:

        Name  Marks  Attendance    City
0   Samadhan     85          90    Pune
1    Pradnya     68          85  Mumbai
2   Samiksha     92          75    Pune
3      Rahul     76          88  Nashik
4       Neha     55          60  Mumbai

3. Using the AND Operator (&)

The AND operator selects rows only when every specified condition evaluates to True. In Pandas, use the ampersand operator & to combine Boolean Series element by element.

Example Requirement

Select students whose marks are greater than 70 and whose attendance is at least 80.

result = df[
    (df["Marks"] > 70) &
    (df["Attendance"] >= 80)
]

print(result)

Expected Output:

        Name  Marks  Attendance    City
0   Samadhan     85          90    Pune
3      Rahul     76          88  Nashik

Explanation

  • Samadhan satisfies both conditions because his marks are 85 and attendance is 90.
  • Rahul satisfies both conditions because his marks are 76 and attendance is 88.
  • Pradnya does not qualify because her marks are not greater than 70.
  • Samiksha does not qualify because her attendance is below 80.
  • Neha does not satisfy either condition.
Key Point: With AND, a row is selected only if all combined conditions are True.

4. Using the OR Operator (|)

The OR operator selects a row when at least one condition is True. If both conditions are True, the row is also included. In Pandas, use the pipe operator |.

Example Requirement

Select students whose marks are greater than 90 or whose attendance is at least 85.

result = df[
    (df["Marks"] > 90) |
    (df["Attendance"] >= 85)
]

print(result)

Expected Output:

        Name  Marks  Attendance    City
0   Samadhan     85          90    Pune
1    Pradnya     68          85  Mumbai
2   Samiksha     92          75    Pune
3      Rahul     76          88  Nashik

Explanation

  • Samadhan qualifies because attendance is 90.
  • Pradnya qualifies because attendance is 85.
  • Samiksha qualifies because marks are 92.
  • Rahul qualifies because attendance is 88.
  • Neha does not qualify because neither condition is True.
Key Point: With OR, at least one condition must be True for a row to be selected.
AND (&) Condition A = True AND Condition B = True Both must be True OR (|) Condition A = True OR Condition B = False At least one must be True

AND requires all conditions to be True, while OR requires at least one condition to be True.

5. Using the NOT Operator (~)

The NOT operator reverses a Boolean condition. In Pandas, the tilde operator ~ inverts each Boolean value in a Boolean Series.

Example Requirement

Select all students whose city is not Pune.

result = df[
    ~(df["City"] == "Pune")
]

print(result[["Name", "City"]])

Expected Output:

      Name    City
1  Pradnya  Mumbai
3    Rahul  Nashik
4     Neha  Mumbai

Explanation

The condition df["City"] == "Pune" identifies rows where the city is Pune. Applying ~ reverses the Boolean mask, so rows where the city is not Pune are selected.

Important: Use ~ to invert a Pandas Boolean Series. Python's not operator cannot directly evaluate an entire Series as a single Boolean value.

6. Combining Three or More Conditions

Pandas allows you to combine three or more conditions in a single filter expression. Connect each condition with the appropriate Boolean operator.

Example Requirement

Select students whose marks are greater than 70, attendance is at least 80, and city is Pune.

result = df[
    (df["Marks"] > 70) &
    (df["Attendance"] >= 80) &
    (df["City"] == "Pune")
]

print(result)

Expected Output:

        Name  Marks  Attendance  City
0   Samadhan     85          90  Pune

Samadhan is the only student who satisfies all three conditions. Although Samiksha lives in Pune and has marks above 70, her attendance is below 80.

7. Combining AND and OR in One Expression

Some filtering requirements involve both AND and OR operators. Use parentheses to group conditions and make the intended logic explicit.

Example Requirement

Select students who either have marks greater than 90 and attendance of at least 80, or live in Pune.

The logic is: (Marks > 90 AND Attendance >= 80) OR City = Pune

result = df[
    (
        (df["Marks"] > 90) &
        (df["Attendance"] >= 80)
    ) |
    (df["City"] == "Pune")
]

print(result[["Name", "Marks", "Attendance", "City"]])

Expected Output:

        Name  Marks  Attendance  City
0   Samadhan     85          90  Pune
2   Samiksha     92          75  Pune

How the Expression Works

  1. The first group checks whether marks are greater than 90 and attendance is at least 80.
  2. The second group checks whether the city is Pune.
  3. The OR operator combines the results of these two groups.
  4. A row is selected if either group evaluates to True.
Best Practice: Put parentheses around every comparison before combining conditions with & or |. This avoids operator-precedence errors and makes the expression easier to read.

8. Filtering with isin()

The isin() method checks whether each value in a Series belongs to a specified collection of values. It is particularly useful when a column can contain several acceptable values.

Example Requirement

Select students who live in either Pune or Nashik.

result = df[
    df["City"].isin(["Pune", "Nashik"])
]

print(result[["Name", "City"]])

Expected Output:

        Name    City
0   Samadhan    Pune
2   Samiksha    Pune
3      Rahul  Nashik

Combining isin() with Another Condition

You can combine isin() with other conditions to narrow down the results further.

result = df[
    (df["City"].isin(["Pune", "Mumbai"])) &
    (df["Marks"] > 70)
]

print(result[["Name", "Marks", "City"]])

Expected Output:

        Name  Marks  City
0   Samadhan     85  Pune
2   Samiksha     92  Pune

9. Filtering Numeric Ranges with between()

The between() method checks whether values fall within a specified range. By default, both endpoints are included.

result = df[
    df["Marks"].between(70, 90)
]

print(result[["Name", "Marks"]])

Expected Output:

        Name  Marks
0   Samadhan     85
3      Rahul     76

This example selects marks from 70 through 90, including both endpoints. In the sample data, Samadhan and Rahul fall within that range.

Remember: Use between() for a range-based condition and isin() when checking membership in a collection of specific values.

10. Using query() for Multiple Conditions

Pandas provides the query() method as another way to filter DataFrame rows. It can make some compound conditions more readable.

result = df.query(
    "Marks > 70 and Attendance >= 80"
)

print(result)

Expected Output:

        Name  Marks  Attendance    City
0   Samadhan     85          90    Pune
3      Rahul     76          88  Nashik

The result matches the earlier AND example. Inside a query() expression, the words and and or can be used to express logical conditions.

Note: Use query() with trusted expressions. Avoid building query strings directly from untrusted user input.

11. Common Mistakes and Best Practices

Incorrect Approach Correct Approach Explanation
df["Marks"] > 70 and df["Attendance"] >= 80 (df["Marks"] > 70) & (df["Attendance"] >= 80) Use element-wise operators to combine Pandas Series.
df["Marks"] > 70 & df["Attendance"] >= 80 (df["Marks"] > 70) & (df["Attendance"] >= 80) Parentheses ensure each comparison is evaluated as a separate condition.
df["City"] == "Pune" or df["City"] == "Nashik" df["City"].isin(["Pune", "Nashik"]) Use isin() to check multiple allowed values.
df[not (df["City"] == "Pune")] df[~(df["City"] == "Pune")] Use the tilde operator to invert a Boolean Series.
Writing a long filter without grouping Group comparisons with parentheses Explicit grouping improves readability and prevents operator-precedence errors.

12. Practical Exercises

Exercise 1: AND Filtering

Select students whose marks are at least 80 and whose attendance is at least 85.

Expected Result: Samadhan.

Exercise 2: OR Filtering

Select students whose marks are greater than 90 or whose city is Mumbai.

Expected Names: Pradnya, Samiksha, and Neha.

Exercise 3: NOT Filtering

Select all students who do not live in Pune.

Expected Names: Pradnya, Rahul, and Neha.

Exercise 4: Combined Conditions

Select students from Pune or Mumbai whose marks are greater than 70.

Expected Names: Samadhan and Samiksha.

Exercise 5: Range Filtering

Select students whose marks are between 60 and 90, including both endpoints.

Expected Names: Samadhan, Pradnya, and Rahul.

13. Quick Revision Questions

  1. Which operator combines Boolean Series using AND?
  2. Which operator combines Boolean Series using OR?
  3. What does the tilde operator do to a Boolean mask?
  4. Why should parentheses be used around individual comparisons?
  5. When is the isin() method useful?
  6. What is the purpose of the between() method?
  7. How can the query() method filter multiple conditions?
  8. What is the difference between AND and OR filtering?

Summary

Multiple-condition filtering allows you to select DataFrame rows using several requirements at the same time. Use & for AND, | for OR, and ~ for NOT when working with Pandas Boolean Series.

Use parentheses around comparisons to create clear and correct filter expressions. The isin() method checks whether values belong to a specified collection, while between() checks whether numeric values fall within a range. The query() method offers an alternative syntax for many filtering operations.

These techniques are useful for filtering student records, analyzing sales, selecting customer segments, and extracting relevant information from large datasets.