Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Element-Wise Operations

Learn how Pandas performs arithmetic, comparisons, logical operations and conditional transformations on individual values in Series and DataFrames.

1. What Are Element-Wise Operations?

Element-wise operations perform a calculation or comparison separately on each corresponding value in a Series or DataFrame. Pandas applies these operations across the data without requiring you to write an explicit loop for every element.

For example, adding 10 to a Series adds 10 to each value. Multiplying a DataFrame by 2 doubles each numeric element.

Learning Objectives

  • Perform arithmetic operations on Series and DataFrames.
  • Understand comparison and logical operations.
  • Use Series and DataFrame arithmetic methods.
  • Understand index alignment during calculations.
  • Handle missing values and division by zero.
  • Apply conditional transformations to data.

2. Creating a Series

Begin by importing Pandas and creating a Series of numbers.

import pandas as pd

numbers = pd.Series([10, 20, 30, 40])

print(numbers)

Output

0    10
1    20
2    30
3    40
dtype: int64

3. Arithmetic Operations on a Series

Standard arithmetic operators work element-wise on numeric Series. Each operation returns a new Series.

Operation Operator Example
Addition + numbers + 5
Subtraction - numbers - 5
Multiplication * numbers * 2
Division / numbers / 10
Power ** numbers ** 2
Remainder % numbers % 3
print(numbers + 5)
print(numbers * 2)

Output

# numbers + 5
0    15
1    25
2    35
3    45
dtype: int64

# numbers * 2
0    20
1    40
2    60
3    80
dtype: int64
Tip: These operations return new results. The original Series is unchanged unless you assign the result back to a variable.

4. Operations Between Two Series

When you perform arithmetic between two Series, Pandas aligns their values by index labels rather than simply relying on their physical positions.

first = pd.Series([10, 20, 30])

second = pd.Series([1, 2, 3])

result = first + second

print(result)

Output

0    11
1    22
2    33
dtype: int64

Both Series use the same index labels, so their corresponding values are added together.

Example: Different Index Labels

first = pd.Series([10, 20], index=["a", "b"])

second = pd.Series([1, 2], index=["b", "c"])

print(first + second)

Output

a     NaN
b    21.0
c     NaN
dtype: float64
Important: Index labels "a" and "c" do not have matching values in both Series, so their results are missing. Matching labels such as "b" are aligned and added.

5. Using Arithmetic Methods

Pandas provides methods such as add(), sub(), mul() and div(). These methods can make index alignment and missing-value handling more explicit.

first = pd.Series([10, 20, 30])

second = pd.Series([1, 2, 3])

print(first.add(second))
print(first.sub(second))
print(first.mul(second))
print(first.div(second))

Output

# Addition
0    11
1    22
2    33
dtype: int64

# Subtraction
0     9
1    18
2    27
dtype: int64

# Multiplication
0    10
1    40
2    90
dtype: int64

# Division
0    10.0
1    10.0
2    10.0
dtype: float64

Filling Missing Results

The fill_value parameter can supply a value for missing entries during supported arithmetic operations.

first = pd.Series([10, 20], index=["a", "b"])

second = pd.Series([5, 8], index=["b", "c"])

result = first.add(second, fill_value=0)

print(result)

Output

a    10.0
b    25.0
c     8.0
dtype: float64

6. Comparison Operations

Comparison operators compare each element and return Boolean values: True or False.

marks = pd.Series([35, 55, 70, 90])

print(marks >= 50)

Output

0    False
1     True
2     True
3     True
dtype: bool
Operator Meaning
== Equal to
!= Not equal to
> Greater than
< Less than
>= Greater than or equal to
<= Less than or equal to

7. Filtering Data with Boolean Conditions

Boolean conditions can be used to select only the rows that satisfy a requirement.

marks = pd.Series([35, 55, 70, 90])

passed_marks = marks[marks >= 50]

print(passed_marks)

Output

1    55
2    70
3    90
dtype: int64

The original index labels are preserved when filtering. Use reset_index(drop=True) if you need a fresh sequential index.

8. Logical Operations

Use & for AND, | for OR and ~ for NOT when combining Boolean conditions on Pandas Series.

marks = pd.Series([35, 55, 70, 90])

# Marks from 50 through 80
result = marks[(marks >= 50) & (marks <= 80)]

print(result)

Output

1    55
2    70
dtype: int64
Remember: Put parentheses around each condition. Use & and | instead of Python's and and or when combining Series conditions.

9. Element-Wise Operations on a DataFrame

Arithmetic can be applied to all compatible numeric cells in a DataFrame.

df = pd.DataFrame({
    "Math": [80, 90, 70],
    "Science": [85, 95, 75]
})

result = df + 5

print(result)

Output

   Math  Science
0    85       90
1    95      100
2    75       80

Each numeric value is increased by five. The original DataFrame remains unchanged because the result is stored in a separate variable.

10. Applying Different Values to Columns

A Series can be used to apply a different value to each DataFrame column. Pandas aligns the Series index with the DataFrame column labels.

df = pd.DataFrame({
    "Math": [80, 90],
    "Science": [70, 85]
})

bonus = pd.Series({
    "Math": 5,
    "Science": 10
})

result = df + bonus

print(result)

Output

   Math  Science
0    85       80
1    95       95
Practical use: This pattern can apply department-specific adjustments, column-specific weights or other values associated with column names.

11. Conditional Transformations with where()

The where() method keeps values that satisfy a condition and replaces values that do not satisfy it.

marks = pd.Series([35, 55, 70, 90])

result = marks.where(marks >= 50, other=0)

print(result)

Output

0     0
1    55
2    70
3    90
dtype: int64

Here, values below 50 are replaced with zero, while the remaining values are preserved.

12. Conditional Transformations with mask()

The mask() method works in the opposite direction: it replaces values where the condition is True.

marks = pd.Series([35, 55, 70, 90])

result = marks.mask(marks < 50, other=0)

print(result)

Output

0     0
1    55
2    70
3    90
dtype: int64
Method Behavior
where(condition) Keeps values where the condition is true.
mask(condition) Replaces values where the condition is true.

13. Handling Division by Zero

Division by zero can produce infinite values or missing results depending on the data types and operation. Check denominators before performing calculations.

numerator = pd.Series([10, 20, 30])
denominator = pd.Series([2, 0, 5])

result = numerator.div(
    denominator.replace(0, float("nan"))
)

print(result)

Output

0     5.0
1     NaN
2     6.0
dtype: float64

The zero denominator is replaced with a missing value before division. This avoids interpreting an invalid calculation as an ordinary numeric result.

14. Vectorized Operations vs Loops

Pandas and NumPy support vectorized operations that perform calculations across entire arrays or Series. For simple numeric transformations, these are generally clearer and often faster than explicit Python loops.

Vectorized Approach

numbers = pd.Series([10, 20, 30, 40])

result = numbers * 2

print(result)

Using a Python Loop

numbers = [10, 20, 30, 40]
result = []

for number in numbers:
    result.append(number * 2)

print(result)

Output

[20, 40, 60, 80]
Best practice: Prefer vectorized arithmetic when it expresses the required calculation directly. Use custom functions when the logic cannot be expressed clearly through built-in operations.

15. Common Mistakes and Best Practices

Common Mistakes

  • Assuming two Series are aligned only by position.
  • Ignoring missing values produced by index alignment.
  • Combining Boolean Series with and instead of &.
  • Forgetting parentheses around individual conditions.
  • Dividing by zero without checking denominators.
  • Overwriting the original data unintentionally.

Best Practices

  • Check Series and DataFrame index labels before arithmetic.
  • Use built-in operators and methods for straightforward calculations.
  • Choose where() or mask() for conditional replacement.
  • Handle missing values according to the business requirement.
  • Validate results before saving transformed data.

16. Practice Exercises

Exercise 1: Apply a Discount

Reduce each price by 10% using a vectorized operation.

prices = pd.Series([100, 250, 500, 1000])

# Write your solution below

Exercise 2: Filter Passing Marks

Select all marks greater than or equal to 50.

marks = pd.Series([35, 55, 42, 80, 95])

# Write your solution below

Exercise 3: Combine Conditions

Select marks between 40 and 80, inclusive.

marks = pd.Series([25, 40, 55, 80, 95])

# Write your solution below

Exercise 4: Handle Missing Results

Add two Series with different index labels and inspect the resulting missing values. Then try using add(fill_value=0).

first = pd.Series([10, 20], index=["a", "b"])
second = pd.Series([5, 8], index=["b", "c"])

# Write your solution below

Summary

Element-wise operations allow Pandas to perform arithmetic, comparisons and conditional transformations on Series and DataFrames. Operators such as +, *, >= and methods such as add() make data calculations straightforward.

Pandas aligns Series and DataFrame values by their index labels during operations. Mismatched labels can produce missing values, so check alignment and handle missing data carefully.

Use vectorized operations for simple calculations, Boolean conditions for filtering, and where() or mask() for conditional replacement.