Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Apply and Lambda Functions

Learn how to define reusable functions, use lambda expressions, and apply custom calculations to Pandas Series and DataFrames.

1. Introduction

Functions allow you to reuse calculations and business rules. Pandas provides the apply() method to apply custom functions to Series values or along DataFrame rows and columns. Lambda expressions provide a concise way to define small functions.

Learning Objectives

  • Define and call custom Python functions.
  • Understand lambda expressions.
  • Use Series.apply().
  • Use DataFrame.apply() with different axes.
  • Create calculated columns using custom logic.
  • Understand when vectorized operations are preferable.
Input Function Result 10, 20, 30 x * 2 20, 40, 60 Reusable logic transforms or analyzes data

A custom function transforms input values into calculated results.

2. Defining a Regular Function

A regular function is defined using the def keyword. It can contain multiple statements and can be reused throughout a program.

def calculate_square(number):
    return number * number

result = calculate_square(5)

print(result)

Output

25

3. Understanding Lambda Functions

A lambda function is a small anonymous function written using the lambda keyword. It contains an expression whose result is returned automatically.

square = lambda number: number * number

print(square(6))

Output

36

The equivalent regular function is:

def square(number):
    return number * number

print(square(6))

Output

36
Remember: Lambda functions are best suited to short expressions. Use a regular function when the logic requires multiple statements or needs a descriptive reusable name.

4. Creating a Pandas Series

The following examples use a Series containing student marks.

import pandas as pd

marks = pd.Series(
    [45, 60, 72, 88, 95],
    name="Marks"
)

print(marks)

Output

0    45
1    60
2    72
3    88
4    95
Name: Marks, dtype: int64

5. Using Series.apply() with a Regular Function

The Series apply() method applies a function to individual values in the Series.

def add_bonus(mark):
    return mark + 5

updated_marks = marks.apply(add_bonus)

print(updated_marks)

Output

0     50
1     65
2     77
3     93
4    100
Name: Marks, dtype: int64

6. Using Series.apply() with Lambda

Lambda expressions can be passed directly to apply() when the transformation is simple.

doubled_marks = marks.apply(
    lambda mark: mark * 2
)

print(doubled_marks)

Output

0     90
1    120
2    144
3    176
4    190
Name: Marks, dtype: int64

7. Applying Conditional Logic

A lambda expression can include a conditional expression to classify values according to a rule.

results = marks.apply(
    lambda mark: "Pass" if mark >= 50 else "Fail"
)

print(results)

Output

0    Fail
1    Pass
2    Pass
3    Pass
4    Pass
Name: Marks, dtype: object

Each mark is checked against the threshold of 50. Values below 50 are classified as Fail, and the remaining values are classified as Pass.

8. Creating a DataFrame

DataFrame functions can operate across columns or across rows. Consider a dataset containing marks in three subjects.

df = pd.DataFrame({
    "Math": [80, 90, 70],
    "Science": [85, 95, 75],
    "English": [78, 88, 82]
})

print(df)

Output

   Math  Science  English
0    80       85       78
1    90       95       88
2    70       75       82

9. DataFrame.apply() with axis=0

The default axis=0 applies the function to each column. For example, calculate the average mark in each subject.

subject_averages = df.apply(
    lambda column: column.mean(),
    axis=0
)

print(subject_averages)

Output

Math       80.000000
Science    85.000000
English    82.666667
dtype: float64

10. DataFrame.apply() with axis=1

Setting axis=1 applies the function to each row. This is useful for calculations that combine multiple columns in the same record.

df["Total"] = df.apply(
    lambda row: row["Math"]
                + row["Science"]
                + row["English"],
    axis=1
)

print(df)

Output

   Math  Science  English  Total
0    80       85       78    243
1    90       95       88    273
2    70       75       82    227

11. Calculating a Grade from Total Marks

A custom function can classify each total using multiple conditions.

def calculate_grade(total):
    if total >= 270:
        return "A"
    elif total >= 240:
        return "B"
    else:
        return "C"

df["Grade"] = df["Total"].apply(calculate_grade)

print(df[["Total", "Grade"]])

Output

   Total Grade
0    243     B
1    273     A
2    227     C

12. Comparing apply() and Vectorized Operations

Simple calculations can often be performed directly on an entire Series or DataFrame column. These vectorized expressions are generally easier to read and often faster than row-wise functions.

# Using apply()
marks_with_bonus = marks.apply(
    lambda mark: mark + 5
)

# Using vectorized arithmetic
marks_vectorized = marks + 5

print(marks_with_bonus.equals(marks_vectorized))

Output

True
Best practice: Prefer vectorized arithmetic for simple mathematical transformations. Use custom functions when the operation requires logic that is not conveniently expressed through built-in Pandas operations.

13. Handling Missing Values

Custom functions should account for missing values when the underlying operation cannot handle them safely.

values = pd.Series([10, 20, None, 40])

def double_if_present(value):
    if pd.isna(value):
        return value
    return value * 2

result = values.apply(double_if_present)

print(result)

Output

0    20.0
1    40.0
2     NaN
3    80.0
dtype: float64

14. Common Mistakes and Best Practices

Common Mistakes

  • Writing long and difficult-to-read lambda expressions.
  • Using the wrong axis for DataFrame operations.
  • Ignoring missing values in custom functions.
  • Using row-wise apply for simple arithmetic.
  • Forgetting that the function's return value becomes the result.

Best Practices

  • Use descriptive names for reusable functions.
  • Use lambda expressions for short transformations.
  • Choose axis=0 for columns and axis=1 for rows.
  • Prefer vectorized calculations where appropriate.
  • Test functions against missing and unexpected values.

15. Practice Exercises

Exercise 1: Apply a Discount

Use a lambda expression to reduce each price by 10 percent.

prices = pd.Series([100, 200, 350, 500])

# Write your solution below

Exercise 2: Classify Marks

Use a function to classify marks of 50 or above as Pass and the remaining marks as Fail.

marks = pd.Series([35, 55, 72, 40, 90])

# Write your solution below

Exercise 3: Calculate Row Totals

Use DataFrame apply() with axis=1 to calculate each student's total marks.

students = pd.DataFrame({
    "Math": [80, 70, 90],
    "Science": [85, 75, 95]
})

# Write your solution below

Summary

Python functions and lambda expressions allow reusable logic to be applied to Pandas data. Series apply() operates on Series values, while DataFrame apply() can process rows or columns.

Use axis=0 for column-wise operations and axis=1 for row-wise operations. Lambda functions are convenient for short expressions, while regular functions are often clearer for more complex logic.

Prefer vectorized calculations for straightforward arithmetic, handle missing values carefully, and select the approach that best matches the task.