Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Apply and Mapping Overview

Learn how to transform Pandas data using functions, apply operations, mapping techniques, and element-wise calculations.

1. Introduction to Apply and Mapping

Pandas provides several ways to apply custom logic to Series and DataFrames. These techniques help transform values, create calculated columns, and perform operations that are not directly available through built-in methods.

Learning Objectives

  • Understand custom functions in Pandas.
  • Use the Series map() method.
  • Apply functions to Series and DataFrame objects.
  • Understand row-wise and column-wise operations.
  • Learn when to use vectorized operations instead.
  • Recognize common performance and data-handling mistakes.
Input Data Function Output Data 10, 20, 30 Transform values 20, 40, 60 Apply custom logic to transform or analyze data

A function receives data, performs a calculation, and produces transformed or analyzed results.

2. Creating Sample Data

Start with a Series of product prices. This dataset will be used to demonstrate different function application techniques.

import pandas as pd

prices = pd.Series(
    [100, 200, 300, 400, 500],
    name="Price"
)

print(prices)

Output

0    100
1    200
2    300
3    400
4    500
Name: Price, dtype: int64

3. Defining a Custom Function

A custom function contains reusable logic. You can define a function using Python's def keyword and then use it with Pandas data.

def add_tax(price):
    return price * 1.10

print(add_tax(100))

Output

110.00000000000001

The function adds ten percent to a price. Python may display a small floating-point precision difference. For formatted output, use round().

print(round(add_tax(100), 2))

Output

110.0

4. Using map() with a Custom Function

The Series map() method applies a function to each value in a Series. It returns a new Series containing the transformed values.

def add_tax(price):
    return price * 1.10

tax_included_prices = prices.map(add_tax)

print(tax_included_prices.round(2))

Output

0    110.0
1    220.0
2    330.0
3    440.0
4    550.0
Name: Price, dtype: float64
Remember: Series map() is useful when each individual value needs to be transformed or replaced.

5. Using map() with a Dictionary

A dictionary can map existing values to new values. This is useful for replacing category codes with descriptive labels.

grades = pd.Series(["A", "B", "C", "A", "B"])

grade_labels = {
    "A": "Excellent",
    "B": "Good",
    "C": "Average"
}

descriptions = grades.map(grade_labels)

print(descriptions)

Output

0    Excellent
1         Good
2      Average
3    Excellent
4         Good
dtype: object

If a value is not present in the dictionary, the mapped result is normally missing. Check your mapping keys when working with real datasets.

6. Using map() with a Lambda Function

A lambda function is a small anonymous function. It is convenient when the transformation is short and does not need a separate function definition.

prices_doubled = prices.map(lambda value: value * 2)

print(prices_doubled)

Output

0     200
1     400
2     600
3     800
4    1000
Name: Price, dtype: int64

7. Using apply() with a Series

The Series apply() method applies a function to each value. It is another option for applying custom logic to Series elements.

def classify_price(price):
    if price >= 400:
        return "High"
    elif price >= 200:
        return "Medium"
    else:
        return "Low"

price_categories = prices.apply(classify_price)

print(price_categories)

Output

0       Low
1    Medium
2    Medium
3      High
4      High
Name: Price, dtype: object

In this example, each price is classified according to a conditional rule.

8. Applying a Function to a DataFrame

DataFrame apply() can apply a function to each column or row. By default, axis=0 applies the function to each column.

df = pd.DataFrame({
    "Math": [80, 90, 75],
    "Science": [85, 95, 70],
    "English": [78, 88, 82]
})

print(df)

Output

   Math  Science  English
0    80       85       78
1    90       95       88
2    75       70       82

Calculate the total of each column:

column_totals = df.apply(sum, axis=0)

print(column_totals)

Output

Math       245
Science    250
English    248
dtype: int64

9. Row-Wise Operations with apply()

Set axis=1 to apply a function to each row. This is useful when a calculation depends on multiple columns in the same record.

df["Total"] = df.apply(
    lambda row: row["Math"]
                + row["Science"]
                + row["English"],
    axis=1
)

print(df)

Output

   Math  Science  English  Total
0    80       85       78    243
1    90       95       88    273
2    75       70       82    227
Tip: When a calculation can be expressed using ordinary arithmetic between columns, direct vectorized operations are often simpler and faster than row-wise apply().

10. Understanding map(), apply(), and Vectorization

Technique Typical Use Example
Series.map() Transform individual values or map labels. prices.map(add_tax)
Series.apply() Apply custom logic to Series elements. prices.apply(classify_price)
DataFrame.apply() Apply a function along rows or columns. df.apply(sum, axis=0)
Vectorized operations Perform calculations directly on Series or columns. prices * 1.10

11. Prefer Vectorized Operations When Possible

Many calculations can be performed directly on an entire Series without writing a custom function. These vectorized expressions are usually easier to read and often more efficient.

vectorized_prices = prices * 1.10

print(vectorized_prices.round(2))

Output

0    110.0
1    220.0
2    330.0
3    440.0
4    550.0
Name: Price, dtype: float64

This produces the same result as applying the tax function to every price individually.

12. Handling Missing Values

Custom functions and mapping operations may encounter missing values. Decide how missing data should be handled before performing a transformation.

values = pd.Series([10, 20, None, 40])

result = values.map(
    lambda value: value * 2
    if pd.notna(value)
    else value
)

print(result)

Output

0    20.0
1    40.0
2     NaN
3    80.0
dtype: float64
Important: A custom function may fail if it performs arithmetic or string operations on missing values without handling them appropriately.

13. Common Mistakes and Best Practices

Common Mistakes

  • Using a dictionary that does not contain all expected keys.
  • Forgetting that row-wise DataFrame operations use axis=1.
  • Ignoring missing values inside custom functions.
  • Using apply() for simple arithmetic unnecessarily.
  • Writing complicated lambda expressions that are difficult to maintain.

Best Practices

  • Use named functions when logic is complex or reused.
  • Use map() for Series value mapping.
  • Use DataFrame apply() when row or column logic is required.
  • Prefer vectorized calculations when available.
  • Test custom functions with missing and unexpected values.

14. Practice Exercises

Exercise 1: Apply a Discount

Create a function that reduces each price by 15 percent. Apply it to a Series of prices.

prices = pd.Series([100, 250, 400, 600])

# Write your solution below

Exercise 2: Map Department Codes

Map department codes such as IT, HR, and FIN to their descriptive department names.

departments = pd.Series(["IT", "HR", "FIN", "IT"])

# Write your solution below

Exercise 3: Calculate Row Totals

Use a DataFrame function to calculate the total marks for each student.

marks = pd.DataFrame({
    "Math": [80, 90, 75],
    "Science": [85, 95, 70]
})

# Write your solution below

Summary

Pandas supports custom data transformations through Series.map(), Series.apply(), and DataFrame.apply(). These methods help apply reusable logic to values, rows, and columns.

Use map() to transform Series values or map categories, and use apply() when custom logic must operate on Series elements or DataFrame rows and columns.

Prefer vectorized operations for straightforward arithmetic because they are generally clearer and often more efficient. Handle missing values explicitly and choose the method that matches the structure of the calculation.