Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

GroupBy Transformations in Pandas

Learn how to calculate group-level values and return them to every original row using Pandas transform().

Learning objectives: Understand transform(), compare it with aggregate() and filter(), calculate group averages and totals, create performance indicators, and normalize values within groups.

1. What Is a GroupBy Transformation?

A GroupBy transformation performs a calculation separately for each group and returns a result aligned with the original rows. Unlike aggregation, which usually produces one result per group, a transformation typically preserves the original number of rows.

Real-world example

A school wants to compare each student's score with the average score of their class. The class average is calculated once for each class, then repeated for every student belonging to that class.

For example, if Class A has an average score of 85, all students in Class A receive 85 in the new class-average column.

How transform() Works Original Data Samadhan — A — 80 Pradnya — A — 90 Samiksha — A — 85 Aarav — B — 60 Diya — B — 70 groupby() Group A average = 85 Group B average = 65 transform("mean") Transformed Data Samadhan — 80 — 85 Pradnya — 90 — 85 Samiksha — 85 — 85 Aarav — 60 — 65 Diya — 70 — 65 Each original row receives its group's calculated value

A group-level result is broadcast back to the rows belonging to that group.

2. Create the Sample DataFrame

We will use student records containing class, score, and attendance values.

import pandas as pd

df = pd.DataFrame({
    "Student": [
        "Samadhan", "Pradnya", "Samiksha",
        "Aarav", "Diya", "Rohan",
        "Meera", "Kabir", "Anaya"
    ],
    "Class": ["A", "A", "A", "B", "B", "B", "C", "C", "C"],
    "Score": [80, 90, 85, 60, 70, 65, 75, 80, 85],
    "Attendance": [95, 98, 92, 70, 80, 75, 88, 90, 94]
})

print(df)

Expected output

    Student Class  Score  Attendance
0  Samadhan     A     80          95
1    Pradnya     A     90          98
2   Samiksha     A     85          92
3      Aarav     B     60          70
4       Diya     B     70          80
5      Rohan     B     65          75
6      Meera     C     75          88
7      Kabir     C     80          90
8      Anaya     C     85          94

3. Understand transform() Syntax

The general syntax is:

df.groupby("group_column")["value_column"].transform("function")

Each part has a specific purpose:

  • df: the original DataFrame.
  • groupby("Class"): separates rows into groups based on class.
  • ["Score"]: selects the numeric column to transform.
  • transform("mean"): calculates each group's mean and returns an aligned result for the original rows.
df["Class_Average"] = df.groupby("Class")["Score"].transform("mean")

print(df[["Student", "Class", "Score", "Class_Average"]])

Expected output

    Student Class  Score  Class_Average
0  Samadhan     A     80           85.0
1    Pradnya     A     90           85.0
2   Samiksha     A     85           85.0
3      Aarav     B     60           65.0
4       Diya     B     70           65.0
5      Rohan     B     65           65.0
6      Meera     C     75           80.0
7      Kabir     C     80           80.0
8      Anaya     C     85           80.0
Important: The transformed result has nine values because the original DataFrame has nine rows. Each value corresponds to its original row.

4. transform() vs agg() vs filter()

These methods solve different problems. Understanding the difference helps you choose the right GroupBy operation.

Method Purpose Typical output
agg() Summarize each group. Usually one row per group.
transform() Calculate group-based values aligned with original rows. Usually the same number of rows as the input.
filter() Keep or remove complete groups based on a condition. Only rows belonging to retained groups.

Example A: Aggregation

print(df.groupby("Class")["Score"].agg("mean"))

Expected output

Class
A    85.0
B    65.0
C    80.0
Name: Score, dtype: float64

Example B: Transformation

print(df.groupby("Class")["Score"].transform("mean"))

Expected output

0    85.0
1    85.0
2    85.0
3    65.0
4    65.0
5    65.0
6    80.0
7    80.0
8    80.0
Name: Score, dtype: float64

Example C: Group filtering

result = df.groupby("Class").filter(
    lambda group: group["Score"].mean() >= 75
)

print(result[["Student", "Class", "Score"]])

Expected output

    Student Class  Score
0  Samadhan     A     80
1    Pradnya     A     90
2   Samiksha     A     85
6      Meera     C     75
7      Kabir     C     80
8      Anaya     C     85
agg("mean") Class A: 85 Class B: 65 Class C: 80 3 group results transform("mean") 85, 85, 85 65, 65, 65 80, 80, 80 9 aligned results filter() Class A retained Class B removed Class C retained 6 rows retained

Aggregation summarizes, transformation maps group calculations back to rows, and filtering selects qualifying groups.

5. Calculate Group Totals with transform()

You can use transform("sum") to calculate each group's total and repeat that total on every row in the group.

df["Class_Total"] = df.groupby("Class")["Score"].transform("sum")

print(df[["Student", "Class", "Score", "Class_Total"]])

Expected output

    Student Class  Score  Class_Total
0  Samadhan     A     80          255
1    Pradnya     A     90          255
2   Samiksha     A     85          255
3      Aarav     B     60          195
4       Diya     B     70          195
5      Rohan     B     65          195
6      Meera     C     75          240
7      Kabir     C     80          240
8      Anaya     C     85          240

Class A has a total score of 255. Therefore, 255 appears on all three rows belonging to Class A.

6. Calculate Group Counts

Use transform("count") to count non-missing values in a selected column for each group. Use transform("size") to count rows in each group, including rows where selected values are missing.

Example A: Count scores in each class

df["Score_Count"] = df.groupby("Class")["Score"].transform("count")

print(df[["Student", "Class", "Score_Count"]])

Expected output

    Student Class  Score_Count
0  Samadhan     A            3
1    Pradnya     A            3
2   Samiksha     A            3
3      Aarav     B            3
4       Diya     B            3
5      Rohan     B            3
6      Meera     C            3
7      Kabir     C            3
8      Anaya     C            3

Example B: Count all rows in each class

df["Class_Size"] = df.groupby("Class")["Score"].transform("size")

In this dataset, every score is present, so count and size both produce 3 for every student. Their results can differ when values are missing.

7. Compare Each Student with the Class Average

A useful application of transformation is to calculate how far each value is from its group average.

df["Class_Average"] = df.groupby("Class")["Score"].transform("mean")

df["Difference_From_Average"] = (
    df["Score"] - df["Class_Average"]
)

print(df[["Student", "Class", "Score", "Class_Average",
          "Difference_From_Average"]])

Expected output

    Student Class  Score  Class_Average  Difference_From_Average
0  Samadhan     A     80           85.0                     -5.0
1    Pradnya     A     90           85.0                      5.0
2   Samiksha     A     85           85.0                      0.0
3      Aarav     B     60           65.0                     -5.0
4       Diya     B     70           65.0                      5.0
5      Rohan     B     65           65.0                      0.0
6      Meera     C     75           80.0                     -5.0
7      Kabir     C     80           80.0                      0.0
8      Anaya     C     85           80.0                      5.0

Positive differences indicate scores above the class average. Negative differences indicate scores below the class average. A difference of zero means the score equals the class average.

8. Create a Group-Based Performance Label

You can use a transformed value inside a condition to classify each row relative to its group.

df["Class_Average"] = df.groupby("Class")["Score"].transform("mean")

df["Performance"] = "At or Below Average"

df.loc[
    df["Score"] > df["Class_Average"],
    "Performance"
] = "Above Average"

print(df[["Student", "Class", "Score", "Class_Average", "Performance"]])

Expected output

    Student Class  Score  Class_Average           Performance
0  Samadhan     A     80           85.0  At or Below Average
1    Pradnya     A     90           85.0        Above Average
2   Samiksha     A     85           85.0  At or Below Average
3      Aarav     B     60           65.0  At or Below Average
4       Diya     B     70           65.0        Above Average
5      Rohan     B     65           65.0  At or Below Average
6      Meera     C     75           80.0  At or Below Average
7      Kabir     C     80           80.0  At or Below Average
8      Anaya     C     85           80.0        Above Average
Tip: The label "At or Below Average" includes scores exactly equal to the group average. Use >= instead of > if those students should be classified as "Above or Equal to Average."

9. Normalize Values Within Each Group

Normalization can express a student's score relative to the average and standard deviation of their class. This helps compare a student's position within their own group.

The following example uses the population standard deviation, calculated with ddof=0.

df["Class_Mean"] = df.groupby("Class")["Score"].transform("mean")

df["Class_Std"] = df.groupby("Class")["Score"].transform(
    lambda values: values.std(ddof=0)
)

df["Class_Z_Score"] = (
    (df["Score"] - df["Class_Mean"]) / df["Class_Std"]
)

print(df[["Student", "Class", "Score", "Class_Z_Score"]].round(2))

Expected output

    Student Class  Score  Class_Z_Score
0  Samadhan     A     80          -1.22
1    Pradnya     A     90           1.22
2   Samiksha     A     85           0.00
3      Aarav     B     60          -1.22
4       Diya     B     70           1.22
5      Rohan     B     65           0.00
6      Meera     C     75          -1.22
7      Kabir     C     80           0.00
8      Anaya     C     85           1.22

A positive z-score means the student's score is above the class mean, while a negative z-score means it is below the mean. A zero indicates that it equals the mean.

Important: If a group has no variation, its standard deviation is zero. Dividing by zero can produce missing or infinite values. Handle such groups explicitly if they occur in your real dataset.

10. Transform Multiple Columns

You can transform multiple numeric columns together by selecting them as a list.

group_means = df.groupby("Class")[["Score", "Attendance"]].transform("mean")

print(group_means)

Expected output

   Score  Attendance
0   85.0   95.000000
1   85.0   95.000000
2   85.0   95.000000
3   65.0   75.000000
4   65.0   75.000000
5   65.0   75.000000
6   80.0   90.666667
7   80.0   90.666667
8   80.0   90.666667

The output has the same index and row count as the original DataFrame. Each selected column receives its corresponding group mean.

Add the transformed columns to the DataFrame

df[["Class_Average", "Attendance_Average"]] = (
    df.groupby("Class")[["Score", "Attendance"]].transform("mean")
)

11. Use a Custom Transformation Function

Besides built-in operations such as mean and sum, transform() can use a function that returns a compatible result for each group.

Example: Subtract each class's minimum score

df["Above_Class_Minimum"] = df.groupby("Class")["Score"].transform(
    lambda values: values - values.min()
)

print(df[["Student", "Class", "Score", "Above_Class_Minimum"]])

Expected output

    Student Class  Score  Above_Class_Minimum
0  Samadhan     A     80                    0
1    Pradnya     A     90                   10
2   Samiksha     A     85                    5
3      Aarav     B     60                    0
4       Diya     B     70                   10
5      Rohan     B     65                    5
6      Meera     C     75                    0
7      Kabir     C     80                    5
8      Anaya     C     85                   10

The smallest score in each class becomes zero after subtracting the group minimum. Other scores show how far they are above that minimum.

Note: A transformation function should return a scalar that can be broadcast to the group rows or a result compatible with the group's shape and alignment. A function returning an unrelated structure may fail or produce unintended results.

12. Use transform() with Missing Values

Group transformations can help fill missing numeric values using a group's average. This is a common data-cleaning technique.

scores = pd.DataFrame({
    "Student": ["Samadhan", "Pradnya", "Samiksha", "Aarav"],
    "Class": ["A", "A", "A", "B"],
    "Score": [80, None, 90, None]
})

scores["Class_Average"] = scores.groupby("Class")["Score"].transform("mean")

scores["Score_Filled"] = scores["Score"].fillna(scores["Class_Average"])

print(scores)

Expected output

    Student Class  Score  Class_Average  Score_Filled
0  Samadhan     A   80.0           85.0          80.0
1    Pradnya     A    NaN           85.0          85.0
2   Samiksha     A   90.0           85.0          90.0
3      Aarav     B    NaN            NaN           NaN

Class A has two available scores, so its mean is 85 and the missing score is filled with that mean. Class B has no available scores, so its group mean remains missing and cannot fill Aarav's score.

Data quality reminder: Filling missing values with group means is not always appropriate. Consider why the values are missing and whether the imputation method makes sense for your analysis.

13. Common Mistakes and Best Practices

Mistake 1: Expecting one row per group

transform("mean") repeats the mean across the original rows. Use agg("mean") if you need a summary with one row per group.

Mistake 2: Using the wrong column

Make sure the selected column contains values suitable for the calculation. For example, calculating a mean requires meaningful numeric data.

Mistake 3: Forgetting index alignment

Transformations are designed to align with the original rows. Preserve that alignment when assigning results back to the DataFrame.

Mistake 4: Confusing transform() with filter()

Use transform() to create row-aligned values. Use filter() when the objective is to retain or exclude entire groups.

Best practice: Use built-in operations when possible

Methods such as mean, sum, min, max, and count clearly communicate intent. Use a custom function when the calculation needs additional logic.

14. Practical Mini Project: Student Performance Analysis

Combine group transformations to calculate class averages, score differences, and performance labels.

import pandas as pd

df = pd.DataFrame({
    "Student": [
        "Samadhan", "Pradnya", "Samiksha",
        "Aarav", "Diya", "Rohan",
        "Meera", "Kabir", "Anaya"
    ],
    "Class": ["A", "A", "A", "B", "B", "B", "C", "C", "C"],
    "Score": [80, 90, 85, 60, 70, 65, 75, 80, 85],
    "Attendance": [95, 98, 92, 70, 80, 75, 88, 90, 94]
})

df["Class_Average"] = df.groupby("Class")["Score"].transform("mean")

df["Class_Total"] = df.groupby("Class")["Score"].transform("sum")

df["Score_Difference"] = df["Score"] - df["Class_Average"]

df["Performance"] = df["Score"].ge(df["Class_Average"]).map({
    True: "At or Above Average",
    False: "Below Average"
})

df["Attendance_Average"] = (
    df.groupby("Class")["Attendance"].transform("mean")
)

print(df.round(2).to_string(index=False))

Expected output

  Student Class  Score  Attendance  Class_Average  Class_Total  Score_Difference         Performance  Attendance_Average
 Samadhan     A     80          95           85.0          255              -5.0     Below Average               95.00
   Pradnya     A     90          98           85.0          255               5.0 At or Above Average         95.00
  Samiksha     A     85          92           85.0          255               0.0 At or Above Average         95.00
     Aarav     B     60          70           65.0          195              -5.0     Below Average               75.00
      Diya     B     70          80           65.0          195               5.0 At or Above Average         75.00
     Rohan     B     65          75           65.0          195               0.0 At or Above Average         75.00
     Meera     C     75          88           80.0          240              -5.0     Below Average               90.67
     Kabir     C     80          90           80.0          240               0.0 At or Above Average         90.67
     Anaya     C     85          94           80.0          240               5.0 At or Above Average         90.67

This project demonstrates how transformations can enrich the original dataset with group-level context while retaining every student's record.

15. Practice Exercises

Exercise 1: Group average

Create a column named Class_Average containing the average score for each student's class.

Hint: Use groupby("Class")["Score"].transform("mean").

Exercise 2: Group maximum

Create a column containing the highest score in each class.

Hint: Use transform("max").

Exercise 3: Difference from maximum

Create a column that subtracts each student's score from their class's highest score.

Hint: Subtract the original score from the transformed group maximum.

Exercise 4: Group row count

Create a column showing how many student records belong to each class.

Hint: Use groupby("Class")["Student"].transform("count").

Exercise 5: Multiple columns

Calculate class-level averages for both Score and Attendance in one transformation.

Hint: Select both columns using a list before calling transform("mean").

Exercise 6: Fill missing values

Use the class average to fill missing scores in the separate DataFrame from Section 12.

Hint: Calculate the transformed mean first, then use fillna().

16. Revision Questions

  1. What is a GroupBy transformation?
  2. How does transform() differ from agg()?
  3. Why does transform("mean") return repeated values?
  4. How do you calculate a group total for every row?
  5. What is the difference between count and size?
  6. How can you compare each value with its group average?
  7. How can group transformations help fill missing values?
  8. When should you use filter() instead of transform()?
  9. What can happen when a group's standard deviation is zero?
  10. How can you transform multiple numeric columns at once?

Summary

The Pandas groupby().transform() method performs calculations separately for each group and returns results aligned with the original rows. It is useful for adding group averages, totals, counts, differences from group statistics, normalized values, performance labels, and group-based imputations without reducing the original DataFrame to one row per group.

Key methods: groupby(), transform(), mean(), sum(), count(), size, min(), max(), std(), and fillna().