Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Reshaping and Pivoting in Pandas

Learn how to change the structure of tabular data, transform rows into columns, convert columns into rows, and reorganize datasets for analysis.

Learning objectives: Understand reshaping, pivot tables, melting, stacking, unstacking, and cross-tabulation.

1. What Is Reshaping in Pandas?

Reshaping is the process of changing the layout of a DataFrame without necessarily changing the underlying information. You can convert data from long format to wide format, wide format to long format, or reorganize index levels.

Real-world example

Samadhan stores student scores in separate rows for each subject. Pradnya wants a report where each student occupies one row and each subject becomes a column. Pandas reshaping methods can create this structure.

Long Format to Wide Format Long Format Student | Subject | Score Samadhan | Math | 85 Samadhan | Science | 90 Pradnya | Math | 92 pivot() Reshape Wide Format Student | Math | Science Samadhan | 85 | 90 Pradnya | 92 | NaN The layout changes to make comparisons easier.

Reshaping changes how values are arranged into rows, columns, and indexes.

2. Create a Sample Dataset

We will use a student score dataset throughout this module.

import pandas as pd

df = pd.DataFrame({
    "Student": [
        "Samadhan",
        "Samadhan",
        "Pradnya",
        "Pradnya",
        "Samiksha",
        "Samiksha"
    ],
    "Subject": [
        "Math",
        "Science",
        "Math",
        "Science",
        "Math",
        "Science"
    ],
    "Score": [85, 90, 92, 88, 78, 95]
})

print(df)

Expected output

    Student  Subject  Score
0  Samadhan     Math     85
1  Samadhan  Science     90
2   Pradnya     Math     92
3   Pradnya  Science     88
4  Samiksha     Math     78
5  Samiksha  Science     95

This is a long-format dataset because each row represents one student-subject combination.

3. Main Reshaping Methods

Method Purpose Typical use
pivot() Convert unique row categories into columns. Student scores by subject.
pivot_table() Reshape data while aggregating duplicate combinations. Average score by student and subject.
melt() Convert wide data into long format. Combine multiple subject columns into rows.
stack() Move column labels into an inner index level. Reshape wide data into a hierarchical structure.
unstack() Move an index level into columns. Turn subject index values into columns.
crosstab() Create frequency or summarized tables. Count records by category.

4. Wide Format and Long Format

Data can represent the same information in different layouts.

Long format

    Student  Subject  Score
0  Samadhan     Math     85
1  Samadhan  Science     90
2   Pradnya     Math     92
3   Pradnya  Science     88

Wide format

Subject   Student  Math  Science
0        Samadhan    85       90
1         Pradnya    92       88

Long format is often useful for data processing and visualization. Wide format can make side-by-side comparisons easier.

5. Why Reshaping Is Important

  • Prepare datasets for reports and dashboards.
  • Compare values across categories.
  • Transform imported spreadsheets into analysis-friendly structures.
  • Prepare data for visualization libraries.
  • Summarize repeated observations.
  • Standardize datasets received from different sources.

6. A Simple Reshaping Workflow

1. Inspect Rows and columns 2. Select Reshape method 3. Transform Pivot or melt 4. Validate Check results

Inspect the structure first, choose the appropriate operation, then validate the result.

7. Common Mistakes

Mistake 1: Using pivot() with duplicate combinations

pivot() requires each index-column combination to identify a single value. If combinations repeat, use pivot_table() with an appropriate aggregation function.

Mistake 2: Confusing pivot and melt

pivot() turns category values into columns. melt() turns selected columns into rows.

Mistake 3: Ignoring missing combinations

When a student has no value for a particular subject, reshaping may produce a missing value. Inspect the output before performing calculations.

Mistake 4: Not checking the resulting index

Operations such as stack() and unstack() can change index structure. Use index, columns, and head() to inspect the result.

8. Practice Exercises

Exercise 1: Convert long to wide

Use the sample student dataset to create one row per student and one column per subject.

Hint: Use pivot().

Exercise 2: Summarize repeated scores

Create a dataset with repeated student-subject combinations and calculate the average score.

Hint: Use pivot_table().

Exercise 3: Convert wide to long

Create a DataFrame with Student, Math, and Science columns and convert the subject columns into rows.

Hint: Use melt().

Exercise 4: Reshape an index

Create a DataFrame with a two-level index and move one level into columns.

Hint: Use unstack().

9. Revision Questions

  1. What is reshaping in Pandas?
  2. What is the difference between long format and wide format?
  3. When should you use pivot()?
  4. How does pivot_table() handle repeated combinations?
  5. What does melt() do?
  6. What is the purpose of stack() and unstack()?
  7. What type of information can crosstab() summarize?
  8. Why should you validate a reshaped DataFrame?

Summary

Reshaping reorganizes the structure of Pandas data to support analysis, reporting, and visualization. Use pivot() to create columns from categories, pivot_table() to aggregate repeated combinations, melt() to convert wide data to long format, stack() and unstack() to reorganize index levels, and crosstab() to summarize category relationships.

Key concepts: Long format, wide format, pivot(), pivot_table(), melt(), stack(), unstack(), and crosstab().