Reshaping and Pivoting in Pandas
Learn how to change the structure of tabular data, transform rows into columns, convert columns into rows, and reorganize datasets for analysis.
Learning objectives: Understand reshaping, pivot tables, melting, stacking, unstacking, and cross-tabulation.
1. What Is Reshaping in Pandas?
Reshaping is the process of changing the layout of a DataFrame without necessarily changing the underlying information. You can convert data from long format to wide format, wide format to long format, or reorganize index levels.
Real-world example
Samadhan stores student scores in separate rows for each subject. Pradnya wants a report where each student occupies one row and each subject becomes a column. Pandas reshaping methods can create this structure.
Reshaping changes how values are arranged into rows, columns, and indexes.
2. Create a Sample Dataset
We will use a student score dataset throughout this module.
import pandas as pd
df = pd.DataFrame({
"Student": [
"Samadhan",
"Samadhan",
"Pradnya",
"Pradnya",
"Samiksha",
"Samiksha"
],
"Subject": [
"Math",
"Science",
"Math",
"Science",
"Math",
"Science"
],
"Score": [85, 90, 92, 88, 78, 95]
})
print(df)
Expected output
Student Subject Score 0 Samadhan Math 85 1 Samadhan Science 90 2 Pradnya Math 92 3 Pradnya Science 88 4 Samiksha Math 78 5 Samiksha Science 95
This is a long-format dataset because each row represents one student-subject combination.
3. Main Reshaping Methods
| Method | Purpose | Typical use |
|---|---|---|
pivot() |
Convert unique row categories into columns. | Student scores by subject. |
pivot_table() |
Reshape data while aggregating duplicate combinations. | Average score by student and subject. |
melt() |
Convert wide data into long format. | Combine multiple subject columns into rows. |
stack() |
Move column labels into an inner index level. | Reshape wide data into a hierarchical structure. |
unstack() |
Move an index level into columns. | Turn subject index values into columns. |
crosstab() |
Create frequency or summarized tables. | Count records by category. |
4. Wide Format and Long Format
Data can represent the same information in different layouts.
Long format
Student Subject Score 0 Samadhan Math 85 1 Samadhan Science 90 2 Pradnya Math 92 3 Pradnya Science 88
Wide format
Subject Student Math Science 0 Samadhan 85 90 1 Pradnya 92 88
Long format is often useful for data processing and visualization. Wide format can make side-by-side comparisons easier.
5. Why Reshaping Is Important
- Prepare datasets for reports and dashboards.
- Compare values across categories.
- Transform imported spreadsheets into analysis-friendly structures.
- Prepare data for visualization libraries.
- Summarize repeated observations.
- Standardize datasets received from different sources.
6. A Simple Reshaping Workflow
Inspect the structure first, choose the appropriate operation, then validate the result.
7. Common Mistakes
Mistake 1: Using pivot() with duplicate combinations
pivot() requires each index-column combination to identify a single value. If combinations repeat, use pivot_table() with an appropriate aggregation function.
Mistake 2: Confusing pivot and melt
pivot() turns category values into columns. melt() turns selected columns into rows.
Mistake 3: Ignoring missing combinations
When a student has no value for a particular subject, reshaping may produce a missing value. Inspect the output before performing calculations.
Mistake 4: Not checking the resulting index
Operations such as stack() and unstack() can change index structure. Use index, columns, and head() to inspect the result.
8. Practice Exercises
Exercise 1: Convert long to wide
Use the sample student dataset to create one row per student and one column per subject.
Hint: Use pivot().
Exercise 2: Summarize repeated scores
Create a dataset with repeated student-subject combinations and calculate the average score.
Hint: Use pivot_table().
Exercise 3: Convert wide to long
Create a DataFrame with Student, Math, and Science columns and convert the subject columns into rows.
Hint: Use melt().
Exercise 4: Reshape an index
Create a DataFrame with a two-level index and move one level into columns.
Hint: Use unstack().
9. Revision Questions
- What is reshaping in Pandas?
- What is the difference between long format and wide format?
- When should you use
pivot()? - How does
pivot_table()handle repeated combinations? - What does
melt()do? - What is the purpose of
stack()andunstack()? - What type of information can
crosstab()summarize? - Why should you validate a reshaped DataFrame?
Summary
Reshaping reorganizes the structure of Pandas data to support analysis, reporting, and visualization. Use pivot() to create columns from categories, pivot_table() to aggregate repeated combinations, melt() to convert wide data to long format, stack() and unstack() to reorganize index levels, and crosstab() to summarize category relationships.
Key concepts: Long format, wide format, pivot(), pivot_table(), melt(), stack(), unstack(), and crosstab().