Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Stack and Unstack in Pandas

Learn how to reshape DataFrames using the Pandas stack() and unstack() operations. Understand MultiIndex structures, row and column transformations, and practical techniques for organizing complex datasets.

1. Introduction to Stack and Unstack

Pandas provides stack() and unstack() to reshape data between row-oriented and column-oriented structures. These operations are especially useful when working with hierarchical indexes, grouped data, and tables containing multiple categories.

Stack

The stack() operation moves a column level into the row index. It generally makes a DataFrame taller and produces a Series when stacking its only column level.

Unstack

The unstack() operation moves an index level into columns. It generally makes a DataFrame wider and can convert a Series with a MultiIndex into a DataFrame.

Wide DataFrame Values in columns stack() Longer Structure Values in index unstack() Back to columns Key Difference stack(): column level → row index unstack(): row index level → columns

Stack and unstack reshape the structure of a dataset by moving levels between columns and the index.

2. Creating a Sample DataFrame

Consider a DataFrame containing sales figures for two products across two regions.

import pandas as pd

df = pd.DataFrame({
    "Product": ["Laptop", "Phone"],
    "North": [100, 150],
    "South": [120, 170]
})

print(df)

Output

  Product  North  South
0  Laptop    100    120
1   Phone    150    170

The region names are stored as columns. We can use set_index() and stack() to move the product identifier into the index and the region columns into index entries.

3. Using stack()

The stack() method moves the column labels into an additional level of the row index.

indexed_df = df.set_index("Product")

stacked = indexed_df.stack()

print(stacked)

Output

Product
Laptop  North    100
        South    120
Phone   North    150
        South    170
dtype: int64

The result is a Series with a MultiIndex. The first index level identifies the product, while the second identifies the region. The sales figures are stored as the Series values.

Remember: stack() is most useful when column labels represent a category that should become part of the row index.

4. Understanding MultiIndex

A MultiIndex is a hierarchical index with two or more levels. It allows each observation to be identified by a combination of labels.

stacked = df.set_index("Product").stack()

print(stacked.index)

The index contains pairs such as ("Laptop", "North") and ("Phone", "South"). Each pair uniquely identifies a product-region combination in this example.

MultiIndex Levels

  • Level 1: Product, such as Laptop or Phone.
  • Level 2: Region, such as North or South.
  • Value: Sales associated with that product and region.

Hierarchical indexes are useful for representing data grouped by multiple categories without creating a separate column for every possible category combination.

5. Using unstack()

The unstack() method moves an index level into columns. It is commonly used to convert a MultiIndex Series back into a DataFrame with a wider layout.

stacked = df.set_index("Product").stack()

unstacked = stacked.unstack()

print(unstacked)

Output

        North  South
Product
Laptop    100    120
Phone     150    170

By default, unstack() moves the last index level into columns. In this example, the region labels become column names again.

Key point: Stacking followed by unstacking can restore the original arrangement when the relevant labels and values have not been altered and no information has been lost.

6. Choosing an Index Level with unstack()

When an index has multiple levels, you can specify which level should be moved into columns.

stacked = df.set_index("Product").stack()

result = stacked.unstack(level=0)

print(result)

The level parameter identifies the index level to unstack. In this example, level=0 selects the first index level, which contains the product labels.

Note: Changing the selected level changes the shape and orientation of the output. Inspect the index levels before selecting a numeric level.

7. Stack and Unstack with MultiIndex Columns

DataFrames can also have hierarchical column labels. Stacking such columns can move one or more column levels into the row index.

columns = pd.MultiIndex.from_tuples([
    ("Sales", "North"),
    ("Sales", "South"),
    ("Profit", "North"),
    ("Profit", "South")
])

data = [
    [100, 120, 20, 25],
    [150, 170, 30, 35]
]

multi_df = pd.DataFrame(
    data,
    index=["Laptop", "Phone"],
    columns=columns
)

print(multi_df)

Output

       Sales       Profit
       North South North South
Laptop   100   120    20    25
Phone    150   170    30    35

You can stack a selected column level to move its labels into the row index.

stacked_df = multi_df.stack(level=1)

print(stacked_df)

Here, level=1 selects the second column level, containing the region labels. The result groups sales and profit values under each region for every product.

8. Handling Missing Values

Depending on the data and the stacking options used, missing combinations may appear during reshaping. These missing values are represented by NaN.

df_missing = pd.DataFrame({
    "North": [100, None],
    "South": [120, 170]
}, index=["Laptop", "Phone"])

stacked = df_missing.stack()

print(stacked)

In current Pandas versions, stack() behavior and its available options depend on the version and the column structure. The default behavior commonly omits missing values when stacking ordinary columns.

If you need to preserve missing entries, consult the documentation for your installed Pandas version and use its supported stacking options. After reshaping, check the resulting rows to confirm that the data matches your requirements.

Important: Do not assume that every missing combination will be preserved by default. Verify the output before using it for calculations or reporting.

9. Resetting the Index After Reshaping

Stacking and unstacking can produce hierarchical indexes. If you need a regular tabular DataFrame, use reset_index() to turn index levels into ordinary columns.

stacked = df.set_index("Product").stack()

result = stacked.rename("Sales").reset_index()

print(result)

Output

  Product level_1  Sales
0  Laptop   North    100
1  Laptop   South    120
2   Phone   North    150
3   Phone   South    170

The level_1 column contains the original region labels. Rename it to a more descriptive name when preparing the final dataset.

result = result.rename(columns={"level_1": "Region"})

print(result)

Final Output

  Product Region  Sales
0  Laptop  North    100
1  Laptop  South    120
2   Phone  North    150
3   Phone  South    170

10. Stack vs. Unstack

Feature stack() unstack()
Operation Moves a column level into the row index. Moves an index level into columns.
Typical shape Produces a taller structure. Produces a wider structure.
Common result A Series with a MultiIndex when stacking a single column level. A DataFrame when unstacking a Series with a MultiIndex.
Common use Representing category measurements as indexed observations. Displaying index categories as separate columns.

11. Common Mistakes and Best Practices

Common Mistakes

  • Confusing stack() with melt().
  • Assuming that stacking always returns a DataFrame.
  • Selecting the wrong index or column level.
  • Ignoring missing values created or omitted during reshaping.
  • Forgetting that unstacking may require unique index combinations or may aggregate the structure into a different shape.
  • Assuming the original layout will always be restored after modifying the reshaped data.

Best Practices

  • Inspect the index using index and nlevels.
  • Use the level parameter when the intended level is not the default.
  • Check the output with head() and shape.
  • Use reset_index() when you need a flat DataFrame.
  • Test the result for missing values and expected row counts.

12. Revision Questions

  1. What is the purpose of stack() in Pandas?
  2. How does unstack() differ from stack()?
  3. What is a MultiIndex?
  4. What does unstack(level=0) do?
  5. How can you convert index levels into ordinary columns?
  6. What should you check when missing values appear during reshaping?
  7. Why might stacking and then unstacking fail to reproduce the exact original DataFrame?

13. Practical Exercises

Exercise 1: Regional Sales

Create a DataFrame with products as rows and regions as columns. Move the product names into the index and use stack() to reshape the regional sales into a Series.

Goal: Understand how column labels become an index level.

Exercise 2: Restore the Wide Format

Use unstack() on the Series created in Exercise 1. Compare the result with the original DataFrame.

Goal: Understand how index levels move back into columns.

Exercise 3: Flatten a MultiIndex

Convert a stacked Series into a regular DataFrame using reset_index(). Rename the generated columns to descriptive names.

Goal: Prepare reshaped data for reporting and analysis.

Lesson Summary

The Pandas stack() and unstack() operations reshape data by moving levels between columns and the row index. Stacking is useful for converting column categories into indexed observations, while unstacking moves index categories into columns. MultiIndex structures allow data to be organized across multiple levels. By choosing the correct level, inspecting missing values, and using reset_index() when necessary, you can reshape complex DataFrames into structures that are easier to analyze and present.