Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Index Management in Pandas

Learn how to set meaningful row indexes, reset the default integer index, rename index labels, and organize DataFrame records for easier analysis.

1. What Is an Index in Pandas?

Every Pandas DataFrame has an index that identifies its rows. By default, Pandas creates an integer index starting from zero. You can replace this default index with meaningful labels, such as student names, employee IDs, or product codes.

For example, Samadhan may want to retrieve a student's marks directly by name rather than remembering the row number. Setting the student name as the index can make this task easier.

Create a Student DataFrame

import pandas as pd

df = pd.DataFrame({
    "StudentID": [101, 102, 103, 104],
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Amit"],
    "City": ["Pune", "Mumbai", "Pune", "Nashik"],
    "Marks": [85, 92, 78, 88]
})

print(df)

Output

   StudentID      Name    City  Marks
0        101  Samadhan    Pune     85
1        102   Pradnya  Mumbai     92
2        103  Samiksha    Pune     78
3        104      Amit  Nashik     88

The leftmost numbers 0, 1, 2, and 3 are the default row index. They are separate from the StudentID column.

2. Set a Column as the Index

Use set_index() to make an existing column the row index.

indexed_df = df.set_index("Name")

print(indexed_df)

Output

          StudentID    City  Marks
Name
Samadhan        101    Pune     85
Pradnya         102  Mumbai     92
Samiksha        103    Pune     78
Amit            104  Nashik     88

The Name column is now the index. It no longer appears as an ordinary data column in this result.

Remember: df.set_index("Name") returns a new DataFrame by default. It does not replace df unless you assign the result back.

Keep the original DataFrame unchanged

indexed_df = df.set_index("Name")

print(df.columns.tolist())
print(indexed_df.columns.tolist())

Output

['StudentID', 'Name', 'City', 'Marks']
['StudentID', 'City', 'Marks']

3. Visual Guide: set_index()

The following illustration shows how the Name column becomes the row index while the other columns remain in the DataFrame.

Before set_index() After set_index("Name") Index Name City Marks Name Index City Marks 0 Samadhan Pune 85 1 Pradnya Mumbai 92 2 Samiksha Pune 78 3 Amit Nashik 88 Samadhan Pune 85 Pradnya Mumbai 92 Samiksha Pune 78 Amit Nashik 88 indexed_df = df.set_index("Name")

Figure 1: set_index() moves the selected column into the index.

4. Access Data Using the New Index

After setting Name as the index, you can retrieve a student's information directly by name with loc[].

indexed_df = df.set_index("Name")

print(indexed_df.loc["Samadhan"])

Output

StudentID    101
City        Pune
Marks         85
Name: Samadhan, dtype: object

Retrieve one value

print(indexed_df.loc["Pradnya", "Marks"])

Output

92

This is useful when the index contains a meaningful identifier that you already know.

5. Reset the Index

Use reset_index() when you want to restore a default integer index. By default, the existing index becomes a regular column.

indexed_df = df.set_index("Name")

restored_df = indexed_df.reset_index()

print(restored_df)

Output

       Name  StudentID    City  Marks
0  Samadhan        101    Pune     85
1   Pradnya        102  Mumbai     92
2  Samiksha        103    Pune     78
3      Amit        104  Nashik     88

The Name values return to a normal column, and Pandas creates the default integer index again.

Remember: reset_index() returns a DataFrame. Assign the result if you want to keep the reset version in a variable.

6. Drop the Old Index During Reset

Sometimes you want a new default integer index but do not need the old index as a column. Use drop=True.

indexed_df = df.set_index("Name")

reset_df = indexed_df.reset_index(drop=True)

print(reset_df)

Output

   StudentID    City  Marks
0        101    Pune     85
1        102  Mumbai     92
2        103    Pune     78
3        104  Nashik     88

Because drop=True is used, the old Name index is discarded instead of being added back as a column.

Be careful: Use drop=True only when you no longer need the existing index values.

7. Rename Index Labels

Use rename() with an index mapping to change specific row labels without changing the data values.

indexed_df = df.set_index("Name")

renamed_df = indexed_df.rename(
    index={
        "Samadhan": "Samadhan Patil",
        "Amit": "Amit Sharma"
    }
)

print(renamed_df)

Output

                StudentID    City  Marks
Name
Samadhan Patil        101    Pune     85
Pradnya                102  Mumbai     92
Samiksha               103    Pune     78
Amit Sharma            104  Nashik     88

Only the specified index labels are renamed. Other labels remain unchanged.

Rename the index name itself

The index can also have a name displayed above its labels. Use rename_axis() to change that name.

indexed_df = df.set_index("Name")

indexed_df = indexed_df.rename_axis("StudentName")

print(indexed_df.index.name)

Output

StudentName
Important distinction: rename(index=...) changes index labels. rename_axis() changes the index's name, not the individual labels.

8. Set a Custom Index

If you already have unique student IDs, you can use those IDs as the index.

student_df = df.set_index("StudentID")

print(student_df)

Output

                Name    City  Marks
StudentID
101         Samadhan    Pune     85
102          Pradnya  Mumbai     92
103         Samiksha    Pune     78
104             Amit  Nashik     88

Access a record by StudentID

print(student_df.loc[103])

Output

Name    Samiksha
City        Pune
Marks         78
Name: 103, dtype: object

IDs are often more suitable than names for indexing because names may not be unique.

9. Use Multiple Columns as an Index

Sometimes one column is not enough to identify a record. You can create a MultiIndex using more than one column.

multi_df = df.set_index(["City", "Name"])

print(multi_df)

Output

                 StudentID  Marks
City   Name
Pune   Samadhan        101     85
Mumbai Pradnya         102     92
Pune   Samiksha        103     78
Nashik Amit            104     88

Access data with a MultiIndex

print(multi_df.loc[("Pune", "Samadhan")])

Output

StudentID    101
Marks         85
Name: (Pune, Samadhan), dtype: int64

MultiIndex is useful for data organized by several dimensions, such as city and student, department and employee, or year and month.

10. Visual Guide: set_index() vs reset_index()

Default Index 0, 1, 2, 3 Name is a column Named Index Samadhan, Pradnya... Name becomes index Reset Index 0, 1, 2, 3 Name restored as column set_index() reset_index() set_index() → Meaningful labels reset_index() → Default integer index

Figure 2: set_index() and reset_index() let you change the structure of a DataFrame's row labels.

11. Index Management Quick Reference

Method Purpose Example
set_index() Use an existing column as the index. df.set_index("Name")
reset_index() Restore the default integer index. df.reset_index()
reset_index(drop=True) Reset without keeping the old index as a column. df.reset_index(drop=True)
rename(index=...) Rename selected index labels. df.rename(index={0: 10})
rename_axis() Change the index name. df.rename_axis("Student")
index Inspect the current row index. df.index

12. Common Mistakes to Avoid

Mistake 1: Assuming set_index() changes the original

By default, set_index() returns a new DataFrame. Assign the result if you want to keep it.

df = df.set_index("Name")

Mistake 2: Losing the old index accidentally

When using reset_index(drop=True), the old index is discarded. If you need it, use reset_index() without drop=True.

Mistake 3: Assuming every index is unique

Pandas permits duplicate index labels. If a label appears more than once, selecting it may return multiple rows.

Mistake 4: Confusing an index name with its labels

Use rename() to change individual labels and rename_axis() to change the index name.

13. Practice Exercises

Exercise 1: Set and Reset the Index

  1. Set Name as the index.
  2. Display the resulting DataFrame.
  3. Reset the index and restore Name as a column.

Exercise 2: Custom Index

  1. Set StudentID as the index.
  2. Retrieve the record for StudentID 103.
  3. Retrieve the marks for StudentID 102.

Exercise 3: Rename Index Labels

  1. Set Name as the index.
  2. Rename Samadhan to Samadhan Patil.
  3. Rename the index name to StudentName.

Suggested solution

# Set and reset the index
indexed_df = df.set_index("Name")
print(indexed_df)

restored_df = indexed_df.reset_index()
print(restored_df)

# Custom StudentID index
student_df = df.set_index("StudentID")
print(student_df.loc[103])
print(student_df.loc[102, "Marks"])

# Rename labels and the index name
named_df = df.set_index("Name")

named_df = named_df.rename(
    index={"Samadhan": "Samadhan Patil"}
)

named_df = named_df.rename_axis("StudentName")

print(named_df)

14. Revision Questions

  1. What is an index in a Pandas DataFrame?
  2. What does set_index() do?
  3. What does reset_index() do?
  4. What is the purpose of drop=True in reset_index()?
  5. How do you select a row after setting Name as the index?
  6. What is the difference between rename() and rename_axis()?
  7. Can a DataFrame contain duplicate index labels?
  8. When might a StudentID index be more useful than a Name index?

Summary

Index management helps you organize DataFrame rows using meaningful labels. Use set_index() to promote a column to the index and reset_index() to restore the default integer index.

You can also rename specific index labels, change the index name, or create a MultiIndex from multiple columns. Choose an index that makes your data easier to identify and analyze.