Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Indexes, Rows and Columns

Learn how Pandas identifies rows and columns, works with custom indexes, selects rows using labels and positions, and accesses individual values in a DataFrame.

1. Learning Objectives

After completing this lesson, you will be able to:

  • Understand DataFrame indexes, rows, and columns.
  • Inspect row indexes and column labels.
  • Create a DataFrame with a custom index.
  • Select rows using loc and iloc.
  • Access individual cells and ranges of rows.
  • Rename index labels and column names.
  • Understand the difference between row labels and row positions.

2. Understanding Rows, Columns and Indexes

A Pandas DataFrame organizes information into rows and columns. Each column has a label, and each row has an index label.

By default, Pandas creates integer index labels beginning with zero. You can also assign meaningful labels such as student IDs or employee codes.

DataFrame Structure Column labels Index Name Marks City 0 Samadhan 85 Pune 1 Pradnya 92 Mumbai Each row has an index label and each column has a name.

Figure 1: The DataFrame index identifies rows; column labels identify fields.

Remember: An index label and a row position are not always the same. For example, a row can have label 100 while being at position 0.

3. Inspecting Indexes and Columns

Pandas provides the index and columns attributes to inspect the labels associated with a DataFrame.

import pandas as pd

data = {
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Marks": [85, 92, 78],
    "City": ["Pune", "Mumbai", "Nashik"]
}

df = pd.DataFrame(data)

print("Index:")
print(df.index)

print("Columns:")
print(df.columns)

Example output:

Index:
RangeIndex(start=0, stop=3, step=1)

Columns:
Index(['Name', 'Marks', 'City'], dtype='object')

The DataFrame has three rows, with default index labels 0, 1, and 2. Its column labels are Name, Marks, and City.

4. Creating a DataFrame with a Custom Index

You can assign custom row labels using the index parameter when creating a DataFrame.

import pandas as pd

data = {
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Marks": [85, 92, 78]
}

df = pd.DataFrame(
    data,
    index=["ST101", "ST102", "ST103"]
)

print(df)

Output:

           Name  Marks
ST101  Samadhan     85
ST102   Pradnya     92
ST103  Samiksha     78

In this example, ST101, ST102, and ST103 are index labels rather than ordinary data columns.

Practical use: Custom indexes can make records easier to identify when using student IDs, employee IDs, or product codes.

5. Selecting Rows with loc

The loc accessor selects rows using index labels. It can also select particular columns by name.

import pandas as pd

df = pd.DataFrame(
    {
        "Name": ["Samadhan", "Pradnya", "Samiksha"],
        "Marks": [85, 92, 78]
    },
    index=["ST101", "ST102", "ST103"]
)

print(df.loc["ST102"])

Output:

Name     Pradnya
Marks         92
Name: ST102, dtype: object

Select a Particular Cell

print(df.loc["ST102", "Marks"])

Output:

92

The expression df.loc["ST102", "Marks"] retrieves the Marks value for the row labelled ST102.

Important: loc uses index labels, not row positions. A label that does not exist in the index raises a KeyError.

6. Selecting Rows with iloc

The iloc accessor selects rows and columns using integer positions. Position numbering starts at zero.

print(df.iloc[0])
print(df.iloc[1])

Output for the first row:

Name     Samadhan
Marks          85
Name: ST101, dtype: object

Select a Cell by Position

print(df.iloc[1, 1])

Output:

92

Here, df.iloc[1, 1] selects the value in the second row and second column because positions start at zero.

Expression Meaning
df.iloc[0] Select the first row.
df.iloc[1] Select the second row.
df.iloc[1, 1] Select the second row, second column.
df.iloc[:, 0] Select every row from the first column.

7. Selecting a Range of Rows

You can use slicing with iloc to select a range of rows or columns.

print(df.iloc[0:2])

Output:

           Name  Marks
ST101  Samadhan     85
ST102   Pradnya     92

The slice 0:2 includes positions 0 and 1. The ending position, 2, is excluded.

Select Rows and Specific Columns

print(df.iloc[0:2, 0:1])

This selects the first two rows and only the first column. The result remains a DataFrame.

Tip: With iloc slicing, the start position is included and the stop position is excluded.

8. Selecting Rows and Columns Together

The loc accessor allows you to select row labels and column labels in the same expression.

print(df.loc["ST101", "Name"])

print(df.loc[
    ["ST101", "ST103"],
    ["Name", "Marks"]
])

The first expression retrieves a single value. The second selects two labelled rows and two columns.

Output of the second expression:

           Name  Marks
ST101  Samadhan     85
ST103  Samiksha     78
Accessor Rows selected by Columns selected by
loc Index labels Column labels
iloc Integer positions Integer positions

9. Renaming Columns and Index Labels

The rename() method lets you change column names or index labels without manually rebuilding the DataFrame.

Rename Columns

renamed_df = df.rename(
    columns={
        "Name": "StudentName",
        "Marks": "TotalMarks"
    }
)

print(renamed_df)

The returned DataFrame contains the new column names. The original DataFrame remains unchanged unless you explicitly assign the result back to it.

Rename Index Labels

renamed_index_df = df.rename(
    index={"ST101": "ST001"}
)

print(renamed_index_df)
Remember: Use columns to rename column labels and index to rename row index labels.

10. Resetting and Setting an Index

You can use reset_index() to replace the current index with a default integer index. The old index can be added back as a regular column.

reset_df = df.reset_index()

print(reset_df)

By default, the old index becomes a column named index when it does not already have a name.

Set a Column as the Index

data = {
    "StudentID": ["ST101", "ST102", "ST103"],
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Marks": [85, 92, 78]
}

students_df = pd.DataFrame(data)

indexed_df = students_df.set_index("StudentID")

print(indexed_df)

The set_index() method makes StudentID the row index. By default, the StudentID column is no longer an ordinary data column in the returned DataFrame.

Important: reset_index() and set_index() return a new DataFrame by default. Assign the result if you want to keep the change.

11. Practical Exercise

CIIT TRAINING INSTITUTE: Student Records

Create a DataFrame with student IDs, names, marks, and cities for Samadhan, Pradnya, and Samiksha.

Complete these tasks:

  1. Set StudentID as the index.
  2. Display the row for ST102 using loc.
  3. Display the first row using iloc.
  4. Retrieve the marks for ST103.
  5. Select the Name and City columns.
  6. Rename the Marks column to TotalMarks.
  7. Reset the index and inspect the result.

Practice Code

import pandas as pd

data = {
    "StudentID": ["ST101", "ST102", "ST103"],
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Marks": [85, 92, 78],
    "City": ["Pune", "Mumbai", "Nashik"]
}

df = pd.DataFrame(data)

df = df.set_index("StudentID")

print("Student ST102:")
print(df.loc["ST102"])

print("\nFirst row:")
print(df.iloc[0])

print("\nST103 marks:")
print(df.loc["ST103", "Marks"])

print("\nName and City:")
print(df[["Name", "City"]])

print("\nRenamed columns:")
print(df.rename(columns={"Marks": "TotalMarks"}))

print("\nReset index:")
print(df.reset_index())

Revision Questions

  1. What is a DataFrame index?
  2. What is the difference between an index label and a row position?
  3. How do loc and iloc differ?
  4. How can you select a single cell?
  5. How can you rename a column?
  6. What does reset_index() do?
  7. How can you set a column as the DataFrame index?

Summary

In this lesson, we explored how Pandas DataFrames organize records into rows and columns. We inspected indexes and column labels and created a DataFrame with custom index labels.

We learned to select rows and cells using loc and iloc, select ranges of rows and columns, rename labels, and use reset_index() and set_index().

Understanding indexes, rows, and columns is essential for selecting the correct data and preparing datasets for analysis.