Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Renaming and Replacing Data in Pandas

Learn how to rename DataFrame columns and index labels, replace values, standardize inconsistent text, and prepare clean data for analysis.

1. Introduction to Renaming and Replacing Data

Real-world datasets often contain unclear column names, inconsistent labels, spelling variations, and values that need standardization. Pandas provides methods to correct these issues without manually rebuilding the entire DataFrame.

Example: Student Results

Samadhan receives a student dataset with columns named stud_name, sub, and score. The subject column also contains inconsistent values such as python, PYTHON, and Py.

Renaming can make the column names more readable, while replacing can standardize the values stored inside the columns.

Before cleaning stud_name sub score Samadhan python 85 Pradnya PYTHON 92 Samiksha Py 88 Clean After cleaning Student Subject Marks Samadhan Python 85 Pradnya Python 92 Samiksha Python 88 Clear column names + consistent values = easier analysis Renaming changes labels; replacing changes data values.

A cleaned DataFrame uses descriptive column names and consistent values.

2. Create a Sample DataFrame

Start with a DataFrame containing short column names and inconsistent subject and status values.

import pandas as pd

df = pd.DataFrame({
    "stud_name": [
        "Samadhan",
        "Pradnya",
        "Samiksha",
        "Aarav",
        "Neha"
    ],
    "sub": [
        "python",
        "PYTHON",
        "Py",
        "sql",
        "SQL"
    ],
    "score": [85, 92, 88, 79, 91],
    "status": [
        "pass",
        "PASS",
        "Pass",
        "fail",
        "FAIL"
    ]
})

print(df)

Expected Output

   stud_name     sub  score status
0   Samadhan  python     85   pass
1    Pradnya  PYTHON     92   PASS
2   Samiksha      Py     88   Pass
3      Aarav     sql     79   fail
4       Neha     SQL     91   FAIL

Notice that the same subject or status is represented using different spellings and capitalization. These inconsistencies can make filtering and grouping more difficult.

3. Rename Columns Using rename()

The rename() method changes selected row or column labels. To rename columns, provide a dictionary that maps each existing column name to its new name.

df_renamed = df.rename(columns={
    "stud_name": "Student",
    "sub": "Subject",
    "score": "Marks"
})

print(df_renamed)

Expected Output

    Student Subject  Marks status
0  Samadhan  python     85   pass
1   Pradnya  PYTHON     92   PASS
2  Samiksha      Py     88   Pass
3     Aarav     sql     79   fail
4      Neha     SQL     91   FAIL

Only the specified column names change. The values in the columns remain unchanged, and the original DataFrame is not modified by this assignment.

Rename Only One Column

df_one_column = df.rename(
    columns={"score": "Marks"}
)

print(df_one_column.columns.tolist())

Expected Output:

['stud_name', 'sub', 'Marks', 'status']
Remember: With rename(), you only need to include the labels you want to change. Other labels remain as they are.

4. Rename All Columns at Once

If every column needs a new name, you can assign a complete list to df.columns. The number of names must match the number of columns.

df_all_names = df.copy()

df_all_names.columns = [
    "Student",
    "Subject",
    "Marks",
    "Status"
]

print(df_all_names.columns.tolist())

Expected Output:

['Student', 'Subject', 'Marks', 'Status']

When Should You Use Each Approach?

Approach Best Use Important Detail
rename(columns={...}) Change selected column names. Unspecified columns remain unchanged.
df.columns = [...] Replace all column names. The list length must equal the number of columns.
df.rename(...) Rename row index labels with index=. Use the appropriate argument for rows or columns.

5. Rename Index Labels

A DataFrame index identifies rows. The index parameter of rename() lets you change specific index labels without changing the data values.

df_index = df.rename(index={
    0: "Student_A",
    1: "Student_B"
})

print(df_index.head(2))

Expected Output

           stud_name     sub  score status
Student_A   Samadhan  python     85   pass
Student_B    Pradnya  PYTHON     92   PASS

The index labels change, but the student names and other values remain the same.

Important: Index labels do not have to be sequential numbers. Renaming an index label does not move a row or change the row's underlying data.

6. Replace Values Using replace()

The replace() method substitutes specified values with new values. It can operate on a complete DataFrame or on a selected column.

Replace a Value in One Column

df_subject = df.copy()

df_subject["sub"] = df_subject["sub"].replace(
    "Py",
    "Python"
)

print(df_subject["sub"])

Expected Output:

0    python
1    PYTHON
2    Python
3       sql
4       SQL
Name: sub, dtype: object

Only the exact value Py was replaced. The lowercase and uppercase spellings remain unchanged.

Replace Multiple Values

df_status = df.copy()

df_status["status"] = df_status["status"].replace({
    "pass": "Passed",
    "PASS": "Passed",
    "Pass": "Passed",
    "fail": "Failed",
    "FAIL": "Failed"
})

print(df_status["status"])

Expected Output:

0    Passed
1    Passed
2    Passed
3    Failed
4    Failed
Name: status, dtype: object

Passing a dictionary makes it possible to replace several different values in one operation.

7. Standardize Text with str.strip() and str.lower()

Some inconsistencies are caused by capitalization or extra spaces. String methods can normalize these values before you apply replacement rules.

Remove Extra Spaces and Normalize Case

subjects = pd.Series([
    " Python ",
    "PYTHON",
    "Py",
    " sql ",
    "SQL"
])

subjects_clean = subjects.str.strip().str.lower()

print(subjects_clean)

Expected Output:

0    python
1    python
2        py
3       sql
4       sql
dtype: object

str.strip() removes leading and trailing whitespace. str.lower() converts text to lowercase. The value py still needs an explicit mapping because lowercasing does not expand abbreviations.

Normalize and Replace Abbreviations

subjects_clean = (
    subjects.str.strip()
            .str.lower()
            .replace({
                "py": "python"
            })
)

print(subjects_clean)

Expected Output:

0    python
1    python
2    python
3       sql
4       sql
dtype: object
Best practice: Normalize spaces and capitalization before mapping known abbreviations or spelling variations.

8. Replace Values Across a DataFrame

You can call replace() on the whole DataFrame when the replacement rules should apply to matching values across its columns.

df_general = pd.DataFrame({
    "Student": ["Samadhan", "Pradnya", "Samiksha"],
    "Status": ["pass", "fail", "pass"],
    "Result": ["pass", "pass", "fail"]
})

df_general = df_general.replace({
    "pass": "Passed",
    "fail": "Failed"
})

print(df_general)

Expected Output

    Student  Status  Result
0  Samadhan  Passed  Passed
1   Pradnya  Failed  Passed
2  Samiksha  Passed  Failed

The replacement rules apply to matching values in both Status and Result. Student names remain unchanged because they do not match either replacement key.

9. Replace Values Using Regular Expressions

Regular expressions can identify text patterns. With regex=True, the replacement pattern is interpreted as a regular expression rather than only as a literal string.

Remove a Prefix from Product Codes

codes = pd.Series([
    "ID-101",
    "ID-102",
    "ID-103"
])

clean_codes = codes.replace(
    r"^ID-",
    "",
    regex=True
)

print(clean_codes)

Expected Output

0    101
1    102
2    103
dtype: object

The pattern ^ID- matches the text ID- at the beginning of each string. The prefix is removed while the remaining characters are preserved.

Use regex carefully: A broad pattern may change more values than intended. Test the replacement on a copy or inspect the affected rows before using it in a production workflow.

10. Replace Values Using Conditions

When a replacement depends on a condition rather than an exact text match, use Boolean indexing or loc.

Replace Marks Below 40 with a Status

marks_df = pd.DataFrame({
    "Student": ["Samadhan", "Pradnya", "Samiksha"],
    "Marks": [85, 32, 48]
})

marks_df["Status"] = "Pass"

marks_df.loc[marks_df["Marks"] < 40, "Status"] = "Fail"

print(marks_df)

Expected Output

    Student  Marks Status
0  Samadhan     85   Pass
1   Pradnya     32   Fail
2  Samiksha     48   Pass

This example creates a new status column and updates it for rows where marks are below 40. It does not replace the numeric marks themselves.

Replace Numeric Values with a New Numeric Value

marks_df["Marks"] = marks_df["Marks"].replace(32, 35)

print(marks_df["Marks"].tolist())

Expected Output:

[85, 35, 48]

11. Rename and Replace: Understanding the Difference

Operation Method What Changes? Example
Rename a column rename(columns=...) Column label score to Marks
Rename an index rename(index=...) Row label 0 to Student_A
Replace a value replace() Cell values pass to Passed
Normalize text str.strip(), str.lower() Text formatting " PYTHON " to "python"
Conditional update loc Values in rows matching a condition Set status to Fail when marks are below 40
Original DataFrame stud_name status Pradnya pass Rename column Change the label Student Status Pradnya Passed Replace value Change the cell content Rename changes labels. Replace changes matching values. Choose the method that matches the intended change.

Column labels and cell values are different parts of a DataFrame.

12. Complete Practical Example

Pradnya wants to clean the student dataset in one workflow. The task includes renaming columns, normalizing text, replacing abbreviations, and standardizing result labels.

import pandas as pd

df = pd.DataFrame({
    "stud_name": ["Samadhan", "Pradnya", "Samiksha", "Aarav"],
    "sub": ["python", "PYTHON", "Py", "sql"],
    "score": [85, 92, 88, 79],
    "status": ["pass", "PASS", "Pass", "fail"]
})

# Step 1: Rename selected columns
clean_df = df.rename(columns={
    "stud_name": "Student",
    "sub": "Subject",
    "score": "Marks",
    "status": "Status"
})

# Step 2: Normalize text formatting
clean_df["Subject"] = (
    clean_df["Subject"]
    .str.strip()
    .str.lower()
)

clean_df["Status"] = (
    clean_df["Status"]
    .str.strip()
    .str.lower()
)

# Step 3: Replace known subject abbreviations
clean_df["Subject"] = clean_df["Subject"].replace({
    "py": "python"
})

# Step 4: Standardize status labels
clean_df["Status"] = clean_df["Status"].replace({
    "pass": "Passed",
    "fail": "Failed"
})

print(clean_df)

Expected Output

    Student Subject  Marks  Status
0  Samadhan  python     85  Passed
1   Pradnya  python     92  Passed
2  Samiksha  python     88  Passed
3     Aarav     sql     79  Failed

The result has readable column names, standardized subject values, and consistent status labels. The number of rows remains four because this workflow changes labels and values without removing records.

13. Common Mistakes to Avoid

  • Using the wrong old column name: The keys in the rename dictionary must match existing labels.
  • Assigning the wrong number of column names: A complete replacement list must match the column count.
  • Expecting replace() to ignore capitalization: Exact string replacements are case-sensitive.
  • Forgetting whitespace: Values such as "SQL" and " SQL " are not identical strings.
  • Replacing abbreviations without checking meaning: A short label may have more than one valid interpretation.
  • Changing every matching value unintentionally: Whole-DataFrame replacement affects matching values in all applicable columns.
  • Overwriting the original data too early: Use df.copy() when you want to preserve an unmodified version for comparison.

14. Quick Revision

Code Purpose
df.rename(columns={"score": "Marks"}) Rename a selected column.
df.rename(index={0: "Student_A"}) Rename an index label.
df.columns = ["A", "B", "C"] Replace all column names when the DataFrame has three columns.
series.replace("pass", "Passed") Replace a matching value.
series.replace({"pass": "Passed", "fail": "Failed"}) Apply multiple replacement rules.
series.str.strip() Remove leading and trailing whitespace.
series.str.lower() Convert text to lowercase.
df.loc[condition, "Status"] = "Failed" Update values for rows matching a condition.

15. Practice Questions and Exercises

Concept Questions

  1. What is the difference between rename() and replace()?
  2. How do you rename only one column?
  3. What happens if the list assigned to df.columns has the wrong length?
  4. Why should you use str.strip() before standardizing text?
  5. How can you replace multiple values in one operation?
  6. When would conditional replacement with loc be more suitable than replace()?

Hands-On Exercises

  1. Create a DataFrame with columns emp_name, dept, and salary. Rename them to Employee, Department, and Salary.
  2. Replace all occurrences of "HR" with "Human Resources" in the department column.
  3. Normalize a text column using str.strip() and str.lower().
  4. Map the abbreviations "py" and "pyt" to "python".
  5. Create a status column that contains "High" for salaries above 50000 and "Standard" otherwise.
  6. Rename two index labels and display the resulting DataFrame.
  7. Build a complete cleaning workflow and verify the output with head(), columns, and value_counts().
Learning goal: You should now be able to rename labels, replace exact values, normalize text, and apply condition-based updates to a DataFrame.

Summary

Renaming and replacing are essential data-cleaning operations. Use rename() to change row or column labels, replace() to substitute matching values, and string methods such as str.strip() and str.lower() to normalize text. Use Boolean conditions and loc when changes depend on a rule. Before replacing data, confirm the meaning of the values and inspect the results to ensure the cleaning process preserves valid records.