Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

String and Categorical Data in Pandas

Learn how to work with text values and categorical variables in Pandas. Explore string manipulation, text cleaning, category management, memory optimization, and practical data analysis techniques.

1. Introduction to String and Categorical Data

Data in a Pandas DataFrame can contain numbers, dates, text, and categorical values. String and categorical data are especially important when working with names, descriptions, locations, product types, customer groups, and survey responses.

Pandas provides string operations for cleaning and transforming text. It also supports categorical data types for representing values that belong to a limited set of categories.

String Data

String data represents text, such as student names, email addresses, city names, and product descriptions. String operations can change capitalization, remove extra spaces, search for text, replace values, and split text into parts.

Categorical Data

Categorical data represents values that belong to a defined set of categories, such as department names, membership levels, product groups, or grades.

2. Understanding String and Categorical Values

Consider a dataset containing student names, departments, and performance grades. Names are text values, while departments and grades may be treated as categorical variables.

String Data Categorical Data Student Names Asha Ravi Neha Department Categories IT HR Finance Text can be transformed; categories can be organized and analyzed.

String operations work with text, while categorical data represents values from a defined set of categories.

3. Creating a Sample DataFrame

First, create a DataFrame containing student names, departments, and grades.

import pandas as pd

df = pd.DataFrame({
    "Student": ["Asha", "Ravi", "Neha", "Amit"],
    "Department": ["IT", "HR", "IT", "Finance"],
    "Grade": ["A", "B", "A", "C"]
})

print(df)

Expected Output

  Student Department Grade
0    Asha         IT     A
1    Ravi         HR     B
2    Neha         IT     A
3    Amit    Finance     C

The Student column contains text. The Department and Grade columns contain a limited set of repeated values and may be suitable for categorical representation.

4. Understanding String Data Types

Pandas supports text data using string-related data types. Depending on how a Series is created, text may be represented using the dedicated string dtype or an object dtype.

names = pd.Series(
    ["Asha", "Ravi", "Neha"],
    dtype="string"
)

print(names)
print(names.dtype)

Expected Output

0    Asha
1    Ravi
2    Neha
dtype: string

string

Using dtype="string" explicitly creates a Series with Pandas' dedicated string data type. This provides a clear way to represent text and handle missing string values.

Tip: Inspect column types with df.dtypes before applying transformations, especially when working with data imported from CSV or Excel files.

5. Accessing String Methods with .str

Pandas provides the .str accessor to apply string operations to every value in a Series. These operations can transform or inspect text without using an explicit loop.

names = pd.Series(["asha", "RAVI", "neha"])

print(names.str.upper())

Expected Output

0    ASHA
1    RAVI
2    NEHA
dtype: object

The upper() method converts each string to uppercase. Other commonly used methods include lower(), title(), strip(), and replace().

6. Cleaning Text Values

Real-world datasets often contain inconsistent capitalization, leading spaces, trailing spaces, or unwanted characters. Cleaning text improves consistency and makes comparisons more reliable.

names = pd.Series([
    "  asha ",
    "RAVI  ",
    "  neha"
], dtype="string")

clean_names = names.str.strip().str.title()

print(clean_names)

Expected Output

0    Asha
1    Ravi
2    Neha
dtype: string

The strip() method removes leading and trailing whitespace. The title() method capitalizes the first letter of each word.

Important: Text cleaning should match the data's purpose. For example, changing capitalization may be appropriate for names but inappropriate for case-sensitive identifiers.

7. Searching Within String Values

Use str.contains() to check whether each string contains a specified pattern. The result is a Boolean Series indicating which rows match.

df = pd.DataFrame({
    "Student": ["Asha", "Ravi", "Anisha", "Neha"]
})

result = df["Student"].str.contains(
    "a",
    case=False,
    na=False
)

print(result)

Expected Output

0     True
1     True
2     True
3    False
Name: Student, dtype: bool

The case=False parameter makes the search case-insensitive. The na=False parameter treats missing values as non-matches.

You can use the resulting Boolean Series to filter a DataFrame.

matching_students = df[
    df["Student"].str.contains("a", case=False, na=False)
]

print(matching_students)

Expected Output

   Student
0     Asha
1     Ravi
2  Anisha

8. Replacing and Splitting Strings

String methods can replace text and split a string into multiple parts. These operations are useful when standardizing values or separating combined information.

Replacing Text

departments = pd.Series(
    ["Information Technology", "Human Resources", "Information Technology"]
)

short_names = departments.str.replace(
    "Information Technology",
    "IT",
    regex=False
)

print(short_names)

Expected Output

0                  IT
1    Human Resources
2                  IT
dtype: object

Splitting Text

full_names = pd.Series([
    "Asha Patil",
    "Ravi Sharma",
    "Neha Joshi"
])

parts = full_names.str.split(" ", expand=True)

print(parts)

Expected Output

      0       1
0  Asha   Patil
1  Ravi  Sharma
2  Neha   Joshi

When expand=True is specified, the split operation returns a DataFrame with a separate column for each split part.

9. Introduction to Categorical Data

A categorical variable contains values that belong to a limited collection of categories. Examples include department names, membership levels, product types, and grades.

You can convert a column to the Pandas category dtype using astype("category").

df = pd.DataFrame({
    "Department": ["IT", "HR", "IT", "Finance", "HR"]
})

df["Department"] = df["Department"].astype("category")

print(df)
print(df["Department"].dtype)

Expected Output

  Department
0         IT
1         HR
2         IT
3    Finance
4         HR

category

The category dtype stores the category definitions separately from the encoded values. This can be memory-efficient when a column contains many repeated values.

10. Understanding Categories and Category Codes

A categorical Series maintains a collection of categories and represents each observation using an internal code. You can inspect the categories and codes using the .cat accessor.

department = pd.Series(
    ["IT", "HR", "IT", "Finance"],
    dtype="category"
)

print(department.cat.categories)
print(department.cat.codes)

Expected Output

Index(['Finance', 'HR', 'IT'], dtype='object')

0    2
1    1
2    2
3    0
dtype: int8

Category codes correspond to the category order. By default, Pandas commonly sorts inferred string categories lexically. Codes are implementation details for representing categories; they should not be interpreted as meaningful numerical rankings.

Important: Never assume that a higher category code means a higher priority or better value. Define an explicit category order when the categories have a meaningful ranking.

11. Defining Ordered Categories

Some categorical variables have a natural order. For example, grades may follow the order Low, Medium, and High. You can define this order explicitly with pd.Categorical().

priority = pd.Series(
    pd.Categorical(
        ["Medium", "Low", "High", "Medium"],
        categories=["Low", "Medium", "High"],
        ordered=True
    )
)

print(priority)
print(priority.sort_values())

Expected Output

0    Medium
1       Low
2      High
3    Medium
dtype: category

Categories (3, object): ['Low' < 'Medium' < 'High']

1       Low
0    Medium
3    Medium
2      High
dtype: category

The declared order controls sorting and comparisons between categorical values. Without an explicit order, categories should not be assumed to represent a ranking.

12. Adding and Removing Categories

Pandas provides methods to manage the set of categories. You can add a new category before assigning it to values, or remove categories that are no longer required.

Adding a Category

department = pd.Series(
    ["IT", "HR", "IT"],
    dtype="category"
)

department = department.cat.add_categories(["Finance"])

print(department.cat.categories)

Expected Output

Index(['HR', 'IT', 'Finance'], dtype='object')

Removing Unused Categories

department = department.cat.remove_unused_categories()

print(department.cat.categories)

Expected Output

Index(['HR', 'IT'], dtype='object')

The unused category Finance is removed because no observation currently has that value.

13. Comparing String and Categorical Data

Feature String Data Categorical Data
Purpose Stores and processes text values. Represents values from a defined set of categories.
Typical examples Names, addresses, descriptions. Departments, grades, membership types.
Common operations Searching, replacing, splitting, changing case. Category management, ordering, grouping.
Memory behavior Depends on the dtype and underlying representation. Can save memory when many values repeat.
Ordering Text comparison follows string rules. Can have an explicitly defined category order.

14. Common Mistakes and Best Practices

Common Mistakes

  • Applying string methods to values that are not strings without checking their types.
  • Forgetting to remove unwanted spaces before comparing text values.
  • Assuming categorical codes represent meaningful numeric rankings.
  • Assigning a category value that is not present in the category definitions.
  • Assuming that categorical dtype is always more memory-efficient.
  • Changing text capitalization when case sensitivity matters.

Best Practices

  • Inspect data types using df.dtypes.
  • Clean inconsistent text before grouping or comparing values.
  • Use the .str accessor for vectorized string operations.
  • Convert repeated, limited-value columns to category when appropriate.
  • Define category ordering explicitly when categories have a natural ranking.
  • Check missing values before applying text or category transformations.

15. Revision Questions

  1. What is the difference between string data and categorical data?
  2. What is the purpose of the Pandas .str accessor?
  3. How does str.strip() help clean text?
  4. What does str.contains() return?
  5. How can you convert a column to categorical dtype?
  6. What is the difference between categories and category codes?
  7. How can you define an ordered categorical variable?
  8. How can you remove unused categories?
  9. Why is categorical dtype sometimes more memory-efficient?

16. Practical Exercises

Exercise 1: Clean Student Names

Create a Series containing names with inconsistent capitalization and extra spaces. Use string methods to clean the values.

Goal: Practice strip(), lower(), and title().

Exercise 2: Search Department Names

Create a DataFrame containing department names. Filter rows where the department contains a specified word.

Goal: Practice str.contains() and Boolean filtering.

Exercise 3: Convert to Categorical Data

Convert a department column into categorical dtype and inspect its categories and category codes.

Goal: Understand categorical representation.

Exercise 4: Create an Ordered Category

Create a priority column with Low, Medium, and High values. Define the category order and sort the DataFrame by priority.

Goal: Practice ordered categorical data.

Summary

Pandas supports string and categorical data for working with text and repeated category values. The .str accessor provides operations for cleaning, searching, replacing, and splitting text. The category dtype represents a limited collection of values and can improve memory efficiency when categories repeat frequently. The .cat accessor supports category inspection, ordering, and category management. Choosing the correct data type and cleaning values consistently helps make data analysis more accurate and efficient.