Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Categorical Data in Pandas

Learn how to represent, organize, order, analyze, and optimize categorical data in Pandas. Explore category creation, category codes, ordered categories, category management, grouping, and practical examples.

1. Introduction to Categorical Data

Categorical data represents values that belong to a limited set of possible categories. Examples include departments, product types, membership levels, colors, grades, and customer segments.

Pandas provides the category data type to represent this kind of information. Instead of treating every value as an independent text string, Pandas stores a collection of categories and codes that refer to those categories.

Examples of Categorical Variables

  • Department: IT, HR, Finance, Marketing.
  • Membership: Bronze, Silver, Gold.
  • Grade: Low, Medium, High.
  • Product type: Laptop, Phone, Tablet.
Key idea: Categorical data is particularly useful when a column contains a limited set of repeated values. It can reduce memory usage and can represent meaningful category ordering.

2. Understanding Categorical Data

Consider a dataset containing department names. The same department may appear many times. Converting the column to categorical dtype allows Pandas to represent repeated values using shared categories.

Original Values Category Codes IT HR IT Finance HR 2 1 2 0 1 Codes refer to categories; they do not automatically represent rankings.

Repeated values can share category definitions and be represented by category codes.

Category codes are determined by the category order. Unless you define an order explicitly, the codes should not be interpreted as meaningful numerical rankings.

3. Creating a Categorical Series

You can create a categorical Series by passing dtype="category" when creating a Series.

import pandas as pd

department = pd.Series(
    ["IT", "HR", "IT", "Finance", "HR"],
    dtype="category"
)

print(department)
print(department.dtype)

Expected Output

0         IT
1         HR
2         IT
3    Finance
4         HR
dtype: category

Categories (3, object): ['Finance', 'HR', 'IT']
category

Pandas infers the categories from the values. In this example, there are five observations but only three unique categories.

4. Converting a DataFrame Column to Category

Existing columns can be converted to categorical dtype using astype("category").

df = pd.DataFrame({
    "Employee": ["Asha", "Ravi", "Neha", "Amit"],
    "Department": ["IT", "HR", "IT", "Finance"]
})

df["Department"] = df["Department"].astype("category")

print(df)
print(df.dtypes)

Expected Output

  Employee Department
0     Asha         IT
1     Ravi         HR
2     Neha         IT
3     Amit    Finance

Employee         object
Department     category
dtype: object

Only the Department column is converted. The Employee column retains its original dtype. Depending on how a text column was created, its original dtype may be object or a string dtype.

5. Inspecting Categories

The .cat accessor provides methods for inspecting the categories of a categorical Series.

department = pd.Series(
    ["IT", "HR", "IT", "Finance"],
    dtype="category"
)

print(department.cat.categories)
print(department.cat.codes)
print(department.cat.ordered)

Expected Output

Index(['Finance', 'HR', 'IT'], dtype='object')

0    2
1    1
2    2
3    0
dtype: int8

False

The categories property returns the available category labels. The codes property returns the integer code associated with each observation. The ordered property indicates whether the categorical dtype has an explicit order.

Important: Category codes are implementation representations. Code values can change if the category order changes, so do not use them as permanent identifiers.

6. Defining Categories Explicitly

You can specify the permitted categories when creating a categorical Series. This is useful when you want a consistent set of categories even if some categories are not present in the current data.

department = pd.Series(
    pd.Categorical(
        ["IT", "HR", "IT"],
        categories=["IT", "HR", "Finance", "Marketing"]
    )
)

print(department)
print(department.cat.categories)

Expected Output

0    IT
1    HR
2    IT
dtype: category

Categories (4, object): ['IT', 'HR', 'Finance', 'Marketing']

Although only IT and HR appear in the observations, Finance and Marketing remain part of the category definitions.

Practical use: Explicit categories are helpful when preparing reports that must use the same set of categories across multiple datasets or reporting periods.

7. Creating Ordered Categories

Some categories have a meaningful order. For example, priority levels can be ordered as Low, Medium, and High. Use ordered=True to define that sequence.

priority = pd.Series(
    pd.Categorical(
        ["Medium", "Low", "High", "Medium"],
        categories=["Low", "Medium", "High"],
        ordered=True
    )
)

print(priority)
print(priority.sort_values())

Expected Output

0    Medium
1       Low
2      High
3    Medium
dtype: category

Categories (3, object): ['Low' < 'Medium' < 'High']

1       Low
0    Medium
3    Medium
2      High
dtype: category

Sorting follows the declared category order rather than ordinary alphabetical order. This is useful for grades, satisfaction levels, priority rankings, and product sizes.

8. Comparing Ordered Categories

Ordered categorical values can be compared according to their declared order.

priority = pd.Series(
    pd.Categorical(
        ["Low", "Medium", "High"],
        categories=["Low", "Medium", "High"],
        ordered=True
    )
)

print(priority > "Low")

Expected Output

0    False
1     True
2     True
dtype: bool

The comparison is based on the declared category order. An unordered categorical variable does not support all ordering comparisons.

9. Adding New Categories

Use cat.add_categories() to add new permitted values to an existing categorical Series.

department = pd.Series(
    ["IT", "HR", "IT"],
    dtype="category"
)

department = department.cat.add_categories(["Finance"])

print(department.cat.categories)

Expected Output

Index(['HR', 'IT', 'Finance'], dtype='object')

Adding a category makes it available for future assignments. It does not automatically add a new observation to the Series.

Assigning a New Category

department.loc[1] = "Finance"

print(department)

Expected Output

0         IT
1    Finance
2         IT
dtype: category

Categories (3, object): ['HR', 'IT', 'Finance']

The assignment succeeds because Finance was added to the permitted category set before the assignment.

10. Renaming Categories

Use cat.rename_categories() to change category labels. This updates the labels while preserving the categorical structure.

department = pd.Series(
    ["IT", "HR", "IT"],
    dtype="category"
)

department = department.cat.rename_categories({
    "IT": "Information Technology",
    "HR": "Human Resources"
})

print(department)

Expected Output

0    Information Technology
1           Human Resources
2    Information Technology
dtype: category

Categories (2, object): ['Human Resources', 'Information Technology']

Renaming categories changes the labels associated with those categories, not the number of observations.

11. Removing Categories

Pandas provides methods to remove categories that are no longer needed. The appropriate method depends on whether the category is unused or should be removed entirely.

Removing Unused Categories

department = pd.Series(
    pd.Categorical(
        ["IT", "HR", "IT"],
        categories=["IT", "HR", "Finance"]
    )
)

department = department.cat.remove_unused_categories()

print(department.cat.categories)

Expected Output

Index(['IT', 'HR'], dtype='object')

Finance is removed because it does not appear in any observation.

Removing a Specific Category

department = pd.Series(
    ["IT", "HR", "IT"],
    dtype="category"
)

department = department.cat.remove_categories(["HR"])

print(department)

Expected Output

0     IT
1    NaN
2     IT
dtype: category

Categories (1, object): ['IT']

When a category is removed, observations that used that category become missing values. Review the result before continuing analysis.

12. Reordering Categories

Use cat.reorder_categories() to change the order of existing categories. The new list must contain every existing category exactly once.

priority = pd.Series(
    pd.Categorical(
        ["Low", "High", "Medium"],
        categories=["Low", "Medium", "High"],
        ordered=True
    )
)

priority = priority.cat.reorder_categories(
    ["High", "Medium", "Low"],
    ordered=True
)

print(priority.sort_values())

Expected Output

1      High
2    Medium
0       Low
dtype: category

Categories (3, object): ['High' < 'Medium' < 'Low']

The category order is now High, Medium, Low. Sorting follows the updated order.

13. Counting Category Frequencies

The value_counts() method counts how often each category appears. It is useful for examining distributions.

department = pd.Series(
    ["IT", "HR", "IT", "Finance", "HR", "IT"],
    dtype="category"
)

print(department.value_counts())

Expected Output

IT         3
HR         2
Finance    1
Name: count, dtype: int64

IT occurs three times, HR occurs twice, and Finance occurs once. The precise output formatting can vary by Pandas version.

Including Unused Categories

department = pd.Series(
    pd.Categorical(
        ["IT", "HR", "IT"],
        categories=["IT", "HR", "Finance"]
    )
)

print(department.value_counts(sort=False))

Expected Output

IT         2
HR         1
Finance    0
Name: count, dtype: int64

Because Finance is a defined category, it can appear in the frequency output with a count of zero even though no row currently contains it.

14. Grouping by Categorical Data

Categorical columns can be used with groupby() to summarize numeric information by category.

df = pd.DataFrame({
    "Department": ["IT", "HR", "IT", "Finance", "HR"],
    "Salary": [60000, 45000, 70000, 50000, 48000]
})

df["Department"] = df["Department"].astype("category")

result = df.groupby(
    "Department",
    observed=True
)["Salary"].mean()

print(result)

Expected Output

Department
Finance    50000.0
HR         46500.0
IT         65000.0
Name: Salary, dtype: float64

The result displays the average salary for each department that appears in the dataset. The observed=True argument limits the result to categories observed in the data.

15. Memory Efficiency of Categorical Data

Categorical dtype can reduce memory usage when a column contains many repeated values and relatively few unique categories. However, it is not guaranteed to use less memory for every dataset.

import pandas as pd

departments = ["IT", "HR", "Finance", "IT", "HR"] * 1000

text_series = pd.Series(departments)
category_series = text_series.astype("category")

print("Text memory:")
print(text_series.memory_usage(deep=True))

print("Category memory:")
print(category_series.memory_usage(deep=True))

This example compares the memory usage of a regular Series with a categorical Series. The exact byte counts depend on the data and the installed Pandas version.

Best practice: Measure memory usage with memory_usage(deep=True) before and after conversion instead of assuming that categorical dtype will always be more efficient.

16. String Data vs. Categorical Data

Feature String Data Categorical Data
Purpose Stores and processes text values. Represents a limited set of categories.
Typical examples Names, descriptions, addresses. Departments, grades, membership levels.
Operations Searching, replacing, splitting, changing case. Category ordering, counting, grouping, category management.
Ordering Uses string comparison rules. Can have an explicitly defined category order.
Memory Depends on the underlying representation and values. Can save memory when values repeat frequently.

17. Common Mistakes and Best Practices

Common Mistakes

  • Assuming category codes represent meaningful rankings.
  • Assigning a value that is not in the category definitions.
  • Removing a category without checking the resulting missing values.
  • Reordering categories without including all existing categories.
  • Assuming categorical dtype always saves memory.
  • Forgetting that unused categories may remain after filtering data.

Best Practices

  • Use categorical dtype for columns with a limited set of repeated values when appropriate.
  • Define categories explicitly when consistent category sets are required.
  • Use ordered categories for meaningful rankings.
  • Inspect categories with .cat.categories.
  • Check category counts with value_counts().
  • Measure memory usage before and after conversion.

18. Revision Questions

  1. What is categorical data in Pandas?
  2. How can you convert a DataFrame column to categorical dtype?
  3. What is the difference between categories and category codes?
  4. How do ordered categories affect sorting?
  5. How can you add a new category to a Series?
  6. What happens when you remove a category used by existing observations?
  7. How can you remove unused categories?
  8. How can you count the occurrences of each category?
  9. Why can categorical dtype reduce memory usage?

19. Practical Exercises

Exercise 1: Convert Department Values

Create a DataFrame containing employee names and department names. Convert the department column to categorical dtype and inspect its categories.

Goal: Practice categorical conversion and inspection.

Exercise 2: Define an Order

Create a priority column with Low, Medium, and High values. Define the category order and sort the DataFrame by priority.

Goal: Practice ordered categorical data.

Exercise 3: Manage Categories

Add a new department category, assign it to a row, and then remove unused categories.

Goal: Practice category management.

Exercise 4: Analyze Category Frequencies

Count how many employees belong to each department. Include defined categories even if some of them have zero observations.

Goal: Practice value_counts() with categorical data.

Summary

Pandas categorical data represents values from a limited collection of categories. You can create categorical Series, convert existing columns, inspect category labels and codes, define explicit category sets, and create ordered categories for meaningful rankings. The .cat accessor supports adding, renaming, reordering, and removing categories. Categorical columns can also be used for frequency counts, grouping, and memory optimization. Always inspect category definitions and verify the resulting data after making changes.