Categorical Data in Pandas
Learn how to represent, organize, order, analyze, and optimize categorical data in Pandas. Explore category creation, category codes, ordered categories, category management, grouping, and practical examples.
1. Introduction to Categorical Data
Categorical data represents values that belong to a limited set of possible categories. Examples include departments, product types, membership levels, colors, grades, and customer segments.
Pandas provides the category data type to represent
this kind of information. Instead of treating every value as an
independent text string, Pandas stores a collection of categories
and codes that refer to those categories.
Examples of Categorical Variables
- Department: IT, HR, Finance, Marketing.
- Membership: Bronze, Silver, Gold.
- Grade: Low, Medium, High.
- Product type: Laptop, Phone, Tablet.
2. Understanding Categorical Data
Consider a dataset containing department names. The same department may appear many times. Converting the column to categorical dtype allows Pandas to represent repeated values using shared categories.
Repeated values can share category definitions and be represented by category codes.
Category codes are determined by the category order. Unless you define an order explicitly, the codes should not be interpreted as meaningful numerical rankings.
3. Creating a Categorical Series
You can create a categorical Series by passing
dtype="category" when creating a Series.
import pandas as pd
department = pd.Series(
["IT", "HR", "IT", "Finance", "HR"],
dtype="category"
)
print(department)
print(department.dtype)
Expected Output
0 IT
1 HR
2 IT
3 Finance
4 HR
dtype: category
Categories (3, object): ['Finance', 'HR', 'IT']
category
Pandas infers the categories from the values. In this example, there are five observations but only three unique categories.
4. Converting a DataFrame Column to Category
Existing columns can be converted to categorical dtype using
astype("category").
df = pd.DataFrame({
"Employee": ["Asha", "Ravi", "Neha", "Amit"],
"Department": ["IT", "HR", "IT", "Finance"]
})
df["Department"] = df["Department"].astype("category")
print(df)
print(df.dtypes)
Expected Output
Employee Department
0 Asha IT
1 Ravi HR
2 Neha IT
3 Amit Finance
Employee object
Department category
dtype: object
Only the Department column is converted. The
Employee column retains its original dtype.
Depending on how a text column was created, its original dtype
may be object or a string dtype.
5. Inspecting Categories
The .cat accessor provides methods for inspecting
the categories of a categorical Series.
department = pd.Series(
["IT", "HR", "IT", "Finance"],
dtype="category"
)
print(department.cat.categories)
print(department.cat.codes)
print(department.cat.ordered)
Expected Output
Index(['Finance', 'HR', 'IT'], dtype='object')
0 2
1 1
2 2
3 0
dtype: int8
False
The categories property returns the available category
labels. The codes property returns the integer code
associated with each observation. The ordered
property indicates whether the categorical dtype has an explicit order.
6. Defining Categories Explicitly
You can specify the permitted categories when creating a categorical Series. This is useful when you want a consistent set of categories even if some categories are not present in the current data.
department = pd.Series(
pd.Categorical(
["IT", "HR", "IT"],
categories=["IT", "HR", "Finance", "Marketing"]
)
)
print(department)
print(department.cat.categories)
Expected Output
0 IT
1 HR
2 IT
dtype: category
Categories (4, object): ['IT', 'HR', 'Finance', 'Marketing']
Although only IT and HR appear in the observations, Finance and Marketing remain part of the category definitions.
7. Creating Ordered Categories
Some categories have a meaningful order. For example, priority levels
can be ordered as Low, Medium, and High. Use
ordered=True to define that sequence.
priority = pd.Series(
pd.Categorical(
["Medium", "Low", "High", "Medium"],
categories=["Low", "Medium", "High"],
ordered=True
)
)
print(priority)
print(priority.sort_values())
Expected Output
0 Medium
1 Low
2 High
3 Medium
dtype: category
Categories (3, object): ['Low' < 'Medium' < 'High']
1 Low
0 Medium
3 Medium
2 High
dtype: category
Sorting follows the declared category order rather than ordinary alphabetical order. This is useful for grades, satisfaction levels, priority rankings, and product sizes.
8. Comparing Ordered Categories
Ordered categorical values can be compared according to their declared order.
priority = pd.Series(
pd.Categorical(
["Low", "Medium", "High"],
categories=["Low", "Medium", "High"],
ordered=True
)
)
print(priority > "Low")
Expected Output
0 False
1 True
2 True
dtype: bool
The comparison is based on the declared category order. An unordered categorical variable does not support all ordering comparisons.
9. Adding New Categories
Use cat.add_categories() to add new permitted values
to an existing categorical Series.
department = pd.Series(
["IT", "HR", "IT"],
dtype="category"
)
department = department.cat.add_categories(["Finance"])
print(department.cat.categories)
Expected Output
Index(['HR', 'IT', 'Finance'], dtype='object')
Adding a category makes it available for future assignments. It does not automatically add a new observation to the Series.
Assigning a New Category
department.loc[1] = "Finance"
print(department)
Expected Output
0 IT
1 Finance
2 IT
dtype: category
Categories (3, object): ['HR', 'IT', 'Finance']
The assignment succeeds because Finance was added to the permitted category set before the assignment.
10. Renaming Categories
Use cat.rename_categories() to change category labels.
This updates the labels while preserving the categorical structure.
department = pd.Series(
["IT", "HR", "IT"],
dtype="category"
)
department = department.cat.rename_categories({
"IT": "Information Technology",
"HR": "Human Resources"
})
print(department)
Expected Output
0 Information Technology
1 Human Resources
2 Information Technology
dtype: category
Categories (2, object): ['Human Resources', 'Information Technology']
Renaming categories changes the labels associated with those categories, not the number of observations.
11. Removing Categories
Pandas provides methods to remove categories that are no longer needed. The appropriate method depends on whether the category is unused or should be removed entirely.
Removing Unused Categories
department = pd.Series(
pd.Categorical(
["IT", "HR", "IT"],
categories=["IT", "HR", "Finance"]
)
)
department = department.cat.remove_unused_categories()
print(department.cat.categories)
Expected Output
Index(['IT', 'HR'], dtype='object')
Finance is removed because it does not appear in any observation.
Removing a Specific Category
department = pd.Series(
["IT", "HR", "IT"],
dtype="category"
)
department = department.cat.remove_categories(["HR"])
print(department)
Expected Output
0 IT
1 NaN
2 IT
dtype: category
Categories (1, object): ['IT']
When a category is removed, observations that used that category become missing values. Review the result before continuing analysis.
12. Reordering Categories
Use cat.reorder_categories() to change the order of
existing categories. The new list must contain every existing
category exactly once.
priority = pd.Series(
pd.Categorical(
["Low", "High", "Medium"],
categories=["Low", "Medium", "High"],
ordered=True
)
)
priority = priority.cat.reorder_categories(
["High", "Medium", "Low"],
ordered=True
)
print(priority.sort_values())
Expected Output
1 High
2 Medium
0 Low
dtype: category
Categories (3, object): ['High' < 'Medium' < 'Low']
The category order is now High, Medium, Low. Sorting follows the updated order.
13. Counting Category Frequencies
The value_counts() method counts how often each
category appears. It is useful for examining distributions.
department = pd.Series(
["IT", "HR", "IT", "Finance", "HR", "IT"],
dtype="category"
)
print(department.value_counts())
Expected Output
IT 3
HR 2
Finance 1
Name: count, dtype: int64
IT occurs three times, HR occurs twice, and Finance occurs once. The precise output formatting can vary by Pandas version.
Including Unused Categories
department = pd.Series(
pd.Categorical(
["IT", "HR", "IT"],
categories=["IT", "HR", "Finance"]
)
)
print(department.value_counts(sort=False))
Expected Output
IT 2
HR 1
Finance 0
Name: count, dtype: int64
Because Finance is a defined category, it can appear in the frequency output with a count of zero even though no row currently contains it.
14. Grouping by Categorical Data
Categorical columns can be used with groupby() to
summarize numeric information by category.
df = pd.DataFrame({
"Department": ["IT", "HR", "IT", "Finance", "HR"],
"Salary": [60000, 45000, 70000, 50000, 48000]
})
df["Department"] = df["Department"].astype("category")
result = df.groupby(
"Department",
observed=True
)["Salary"].mean()
print(result)
Expected Output
Department
Finance 50000.0
HR 46500.0
IT 65000.0
Name: Salary, dtype: float64
The result displays the average salary for each department that
appears in the dataset. The observed=True argument
limits the result to categories observed in the data.
15. Memory Efficiency of Categorical Data
Categorical dtype can reduce memory usage when a column contains many repeated values and relatively few unique categories. However, it is not guaranteed to use less memory for every dataset.
import pandas as pd
departments = ["IT", "HR", "Finance", "IT", "HR"] * 1000
text_series = pd.Series(departments)
category_series = text_series.astype("category")
print("Text memory:")
print(text_series.memory_usage(deep=True))
print("Category memory:")
print(category_series.memory_usage(deep=True))
This example compares the memory usage of a regular Series with a categorical Series. The exact byte counts depend on the data and the installed Pandas version.
memory_usage(deep=True) before and after conversion
instead of assuming that categorical dtype will always be more efficient.
16. String Data vs. Categorical Data
| Feature | String Data | Categorical Data |
|---|---|---|
| Purpose | Stores and processes text values. | Represents a limited set of categories. |
| Typical examples | Names, descriptions, addresses. | Departments, grades, membership levels. |
| Operations | Searching, replacing, splitting, changing case. | Category ordering, counting, grouping, category management. |
| Ordering | Uses string comparison rules. | Can have an explicitly defined category order. |
| Memory | Depends on the underlying representation and values. | Can save memory when values repeat frequently. |
17. Common Mistakes and Best Practices
Common Mistakes
- Assuming category codes represent meaningful rankings.
- Assigning a value that is not in the category definitions.
- Removing a category without checking the resulting missing values.
- Reordering categories without including all existing categories.
- Assuming categorical dtype always saves memory.
- Forgetting that unused categories may remain after filtering data.
Best Practices
- Use categorical dtype for columns with a limited set of repeated values when appropriate.
- Define categories explicitly when consistent category sets are required.
- Use ordered categories for meaningful rankings.
- Inspect categories with
.cat.categories. - Check category counts with
value_counts(). - Measure memory usage before and after conversion.
18. Revision Questions
- What is categorical data in Pandas?
- How can you convert a DataFrame column to categorical dtype?
- What is the difference between categories and category codes?
- How do ordered categories affect sorting?
- How can you add a new category to a Series?
- What happens when you remove a category used by existing observations?
- How can you remove unused categories?
- How can you count the occurrences of each category?
- Why can categorical dtype reduce memory usage?
19. Practical Exercises
Exercise 1: Convert Department Values
Create a DataFrame containing employee names and department names. Convert the department column to categorical dtype and inspect its categories.
Goal: Practice categorical conversion and inspection.
Exercise 2: Define an Order
Create a priority column with Low, Medium, and High values. Define the category order and sort the DataFrame by priority.
Goal: Practice ordered categorical data.
Exercise 3: Manage Categories
Add a new department category, assign it to a row, and then remove unused categories.
Goal: Practice category management.
Exercise 4: Analyze Category Frequencies
Count how many employees belong to each department. Include defined categories even if some of them have zero observations.
Goal: Practice value_counts() with categorical data.
Summary
Pandas categorical data represents values from a limited collection
of categories. You can create categorical Series, convert existing
columns, inspect category labels and codes, define explicit category
sets, and create ordered categories for meaningful rankings. The
.cat accessor supports adding, renaming, reordering,
and removing categories. Categorical columns can also be used for
frequency counts, grouping, and memory optimization. Always inspect
category definitions and verify the resulting data after making changes.