Data Types and Type Conversion in Pandas
Understand Pandas data types, inspect column types, convert strings to numbers, work with dates, handle invalid values, and prepare datasets for accurate analysis.
1. What Are Data Types?
A data type describes the kind of value stored in a column. A column might contain text, integers, decimal numbers, dates, Boolean values, or categorical labels.
Pandas uses data types to determine how values are stored, compared, calculated, and processed. Incorrect data types can cause calculation errors or prevent mathematical operations.
Example: Student Records
Samadhan receives a dataset in which marks are stored as text. The values look numeric, but Pandas may treat them as strings. Before calculating the average marks, he needs to convert the column to a suitable numeric type.
Correct data types enable reliable calculations, comparisons, and analysis.
2. Common Pandas Data Types
| Data Type | Description | Example |
|---|---|---|
int64 |
Integer values without a fractional part. | 10, 250 |
float64 |
Numeric values that can include decimals. | 85.5, 19.99 |
object |
Often used for text or mixed Python objects. | "Python" |
string |
Pandas string extension dtype for text. | "Samadhan" |
bool |
Boolean values. | True, False |
datetime64[ns] |
Date and time values. | 2026-10-29 |
category |
Repeated values stored as categorical data. | "Beginner", "Advanced" |
Int64 |
Nullable integer extension dtype that supports missing values. | 10, <NA> |
int64 and
Int64 are not identical. The capitalized nullable
extension dtype Int64 can represent missing values
using pd.NA.
3. Create a Sample DataFrame
This example contains marks stored as text, salary values with currency symbols, and dates stored as strings.
import pandas as pd
df = pd.DataFrame({
"Student": [
"Samadhan",
"Pradnya",
"Samiksha",
"Aarav"
],
"Marks": [
"85",
"92",
"88",
"79"
],
"Salary": [
"50000",
"62000",
"58000",
"45000"
],
"JoinDate": [
"2025-01-10",
"2025-02-15",
"2025-03-20",
"2025-04-05"
]
})
print(df)
Expected Output
Student Marks Salary JoinDate 0 Samadhan 85 50000 2025-01-10 1 Pradnya 92 62000 2025-02-15 2 Samiksha 88 58000 2025-03-20 3 Aarav 79 45000 2025-04-05
The values in Marks, Salary, and
JoinDate look numeric or date-like, but they were
provided as strings. We will inspect and convert them.
4. Inspect Data Types with dtypes
The dtypes attribute displays the data type of
every column. It is a useful first step when investigating
unexpected calculations or conversion problems.
print(df.dtypes)
Expected Output
Student object Marks object Salary object JoinDate object dtype: object
All four columns have the object dtype in this
example because the columns were created from strings.
Inspect One Column
print(df["Marks"].dtype)
Expected Output:
object
Inspect DataFrame Information
df.info()
The output includes the number of rows, column names, non-null counts, data types, and memory usage. Exact memory usage and formatting may vary between Pandas versions.
5. Convert a Column Using astype()
The astype() method converts values to a specified
data type when the values are compatible with that type.
Convert Marks to Integers
df["Marks"] = df["Marks"].astype("int64")
print(df["Marks"].dtype)
print(df["Marks"])
Expected Output
int64 0 85 1 92 2 88 3 79 Name: Marks, dtype: int64
The marks column is now numeric, so mathematical calculations can be performed.
Calculate the Average
print(df["Marks"].mean())
Expected Output:
86.0
Convert Salary to Float
df["Salary"] = df["Salary"].astype("float64")
print(df["Salary"].dtype)
Expected Output:
float64
astype() raises an
error if a value cannot be converted to the requested type.
For messy numeric text, consider pd.to_numeric().
6. Convert Text to Numbers with to_numeric()
The pd.to_numeric() function converts compatible
values into numeric types. It also provides an
errors parameter for handling invalid values.
Example with Invalid Data
marks = pd.Series([
"85",
"92",
"Not Available",
"79"
])
numeric_marks = pd.to_numeric(
marks,
errors="coerce"
)
print(numeric_marks)
Expected Output
0 85.0 1 92.0 2 NaN 3 79.0 dtype: float64
The invalid value "Not Available" becomes
NaN. Because the result contains a missing numeric
value, Pandas uses a floating-point dtype in this example.
Understanding the errors Parameter
| Option | Behavior |
|---|---|
errors="raise" |
Raises an error when a value cannot be converted. This is the default. |
errors="coerce" |
Converts invalid values to NaN. |
errors="ignore" |
Legacy behavior that may return the original input when conversion fails. Prefer explicit handling with coerce or validation. |
errors="coerce"
when cleaning untrusted numeric input, then inspect the values
that became missing. Do not silently discard conversion failures.
7. Find Values That Cannot Be Converted
When using errors="coerce", compare the original
values with the converted results to identify invalid entries.
raw_marks = pd.Series([
"85",
"92",
"Not Available",
"79",
"unknown"
])
converted_marks = pd.to_numeric(
raw_marks,
errors="coerce"
)
invalid_mask = converted_marks.isna() & raw_marks.notna()
print(raw_marks[invalid_mask])
Expected Output
2 Not Available 4 unknown dtype: object
This method identifies non-missing original values that failed numeric conversion. It helps you decide whether the values should be corrected, mapped to missing values, or investigated further.
Count Invalid Values
print("Invalid values:", invalid_mask.sum())
Expected Output:
Invalid values: 2
8. Handle Missing Values During Conversion
A column may contain both invalid text and legitimate missing values. After converting to numeric values, choose an appropriate missing-value strategy for the analysis.
Example
marks = pd.Series([
"85",
"92",
"Not Available",
None,
"79"
])
numeric_marks = pd.to_numeric(
marks,
errors="coerce"
)
print(numeric_marks)
Expected Output
0 85.0 1 92.0 2 NaN 3 NaN 4 79.0 dtype: float64
Option A: Remove Rows with Missing Marks
valid_marks = numeric_marks.dropna() print(valid_marks.tolist())
Expected Output:
[85.0, 92.0, 79.0]
Option B: Fill Missing Values with a Chosen Value
filled_marks = numeric_marks.fillna(0) print(filled_marks.tolist())
Expected Output:
[85.0, 92.0, 0.0, 0.0, 79.0]
9. Use Nullable Integer Types
Traditional NumPy integer types such as int64
cannot represent missing values directly in a standard integer
array. Pandas provides the nullable extension dtype
Int64, which supports pd.NA.
marks = pd.Series([
85,
92,
None,
79
])
marks_nullable = marks.astype("Int64")
print(marks_nullable)
print(marks_nullable.dtype)
Expected Output
0 85 1 92 2 <NA> 3 79 dtype: Int64 Int64
The values remain integer-like, and the missing entry is
represented by <NA> instead of forcing the
entire column to use floating-point numbers.
Int64 is a Pandas nullable integer dtype;
int64 is a standard NumPy integer dtype.
10. Convert Strings to Dates with to_datetime()
Dates stored as strings cannot reliably support date calculations
until they are converted into datetime values. Use
pd.to_datetime().
Convert the JoinDate Column
df["JoinDate"] = pd.to_datetime(
df["JoinDate"]
)
print(df["JoinDate"].dtype)
Expected Output:
datetime64[ns]
Extract the Year
df["JoinYear"] = df["JoinDate"].dt.year print(df[["Student", "JoinYear"]])
Expected Output
Student JoinYear 0 Samadhan 2025 1 Pradnya 2025 2 Samiksha 2025 3 Aarav 2025
Extract Month and Day
df["JoinMonth"] = df["JoinDate"].dt.month df["JoinDay"] = df["JoinDate"].dt.day print(df[["Student", "JoinMonth", "JoinDay"]])
Expected Output
Student JoinMonth JoinDay 0 Samadhan 1 10 1 Pradnya 2 15 2 Samiksha 3 20 3 Aarav 4 5
11. Handle Invalid Dates
Some datasets contain malformed dates, unknown labels, or
inconsistent date formats. With errors="coerce",
invalid dates become NaT, Pandas' missing datetime
value.
dates = pd.Series([
"2025-01-10",
"2025-02-15",
"not-a-date",
"2025-04-05"
])
parsed_dates = pd.to_datetime(
dates,
errors="coerce"
)
print(parsed_dates)
Expected Output
0 2025-01-10 1 2025-02-15 2 NaT 3 2025-04-05 dtype: datetime64[ns]
The invalid date becomes NaT. You can inspect the
affected records before deciding how to handle them.
Find Invalid Date Entries
invalid_dates = dates[parsed_dates.isna() & dates.notna()] print(invalid_dates)
Expected Output:
2 not-a-date dtype: object
03/04/2025, specify the intended
format when known. This avoids interpreting the day and month
incorrectly.
Parse a Known Date Format
dates = pd.Series([
"31/01/2025",
"15/02/2025"
])
parsed_dates = pd.to_datetime(
dates,
format="%d/%m/%Y"
)
print(parsed_dates)
Expected Output:
0 2025-01-31 1 2025-02-15 dtype: datetime64[ns]
12. Convert Values to Boolean
Boolean columns contain True or False.
They are useful for flags, conditions, and filtering.
Create a Boolean Column from a Condition
marks = pd.Series([85, 35, 72, 28]) passed = marks >= 40 print(passed)
Expected Output
0 True 1 False 2 True 3 False dtype: bool
A comparison produces Boolean values automatically. This is generally safer than converting arbitrary text directly to Boolean values.
astype(bool) does not
interpret strings such as "False" as the Boolean
value False. A non-empty string is truthy in Python.
Map known text labels explicitly when converting text flags.
Convert Text Flags Explicitly
flags = pd.Series([
"True",
"False",
"True"
])
flag_map = {
"True": True,
"False": False
}
boolean_flags = flags.map(flag_map)
print(boolean_flags)
print(boolean_flags.dtype)
Expected Output
0 True 1 False 2 True dtype: bool bool
13. Convert Repeated Text to Category
Categorical data is useful for columns containing a limited set of repeated labels, such as department, product type, or student level. It can reduce memory use when many rows repeat the same values.
levels = pd.Series([
"Beginner",
"Advanced",
"Beginner",
"Intermediate",
"Advanced"
])
levels = levels.astype("category")
print(levels.dtype)
print(levels.cat.categories.tolist())
Expected Output
category ['Advanced', 'Beginner', 'Intermediate']
Pandas stores the unique categories and associates each row with a category code. Category ordering is not automatically a meaningful ranking; define an ordered categorical explicitly when the order matters.
Create an Ordered Category
level_type = pd.CategoricalDtype(
categories=[
"Beginner",
"Intermediate",
"Advanced"
],
ordered=True
)
levels_ordered = levels.astype(level_type)
print(levels_ordered.cat.ordered)
Expected Output:
True
14. Convert Multiple Columns Together
You can convert several columns in one workflow. For example, convert marks to integers and salary to floating-point numbers.
df = pd.DataFrame({
"Student": ["Samadhan", "Pradnya", "Samiksha"],
"Marks": ["85", "92", "88"],
"Salary": ["50000.50", "62000.75", "58000.25"]
})
df = df.astype({
"Marks": "int64",
"Salary": "float64"
})
print(df.dtypes)
Expected Output
Student object Marks int64 Salary float64 dtype: object
Passing a dictionary to astype() lets you specify
a different target type for each selected column.
Inspect the Converted DataFrame
print(df)
print("Average marks:", df["Marks"].mean())
print("Average salary:", df["Salary"].mean())
Expected Output
Student Marks Salary 0 Samadhan 85 50000.50 1 Pradnya 92 62000.75 2 Samiksha 88 58000.25 Average marks: 88.33333333333333 Average salary: 56667.166666666664
15. Convert Currency Text to Numeric Values
Financial datasets sometimes store amounts as text containing currency symbols or thousands separators. Remove known formatting characters before converting the values.
salary = pd.Series([
"$50,000",
"$62,500",
"$58,750"
])
salary_clean = salary.str.replace(
"$",
"",
regex=False
).str.replace(
",",
"",
regex=False
)
salary_numeric = pd.to_numeric(salary_clean)
print(salary_numeric)
Expected Output
0 50000 1 62500 2 58750 dtype: int64
The currency symbol and thousands separators are removed before conversion. The result is a numeric Series that supports arithmetic operations.
16. Verify Data Types After Conversion
A successful assignment does not guarantee that the resulting values are appropriate for the analysis. Check data types, missing values, and sample records after conversion.
print(df.dtypes)
print("\nMissing values:")
print(df.isna().sum())
print("\nSample records:")
print(df.head())
The output displays the current data type of each column, the number of missing values, and the first five records. Use these checks to identify unexpected conversions or newly introduced missing values.
Validation Checklist
- Are numeric columns actually numeric?
- Are date columns stored as datetime values?
- Did invalid conversions create unexpected missing values?
- Were missing values handled according to the business rules?
- Do the converted values still represent the original information correctly?
17. Complete Practical Data Type Cleaning Workflow
This example combines type inspection, numeric conversion, date conversion, invalid-value detection, and validation.
import pandas as pd
df = pd.DataFrame({
"Student": [
"Samadhan",
"Pradnya",
"Samiksha",
"Aarav"
],
"Marks": [
"85",
"92",
"not available",
"79"
],
"JoinDate": [
"2025-01-10",
"2025-02-15",
"invalid-date",
"2025-04-05"
]
})
print("Original data types:")
print(df.dtypes)
# Convert marks to numeric values.
df["Marks"] = pd.to_numeric(
df["Marks"],
errors="coerce"
)
# Convert join dates to datetime values.
df["JoinDate"] = pd.to_datetime(
df["JoinDate"],
errors="coerce"
)
print("\nCleaned data:")
print(df)
print("\nUpdated data types:")
print(df.dtypes)
print("\nMissing values:")
print(df.isna().sum())
Expected Output
Original data types:
Student object
Marks object
JoinDate object
dtype: object
Cleaned data:
Student Marks JoinDate
0 Samadhan 85.0 2025-01-10
1 Pradnya 92.0 2025-02-15
2 Samiksha NaN NaT
3 Aarav 79.0 2025-04-05
Updated data types:
Student object
Marks float64
JoinDate datetime64[ns]
dtype: object
Missing values:
Student 0
Marks 1
JoinDate 1
dtype: int64
The invalid mark becomes NaN, and the invalid date
becomes NaT. Both conversions succeed without
stopping the workflow, while the resulting missing values remain
visible for further investigation.
18. Common Mistakes to Avoid
- Converting numeric text without checking it: Values containing letters or unexpected symbols may fail conversion.
-
Confusing text with numbers:
The string
"100"is not the same data type as the integer100. -
Using int64 when missing values are present:
Consider cleaning missing values first or using nullable
Int64. -
Ignoring conversion failures:
Investigate values converted to
NaNorNaT. - Parsing ambiguous dates without a format: Specify the intended format when the input convention is known.
- Replacing unknown values with zero automatically: Zero may be a valid measurement rather than a missing-value marker.
-
Using Boolean conversion on text flags:
Map text labels such as
"True"and"False"explicitly. - Assuming categories are automatically ordered: Define an ordered categorical when the sequence has meaning.
19. Quick Revision
| Method or Attribute | Purpose |
|---|---|
df.dtypes |
Displays the data type of every column. |
df["Marks"].dtype |
Displays the data type of one column. |
df.info() |
Displays DataFrame structure, non-null counts, data types, and memory information. |
series.astype("int64") |
Converts compatible values to a standard integer type. |
series.astype("Int64") |
Converts values to the nullable integer extension dtype. |
pd.to_numeric(series, errors="coerce") |
Converts numeric values and turns invalid entries into NaN. |
pd.to_datetime(series) |
Converts compatible date strings into datetime values. |
pd.to_datetime(series, errors="coerce") |
Converts invalid dates to NaT. |
series.isna() |
Identifies missing values. |
series.astype("category") |
Converts a column to categorical data. |
series.dt.year |
Extracts the year from a datetime Series. |
20. Practice Questions and Exercises
Concept Questions
- What is a data type, and why does it matter in Pandas?
- What is the difference between
int64andInt64? - When should you use
astype()instead ofpd.to_numeric()? - What happens when
errors="coerce"is used for numeric conversion? - What is the difference between
NaNandNaT? - Why should ambiguous date strings be parsed with an explicit format?
- Why is direct Boolean conversion of text such as
"False"potentially misleading?
Hands-On Exercises
- Create a DataFrame with marks stored as strings and convert them to integers.
- Use
pd.to_numeric()to convert a column containing invalid numeric text. - Identify the original values that became missing after numeric conversion.
- Convert a date column containing valid and invalid strings to datetime values.
- Extract the year and month from the converted date column.
- Convert a text column containing
"True"and"False"into Boolean values using an explicit mapping. - Convert a repeated category column to the
categorydtype. - Write a complete workflow that checks data types, converts values, counts missing values, and prints the cleaned DataFrame.
Summary
Correct data types are essential for accurate data analysis.
Use dtypes and info() to inspect
columns, astype() to convert compatible values,
pd.to_numeric() for numeric conversion,
and pd.to_datetime() for date conversion.
Use nullable types when missing values must be preserved,
and investigate invalid values produced by coercion. Always
validate the converted columns and choose a missing-value
strategy that preserves the meaning of the original data.