Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Data Types and Type Conversion in Pandas

Understand Pandas data types, inspect column types, convert strings to numbers, work with dates, handle invalid values, and prepare datasets for accurate analysis.

1. What Are Data Types?

A data type describes the kind of value stored in a column. A column might contain text, integers, decimal numbers, dates, Boolean values, or categorical labels.

Pandas uses data types to determine how values are stored, compared, calculated, and processed. Incorrect data types can cause calculation errors or prevent mathematical operations.

Example: Student Records

Samadhan receives a dataset in which marks are stored as text. The values look numeric, but Pandas may treat them as strings. Before calculating the average marks, he needs to convert the column to a suitable numeric type.

Raw dataset Student Marks Pradnya "92" Samiksha "88" Text values Convert astype() to_numeric() to_datetime() Validate Clean dataset Student Marks Pradnya 92 Samiksha 88 Numeric values Inspect → Convert → Validate Conversion must preserve the meaning of the original data.

Correct data types enable reliable calculations, comparisons, and analysis.

2. Common Pandas Data Types

Data Type Description Example
int64 Integer values without a fractional part. 10, 250
float64 Numeric values that can include decimals. 85.5, 19.99
object Often used for text or mixed Python objects. "Python"
string Pandas string extension dtype for text. "Samadhan"
bool Boolean values. True, False
datetime64[ns] Date and time values. 2026-10-29
category Repeated values stored as categorical data. "Beginner", "Advanced"
Int64 Nullable integer extension dtype that supports missing values. 10, <NA>
Important: int64 and Int64 are not identical. The capitalized nullable extension dtype Int64 can represent missing values using pd.NA.

3. Create a Sample DataFrame

This example contains marks stored as text, salary values with currency symbols, and dates stored as strings.

import pandas as pd

df = pd.DataFrame({
    "Student": [
        "Samadhan",
        "Pradnya",
        "Samiksha",
        "Aarav"
    ],
    "Marks": [
        "85",
        "92",
        "88",
        "79"
    ],
    "Salary": [
        "50000",
        "62000",
        "58000",
        "45000"
    ],
    "JoinDate": [
        "2025-01-10",
        "2025-02-15",
        "2025-03-20",
        "2025-04-05"
    ]
})

print(df)

Expected Output

    Student Marks Salary    JoinDate
0  Samadhan    85  50000  2025-01-10
1   Pradnya    92  62000  2025-02-15
2  Samiksha    88  58000  2025-03-20
3     Aarav    79  45000  2025-04-05

The values in Marks, Salary, and JoinDate look numeric or date-like, but they were provided as strings. We will inspect and convert them.

4. Inspect Data Types with dtypes

The dtypes attribute displays the data type of every column. It is a useful first step when investigating unexpected calculations or conversion problems.

print(df.dtypes)

Expected Output

Student     object
Marks       object
Salary      object
JoinDate    object
dtype: object

All four columns have the object dtype in this example because the columns were created from strings.

Inspect One Column

print(df["Marks"].dtype)

Expected Output:

object

Inspect DataFrame Information

df.info()

The output includes the number of rows, column names, non-null counts, data types, and memory usage. Exact memory usage and formatting may vary between Pandas versions.

5. Convert a Column Using astype()

The astype() method converts values to a specified data type when the values are compatible with that type.

Convert Marks to Integers

df["Marks"] = df["Marks"].astype("int64")

print(df["Marks"].dtype)
print(df["Marks"])

Expected Output

int64
0    85
1    92
2    88
3    79
Name: Marks, dtype: int64

The marks column is now numeric, so mathematical calculations can be performed.

Calculate the Average

print(df["Marks"].mean())

Expected Output:

86.0

Convert Salary to Float

df["Salary"] = df["Salary"].astype("float64")

print(df["Salary"].dtype)

Expected Output:

float64
Limitation: astype() raises an error if a value cannot be converted to the requested type. For messy numeric text, consider pd.to_numeric().

6. Convert Text to Numbers with to_numeric()

The pd.to_numeric() function converts compatible values into numeric types. It also provides an errors parameter for handling invalid values.

Example with Invalid Data

marks = pd.Series([
    "85",
    "92",
    "Not Available",
    "79"
])

numeric_marks = pd.to_numeric(
    marks,
    errors="coerce"
)

print(numeric_marks)

Expected Output

0    85.0
1    92.0
2     NaN
3    79.0
dtype: float64

The invalid value "Not Available" becomes NaN. Because the result contains a missing numeric value, Pandas uses a floating-point dtype in this example.

Understanding the errors Parameter

Option Behavior
errors="raise" Raises an error when a value cannot be converted. This is the default.
errors="coerce" Converts invalid values to NaN.
errors="ignore" Legacy behavior that may return the original input when conversion fails. Prefer explicit handling with coerce or validation.
Practical tip: Use errors="coerce" when cleaning untrusted numeric input, then inspect the values that became missing. Do not silently discard conversion failures.

7. Find Values That Cannot Be Converted

When using errors="coerce", compare the original values with the converted results to identify invalid entries.

raw_marks = pd.Series([
    "85",
    "92",
    "Not Available",
    "79",
    "unknown"
])

converted_marks = pd.to_numeric(
    raw_marks,
    errors="coerce"
)

invalid_mask = converted_marks.isna() & raw_marks.notna()

print(raw_marks[invalid_mask])

Expected Output

2    Not Available
4          unknown
dtype: object

This method identifies non-missing original values that failed numeric conversion. It helps you decide whether the values should be corrected, mapped to missing values, or investigated further.

Count Invalid Values

print("Invalid values:", invalid_mask.sum())

Expected Output:

Invalid values: 2

8. Handle Missing Values During Conversion

A column may contain both invalid text and legitimate missing values. After converting to numeric values, choose an appropriate missing-value strategy for the analysis.

Example

marks = pd.Series([
    "85",
    "92",
    "Not Available",
    None,
    "79"
])

numeric_marks = pd.to_numeric(
    marks,
    errors="coerce"
)

print(numeric_marks)

Expected Output

0    85.0
1    92.0
2     NaN
3     NaN
4    79.0
dtype: float64

Option A: Remove Rows with Missing Marks

valid_marks = numeric_marks.dropna()

print(valid_marks.tolist())

Expected Output:

[85.0, 92.0, 79.0]

Option B: Fill Missing Values with a Chosen Value

filled_marks = numeric_marks.fillna(0)

print(filled_marks.tolist())

Expected Output:

[85.0, 92.0, 0.0, 0.0, 79.0]
Important: Filling missing marks with zero changes the meaning of the data. Zero may mean a real score, not an unknown value. Choose a strategy based on the dataset's meaning and the intended analysis.

9. Use Nullable Integer Types

Traditional NumPy integer types such as int64 cannot represent missing values directly in a standard integer array. Pandas provides the nullable extension dtype Int64, which supports pd.NA.

marks = pd.Series([
    85,
    92,
    None,
    79
])

marks_nullable = marks.astype("Int64")

print(marks_nullable)
print(marks_nullable.dtype)

Expected Output

0      85
1      92
2    <NA>
3      79
dtype: Int64
Int64

The values remain integer-like, and the missing entry is represented by <NA> instead of forcing the entire column to use floating-point numbers.

Remember the capitalization: Int64 is a Pandas nullable integer dtype; int64 is a standard NumPy integer dtype.

10. Convert Strings to Dates with to_datetime()

Dates stored as strings cannot reliably support date calculations until they are converted into datetime values. Use pd.to_datetime().

Convert the JoinDate Column

df["JoinDate"] = pd.to_datetime(
    df["JoinDate"]
)

print(df["JoinDate"].dtype)

Expected Output:

datetime64[ns]

Extract the Year

df["JoinYear"] = df["JoinDate"].dt.year

print(df[["Student", "JoinYear"]])

Expected Output

    Student  JoinYear
0  Samadhan      2025
1   Pradnya      2025
2  Samiksha      2025
3     Aarav      2025

Extract Month and Day

df["JoinMonth"] = df["JoinDate"].dt.month
df["JoinDay"] = df["JoinDate"].dt.day

print(df[["Student", "JoinMonth", "JoinDay"]])

Expected Output

    Student  JoinMonth  JoinDay
0  Samadhan          1       10
1   Pradnya          2       15
2  Samiksha          3       20
3     Aarav          4        5

11. Handle Invalid Dates

Some datasets contain malformed dates, unknown labels, or inconsistent date formats. With errors="coerce", invalid dates become NaT, Pandas' missing datetime value.

dates = pd.Series([
    "2025-01-10",
    "2025-02-15",
    "not-a-date",
    "2025-04-05"
])

parsed_dates = pd.to_datetime(
    dates,
    errors="coerce"
)

print(parsed_dates)

Expected Output

0   2025-01-10
1   2025-02-15
2          NaT
3   2025-04-05
dtype: datetime64[ns]

The invalid date becomes NaT. You can inspect the affected records before deciding how to handle them.

Find Invalid Date Entries

invalid_dates = dates[parsed_dates.isna() & dates.notna()]

print(invalid_dates)

Expected Output:

2    not-a-date
dtype: object
Date-format caution: When dates use ambiguous formats such as 03/04/2025, specify the intended format when known. This avoids interpreting the day and month incorrectly.

Parse a Known Date Format

dates = pd.Series([
    "31/01/2025",
    "15/02/2025"
])

parsed_dates = pd.to_datetime(
    dates,
    format="%d/%m/%Y"
)

print(parsed_dates)

Expected Output:

0   2025-01-31
1   2025-02-15
dtype: datetime64[ns]

12. Convert Values to Boolean

Boolean columns contain True or False. They are useful for flags, conditions, and filtering.

Create a Boolean Column from a Condition

marks = pd.Series([85, 35, 72, 28])

passed = marks >= 40

print(passed)

Expected Output

0     True
1    False
2     True
3    False
dtype: bool

A comparison produces Boolean values automatically. This is generally safer than converting arbitrary text directly to Boolean values.

Important: astype(bool) does not interpret strings such as "False" as the Boolean value False. A non-empty string is truthy in Python. Map known text labels explicitly when converting text flags.

Convert Text Flags Explicitly

flags = pd.Series([
    "True",
    "False",
    "True"
])

flag_map = {
    "True": True,
    "False": False
}

boolean_flags = flags.map(flag_map)

print(boolean_flags)
print(boolean_flags.dtype)

Expected Output

0     True
1    False
2     True
dtype: bool
bool

13. Convert Repeated Text to Category

Categorical data is useful for columns containing a limited set of repeated labels, such as department, product type, or student level. It can reduce memory use when many rows repeat the same values.

levels = pd.Series([
    "Beginner",
    "Advanced",
    "Beginner",
    "Intermediate",
    "Advanced"
])

levels = levels.astype("category")

print(levels.dtype)
print(levels.cat.categories.tolist())

Expected Output

category
['Advanced', 'Beginner', 'Intermediate']

Pandas stores the unique categories and associates each row with a category code. Category ordering is not automatically a meaningful ranking; define an ordered categorical explicitly when the order matters.

Create an Ordered Category

level_type = pd.CategoricalDtype(
    categories=[
        "Beginner",
        "Intermediate",
        "Advanced"
    ],
    ordered=True
)

levels_ordered = levels.astype(level_type)

print(levels_ordered.cat.ordered)

Expected Output:

True

14. Convert Multiple Columns Together

You can convert several columns in one workflow. For example, convert marks to integers and salary to floating-point numbers.

df = pd.DataFrame({
    "Student": ["Samadhan", "Pradnya", "Samiksha"],
    "Marks": ["85", "92", "88"],
    "Salary": ["50000.50", "62000.75", "58000.25"]
})

df = df.astype({
    "Marks": "int64",
    "Salary": "float64"
})

print(df.dtypes)

Expected Output

Student     object
Marks        int64
Salary     float64
dtype: object

Passing a dictionary to astype() lets you specify a different target type for each selected column.

Inspect the Converted DataFrame

print(df)
print("Average marks:", df["Marks"].mean())
print("Average salary:", df["Salary"].mean())

Expected Output

    Student  Marks    Salary
0  Samadhan     85  50000.50
1   Pradnya     92  62000.75
2  Samiksha     88  58000.25

Average marks: 88.33333333333333
Average salary: 56667.166666666664

15. Convert Currency Text to Numeric Values

Financial datasets sometimes store amounts as text containing currency symbols or thousands separators. Remove known formatting characters before converting the values.

salary = pd.Series([
    "$50,000",
    "$62,500",
    "$58,750"
])

salary_clean = salary.str.replace(
    "$",
    "",
    regex=False
).str.replace(
    ",",
    "",
    regex=False
)

salary_numeric = pd.to_numeric(salary_clean)

print(salary_numeric)

Expected Output

0    50000
1    62500
2    58750
dtype: int64

The currency symbol and thousands separators are removed before conversion. The result is a numeric Series that supports arithmetic operations.

Production consideration: Financial data can include different currency symbols, decimal separators, negative amount conventions, and accounting formats. Define and validate the expected format instead of removing arbitrary characters.

16. Verify Data Types After Conversion

A successful assignment does not guarantee that the resulting values are appropriate for the analysis. Check data types, missing values, and sample records after conversion.

print(df.dtypes)

print("\nMissing values:")
print(df.isna().sum())

print("\nSample records:")
print(df.head())

The output displays the current data type of each column, the number of missing values, and the first five records. Use these checks to identify unexpected conversions or newly introduced missing values.

Validation Checklist

  • Are numeric columns actually numeric?
  • Are date columns stored as datetime values?
  • Did invalid conversions create unexpected missing values?
  • Were missing values handled according to the business rules?
  • Do the converted values still represent the original information correctly?

17. Complete Practical Data Type Cleaning Workflow

This example combines type inspection, numeric conversion, date conversion, invalid-value detection, and validation.

import pandas as pd

df = pd.DataFrame({
    "Student": [
        "Samadhan",
        "Pradnya",
        "Samiksha",
        "Aarav"
    ],
    "Marks": [
        "85",
        "92",
        "not available",
        "79"
    ],
    "JoinDate": [
        "2025-01-10",
        "2025-02-15",
        "invalid-date",
        "2025-04-05"
    ]
})

print("Original data types:")
print(df.dtypes)

# Convert marks to numeric values.
df["Marks"] = pd.to_numeric(
    df["Marks"],
    errors="coerce"
)

# Convert join dates to datetime values.
df["JoinDate"] = pd.to_datetime(
    df["JoinDate"],
    errors="coerce"
)

print("\nCleaned data:")
print(df)

print("\nUpdated data types:")
print(df.dtypes)

print("\nMissing values:")
print(df.isna().sum())

Expected Output

Original data types:
Student     object
Marks       object
JoinDate    object
dtype: object

Cleaned data:
    Student  Marks   JoinDate
0  Samadhan   85.0 2025-01-10
1   Pradnya   92.0 2025-02-15
2  Samiksha    NaN        NaT
3     Aarav   79.0 2025-04-05

Updated data types:
Student             object
Marks              float64
JoinDate     datetime64[ns]
dtype: object

Missing values:
Student     0
Marks       1
JoinDate    1
dtype: int64

The invalid mark becomes NaN, and the invalid date becomes NaT. Both conversions succeed without stopping the workflow, while the resulting missing values remain visible for further investigation.

18. Common Mistakes to Avoid

  • Converting numeric text without checking it: Values containing letters or unexpected symbols may fail conversion.
  • Confusing text with numbers: The string "100" is not the same data type as the integer 100.
  • Using int64 when missing values are present: Consider cleaning missing values first or using nullable Int64.
  • Ignoring conversion failures: Investigate values converted to NaN or NaT.
  • Parsing ambiguous dates without a format: Specify the intended format when the input convention is known.
  • Replacing unknown values with zero automatically: Zero may be a valid measurement rather than a missing-value marker.
  • Using Boolean conversion on text flags: Map text labels such as "True" and "False" explicitly.
  • Assuming categories are automatically ordered: Define an ordered categorical when the sequence has meaning.

19. Quick Revision

Method or Attribute Purpose
df.dtypes Displays the data type of every column.
df["Marks"].dtype Displays the data type of one column.
df.info() Displays DataFrame structure, non-null counts, data types, and memory information.
series.astype("int64") Converts compatible values to a standard integer type.
series.astype("Int64") Converts values to the nullable integer extension dtype.
pd.to_numeric(series, errors="coerce") Converts numeric values and turns invalid entries into NaN.
pd.to_datetime(series) Converts compatible date strings into datetime values.
pd.to_datetime(series, errors="coerce") Converts invalid dates to NaT.
series.isna() Identifies missing values.
series.astype("category") Converts a column to categorical data.
series.dt.year Extracts the year from a datetime Series.

20. Practice Questions and Exercises

Concept Questions

  1. What is a data type, and why does it matter in Pandas?
  2. What is the difference between int64 and Int64?
  3. When should you use astype() instead of pd.to_numeric()?
  4. What happens when errors="coerce" is used for numeric conversion?
  5. What is the difference between NaN and NaT?
  6. Why should ambiguous date strings be parsed with an explicit format?
  7. Why is direct Boolean conversion of text such as "False" potentially misleading?

Hands-On Exercises

  1. Create a DataFrame with marks stored as strings and convert them to integers.
  2. Use pd.to_numeric() to convert a column containing invalid numeric text.
  3. Identify the original values that became missing after numeric conversion.
  4. Convert a date column containing valid and invalid strings to datetime values.
  5. Extract the year and month from the converted date column.
  6. Convert a text column containing "True" and "False" into Boolean values using an explicit mapping.
  7. Convert a repeated category column to the category dtype.
  8. Write a complete workflow that checks data types, converts values, counts missing values, and prints the cleaned DataFrame.
Learning goal: You should now be able to inspect data types, choose the appropriate conversion method, identify invalid values, and verify the result before continuing with further analysis.

Summary

Correct data types are essential for accurate data analysis. Use dtypes and info() to inspect columns, astype() to convert compatible values, pd.to_numeric() for numeric conversion, and pd.to_datetime() for date conversion. Use nullable types when missing values must be preserved, and investigate invalid values produced by coercion. Always validate the converted columns and choose a missing-value strategy that preserves the meaning of the original data.