Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Data Information and Types

Understand DataFrame structure, dimensions, column data types, missing values, and memory usage.

1. Introduction

When we load a dataset into Pandas, we need to understand its structure before performing calculations or modifying values. A dataset may contain thousands of rows, many columns, missing values, or columns stored with unexpected data types.

Pandas provides several methods and attributes to inspect these details without manually checking every record.

What will we learn?

  • info() — inspect the overall DataFrame structure.
  • dtypes — examine the type of each column.
  • shape — find the number of rows and columns.
  • columns — inspect column labels.
  • index — inspect row labels.
  • memory_usage() — estimate memory used by columns.
  • isna().sum() — count missing values.
Best practice: Use these inspection tools immediately after loading a new dataset. They help you identify structural problems before cleaning or analysis.

2. Create a Sample DataFrame

We will use student information to understand each inspection method with the same example.

import pandas as pd

data = {
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav", "Neha"],
    "Marks": [85, 92, 88, 76, 95],
    "Age": [21, 22, 20, 23, 21],
    "City": ["Pune", "Mumbai", "Nashik", "Pune", "Mumbai"]
}

df = pd.DataFrame(data)

print(df)

Output

       Name  Marks  Age    City
0  Samadhan     85   21    Pune
1   Pradnya     92   22  Mumbai
2  Samiksha     88   20  Nashik
3     Aarav     76   23    Pune
4      Neha     95   21  Mumbai

Our DataFrame contains five records and four columns. We can now inspect its dimensions, structure, and data types.

3. Understanding DataFrame Structure

A DataFrame is a two-dimensional labeled data structure. It contains rows, columns, labels, and values.

DataFrame Structure Index Name Marks Age City 0 Samadhan 85 21 Pune 1 Pradnya 92 22 Mumbai 2 Samiksha 88 20 Nashik 3 Aarav 76 23 Pune Rows + Columns + Index + Values

Image 1: The structure of a Pandas DataFrame.

Important components

  • Rows: Individual records.
  • Columns: Fields such as Name, Marks, Age, and City.
  • Index: Labels used to identify rows.
  • Values: The actual data stored in each cell.

4. Using shape to Count Rows and Columns

The shape attribute returns a tuple containing the number of rows and columns, in that order.

print(df.shape)

Output

(5, 4)

The first number, 5, represents rows. The second number, 4, represents columns.

Get row and column counts separately

print("Rows:", df.shape[0])
print("Columns:", df.shape[1])

Output

Rows: 5
Columns: 4
Remember: shape is an attribute, not a method. Write df.shape, not df.shape().

5. Inspect Column Names with columns

The columns attribute returns the labels used for the DataFrame's columns.

print(df.columns)

The output contains these labels:

Index(['Name', 'Marks', 'Age', 'City'], dtype='object')

Convert column labels into a Python list

print(df.columns.tolist())

Output

['Name', 'Marks', 'Age', 'City']

Check whether a column exists

if "Marks" in df.columns:
    print("Marks column exists")

Output

Marks column exists

This check is useful when working with datasets whose column names may change between files.

6. Inspect the Index

The index attribute returns the row labels of a DataFrame. Pandas normally creates a default integer index when you construct a DataFrame without specifying one.

print(df.index)

In this example, the index is a range beginning at 0 and ending before 5.

Check the number of rows

print(len(df.index))

Output

5
Important: The index is not automatically a normal data column. It can be converted into a column using reset_index() when required.

7. Understanding dtypes

The dtypes attribute displays the data type of each column. This helps you verify that values are stored in an appropriate format.

print(df.dtypes)

Typical output

Name     object
Marks     int64
Age       int64
City     object
dtype: object
Data type Meaning Example
int64 Integer numbers 85, 92, 76
float64 Floating-point numbers 85.5, 92.0
object Often text or mixed Python objects "Pune", "Mumbai"
bool True or False values True, False
datetime64 Date and time values 2026-01-15

Actual dtype names can vary with the values, the way data was imported, and the Pandas version.

8. Complete Information with info()

The info() method prints a summary of the DataFrame. It includes information about the index, columns, non-null values, data types, and memory usage.

df.info()

Example output

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 4 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   Name    5 non-null      object
 1   Marks   5 non-null      int64
 2   Age     5 non-null      int64
 3   City    5 non-null      object
dtypes: int64(2), object(2)

This output is illustrative. Your actual output may also display memory usage and formatting details that vary by Pandas version and configuration.

What should you look for?

  • RangeIndex: The row index information.
  • Data columns: The number of columns.
  • Non-Null Count: The number of non-missing entries per column.
  • Dtype: The type of each column.
  • Memory usage: An estimate of the memory used.

9. Image 2: Reading DataFrame Information

This diagram shows how the main inspection attributes and methods help you understand different aspects of a dataset.

Pandas DataFrame df.shape Rows and columns df.dtypes Column data types df.info() Structure and null counts df.columns Column labels df.isna().sum() Missing values df.memory_usage() Memory per column Understand Dataset Quality and Structure

Image 2: Each Pandas inspection command answers a different question.

10. Count Missing Values

Missing values can affect calculations and data quality. Use isna() to identify missing entries and sum() to count them by column.

df_missing = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav"],
    "Marks": [85, None, 88, 76],
    "City": ["Pune", "Mumbai", None, "Pune"]
})

print(df_missing.isna().sum())

Output

Name     0
Marks    1
City     1
dtype: int64

There is one missing value in the Marks column and one in the City column.

Remember: Counting missing values does not fill or remove them. It only reports where missing data exists.

11. Check Memory Usage

The memory_usage() method reports estimated memory usage for each DataFrame column. This becomes useful when working with large datasets.

print(df.memory_usage())

By default, the index is included in the returned Series. Exact byte counts depend on the column data types and index.

Include object-column contents

print(df.memory_usage(deep=True))

Setting deep=True asks Pandas to inspect deeper memory usage for object-backed data. This can provide a more useful estimate for text-heavy DataFrames.

Note: Memory figures are estimates, not a complete measurement of every memory cost associated with your Python application.

12. Inspect a Single Column

You can inspect one column rather than the entire DataFrame. This is useful when you only need to understand a specific field.

Check the Marks data type

print(df["Marks"].dtype)

Count non-missing values

print(df["Marks"].count())

Inspect unique values

print(df["City"].unique())

These examples help you examine a column's type, the number of non-missing values, and its distinct values.

13. Complete Inspection Workflow

The following example combines the most important checks into one reusable workflow.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav"],
    "Marks": [85, 92, None, 76],
    "Age": [21, 22, 20, 23],
    "City": ["Pune", "Mumbai", "Nashik", "Pune"]
})

print("Shape:")
print(df.shape)

print("\\nColumns:")
print(df.columns.tolist())

print("\\nData types:")
print(df.dtypes)

print("\\nMissing values:")
print(df.isna().sum())

print("\\nDataFrame information:")
df.info()

print("\\nMemory usage:")
print(df.memory_usage(deep=True))

Use this workflow when you need an initial overview of an unfamiliar dataset before cleaning or analysis.

14. Common Mistakes to Avoid

Mistake 1: Calling shape like a function

# Incorrect
df.shape()

# Correct
df.shape

Mistake 2: Confusing dtypes with dtype

Use df.dtypes for all DataFrame columns and df["Marks"].dtype for one selected column.

Mistake 3: Assuming info() returns a DataFrame

info() prints a structural summary. It is primarily intended for displaying information, rather than returning a table for further calculations.

Mistake 4: Ignoring missing values

A column can contain missing values even when the DataFrame loads successfully. Always check non-null counts and isna().sum().

Mistake 5: Treating memory estimates as exact application usage

memory_usage() measures DataFrame-related memory estimates, not the entire runtime memory of your application.

15. Hands-on Practice Exercise

Pradnya has received a dataset containing student details. Write Pandas statements to complete the following tasks.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav"],
    "Marks": [85, None, 88, 76],
    "Age": [21, 22, 20, 23],
    "City": ["Pune", "Mumbai", None, "Pune"]
})

Tasks

  1. Find the number of rows and columns.
  2. Display all column names as a Python list.
  3. Display the data type of every column.
  4. Display the DataFrame's structural information.
  5. Count missing values in every column.
  6. Display memory usage for each column.
  7. Check whether the Marks column exists.

Suggested solution

print(df.shape)
print(df.columns.tolist())
print(df.dtypes)

df.info()

print(df.isna().sum())
print(df.memory_usage(deep=True))

print("Marks" in df.columns)

16. Revision Questions

  1. What does df.shape return?
  2. How do you get all DataFrame column names?
  3. What is the difference between dtype and dtypes?
  4. What information does df.info() provide?
  5. How can you count missing values in each column?
  6. What does memory_usage() measure?
  7. Why might memory_usage(deep=True) be useful?
  8. Does info() automatically clean a dataset?
  9. How can you check whether a column exists?
  10. Why should you inspect data types before performing calculations?

Summary

Pandas provides several tools to understand a DataFrame's structure and quality. Use shape for dimensions, columns for column labels, index for row labels, and dtypes for column types.

Use info() to view structural information, isna().sum() to count missing values, and memory_usage() to estimate memory consumption.

These methods help you understand your dataset before moving on to cleaning, transformation, and statistical analysis.