Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Data Inspection in Pandas

Learn how to explore a dataset, inspect rows and columns, understand data types, identify missing values, and summarize data.

1. What Is Data Inspection?

Data inspection is the process of examining a dataset before performing analysis, visualization, or machine learning. It helps us understand what information is available and whether the data is suitable for our task.

For example, suppose Samadhan receives a CSV file containing student names, marks, ages, and cities. Before calculating results, he should check the records, column names, data types, and missing values.

Why do we inspect data?

  • Understand the size and structure of the dataset.
  • Preview records without printing the entire dataset.
  • Check whether columns contain the expected data types.
  • Find missing or incomplete values.
  • Understand numerical values through statistical summaries.
  • Identify potential data-quality problems before analysis.
Best practice: After loading a dataset, begin with head(), shape, info(), and isna().sum(). These give you a useful first overview.

2. Create Our Sample Dataset

We will use one DataFrame throughout this lesson so that every inspection method works on the same data.

import pandas as pd

data = {
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav", "Neha"],
    "Marks": [85, 92, 88, 76, 95],
    "Age": [21, 22, 20, 23, 21],
    "City": ["Pune", "Mumbai", "Nashik", "Pune", "Mumbai"]
}

df = pd.DataFrame(data)

print(df)

Output

       Name  Marks  Age    City
0  Samadhan     85   21    Pune
1   Pradnya     92   22  Mumbai
2  Samiksha     88   20  Nashik
3     Aarav     76   23    Pune
4      Neha     95   21  Mumbai

Our dataset contains five student records and four columns.

Column Meaning Expected data
Name Student name Text
Marks Student marks Numeric
Age Student age Numeric
City Student city Text

3. Visual Guide: Understanding a DataFrame

A DataFrame is a two-dimensional table with rows and columns. Each row represents a record, while each column represents a variable or field.

Pandas DataFrame Index Name Marks Age City 0 Samadhan 85 21 Pune 1 Pradnya 92 22 Mumbai 2 Samiksha 88 20 Nashik 3 Aarav 76 23 Pune 4 Neha 95 21 Mumbai Rows = records | Columns = fields | Index = row labels

Image 1: A DataFrame consists of row indexes, column labels, and the values stored in each cell.

Important: The index shown as 0, 1, 2, 3, and 4 is the default row index. It is not automatically a separate data column.

4. Preview Rows with head() and tail()

When a dataset contains thousands or millions of records, printing the entire DataFrame is usually unnecessary. Pandas lets you preview selected rows.

4.1 The head() method

head() displays the first five rows by default. You can provide a number to control how many rows are returned.

print(df.head())
print(df.head(3))

Output of df.head(3)

       Name  Marks  Age    City
0  Samadhan     85   21    Pune
1   Pradnya     92   22  Mumbai
2  Samiksha     88   20  Nashik

4.2 The tail() method

tail() displays the last five rows by default. It is useful for checking the end of a file or dataset.

print(df.tail(2))

Output

    Name  Marks  Age    City
3  Aarav     76   23    Pune
4   Neha     95   21  Mumbai

Learn these methods in detail on the dedicated Head and Tail lesson .

5. Understand DataFrame Shape and Columns

5.1 The shape attribute

The shape attribute returns a tuple containing the number of rows followed by the number of columns.

print(df.shape)

Output

(5, 4)

This means the DataFrame has five rows and four columns. Notice that shape is an attribute, so we use df.shape, not df.shape().

5.2 Count rows and columns separately

rows = df.shape[0]
columns = df.shape[1]

print("Rows:", rows)
print("Columns:", columns)

Output

Rows: 5
Columns: 4

5.3 Inspect column names

print(df.columns)

This returns the column labels: Name, Marks, Age, and City.

5.4 Check the index

print(df.index)

The default index is a range of labels starting at 0 and ending before 5. The exact representation may vary by Pandas version.

6. Inspect Data Types with dtypes

A DataFrame can contain different kinds of information. For example, student names are text, whereas marks and ages are numeric values.

print(df.dtypes)

Typical output

Name     object
Marks     int64
Age       int64
City     object
dtype: object

The exact dtype names can differ depending on the data and Pandas version. In this example, Name and City are commonly represented as object, while Marks and Age are integers.

Data type Meaning Typical example
int64 Integer values 85, 92, 76
float64 Decimal or floating-point values 85.5, 92.0
object Often used for text or mixed Python objects "Pune", "Mumbai"
bool Boolean values True, False
datetime64 Date and time values 2026-01-15
Common mistake: A column containing numbers may actually be stored as text. Always inspect data types before doing calculations.

7. Get Complete Structural Information with info()

The info() method provides a compact summary of the DataFrame. It reports the index range, number of columns, column names, non-null counts, data types, and memory usage.

df.info()

Example output

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 4 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   Name    5 non-null      object
 1   Marks   5 non-null      int64
 2   Age     5 non-null      int64
 3   City    5 non-null      object
dtypes: int64(2), object(2)

This is a simplified illustration of the output. The actual display may include different spacing, memory information, and additional details depending on the installed version.

What should you check?

  • Entries: How many rows are present?
  • Columns: Are the expected fields available?
  • Non-Null Count: Are any values missing?
  • Dtype: Are columns stored using appropriate types?
  • Memory Usage: How much memory does the DataFrame use?

Read the Data Information and Types lesson for more examples.

8. Visual Guide: The Data Inspection Workflow

A useful inspection workflow moves from a quick preview to structural checks, missing-value checks, and statistical summaries. The workflow helps you decide what to investigate next.

1. Load the Dataset 2. Preview Rows: head() / tail() 3. Inspect Shape, Columns and Data Types 4. Check Missing Values 5. Review Descriptive Statistics Ready for Cleaning and Analysis

Image 2: Follow a consistent inspection sequence before analyzing or transforming your dataset.

Remember: Inspection does not automatically clean or modify your data. It helps you discover what needs attention.

9. Check Missing Values

Missing values are common in real-world datasets. A missing mark, age, or city can affect calculations and reports.

Pandas provides isna() to identify missing values and sum() to count them.

data = {
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav"],
    "Marks": [85, None, 88, 76],
    "City": ["Pune", "Mumbai", None, "Pune"]
}

df_missing = pd.DataFrame(data)

print(df_missing.isna())
print(df_missing.isna().sum())

Output of df_missing.isna().sum()

Name     0
Marks    1
City     1
dtype: int64

This output indicates that the Marks column has one missing value and the City column has one missing value.

Tip: Inspect missing values before deciding whether to fill them, remove rows, or leave them unchanged. The appropriate choice depends on the dataset and the task.

Continue with the Missing Values lesson to learn about handling incomplete data.

10. Descriptive Statistics with describe()

The describe() method calculates useful summary statistics for numerical columns by default.

print(df.describe())

Example output

           Marks        Age
count   5.000000   5.000000
mean   87.200000  21.400000
std     7.395945   1.140175
min    76.000000  20.000000
25%    85.000000  21.000000
50%    88.000000  21.000000
75%    92.000000  22.000000
max    95.000000  23.000000

The values above are illustrative of the sample dataset. Depending on formatting and data types, Pandas may display slightly different decimal precision.

Statistic Meaning
count Number of non-missing values.
mean Arithmetic average.
std Standard deviation, a measure of spread.
min Smallest observed value.
25% First quartile.
50% Median, or second quartile.
75% Third quartile.
max Largest observed value.

You can learn the details in the Descriptive Statistics lesson .

11. Inspect Text and Numerical Columns Separately

Sometimes you want to inspect only numerical columns, or you want to summarize text columns such as student names and cities.

11.1 Numerical columns

print(df.select_dtypes(include="number").describe())

This selects columns with numerical data types before generating descriptive statistics.

11.2 Text columns

print(df.describe(include="object"))

For text columns, the summary can include the number of non-missing values, unique values, the most frequent value, and its frequency.

Note: The available summary fields depend on the data types in the selected columns. Use include="all" when you want Pandas to attempt a summary across all column types.
print(df.describe(include="all"))

12. Complete Data Inspection Example

The following example combines the most useful inspection statements into one simple workflow.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav", "Neha"],
    "Marks": [85, 92, 88, 76, 95],
    "Age": [21, 22, 20, 23, 21],
    "City": ["Pune", "Mumbai", "Nashik", "Pune", "Mumbai"]
})

# 1. Preview records
print(df.head())

# 2. Check dataset dimensions
print("Shape:", df.shape)

# 3. Check column names
print("Columns:", df.columns.tolist())

# 4. Check data types
print(df.dtypes)

# 5. Inspect structure
df.info()

# 6. Count missing values
print(df.isna().sum())

# 7. Generate statistics
print(df.describe())
Practical workflow: Run these checks when you first load a dataset. Use the results to decide whether you need data cleaning, type conversion, filtering, or further analysis.

13. Common Mistakes to Avoid

Mistake 1: Calling attributes as methods

# Incorrect
df.shape()

# Correct
df.shape

Mistake 2: Confusing missing values with zero

A missing value does not mean the value is zero. Inspect missing data explicitly with df.isna().sum().

Mistake 3: Assuming all numeric-looking columns are numeric

Values imported from files may be stored as text. Check df.dtypes and investigate unexpected types.

Mistake 4: Printing a huge dataset

Use head(), tail(), or select specific columns instead of printing every record.

Mistake 5: Treating inspection as data cleaning

Inspection helps reveal problems; it does not automatically fix incorrect values or remove duplicates.

14. Hands-on Practice Exercise

Samiksha is working with a student dataset. Complete the following tasks using Pandas.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav"],
    "Marks": [85, None, 88, 76],
    "Age": [21, 22, 20, 23],
    "City": ["Pune", "Mumbai", None, "Pune"]
})

Tasks

  1. Display the first two records.
  2. Display the last two records.
  3. Print the number of rows and columns.
  4. Display all column names.
  5. Inspect the data types and structure.
  6. Count the missing values in each column.
  7. Calculate descriptive statistics for numerical columns.

Suggested solution

print(df.head(2))
print(df.tail(2))

print("Shape:", df.shape)
print("Columns:", df.columns.tolist())

print(df.dtypes)
df.info()

print(df.isna().sum())
print(df.describe())

15. Revision Questions

  1. What is data inspection, and why is it useful?
  2. What is the difference between head() and tail()?
  3. What does df.shape return?
  4. How do you display all column names?
  5. What information does df.dtypes provide?
  6. What is the purpose of df.info()?
  7. How can you count missing values in every column?
  8. What is the purpose of df.describe()?
  9. How can you inspect numerical columns separately?
  10. Does data inspection automatically modify the dataset?

Lesson Summary

Data inspection is an essential first step in a Pandas workflow. It helps you understand the contents, structure, types, and quality of a dataset before analysis.

Use head() and tail() to preview rows, shape to check dimensions, columns to inspect labels, dtypes and info() to understand structure, isna().sum() to count missing values, and describe() to summarize data.

After completing this overview, study each topic in detail: Head and Tail, Data Information and Types, and Descriptive Statistics.