Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Introduction to Pandas

Learn what Pandas is, why it is important in Python, and how Series and DataFrames help you organize, explore, clean, and analyze structured data.

1. Learning Objectives

By the end of this lesson, you will be able to:

  • Define Pandas and explain its purpose.
  • Understand why data analysis is important.
  • Recognize the difference between Python lists and Pandas structures.
  • Understand Series and DataFrames at a basic level.
  • Identify common uses of Pandas in real-world projects.
  • Write a simple Python program using Pandas.

2. What Is Pandas?

Pandas is an open-source Python library used for data manipulation and analysis. It provides convenient data structures and tools for working with tabular and labelled data.

With Pandas, you can load information from files, inspect rows and columns, filter records, handle missing values, calculate statistics, combine datasets, and prepare data for reports or machine learning.

A Practical Example

Imagine that a teacher has marks for several students. Instead of manually calculating the average, Pandas can store the records in a DataFrame and calculate the average marks using a single instruction.

import pandas as pd

data = {
    "Student": ["Aarav", "Priya", "Rahul", "Neha"],
    "Marks": [75, 88, 92, 65]
}

df = pd.DataFrame(data)

print(df)
print("Average Marks:", df["Marks"].mean())

Output:

  Student  Marks
0   Aarav     75
1   Priya     88
2   Rahul     92
3    Neha     65

Average Marks: 80.0
Tip: The statement import pandas as pd imports the Pandas library and assigns it the commonly used alias pd.

3. Why Do We Need Pandas?

Real-world data often comes from spreadsheets, databases, CSV files, websites, and business applications. This data may contain thousands of records, missing values, duplicate rows, and inconsistent formats.

Pandas provides tools to handle these tasks in a clear and repeatable way.

Common Tasks You Can Perform

  • Read data from CSV, Excel, and JSON files.
  • Inspect the first and last rows of a dataset.
  • Select particular columns or rows.
  • Filter records using conditions.
  • Find and handle missing values.
  • Remove duplicate records.
  • Group data and calculate summaries.
  • Combine multiple datasets.
  • Prepare data for visualization and reporting.

Example: Filter Students Who Passed

import pandas as pd

df = pd.DataFrame({
    "Student": ["Aarav", "Priya", "Rahul", "Neha"],
    "Marks": [75, 88, 92, 35]
})

passed = df[df["Marks"] >= 40]

print(passed)

Output:

  Student  Marks
0   Aarav     75
1   Priya     88
2   Rahul     92

The expression df["Marks"] >= 40 checks every student's marks. Pandas returns only the rows that satisfy the condition.

4. Understanding Series and DataFrames

Pandas mainly provides two data structures that beginners should understand first: Series and DataFrame.

Series

A Series is a one-dimensional labelled data structure. You can think of it as a single column of values with an associated index.

import pandas as pd

marks = pd.Series([75, 88, 92])

print(marks)

Output:

0    75
1    88
2    92
dtype: int64

DataFrame

A DataFrame is a two-dimensional labelled data structure with rows and columns. It is useful for representing tables such as student records, sales reports, and employee information.

students = pd.DataFrame({
    "Name": ["Aarav", "Priya", "Rahul"],
    "Marks": [75, 88, 92]
})

print(students)

Output:

    Name  Marks
0  Aarav     75
1  Priya     88
2  Rahul     92
Series Marks 75 88 92 One-dimensional DataFrame Name Marks Aarav 75 Priya 88 Rahul 92 Rows and columns

Visual 1: A Series represents one labelled dimension; a DataFrame represents a table with rows and columns.

Quick Comparison

Feature Series DataFrame
Dimensions One-dimensional Two-dimensional
Structure One labelled sequence Rows and columns
Typical use One set of values Tabular datasets
Example Student marks column Student names and marks table

5. Pandas vs Python Lists

Python lists are general-purpose containers. Pandas Series and DataFrames provide additional features designed for labelled data, table operations, filtering, and analysis.

Feature Python List Pandas
Data structure Sequence of items Series and DataFrame
Column labels Not built in Available in DataFrames
Filtering tables Usually requires additional logic Convenient conditional selection
Missing data Requires custom handling Built-in missing-data tools
File handling Requires other code or libraries Convenient CSV, Excel and JSON functions
Remember: Python lists remain useful for many tasks. Pandas is especially convenient when you need labelled, tabular data and common data-analysis operations.

6. Where Is Pandas Used?

Business and Sales Analysis

Analyze revenue, compare products, summarize sales by region, and prepare monthly reports.

Education

Study student marks, calculate class averages, identify missing scores, and summarize examination results.

Finance

Organize transaction records, calculate financial summaries, and analyze time-based datasets.

Data Science and Machine Learning

Inspect datasets, clean inconsistent values, transform columns, and prepare structured data for later analysis.

Reporting and Automation

Combine files, generate summaries, and automate repetitive data preparation tasks.

7. Your First Pandas Program

Let's create a small employee dataset and calculate the total and average salary. This example demonstrates how a DataFrame stores records and how Pandas calculates values from a column.

import pandas as pd

employees = pd.DataFrame({
    "Name": ["Amit", "Sneha", "Rohan"],
    "Salary": [30000, 45000, 40000]
})

print("Employee Data:")
print(employees)

print("Total Salary:", employees["Salary"].sum())
print("Average Salary:", employees["Salary"].mean())

Expected Output:

Employee Data:
    Name  Salary
0   Amit   30000
1  Sneha   45000
2  Rohan   40000

Total Salary: 115000
Average Salary: 38333.333333333336

Understanding the Code

  1. import pandas as pd imports the Pandas library.
  2. pd.DataFrame() creates a table from the supplied data.
  3. The Name and Salary keys become column labels.
  4. employees["Salary"] selects the Salary column.
  5. .sum() calculates the total salary.
  6. .mean() calculates the average salary.
Setup reminder: If Python reports ModuleNotFoundError: No module named 'pandas', Pandas is not installed in the Python environment currently being used. We will cover installation in the next lesson.

8. Common Beginner Mistakes

Mistake 1: Forgetting to Import Pandas

You must import Pandas before using the pd alias or calling pd.DataFrame().

Mistake 2: Misspelling Column Names

Column labels must match the names in the DataFrame. For example, Salary and salary are different labels.

import pandas as pd

df = pd.DataFrame({
    "Salary": [30000, 45000]
})

print(df["Salary"])

Mistake 3: Assuming a DataFrame Is a Python List

A DataFrame is a table with labelled columns and an index. You can select columns, filter rows, and use methods such as head(), info(), and describe() to inspect it.

9. Classroom Practical Activity

Activity: Analyze Student Marks

Create a DataFrame with the names and marks of five students. Display the table and calculate the total, average, highest, and lowest marks.

Starter Code

import pandas as pd

students = pd.DataFrame({
    "Name": ["Aarav", "Priya", "Rahul", "Neha", "Karan"],
    "Marks": [72, 85, 91, 64, 78]
})

print(students)

print("Total:", students["Marks"].sum())
print("Average:", students["Marks"].mean())
print("Highest:", students["Marks"].max())
print("Lowest:", students["Marks"].min())

Practice Questions

  1. What is Pandas used for?
  2. What does import pandas as pd mean?
  3. What is a Pandas Series?
  4. What is a DataFrame?
  5. How do you select a column from a DataFrame?
  6. Which method calculates the average of a numerical column?
  7. Name three real-world uses of Pandas.

Summary

Pandas is a Python library for data manipulation and analysis. It provides Series for one-dimensional labelled data and DataFrames for tabular data arranged in rows and columns.

In this lesson, you learned why Pandas is useful, where it is used, how to create a DataFrame, select a column, filter records, and calculate basic statistics. You are now ready to prepare your Python environment and start using Pandas.