Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Histograms, Lines, Scatter and Box Plots

Learn how to analyse numerical data using histograms, line charts, scatter plots and box plots with Pandas and Matplotlib.

1. Introduction to Different Chart Types

Data visualization helps us understand information through charts and graphs. Different chart types are designed for different analytical purposes.

For example, a histogram helps us understand the distribution of student marks, a line chart displays monthly sales trends, a scatter plot identifies relationships between two variables, and a box plot helps identify outliers.

Important: Choosing the correct chart type is important because each chart communicates a different aspect of the dataset.

Charts Covered in This Lesson

Chart Purpose Example
Histogram Shows the frequency distribution of numerical data. Student marks distribution
Line Chart Shows trends and changes over a sequence. Monthly sales trends
Scatter Plot Shows the relationship between two numerical variables. Study hours and marks
Box Plot Shows spread, quartiles and potential outliers. Employee salary distribution

2. Creating Sample Dataset

Let us create a student performance dataset. We will use this dataset to understand different chart types.

import pandas as pd
import matplotlib.pyplot as plt

data = {
    'Student': [
        'Amit', 'Rahul', 'Priya', 'Sneha',
        'Omkar', 'Neha', 'Kiran', 'Pooja',
        'Rohit', 'Anjali'
    ],
    'StudyHours': [2, 3, 4, 5, 6, 7, 3, 8, 5, 9],
    'Marks': [45, 50, 55, 65, 70, 75, 52, 85, 68, 92],
    'Attendance': [65, 70, 75, 80, 85, 90, 72, 95, 82, 98]
}

df = pd.DataFrame(data)

print(df)

Expected Output

  Student  StudyHours  Marks  Attendance
0    Amit           2     45          65
1   Rahul           3     50          70
2   Priya           4     55          75
3   Sneha           5     65          80
4   Omkar           6     70          85
5    Neha           7     75          90
6   Kiran           3     52          72
7   Pooja           8     85          95
8   Rohit           5     68          82
9  Anjali           9     92          98

The dataset contains student names, study hours, marks and attendance percentages.

3. Histogram in Pandas

A histogram is a chart used to represent the frequency distribution of numerical data. It divides values into intervals called bins.

Each bar represents the number of observations that fall within a particular interval.

Example: Student Marks Distribution

df['Marks'].plot(
    kind='hist',
    bins=5,
    color='skyblue',
    edgecolor='black'
)

plt.title('Student Marks Distribution')
plt.xlabel('Marks')
plt.ylabel('Number of Students')
plt.show()

Explanation

  • df['Marks'] selects the Marks column.
  • kind='hist' creates a histogram.
  • bins=5 divides the data range into five intervals.
  • edgecolor='black' adds borders around the bars.
  • plt.xlabel() labels the horizontal axis.
  • plt.ylabel() labels the frequency axis.

Understanding Bins

Bins are intervals used to group numerical values. The number of bins controls how detailed the histogram looks.

# Histogram with 5 bins
df['Marks'].plot(kind='hist', bins=5)
plt.show()

# Histogram with 10 bins
df['Marks'].plot(kind='hist', bins=10)
plt.show()
Use Case: Educational institutes can use histograms to understand how student marks are distributed across different score ranges.

4. Histogram with Density

A density histogram displays the distribution of values using probability density rather than raw frequency. It is useful when comparing distributions with different sample sizes.

Example

df['Marks'].plot(
    kind='hist',
    bins=5,
    density=True,
    color='lightgreen',
    edgecolor='black'
)

plt.title('Marks Density Distribution')
plt.xlabel('Marks')
plt.ylabel('Density')
plt.show()

Frequency vs Density

Frequency Histogram Density Histogram
Shows the number of observations in each bin. Shows probability density.
Uses raw counts. Normalises the distribution.
Useful for understanding sample counts. Useful for comparing distributions.

5. Line Chart in Pandas

A line chart displays data points connected by straight lines. It is commonly used to show trends and changes over time or across an ordered sequence.

Example: Monthly Sales Data

sales_data = {
    'Month': [
        'January', 'February', 'March',
        'April', 'May', 'June'
    ],
    'Sales': [
        12000, 15000, 18000,
        14000, 22000, 25000
    ]
}

sales_df = pd.DataFrame(sales_data)

sales_df.plot(
    x='Month',
    y='Sales',
    kind='line',
    marker='o',
    color='blue'
)

plt.title('Monthly Sales Trend')
plt.xlabel('Month')
plt.ylabel('Sales')
plt.grid(True)
plt.show()

Explanation

  • kind='line' creates a line chart.
  • marker='o' displays circular markers at data points.
  • color='blue' sets the line colour.
  • plt.grid(True) displays grid lines.

Multiple Line Charts

We can display multiple numerical columns in the same line chart.

sales_df['Expenses'] = [
    8000, 9000, 11000, 8500, 13000, 15000
]

sales_df.plot(
    x='Month',
    y=['Sales', 'Expenses'],
    kind='line',
    marker='o'
)

plt.title('Sales and Expenses Trend')
plt.xlabel('Month')
plt.ylabel('Amount')
plt.legend()
plt.grid(True)
plt.show()
Use Case: Businesses use line charts to monitor monthly revenue, expenses, customer growth and product performance.

6. Scatter Plot in Pandas

A scatter plot displays individual data points using their X and Y coordinates. It is useful for identifying relationships, patterns and correlations between two numerical variables.

Example: Study Hours vs Student Marks

df.plot(
    kind='scatter',
    x='StudyHours',
    y='Marks',
    color='green'
)

plt.title('Study Hours vs Marks')
plt.xlabel('Study Hours')
plt.ylabel('Student Marks')
plt.grid(True)
plt.show()

Explanation

  • kind='scatter' creates a scatter plot.
  • x='StudyHours' selects the X-axis variable.
  • y='Marks' selects the Y-axis variable.
  • Each point represents one student's study hours and marks.

Understanding Relationships

Scatter plots can help us understand how two variables are related.

  • Positive relationship: Both variables tend to increase together.
  • Negative relationship: One variable tends to decrease when the other increases.
  • No clear relationship: There is no obvious pattern between the variables.
Important: A scatter plot can suggest a relationship, but it does not prove that one variable causes changes in another.

7. Customising Scatter Plots

We can customise scatter plots using parameters such as colour, marker size and transparency.

Example

df.plot(
    kind='scatter',
    x='StudyHours',
    y='Marks',
    color='purple',
    s=100,
    alpha=0.7
)

plt.title('Student Study Analysis')
plt.xlabel('Study Hours')
plt.ylabel('Marks')
plt.grid(True)
plt.show()

Important Parameters

Parameter Purpose
x Selects the X-axis column.
y Selects the Y-axis column.
color Sets the point colour.
s Controls marker size.
alpha Controls transparency.
marker Sets the marker shape when supported.

8. Box Plot in Pandas

A box plot is used to understand the distribution and spread of numerical data. It displays the median, quartiles and potential outliers.

Box plots are useful when comparing the distribution of values across different groups.

Example: Student Marks Box Plot

df['Marks'].plot(
    kind='box',
    color='blue'
)

plt.title('Student Marks Box Plot')
plt.ylabel('Marks')
plt.grid(True)
plt.show()

Understanding Box Plot Components

  • Minimum: The lowest non-outlier value.
  • Q1: The first quartile or 25th percentile.
  • Median: The middle value or 50th percentile.
  • Q3: The third quartile or 75th percentile.
  • Maximum: The highest non-outlier value.
  • Outliers: Values outside the whisker limits.
Note: In a standard box plot, whiskers generally extend to the most extreme observations within 1.5 times the interquartile range from the quartiles. Points beyond these limits are shown separately as potential outliers.

9. Understanding Interquartile Range (IQR)

The Interquartile Range (IQR) measures the spread of the middle 50% of the data.

Formula

IQR = Q3 - Q1

Where:

  • Q1: 25th percentile.
  • Q3: 75th percentile.

Outlier Boundaries

Lower Bound = Q1 - 1.5 * IQR

Upper Bound = Q3 + 1.5 * IQR

Example: Calculating IQR with Pandas

Q1 = df['Marks'].quantile(0.25)
Q3 = df['Marks'].quantile(0.75)

IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

print("Q1:", Q1)
print("Q3:", Q3)
print("IQR:", IQR)

print("Lower Bound:", lower_bound)
print("Upper Bound:", upper_bound)

Identify Potential Outliers

outliers = df[
    (df['Marks'] < lower_bound) |
    (df['Marks'] > upper_bound)
]

print(outliers)

This code selects observations whose marks are outside the calculated IQR boundaries.

10. Comparing Multiple Columns with Box Plot

Pandas allows us to create box plots for multiple numerical columns together. This helps compare their distributions.

Example

df[['StudyHours', 'Marks', 'Attendance']].plot(
    kind='box',
    figsize=(9, 6)
)

plt.title('Student Performance Comparison')
plt.ylabel('Values')
plt.grid(True)
plt.show()

This chart displays separate box plots for study hours, marks and attendance.

Important: When comparing columns with very different scales, consider using separate charts or appropriate scaling.

11. Creating Histograms for Multiple Columns

We can create histograms for several numerical columns to understand their individual distributions.

Example

df[['StudyHours', 'Marks', 'Attendance']].hist(
    figsize=(10, 7),
    bins=5,
    edgecolor='black'
)

plt.suptitle('Student Data Distribution')
plt.tight_layout()
plt.show()

This creates separate histograms for StudyHours, Marks and Attendance.

12. Complete Practical Example – Student Performance Analysis

In this example, we will create a student dataset and generate four different charts to analyse academic performance.

Complete Python Program

import pandas as pd
import matplotlib.pyplot as plt

# Create student dataset
data = {
    'Student': [
        'Amit', 'Rahul', 'Priya', 'Sneha',
        'Omkar', 'Neha', 'Kiran', 'Pooja',
        'Rohit', 'Anjali'
    ],
    'StudyHours': [2, 3, 4, 5, 6, 7, 3, 8, 5, 9],
    'Marks': [45, 50, 55, 65, 70, 75, 52, 85, 68, 92],
    'Attendance': [65, 70, 75, 80, 85, 90, 72, 95, 82, 98]
}

df = pd.DataFrame(data)

print(df)

# 1. Histogram
df['Marks'].plot(
    kind='hist',
    bins=5,
    edgecolor='black'
)

plt.title('Marks Distribution')
plt.xlabel('Marks')
plt.ylabel('Frequency')
plt.show()

# 2. Line Chart
df['Marks'].plot(
    kind='line',
    marker='o'
)

plt.title('Student Marks Trend')
plt.xlabel('Student Index')
plt.ylabel('Marks')
plt.grid(True)
plt.show()

# 3. Scatter Plot
df.plot(
    kind='scatter',
    x='StudyHours',
    y='Marks'
)

plt.title('Study Hours vs Marks')
plt.xlabel('Study Hours')
plt.ylabel('Marks')
plt.grid(True)
plt.show()

# 4. Box Plot
df[['Marks']].plot(
    kind='box'
)

plt.title('Marks Distribution Box Plot')
plt.ylabel('Marks')
plt.show()

Analysis

  • The histogram shows the distribution of student marks.
  • The line chart displays marks across the student records.
  • The scatter plot helps explore the relationship between study hours and marks.
  • The box plot summarises the spread of student marks and highlights potential outliers.

13. Difference Between Histogram, Line, Scatter and Box Plot

Feature Histogram Line Scatter Box
Purpose Distribution Trend Relationship Spread
Data One numerical variable Ordered sequence Two numerical variables One or more numerical variables
Shows Frequency Yes No No No
Shows Outliers Not directly Not specifically Can reveal unusual points Yes
Common Use Marks distribution Sales trend Study hours vs marks Salary analysis

14. Best Practices for Chart Analysis

  • Select a chart based on the analytical requirement.
  • Use suitable bins when creating histograms.
  • Use line charts for meaningful ordered sequences.
  • Use scatter plots to examine two numerical variables.
  • Use box plots to understand data spread and potential outliers.
  • Add meaningful titles and axis labels.
  • Check missing values before creating charts.
  • Avoid misleading scales and unnecessary visual elements.
  • Investigate outliers before deciding whether to remove them.
  • Always interpret charts in the context of the original data.

Summary

Histograms, line charts, scatter plots and box plots are important tools for analysing numerical data using Pandas. Each chart provides a different way to understand a dataset.

A histogram displays frequency distribution, a line chart identifies trends, a scatter plot explores relationships between numerical variables and a box plot summarises data distribution using quartiles, median and potential outliers.

We also learned how to calculate the Interquartile Range (IQR), identify potential outliers, customise charts and compare multiple numerical columns.

These visualisation techniques are useful in student performance analysis, business reporting, financial analysis and data science. In the next lesson, we will learn how to use Matplotlib and Seaborn for more advanced and customised visualisations.