Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Pandas Data Visualization

Learn how to convert Pandas DataFrame and Series data into meaningful charts and graphs using Python visualization libraries.

1. Introduction to Data Visualization

Data visualization is the process of presenting information using graphical representations such as charts, graphs and plots. It helps us understand large amounts of data quickly and easily.

Pandas provides built-in plotting functionality that allows us to create visual representations directly from DataFrames and Series. It uses the Matplotlib library internally for generating plots.

Important: Data visualization makes it easier to identify patterns, trends, relationships, distributions and unusual values in a dataset.

2. Why Do We Need Data Visualization?

When working with large datasets, reading only rows and columns may not be sufficient to understand the information. Charts help us analyse the data in a visual format.

  • Understand trends: Identify how values change over time.
  • Compare data: Compare sales, marks, revenue or other values.
  • Find patterns: Discover relationships between different columns.
  • Identify outliers: Detect unusually high or low values.
  • Analyse distribution: Understand how data values are spread.
  • Communicate results: Present analytical findings clearly.

3. Python Libraries for Data Visualization

Python provides several libraries for creating charts and visualising data. Pandas works well with these libraries.

Library Purpose Common Usage
Pandas Provides built-in plotting methods for DataFrames and Series. Quick data plotting
Matplotlib Provides detailed control over charts and graphs. Line charts, bar charts and histograms
Seaborn Provides attractive statistical visualisations. Heatmaps, distribution plots and box plots
Plotly Creates interactive charts and visualisations. Interactive dashboards

4. Installing Visualization Libraries

Before creating charts, install the required Python libraries using the pip package manager.

Install Pandas and Matplotlib

pip install pandas matplotlib

Install Seaborn

pip install seaborn

Install Plotly

pip install plotly
You can install all these libraries together using:
pip install pandas matplotlib seaborn plotly

5. Importing Required Libraries

After installation, import Pandas and Matplotlib into your Python program. We can also import Seaborn when statistical charts are required.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

Explanation:

  • pandas is imported using the alias pd.
  • matplotlib.pyplot is imported using the alias plt.
  • seaborn is imported using the alias sns.

6. Creating a Sample DataFrame

Before creating visualisations, let us create a sample DataFrame containing monthly sales information.

import pandas as pd

data = {
    'Month': ['January', 'February', 'March', 'April', 'May'],
    'Sales': [12000, 15000, 18000, 14000, 22000],
    'Expenses': [8000, 9000, 11000, 8500, 13000]
}

df = pd.DataFrame(data)

print(df)

Expected Output

      Month  Sales  Expenses
0   January  12000      8000
1  February  15000      9000
2     March  18000     11000
3     April  14000      8500
4       May  22000     13000

This DataFrame contains monthly sales and expenses. We can use this information to create different types of charts.

7. Common Types of Data Visualization

Different charts are useful for different types of data analysis. The following chart types are commonly used in Pandas projects.

Chart Type Purpose Example
Line Chart Shows trends and changes over time. Monthly sales trends
Bar Chart Compares values between categories. Sales by product
Histogram Displays the distribution of numerical values. Student marks distribution
Scatter Plot Shows the relationship between two numerical variables. Study hours and marks
Box Plot Shows data spread, quartiles and potential outliers. Salary distribution
Pie Chart Shows proportions of different categories. Department-wise employees
Note: The next lessons will cover basic plotting, histograms, line charts, scatter plots, box plots, Matplotlib and Seaborn in detail.

8. Understanding the Pandas plot() Method

Pandas provides the plot() method for creating charts directly from a DataFrame or Series.

General syntax:

DataFrame.plot(kind='chart_type')

Example:

df.plot(x='Month', y='Sales', kind='line')

plt.title('Monthly Sales')
plt.xlabel('Month')
plt.ylabel('Sales')

plt.show()

Here, kind='line' specifies that a line chart should be created. The plt.show() method displays the chart.

Common plot() Parameters

Parameter Description
kind Specifies the type of chart.
x Specifies the column for the X-axis.
y Specifies the column for the Y-axis.
title Sets the chart title.
figsize Sets the chart size.
color Sets the chart colour.
legend Controls whether the legend is displayed.

9. Real-World Applications

Data visualization is used in many industries to understand business information and support decision-making.

  • Education: Analyse student marks, attendance and results.
  • Banking: Analyse transactions and financial activities.
  • Healthcare: Visualise patient records and medical statistics.
  • Business: Compare sales, expenses and revenue.
  • E-commerce: Analyse product performance and customer behaviour.
  • Marketing: Track campaign performance and customer engagement.

10. Best Practices for Data Visualization

  • Choose the correct chart according to the data.
  • Always provide meaningful chart titles.
  • Use clear X-axis and Y-axis labels.
  • Avoid unnecessary colours and decorations.
  • Keep charts simple and easy to understand.
  • Handle missing values before plotting data.
  • Use legends when comparing multiple datasets.
  • Verify the data before presenting the visualisation.

Summary

Data visualization is an important part of data analysis. It helps convert complex datasets into meaningful charts and graphs. Pandas provides built-in plotting functionality through the plot() method, which uses Matplotlib internally.

Python libraries such as Matplotlib, Seaborn and Plotly provide additional features for creating different types of visualisations. Common chart types include line charts, bar charts, histograms, scatter plots, box plots and pie charts.

In the next lesson, we will learn how to create different charts directly from Pandas DataFrames using basic plotting methods.