Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Descriptive Statistics in Pandas

Learn how to summarize data using averages, medians, modes, quartiles, standard deviation, and the describe() method.

1. What Are Descriptive Statistics?

Descriptive statistics are numerical summaries that help us understand the main characteristics of a dataset. They help us identify typical values, variation, minimums, maximums, and the distribution of numerical data.

For example, Samadhan wants to understand the marks obtained by students. Instead of examining each score separately, he can calculate the average, median, minimum, maximum, and standard deviation.

What will we learn?

  • Generate summaries with describe().
  • Calculate mean, median, and mode.
  • Find minimum, maximum, range, and sum.
  • Understand variance and standard deviation.
  • Interpret quartiles and percentiles.
  • Summarize categorical columns.
  • Handle missing values during statistical calculations.

2. Create a Sample DataFrame

We will use a student dataset to demonstrate the different statistical operations.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav", "Neha"],
    "Marks": [85, 92, 88, 76, 95],
    "Age": [21, 22, 20, 23, 21],
    "City": ["Pune", "Mumbai", "Pune", "Nashik", "Mumbai"]
})

print(df)

Output

       Name  Marks  Age    City
0  Samadhan     85   21    Pune
1   Pradnya     92   22  Mumbai
2  Samiksha     88   20    Pune
3     Aarav     76   23  Nashik
4      Neha     95   21  Mumbai

3. The describe() Method

The describe() method calculates a statistical summary for selected columns. By default, it summarizes numerical columns.

print(df.describe())

Output

           Marks        Age
count   5.000000   5.000000
mean   87.200000  21.400000
std     7.395945   1.140175
min    76.000000  20.000000
25%    85.000000  21.000000
50%    88.000000  21.000000
75%    92.000000  22.000000
max    95.000000  23.000000
Statistic Meaning
count Number of non-missing observations.
mean Arithmetic average.
std Standard deviation, which measures spread.
min Smallest value.
25% First quartile.
50% Median or second quartile.
75% Third quartile.
max Largest value.
Tip: Use df.describe() as a quick first summary before performing detailed analysis.

4. Image 1: Understanding a Statistical Summary

This illustration explains how a dataset's minimum, quartiles, median, and maximum describe the distribution of values.

Five-Number Summary Minimum Q1 (25%) Median Q3 (75%) Maximum 76 85 88 92 95 Illustrative summary of the sample Marks column

Image 1: Quartiles and the median help describe the spread of numerical observations.

Note: The drawing is a simplified illustration, not a scale-accurate box plot. Exact quartile values depend on the dataset and quantile calculation method.

5. Mean — Average Value

The mean is calculated by adding all observed values and dividing the total by the number of observed values.

For the Marks column, the values are 85, 92, 88, 76, and 95.

mean_marks = df["Marks"].mean()

print(mean_marks)

Output

87.2

The average marks are 87.2.

Mean of all numerical columns

print(df[["Marks", "Age"]].mean())

Output

Marks    87.2
Age      21.4
dtype: float64
Remember: By default, Pandas skips missing values when calculating the mean of a numerical Series.

6. Median — Middle Value

The median is the middle value after the observations are sorted. For an even number of observations, it is the average of the two middle values.

The sorted Marks values are 76, 85, 88, 92, and 95. The middle value is 88.

print(df["Marks"].median())

Output

88.0

When is median useful?

The median can be useful when the data contains extreme values, because it is often less affected by outliers than the mean.

7. Mode — Most Frequent Value

The mode is the value that occurs most frequently. Pandas returns the mode using the mode() method.

print(df["City"].mode())

Output

0    Mumbai
1      Pune
Name: City, dtype: object

In this dataset, Mumbai and Pune both occur twice, so both are modes. Pandas can return multiple modes when there is a tie.

Mode of the Marks column

print(df["Marks"].mode())

Since every mark in this example occurs only once, each value is tied for the highest frequency. The mode method can therefore return all of them.

8. Minimum, Maximum, Sum, and Range

These operations help identify the smallest and largest observations and calculate the total.

marks = df["Marks"]

print("Minimum:", marks.min())
print("Maximum:", marks.max())
print("Total:", marks.sum())
print("Range:", marks.max() - marks.min())

Output

Minimum: 76
Maximum: 95
Total: 436
Range: 19
Method Purpose
min() Returns the minimum value.
max() Returns the maximum value.
sum() Adds the observed values.
count() Counts non-missing observations.

9. Variance and Standard Deviation

Variance and standard deviation describe how spread out the values are around their mean.

Variance

Variance measures the average squared deviation from the mean, using a denominator determined by the selected variance convention.

print(df["Marks"].var())

Standard deviation

Standard deviation is the square root of variance. It expresses spread in the same units as the original measurements.

print(df["Marks"].std())

By default, Pandas uses the sample variance and sample standard deviation, with ddof=1.

Population standard deviation

print(df["Marks"].std(ddof=0))
Important: Use the sample or population formula according to what your data represents. The default Pandas result is not the population standard deviation.

10. Quartiles and Quantiles

Quartiles divide sorted numerical data into four parts. The first quartile is the 25th percentile, the second quartile is the median, and the third quartile is the 75th percentile.

print(df["Marks"].quantile(0.25))
print(df["Marks"].quantile(0.50))
print(df["Marks"].quantile(0.75))

Pandas calculates quantiles using its configured quantile method. For small datasets, results can be fractional.

Calculate the interquartile range (IQR)

q1 = df["Marks"].quantile(0.25)
q3 = df["Marks"].quantile(0.75)

iqr = q3 - q1

print("Q1:", q1)
print("Q3:", q3)
print("IQR:", iqr)

The interquartile range is the difference between the third and first quartiles. It describes the spread of the middle half of the observations.

11. Image 2: Comparing Mean and Median

The mean uses all observed numerical values, while the median represents the middle position after sorting.

Mean vs Median Marks: 70 75 80 82 85 150 Outlier Median Mean Median Less affected by extreme values Mean Uses every observed value

Image 2: An extreme value can pull the mean away from the median. The diagram is illustrative.

Practical insight: When data contains extreme values, compare the mean and median instead of relying on only one measure of central tendency.

12. Descriptive Statistics for Categorical Data

DataFrames can contain text categories such as cities, departments, or product types. You can summarize categorical columns with describe(include="object").

print(df.describe(include="object"))

The summary can include:

  • count: Number of non-missing entries.
  • unique: Number of distinct values.
  • top: Most frequent value.
  • freq: Frequency of the most frequent value.

Count category frequencies

print(df["City"].value_counts())

Output

City
Pune      2
Mumbai    2
Nashik    1
Name: count, dtype: int64

The exact output label for the count column can differ by Pandas version.

13. Working with Missing Values

Pandas generally skips missing values when calculating descriptive statistics for numerical Series. Always inspect missing values so you understand how many observations contribute to a calculation.

df_missing = pd.DataFrame({
    "Marks": [85, 92, None, 76, 95]
})

print(df_missing["Marks"].mean())
print(df_missing["Marks"].count())
print(df_missing["Marks"].isna().sum())

Output

87.0
4
1

Four values are present, one is missing, and the mean is calculated from the four observed marks.

Remember: Skipping missing values is not the same as filling them. Choose a missing-data strategy based on the context of your analysis.

14. Calculate Statistics for Selected Columns

You can select one or more numerical columns before applying statistical methods.

# Mean marks
print(df["Marks"].mean())

# Mean and median for selected columns
print(df[["Marks", "Age"]].mean())
print(df[["Marks", "Age"]].median())

# Summary for numerical columns
print(df.select_dtypes(include="number").describe())

Selecting columns explicitly makes your code easier to read and helps you avoid applying numerical calculations to inappropriate data.

15. Practical Exercise

Samiksha wants to analyze the performance of five students. Use the following DataFrame to complete the tasks.

import pandas as pd

df = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav", "Neha"],
    "Marks": [85, 92, 88, 76, 95],
    "Age": [21, 22, 20, 23, 21],
    "City": ["Pune", "Mumbai", "Pune", "Nashik", "Mumbai"]
})

Tasks

  1. Generate descriptive statistics for numerical columns.
  2. Calculate the mean and median of Marks.
  3. Find the minimum and maximum marks.
  4. Calculate the range of marks.
  5. Calculate the standard deviation.
  6. Find the mode of City.
  7. Count how many students belong to each city.

Suggested solution

print(df.describe())

print("Mean:", df["Marks"].mean())
print("Median:", df["Marks"].median())

print("Minimum:", df["Marks"].min())
print("Maximum:", df["Marks"].max())

print("Range:", df["Marks"].max() - df["Marks"].min())
print("Standard deviation:", df["Marks"].std())

print(df["City"].mode())
print(df["City"].value_counts())

16. Common Mistakes to Avoid

Mistake 1: Assuming describe() summarizes every column by default

By default, describe() summarizes numerical columns. Use describe(include="all") or describe(include="object") when you want summaries that include categorical data.

Mistake 2: Confusing mean and median

The mean is the arithmetic average. The median is the middle value after sorting.

Mistake 3: Assuming mode always returns one value

Multiple values can share the highest frequency. Pandas can return more than one mode.

Mistake 4: Ignoring missing values

Always inspect how many observations are present before interpreting a statistical result.

Mistake 5: Confusing sample and population standard deviation

Pandas uses ddof=1 by default for std(). Use ddof=0 for the population standard deviation when appropriate.

17. Revision Questions

  1. What are descriptive statistics?
  2. What does describe() return?
  3. What is the difference between mean and median?
  4. What is mode, and can a dataset have multiple modes?
  5. How do you calculate the range of a numerical column?
  6. What do variance and standard deviation measure?
  7. What are quartiles and percentiles?
  8. How do you summarize categorical columns?
  9. How does Pandas handle missing values in numerical calculations by default?
  10. What is the difference between sample and population standard deviation?

Summary

Descriptive statistics help you summarize and understand numerical and categorical data. Use describe() for a quick statistical overview, mean() for averages, median() for the middle value, and mode() for the most frequent value.

Methods such as min(), max(), sum(), var(), std(), quantile(), and value_counts() provide additional insights into data distribution and frequency.

Always interpret statistical results in context and check missing values before drawing conclusions.