Descriptive Statistics in Pandas
Learn how to summarize data using averages, medians, modes, quartiles, standard deviation, and the describe() method.
1. What Are Descriptive Statistics?
Descriptive statistics are numerical summaries that help us understand the main characteristics of a dataset. They help us identify typical values, variation, minimums, maximums, and the distribution of numerical data.
For example, Samadhan wants to understand the marks obtained by students. Instead of examining each score separately, he can calculate the average, median, minimum, maximum, and standard deviation.
What will we learn?
- Generate summaries with
describe(). - Calculate mean, median, and mode.
- Find minimum, maximum, range, and sum.
- Understand variance and standard deviation.
- Interpret quartiles and percentiles.
- Summarize categorical columns.
- Handle missing values during statistical calculations.
2. Create a Sample DataFrame
We will use a student dataset to demonstrate the different statistical operations.
import pandas as pd
df = pd.DataFrame({
"Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav", "Neha"],
"Marks": [85, 92, 88, 76, 95],
"Age": [21, 22, 20, 23, 21],
"City": ["Pune", "Mumbai", "Pune", "Nashik", "Mumbai"]
})
print(df)
Output
Name Marks Age City 0 Samadhan 85 21 Pune 1 Pradnya 92 22 Mumbai 2 Samiksha 88 20 Pune 3 Aarav 76 23 Nashik 4 Neha 95 21 Mumbai
3. The describe() Method
The describe() method calculates a statistical
summary for selected columns. By default, it summarizes
numerical columns.
print(df.describe())
Output
Marks Age count 5.000000 5.000000 mean 87.200000 21.400000 std 7.395945 1.140175 min 76.000000 20.000000 25% 85.000000 21.000000 50% 88.000000 21.000000 75% 92.000000 22.000000 max 95.000000 23.000000
| Statistic | Meaning |
|---|---|
count |
Number of non-missing observations. |
mean |
Arithmetic average. |
std |
Standard deviation, which measures spread. |
min |
Smallest value. |
25% |
First quartile. |
50% |
Median or second quartile. |
75% |
Third quartile. |
max |
Largest value. |
df.describe() as a
quick first summary before performing detailed analysis.
4. Image 1: Understanding a Statistical Summary
This illustration explains how a dataset's minimum, quartiles, median, and maximum describe the distribution of values.
Image 1: Quartiles and the median help describe the spread of numerical observations.
5. Mean — Average Value
The mean is calculated by adding all observed values and dividing the total by the number of observed values.
For the Marks column, the values are 85, 92, 88, 76, and 95.
mean_marks = df["Marks"].mean() print(mean_marks)
Output
87.2
The average marks are 87.2.
Mean of all numerical columns
print(df[["Marks", "Age"]].mean())
Output
Marks 87.2 Age 21.4 dtype: float64
6. Median — Middle Value
The median is the middle value after the observations are sorted. For an even number of observations, it is the average of the two middle values.
The sorted Marks values are 76, 85, 88, 92, and 95. The middle value is 88.
print(df["Marks"].median())
Output
88.0
When is median useful?
The median can be useful when the data contains extreme values, because it is often less affected by outliers than the mean.
7. Mode — Most Frequent Value
The mode is the value that occurs most frequently.
Pandas returns the mode using the mode() method.
print(df["City"].mode())
Output
0 Mumbai 1 Pune Name: City, dtype: object
In this dataset, Mumbai and Pune both occur twice, so both are modes. Pandas can return multiple modes when there is a tie.
Mode of the Marks column
print(df["Marks"].mode())
Since every mark in this example occurs only once, each value is tied for the highest frequency. The mode method can therefore return all of them.
8. Minimum, Maximum, Sum, and Range
These operations help identify the smallest and largest observations and calculate the total.
marks = df["Marks"]
print("Minimum:", marks.min())
print("Maximum:", marks.max())
print("Total:", marks.sum())
print("Range:", marks.max() - marks.min())
Output
Minimum: 76 Maximum: 95 Total: 436 Range: 19
| Method | Purpose |
|---|---|
min() |
Returns the minimum value. |
max() |
Returns the maximum value. |
sum() |
Adds the observed values. |
count() |
Counts non-missing observations. |
9. Variance and Standard Deviation
Variance and standard deviation describe how spread out the values are around their mean.
Variance
Variance measures the average squared deviation from the mean, using a denominator determined by the selected variance convention.
print(df["Marks"].var())
Standard deviation
Standard deviation is the square root of variance. It expresses spread in the same units as the original measurements.
print(df["Marks"].std())
By default, Pandas uses the sample variance and sample standard
deviation, with ddof=1.
Population standard deviation
print(df["Marks"].std(ddof=0))
10. Quartiles and Quantiles
Quartiles divide sorted numerical data into four parts. The first quartile is the 25th percentile, the second quartile is the median, and the third quartile is the 75th percentile.
print(df["Marks"].quantile(0.25)) print(df["Marks"].quantile(0.50)) print(df["Marks"].quantile(0.75))
Pandas calculates quantiles using its configured quantile method. For small datasets, results can be fractional.
Calculate the interquartile range (IQR)
q1 = df["Marks"].quantile(0.25)
q3 = df["Marks"].quantile(0.75)
iqr = q3 - q1
print("Q1:", q1)
print("Q3:", q3)
print("IQR:", iqr)
The interquartile range is the difference between the third and first quartiles. It describes the spread of the middle half of the observations.
11. Image 2: Comparing Mean and Median
The mean uses all observed numerical values, while the median represents the middle position after sorting.
Image 2: An extreme value can pull the mean away from the median. The diagram is illustrative.
12. Descriptive Statistics for Categorical Data
DataFrames can contain text categories such as cities,
departments, or product types. You can summarize categorical
columns with describe(include="object").
print(df.describe(include="object"))
The summary can include:
- count: Number of non-missing entries.
- unique: Number of distinct values.
- top: Most frequent value.
- freq: Frequency of the most frequent value.
Count category frequencies
print(df["City"].value_counts())
Output
City Pune 2 Mumbai 2 Nashik 1 Name: count, dtype: int64
The exact output label for the count column can differ by Pandas version.
13. Working with Missing Values
Pandas generally skips missing values when calculating descriptive statistics for numerical Series. Always inspect missing values so you understand how many observations contribute to a calculation.
df_missing = pd.DataFrame({
"Marks": [85, 92, None, 76, 95]
})
print(df_missing["Marks"].mean())
print(df_missing["Marks"].count())
print(df_missing["Marks"].isna().sum())
Output
87.0 4 1
Four values are present, one is missing, and the mean is calculated from the four observed marks.
14. Calculate Statistics for Selected Columns
You can select one or more numerical columns before applying statistical methods.
# Mean marks print(df["Marks"].mean()) # Mean and median for selected columns print(df[["Marks", "Age"]].mean()) print(df[["Marks", "Age"]].median()) # Summary for numerical columns print(df.select_dtypes(include="number").describe())
Selecting columns explicitly makes your code easier to read and helps you avoid applying numerical calculations to inappropriate data.
15. Practical Exercise
Samiksha wants to analyze the performance of five students. Use the following DataFrame to complete the tasks.
import pandas as pd
df = pd.DataFrame({
"Name": ["Samadhan", "Pradnya", "Samiksha", "Aarav", "Neha"],
"Marks": [85, 92, 88, 76, 95],
"Age": [21, 22, 20, 23, 21],
"City": ["Pune", "Mumbai", "Pune", "Nashik", "Mumbai"]
})
Tasks
- Generate descriptive statistics for numerical columns.
- Calculate the mean and median of Marks.
- Find the minimum and maximum marks.
- Calculate the range of marks.
- Calculate the standard deviation.
- Find the mode of City.
- Count how many students belong to each city.
Suggested solution
print(df.describe())
print("Mean:", df["Marks"].mean())
print("Median:", df["Marks"].median())
print("Minimum:", df["Marks"].min())
print("Maximum:", df["Marks"].max())
print("Range:", df["Marks"].max() - df["Marks"].min())
print("Standard deviation:", df["Marks"].std())
print(df["City"].mode())
print(df["City"].value_counts())
16. Common Mistakes to Avoid
Mistake 1: Assuming describe() summarizes every column by default
By default, describe() summarizes numerical
columns. Use describe(include="all") or
describe(include="object") when you want
summaries that include categorical data.
Mistake 2: Confusing mean and median
The mean is the arithmetic average. The median is the middle value after sorting.
Mistake 3: Assuming mode always returns one value
Multiple values can share the highest frequency. Pandas can return more than one mode.
Mistake 4: Ignoring missing values
Always inspect how many observations are present before interpreting a statistical result.
Mistake 5: Confusing sample and population standard deviation
Pandas uses ddof=1 by default for
std(). Use ddof=0 for the
population standard deviation when appropriate.
17. Revision Questions
- What are descriptive statistics?
- What does
describe()return? - What is the difference between mean and median?
- What is mode, and can a dataset have multiple modes?
- How do you calculate the range of a numerical column?
- What do variance and standard deviation measure?
- What are quartiles and percentiles?
- How do you summarize categorical columns?
- How does Pandas handle missing values in numerical calculations by default?
- What is the difference between sample and population standard deviation?
Summary
Descriptive statistics help you summarize and understand
numerical and categorical data. Use describe()
for a quick statistical overview, mean() for
averages, median() for the middle value, and
mode() for the most frequent value.
Methods such as min(), max(),
sum(), var(), std(),
quantile(), and value_counts()
provide additional insights into data distribution and
frequency.
Always interpret statistical results in context and check missing values before drawing conclusions.