Master NumPy From Scratch

Learn NumPy fundamentals, arrays, indexing, slicing, data types, array operations, broadcasting, mathematical functions, statistics, random numbers, linear algebra, file handling, and practical projects through structured lessons.

COURSE SYLLABUS
1. NumPy Introduction and Installation
Introduction to NumPy NumPy Installation and Setup NumPy Environment and IDE Setup First NumPy Program
2. NumPy Arrays and Array Creation
NumPy Array Fundamentals Array Creation Functions Zeros, Ones and Identity Arrays Range and Linspace Arrays
3. NumPy Array Indexing and Slicing
One-Dimensional Array Indexing Array Slicing Techniques Two-Dimensional Array Indexing Boolean and Fancy Indexing
4. NumPy Data Types and Attributes
NumPy Data Types Array Shape, Size and Dimensions Data Type Conversion and Casting Array Inspection and Attributes
5. NumPy Array Shape and Reshaping
Reshape and Flatten Arrays Transpose and Axis Operations Stacking and Splitting Arrays Expand and Squeeze Dimensions
6. NumPy Array Operations and Broadcasting
Arithmetic Array Operations Comparison and Logical Operations Broadcasting Fundamentals Universal Functions and Vectorization
7. NumPy Mathematical and Statistical Functions
Mathematical Functions Aggregate and Statistical Functions Minimum, Maximum and Percentiles Rounding and Trigonometric Functions
8. NumPy Filtering and Sorting
Array Filtering and Conditions Sorting and Searching Arrays Unique Values and Set Operations Where, Select and Nonzero Functions
9. NumPy Copies and Views
Array Copies and Views Shallow and Deep Copy Concepts Memory Layout and Contiguity Array Iteration and Memory Efficiency
10. NumPy Random Number Generation
Random Generator Fundamentals Random Integers and Sampling Probability Distributions Random Seeds and Simulation
11. NumPy Linear Algebra
Vectors and Matrices Matrix Multiplication and Dot Product Determinants and Matrix Inverse Eigenvalues and Linear Equations
12. NumPy File Input and Output
Importing and Exporting Text Files CSV Data with NumPy Binary Files and NPY Format NPZ Files and Data Persistence
13. NumPy Practical Projects
Student Marks Analysis Project Sales Data Analysis Project Random Simulation Project Image Array Processing Project
14. NumPy Practice and Assessment
NumPy Beginner Practice NumPy Intermediate Exercises NumPy Advanced Challenges Final NumPy Assessment

NumPy Aggregate and Statistical Functions

NumPy provides powerful aggregate and statistical functions for summarizing numerical data. These functions help calculate totals, averages, medians, variances, standard deviations, and other useful measurements. In this lesson, you will learn how to summarize one-dimensional and multidimensional arrays, use the axis parameter, and apply statistical functions to practical data analysis tasks.

1. Introduction to Aggregate Functions

An aggregate function combines multiple values into a summary result. For example, a list of student marks can be summarized by calculating the total marks, average marks, minimum mark, and maximum mark.

Common Aggregate Functions

  • np.sum() calculates the sum of values.
  • np.mean() calculates the arithmetic mean.
  • np.min() returns the minimum value.
  • np.max() returns the maximum value.
  • np.median() calculates the median.
  • np.std() calculates standard deviation.
  • np.var() calculates variance.
import numpy as np marks = np.array([65, 72, 84, 91, 78]) print("Total:", np.sum(marks)) print("Average:", np.mean(marks)) print("Minimum:", np.min(marks)) print("Maximum:", np.max(marks))

Output:

Total: 390 Average: 78.0 Minimum: 65 Maximum: 91

These functions provide a quick overview of the values in an array without requiring you to write manual loops.

2. Calculating Sum and Product

The sum of an array is the total of its elements. The product is the result of multiplying its elements together.

import numpy as np values = np.array([2, 3, 4, 5]) total = np.sum(values) product = np.prod(values) print("Sum:", total) print("Product:", product)

Output:

Sum: 14 Product: 120

Related Functions

Function Purpose Example Result
np.sum() Adds the array elements. 2 + 3 + 4 + 5 = 14
np.prod() Multiplies the array elements. 2 × 3 × 4 × 5 = 120
np.cumsum() Returns cumulative sums. [2, 5, 9, 14]
np.cumprod() Returns cumulative products. [2, 6, 24, 120]
values = np.array([2, 3, 4, 5]) print("Cumulative sum:", np.cumsum(values)) print("Cumulative product:", np.cumprod(values))

Output:

Cumulative sum: [ 2 5 9 14] Cumulative product: [ 2 6 24 120]

3. Mean, Median, and Average

Mean and median are measures of central tendency. They help describe the center of a dataset, but they behave differently when extreme values are present.

Mean

The arithmetic mean is calculated by adding all values and dividing the total by the number of values.

import numpy as np marks = np.array([60, 70, 80, 90, 100]) print("Mean:", np.mean(marks)) print("Average:", np.average(marks))

Output:

Mean: 80.0 Average: 80.0

Median

The median is the middle value after sorting the data. If the dataset contains an even number of values, the median is the average of the two middle values.

values = np.array([10, 20, 30, 40, 1000]) print("Median:", np.median(values)) print("Mean:", np.mean(values))

Output:

Median: 30.0 Mean: 220.0

The extreme value of 1000 increases the mean considerably, while the median remains at 30. This makes the median useful when data contains unusually large or small observations.

Tip: Use the mean when you need the arithmetic average. Consider the median when extreme observations may distort the average.

4. Minimum, Maximum, and Range

Minimum and maximum functions identify the smallest and largest values. The range is the difference between the maximum and minimum.

import numpy as np temperatures = np.array([24, 29, 31, 22, 35, 27]) minimum = np.min(temperatures) maximum = np.max(temperatures) value_range = maximum - minimum print("Minimum:", minimum) print("Maximum:", maximum) print("Range:", value_range)

Output:

Minimum: 22 Maximum: 35 Range: 13

Visualizing the Range

Temperature Range Minimum 22 Maximum 35 Range = 35 - 22 = 13

The range measures the difference between the largest and smallest observations.

5. Variance and Standard Deviation

Variance measures the average squared deviation from the mean. Standard deviation is the square root of variance and expresses dispersion in the same units as the original data.

import numpy as np values = np.array([10, 20, 30, 40, 50]) print("Mean:", np.mean(values)) print("Variance:", np.var(values)) print("Standard deviation:", np.std(values))

Output:

Mean: 30.0 Variance: 200.0 Standard deviation: 14.142135623730951

Population and Sample Calculations

By default, np.var() and np.std() use ddof=0, which calculates population variance and standard deviation. For a sample estimate, use ddof=1.

import numpy as np sample = np.array([10, 20, 30, 40, 50]) population_variance = np.var(sample) sample_variance = np.var(sample, ddof=1) population_std = np.std(sample) sample_std = np.std(sample, ddof=1) print("Population variance:", population_variance) print("Sample variance:", sample_variance) print("Population standard deviation:", round(population_std, 2)) print("Sample standard deviation:", round(sample_std, 2))

Output:

Population variance: 200.0 Sample variance: 250.0 Population standard deviation: 14.14 Sample standard deviation: 15.81
Important: Choose ddof according to the statistical question. Population and sample calculations use different denominators and therefore produce different results.

6. Percentiles and Quantiles

Percentiles indicate the position below which a given percentage of observations falls. Quantiles divide ordered data into specified proportions. They are commonly used to understand distributions and compare values within a dataset.

import numpy as np scores = np.array([45, 55, 60, 70, 80, 90, 95]) print("25th percentile:", np.percentile(scores, 25)) print("50th percentile:", np.percentile(scores, 50)) print("75th percentile:", np.percentile(scores, 75)) print("Quartiles:", np.quantile(scores, [0.25, 0.50, 0.75]))

Output:

25th percentile: 57.5 50th percentile: 70.0 75th percentile: 87.5 Quartiles: [57.5 70. 87.5]
Function Input Scale Example
np.percentile() 0 to 100 np.percentile(scores, 90)
np.quantile() 0 to 1 np.quantile(scores, 0.90)

The default interpolation method is linear. Percentile results can therefore be values that do not appear directly in the original array.

7. Understanding the Axis Parameter

For multidimensional arrays, the axis parameter specifies the direction in which an operation is performed.

  • axis=0 reduces rows and calculates results for each column.
  • axis=1 reduces columns and calculates results for each row.
  • axis=None uses all elements in the array. This is the default for many aggregate functions.
import numpy as np marks = np.array([ [70, 80, 90], [60, 75, 85], [88, 92, 95] ]) print("Total of all marks:", np.sum(marks)) print("Column totals:", np.sum(marks, axis=0)) print("Row totals:", np.sum(marks, axis=1)) print("Column averages:", np.mean(marks, axis=0)) print("Row averages:", np.mean(marks, axis=1))

Output:

Total of all marks: 735 Column totals: [218 247 270] Row totals: [240 220 275] Column averages: [72.66666667 82.33333333 90. ] Row averages: [80. 73.33333333 91.66666667]
Understanding axis=0 and axis=1 axis=0 Reduce rows 70 80 90 60 75 85 88 92 95 axis=1 Reduce columns Row 1 → 240 Row 2 → 220 Row 3 → 275 The selected axis determines which dimension is reduced.

Axis operations summarize a matrix by column or by row.

Remember: For a matrix, axis=0 returns one result per column, while axis=1 returns one result per row.

8. Counting Values and Handling NaN

The np.count_nonzero() function counts values that are considered nonzero. NumPy also provides NaN-aware aggregate functions that can ignore missing numerical values represented by np.nan.

import numpy as np values = np.array([0, 5, 0, 8, 12, 0]) print("Nonzero count:", np.count_nonzero(values)) print("Array size:", values.size)

Output:

Nonzero count: 3 Array size: 6

Handling Missing Numerical Values

import numpy as np values = np.array([10.0, 20.0, np.nan, 40.0]) print("Regular mean:", np.mean(values)) print("NaN-aware mean:", np.nanmean(values)) print("NaN-aware sum:", np.nansum(values)) print("NaN-aware minimum:", np.nanmin(values))

Output:

Regular mean: nan NaN-aware mean: 23.333333333333332 NaN-aware sum: 70.0 NaN-aware minimum: 10.0
Important: NaN-aware functions ignore NaN values, but this does not automatically solve every missing-data problem. Check how missing values should be interpreted in your dataset before calculating statistical summaries.

9. Practical Project: Student Marks Analysis

In this example, we will analyze marks for three students across three subjects. We will calculate each student's total and average, along with the average score for each subject.

import numpy as np students = np.array(["Samadhan", "Pradnya", "Samiksha"]) subjects = np.array(["Python", "Mathematics", "Statistics"]) marks = np.array([ [78, 85, 82], [92, 88, 95], [84, 90, 87] ]) student_totals = np.sum(marks, axis=1) student_averages = np.mean(marks, axis=1) subject_averages = np.mean(marks, axis=0) print("Student Analysis:") for i in range(len(students)): print( students[i], "Total:", student_totals[i], "Average:", round(student_averages[i], 2) ) print("Subject averages:") for i in range(len(subjects)): print(subjects[i], round(subject_averages[i], 2)) print("Highest mark:", np.max(marks)) print("Lowest mark:", np.min(marks))

Output:

Student Analysis: Samadhan Total: 245 Average: 81.67 Pradnya Total: 275 Average: 91.67 Samiksha Total: 261 Average: 87.0 Subject averages: Python 84.67 Mathematics 87.67 Statistics 88.0 Highest mark: 95 Lowest mark: 78

What This Project Demonstrates

  • np.sum(marks, axis=1) calculates each student's total.
  • np.mean(marks, axis=1) calculates each student's average.
  • np.mean(marks, axis=0) calculates each subject's average.
  • np.max(marks) identifies the highest mark in the matrix.
  • np.min(marks) identifies the lowest mark in the matrix.

10. Common Mistakes and Best Practices

Confusing Mean and Median

The mean uses every value in the arithmetic calculation, while the median depends on the middle position of the sorted data. Choose the measure that best represents your dataset.

Using the Wrong Axis

Before calculating row-wise or column-wise summaries, confirm the arrangement of the data and select the correct axis.

Ignoring Missing Values

A regular mean or sum can propagate NaN values. Consider NaN-aware functions when ignoring missing numerical values is appropriate.

Confusing Population and Sample Standard Deviation

The default ddof=0 differs from the sample calculation with ddof=1. Use the setting that matches your analysis.

Rounding Too Early

Keep full precision during intermediate calculations when possible. Round the final result for display or reporting.

11. Practice Exercises

Exercise 1: Basic Aggregates

Create an array containing [12, 18, 24, 30, 36]. Calculate its sum, product, mean, minimum, and maximum.

Exercise 2: Mean and Median

Create an array containing [10, 15, 20, 25, 100]. Compare its mean and median. Explain why they differ.

Exercise 3: Variance and Standard Deviation

Calculate population variance and standard deviation for [5, 10, 15, 20, 25]. Repeat the calculations using ddof=1.

Exercise 4: Axis Operations

Create a 3 by 3 array of numbers. Calculate row totals, column totals, row averages, and column averages.

Exercise 5: Percentiles

Use np.percentile() to calculate the 25th, 50th, and 75th percentiles of a numerical dataset.

Exercise 6: Student Analysis

Create a matrix containing marks for three students in four subjects. Find the total and average for each student and the average for each subject.

Summary

Aggregate and statistical functions turn raw numerical arrays into useful summaries. They help describe totals, central tendency, spread, and the distribution of values.

  • Use np.sum() and np.prod() for totals and products.
  • Use np.mean(), np.average(), and np.median() to describe central tendency.
  • Use np.min(), np.max(), and the difference between them to calculate the range.
  • Use np.var() and np.std() to measure data variability.
  • Use np.percentile() and np.quantile() to study the distribution of values.
  • Use the axis parameter to calculate row-wise or column-wise results.
  • Use NaN-aware functions when missing numerical values should be excluded from a calculation.

These functions are essential tools for data analysis and prepare you for more advanced numerical work with NumPy.