Array Iteration and Memory Efficiency
NumPy provides several ways to process array elements, from ordinary Python loops to specialized iteration tools and vectorized operations. Choosing the right approach can make programs easier to read, faster to execute, and more memory-efficient. In this lesson, you will learn basic iteration, multidimensional traversal, nditer(), ndenumerate(), vectorization, temporary arrays, and practical optimization techniques.
1. What Is Array Iteration?
Array iteration means accessing or processing elements of an array one at a time or in a defined sequence. You may iterate to display values, inspect records, apply special logic, or update selected data.
import numpy as np
numbers = np.array([10, 20, 30, 40, 50])
for value in numbers:
print(value)
Output
10 20 30 40 50
The loop visits each element in sequence. This is simple and readable for small arrays, but Python-level loops can be slower than NumPy operations for large numerical datasets.
Common Iteration Approaches
- Python
forloops for straightforward element processing. - Nested loops for explicit row-and-column traversal.
np.nditer()for flexible iteration over array elements.np.ndenumerate()for values together with their indices.- Vectorized operations for calculations performed across whole arrays.
2. Iterating Through a One-Dimensional Array
A one-dimensional array behaves like a sequence of values. You can use a normal loop whenever each element needs individual processing.
import numpy as np
marks = np.array([78, 85, 92, 67, 88])
for mark in marks:
print("Mark:", mark)
Output
Mark: 78 Mark: 85 Mark: 92 Mark: 67 Mark: 88
Calculate a Result for Each Element
import numpy as np
numbers = np.array([2, 4, 6, 8])
for number in numbers:
square = number ** 2
print(number, "squared =", square)
Output
2 squared = 4 4 squared = 16 6 squared = 36 8 squared = 64
This is useful when each element requires custom logic. If the calculation is a standard numerical operation, vectorization is often simpler and faster.
3. Iterating Through a Two-Dimensional Array
Iterating directly over a two-dimensional array yields one row at a time. Each row is itself a NumPy array.
import numpy as np
matrix = np.array([
[10, 20, 30],
[40, 50, 60],
[70, 80, 90]
])
for row in matrix:
print(row)
Output
[10 20 30] [40 50 60] [70 80 90]
Iterate Through Each Individual Value
import numpy as np
matrix = np.array([
[10, 20, 30],
[40, 50, 60]
])
for row in matrix:
for value in row:
print(value)
Output
10 20 30 40 50 60
Nested loops give explicit control over rows and elements. For multidimensional arrays, NumPy also provides iteration tools that avoid writing a separate nested loop for every dimension.
4. Using np.nditer()
The np.nditer() function provides a flexible way to
iterate through the elements of arrays with one or more dimensions.
It can traverse all elements without requiring a different number
of nested loops for each shape.
import numpy as np
matrix = np.array([
[10, 20, 30],
[40, 50, 60]
])
for value in np.nditer(matrix):
print(value)
Output
10 20 30 40 50 60
The default iteration order follows the array's logical C-style index order, so this example visits each row from left to right.
Iterate Over a Different Shape
import numpy as np
arr = np.arange(1, 13).reshape(3, 4)
for value in np.nditer(arr):
print(value, end=" ")
Output
1 2 3 4 5 6 7 8 9 10 11 12
np.nditer() when you need
element-by-element iteration across arrays with varying numbers
of dimensions. For ordinary arithmetic, prefer vectorized expressions
when they express the operation clearly.
5. Using np.ndenumerate()
Sometimes you need both the index and the value of an element.
np.ndenumerate() yields each element's multidimensional
index together with its value.
import numpy as np
matrix = np.array([
[10, 20, 30],
[40, 50, 60]
])
for index, value in np.ndenumerate(matrix):
print("Index:", index, "Value:", value)
Output
Index: (0, 0) Value: 10 Index: (0, 1) Value: 20 Index: (0, 2) Value: 30 Index: (1, 0) Value: 40 Index: (1, 1) Value: 50 Index: (1, 2) Value: 60
The index is a tuple. For a two-dimensional array, it contains the row index and column index.
Find Values Greater Than a Threshold
import numpy as np
marks = np.array([
[65, 82, 91],
[74, 88, 59]
])
for index, mark in np.ndenumerate(marks):
if mark >= 80:
print("Index:", index, "Mark:", mark)
Output
Index: (0, 1) Mark: 82 Index: (0, 2) Mark: 91 Index: (1, 1) Mark: 88
This is useful when each value must be reported with its position.
If you only need the matching indices or values, Boolean indexing
and functions such as np.where() may be more concise.
6. Vectorization: A More Efficient Alternative
Vectorization means expressing an operation on an entire array instead of explicitly processing every element in a Python loop. NumPy performs many numerical operations through optimized compiled implementations.
Using a Python Loop
import numpy as np
numbers = np.array([1, 2, 3, 4, 5])
squares = []
for number in numbers:
squares.append(number ** 2)
squares = np.array(squares)
print(squares)
Output
[ 1 4 9 16 25]
Using Vectorization
import numpy as np numbers = np.array([1, 2, 3, 4, 5]) squares = numbers ** 2 print(squares)
Output
[ 1 4 9 16 25]
Both approaches produce the same values. The vectorized expression is shorter and generally avoids Python-level iteration overhead.
Vectorized expressions often reduce Python-level loop overhead.
arr * 2, arr ** 2,
np.sqrt(arr), and arr.sum() when they
express the calculation you need.
7. Vectorized Conditions and Selection
You can use Boolean conditions and selection functions to process many elements without writing an explicit loop.
import numpy as np
marks = np.array([45, 72, 88, 36, 91, 64])
passed = marks >= 40
print("Pass conditions:", passed)
print("Passing marks:", marks[passed])
Output
Pass conditions: [ True True True False True True] Passing marks: [45 72 88 91 64]
Apply a Bonus Using np.where()
import numpy as np
marks = np.array([35, 55, 70, 85])
adjusted_marks = np.where(
marks >= 40,
marks + 5,
marks
)
print(adjusted_marks)
Output
[35 60 75 90]
The condition is applied element-wise. Passing marks receive a five-point increase, while marks below 40 remain unchanged.
8. Understanding Memory Usage
A NumPy array's element storage depends largely on its number of
elements and data type. The nbytes attribute reports
the total bytes occupied by the array's element data.
import numpy as np
arr = np.array([10, 20, 30, 40], dtype=np.int64)
print("Number of elements:", arr.size)
print("Bytes per element:", arr.itemsize)
print("Total element bytes:", arr.nbytes)
Output
Number of elements: 4 Bytes per element: 8 Total element bytes: 32
The array contains four elements, each occupying eight bytes, for a total of 32 bytes of element storage.
Memory Calculation
For a regular numeric array, the element-storage size can be calculated as:
Total element bytes = Number of elements × Bytes per element
For example, one million float64 values typically
require about 8,000,000 bytes of element storage, excluding
array-object overhead and any additional arrays created by operations.
nbytes reports the element
storage associated with the array's shape and data type. It does not
report all memory retained by the Python object or necessarily all
memory owned by a larger base array when the array is a view.
9. Choosing an Appropriate Data Type
Data types influence memory consumption. If your data fits safely within a smaller numeric type, using that type can reduce the memory required for the array.
import numpy as np
arr_int64 = np.arange(1000, dtype=np.int64)
arr_int32 = np.arange(1000, dtype=np.int32)
print("int64 bytes:", arr_int64.nbytes)
print("int32 bytes:", arr_int32.nbytes)
Output on Typical Platforms
int64 bytes: 8000 int32 bytes: 4000
In this example, the int32 array uses half the element
storage of the int64 array.
10. Avoiding Unnecessary Temporary Arrays
Some vectorized expressions create intermediate arrays. For large
datasets, these temporary arrays can increase peak memory usage.
You can sometimes reduce temporary allocations by using supported
operations with an out= argument.
Creating an Additional Result Array
import numpy as np numbers = np.arange(1_000_000, dtype=np.float64) result = numbers * 2 print(result[:5])
Output
[0. 2. 4. 6. 8.]
The multiplication creates a new result array.
Reusing an Existing Array
import numpy as np numbers = np.arange(1_000_000, dtype=np.float64) np.multiply(numbers, 2, out=numbers) print(numbers[:5])
Output
[0. 2. 4. 6. 8.]
This example writes the results into the existing array instead of creating a separate full-sized result array. The original values are overwritten.
out= can save memory,
but it may modify data that another part of your program still needs.
Use it only when overwriting the destination array is acceptable.
11. Broadcasting Instead of Repeating Data
Broadcasting allows NumPy to combine compatible arrays of different shapes without explicitly repeating the smaller array to match the larger one.
import numpy as np
marks = np.array([
[70, 80, 90],
[65, 75, 85]
])
bonus = np.array([5, 3, 2])
adjusted = marks + bonus
print(adjusted)
Output
[[75 83 92] [70 78 87]]
The bonus array is applied to each row. NumPy broadcasts the one-dimensional array across the two-dimensional array without requiring you to build a separate repeated bonus matrix.
12. Chunk Processing for Large Arrays
Sometimes a dataset is too large to process comfortably in one operation, or the calculation requires only a small working set. Processing data in chunks can limit the size of temporary results.
import numpy as np
data = np.arange(1, 21, dtype=np.int64)
chunk_size = 5
chunk_sums = []
for start in range(0, data.size, chunk_size):
chunk = data[start:start + chunk_size]
chunk_sums.append(chunk.sum())
print("Chunk sums:", chunk_sums)
print("Total sum:", sum(chunk_sums))
Output
Chunk sums: [np.int64(15), np.int64(40), np.int64(65), np.int64(90)] Total sum: 210
The array is processed in groups of five elements. Each chunk is a basic slice, so it generally shares the original array's data rather than copying all its elements.
Chunking is useful when intermediate calculations would otherwise create large arrays. For simple sums, NumPy can sum the complete array directly; chunking is most valuable when it solves a real memory or processing constraint.
13. Comparing Iteration and Vectorization
| Approach | Best Use | Performance and Memory Notes |
|---|---|---|
Python for loop |
Custom element-by-element logic | Easy to understand, but may have Python overhead. |
np.nditer() |
Flexible iteration over multiple dimensions | Convenient traversal; not automatically faster than vectorization. |
np.ndenumerate() |
Processing values with their indices | Useful for index-aware logic, but still iterates at Python level. |
| Vectorized operations | Array-wide mathematical calculations | Often reduce Python overhead, but may create temporary arrays. |
| Chunk processing | Large data and limited working memory | Can limit temporary memory requirements; introduces loop overhead. |
Operations with out= |
Supported in-place calculations | Can avoid a separate result array when overwriting is safe. |
There is no single fastest approach for every problem. Choose based on the operation, data size, memory constraints, and whether each element requires unique logic.
14. Practical Project: Student Marks Analysis
In this example, CIIT TRANING INSTITUTE analyzes marks for Samadhan, Pradnya, and Samiksha. We will calculate total marks, average marks, and passing status using vectorized operations.
import numpy as np
student_names = np.array([
"Samadhan",
"Pradnya",
"Samiksha"
])
marks = np.array([
[78, 82, 88],
[85, 90, 87],
[92, 95, 89]
])
total_marks = marks.sum(axis=1)
average_marks = marks.mean(axis=1)
passed_all = np.all(marks >= 40, axis=1)
for i, name in enumerate(student_names):
print("Student:", name)
print("Total:", total_marks[i])
print("Average:", round(average_marks[i], 2))
print("Passed all subjects:", passed_all[i])
print()
Output
Student: Samadhan Total: 248 Average: 82.67 Passed all subjects: True Student: Pradnya Total: 262 Average: 87.33 Passed all subjects: True Student: Samiksha Total: 276 Average: 92.0 Passed all subjects: True
The totals, averages, and passing checks are calculated across entire rows using NumPy operations. The loop is used only to format the final report for each student.
15. Common Mistakes
Mistake 1: Using Python Loops for Every Calculation
Many numerical calculations can be expressed directly as array operations. Prefer vectorization when it is clear and appropriate.
Mistake 2: Assuming nditer() Always Improves Performance
np.nditer() simplifies multidimensional traversal,
but Python-level processing still has overhead. It is not a
replacement for vectorization in every numerical calculation.
Mistake 3: Ignoring Temporary Arrays
Several chained expressions may allocate intermediate results.
For large arrays, inspect memory needs and consider supported
out= operations or chunk processing.
Mistake 4: Choosing a Smaller Data Type Without Checking the Range
Smaller types can save memory, but they may overflow or lose precision. Check the required value range before converting.
Mistake 5: Assuming Chunking Is Always Faster
Chunking adds loop overhead. Use it when it helps manage working memory or enables processing of data that otherwise would not fit.
16. Practice Exercises
Exercise 1: Basic Iteration
Create an array containing ten integers and print each element
with a Python for loop.
Exercise 2: Multidimensional Iteration
Create a 3 by 3 matrix and print all values using
np.nditer().
Exercise 3: Index and Value
Use np.ndenumerate() to display the index and value
of every element in a two-dimensional array.
Exercise 4: Vectorization
Create an array of numbers from 1 to 20 and calculate the square of each value without using an explicit Python loop.
Exercise 5: Memory Usage
Create arrays with 1,000 elements using int32 and
int64. Compare their nbytes values.
Exercise 6: Avoid Temporary Arrays
Create a large floating-point array and use
np.multiply() with out= to double
its values in place.
Exercise 7: Chunk Processing
Divide an array into chunks of ten elements and calculate the sum of each chunk.
Exercise 8: Student Analysis
Create a marks matrix for Samadhan, Pradnya, and Samiksha. Calculate row totals, subject averages, and passing status using vectorized operations.
Summary
NumPy supports ordinary loops, np.nditer(), and
np.ndenumerate() for element-wise traversal. These
techniques are useful when individual values or their indices
need custom processing.
Vectorized expressions such as arr ** 2,
arr.sum(), and np.where() often reduce
Python-level loop overhead. Memory efficiency can be improved by
choosing appropriate data types, avoiding unnecessary temporary
arrays, using supported out= operations, and processing
data in chunks when needed.
The best approach depends on the task, data size, memory requirements, and correctness constraints. Prefer clear vectorized code for numerical calculations, and use explicit iteration when it makes specialized logic easier to express.