CSV Data with NumPy
Learn how to read, process, analyse, and export CSV data using NumPy. Explore delimiters, headers, column selection, row-wise and column-wise calculations, filtering numerical records, and saving processed results to CSV files.
1. Learning Objectives
By the end of this lesson, you will be able to:
- Understand the structure of CSV files.
- Load numerical CSV data using np.loadtxt().
- Handle CSV headers and delimiters.
- Inspect rows, columns, and array shapes.
- Calculate totals, averages, minimums, and maximums.
- Filter numerical data using conditions.
- Export processed data using np.savetxt().
- Understand when genfromtxt() is useful for missing values.
2. What Is a CSV File?
CSV stands for Comma-Separated Values. It is a text-based file format in which values are commonly separated by commas. CSV files are often used to exchange tabular data between spreadsheets, databases, and programming applications.
Example file named marks.csv:
The first row contains column headings. Each following row contains numerical marks for one student.
CSV records can be loaded into NumPy for numerical analysis.
3. Loading CSV Data with np.loadtxt()
Use np.loadtxt() to load a CSV file when its data is consistently numerical and can be represented in a regular array.
Output:
The delimiter argument specifies commas, while skiprows=1 skips the textual header. The resulting array contains three rows and three columns.
Loading Values as Integers
Output:
4. Understanding Rows, Columns, and Shape
After loading CSV data, use the shape attribute to identify how many rows and columns are present.
Output:
In this dataset, each row represents one student and each column represents a subject.
Accessing a Row
Output:
Accessing a Column
Output:
5. Selecting Specific Columns
You can select a column by using a colon for all rows and the column index for the desired subject.
Output:
This technique is useful when you want to analyse one subject or one numerical field independently.
6. Calculating Row-Wise Totals and Averages
NumPy aggregation functions can calculate totals and averages for each row. Use axis=1 to perform calculations across columns within each row.
Output:
The first row has a total of 255 and an average of 85. The second row also has a total of 255 and an average of 85. The third row has a total of 272 and an average of approximately 90.67.
7. Calculating Column-Wise Statistics
To calculate statistics for each subject across all students, use axis=0.
Output:
Each result contains one value for each subject column: Python, NumPy, and Statistics.
8. Filtering CSV Data
Once CSV data is loaded into a NumPy array, Boolean indexing can select rows that satisfy a condition.
Select Students with Average Marks at Least 85
Output:
Every row in this example has an average of at least 85, so all three rows are selected.
Select Rows with Python Marks of at Least 80
Output:
9. Saving Processed Data to CSV
After analysing data, you can export the processed results to a new CSV file using np.savetxt().
Output:
The resulting file contains:
The fmt="%.2f" argument writes averages to two decimal places. The header provides a column label for the exported data.
10. Handling Missing Values with np.genfromtxt()
Real-world CSV files may contain missing numerical values. np.loadtxt() is best suited to clean, consistently formatted numerical data. For certain missing-value situations, np.genfromtxt() provides additional options.
Example file named measurements.csv:
Loading Missing Values as NaN
Output:
The missing value is represented by NaN, which stands for Not a Number.
Calculating an Average While Ignoring NaN
Output:
np.nanmean() calculates averages while ignoring NaN values. This is useful when a missing value should not be treated as a valid numerical measurement.
11. Practical Project: Student Marks Analysis
Project Description
At CIIT TRANING INSTITUTE, analyse the marks of Samadhan, Pradnya, and Samiksha. Store the marks in a CSV file, load the numerical records, calculate totals and averages, identify the highest average, and export the summary.
Complete Program
Output:
This project demonstrates a complete workflow: create numerical data, export it, load it, calculate statistics, and save the results in a separate CSV file.
12. Common Mistakes
Mistake 1: Forgetting the Delimiter
Specify delimiter="," when loading comma-separated data.
Mistake 2: Reading the Header as Numbers
Use skiprows=1 when the first row contains textual column headings and the remaining data is numerical.
Mistake 3: Confusing Rows and Columns
Use axis=1 for row-wise operations and axis=0 for column-wise operations.
Mistake 4: Ignoring Missing Values
Inspect the dataset and choose an appropriate missing-data strategy before calculating statistics.
Mistake 5: Using np.loadtxt() for Irregular Data
When numerical data contains missing values, consider np.genfromtxt() and configure its missing-value handling.
13. Practice Exercises
Exercise 1: Load a CSV File
Create a CSV file with three numerical columns. Load it using np.loadtxt() and display its shape.
Exercise 2: Select a Column
Load student marks and display only the NumPy marks column.
Exercise 3: Calculate Row Averages
Calculate each student's average using np.mean() with axis=1.
Exercise 4: Calculate Subject Averages
Calculate the average for each subject using axis=0.
Exercise 5: Filter Records
Select all rows in which the Python marks are at least 80.
Exercise 6: Export a Summary
Save totals and averages to a CSV file using np.savetxt(). Include a header and format averages to two decimal places.
Exercise 7: Handle Missing Values
Create a CSV file with one missing numerical value. Load it using np.genfromtxt() and calculate a NaN-aware column average.
Summary
In this lesson, you learned how to load numerical CSV
data using np.loadtxt(), handle delimiters
and headers, and select individual rows and columns.
You explored row-wise and column-wise statistics,
Boolean filtering, exporting processed results with
np.savetxt(), and handling missing values
with np.genfromtxt().