Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Pandas Core Data Structures

Understand Series, DataFrames, indexes, rows, columns, dimensions, shape, and size — the building blocks of effective data analysis with Python.

1. Learning Objectives

By the end of this lesson, you will be able to:

  • Explain the main data structures in Pandas.
  • Create a Series from a list.
  • Create a DataFrame from a dictionary.
  • Understand row indexes and column labels.
  • Read the shape, size, and dimensions of a DataFrame.
  • Choose an appropriate data structure for a task.

2. What Are Pandas Data Structures?

A data structure organizes information so that a program can store, access, and process it. Pandas provides labelled structures that are especially useful for data analysis.

Series

A Series stores one-dimensional labelled data. It is similar to a single column of values with an index.

DataFrame

A DataFrame stores two-dimensional tabular data with rows and columns. Different columns can contain different data types.

Series Marks 80 92 86 One dimension DataFrame Name Marks Samadhan 80 Pradnya 92 Samiksha 86 Rows and columns

Figure 1: A Series stores one labelled sequence; a DataFrame organizes information into a table.

3. Creating a Pandas Series

You can create a Series using pd.Series(). If you do not provide an index, Pandas normally assigns default integer labels starting from zero.

import pandas as pd

marks = pd.Series([80, 92, 86])

print(marks)

Output:

0    80
1    92
2    86
dtype: int64

Creating a Series with Custom Labels

marks = pd.Series(
    [80, 92, 86],
    index=["Samadhan", "Pradnya", "Samiksha"]
)

print(marks)
print("Pradnya's marks:", marks["Pradnya"])

Output:

Samadhan    80
Pradnya     92
Samiksha    86
dtype: int64

Pradnya's marks: 92
Remember: A Series has values and an index. Its labels can be integers, strings, or other appropriate index values.

4. Creating a DataFrame

The pd.DataFrame() function can create a table from a dictionary, list of dictionaries, or other supported data structures.

Example: CIIT Student Records

import pandas as pd

students = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Course": ["Python", "Pandas", "Data Analysis"],
    "Marks": [80, 92, 86]
})

print(students)

Output:

       Name         Course  Marks
0  Samadhan         Python     80
1   Pradnya         Pandas     92
2  Samiksha  Data Analysis     86

Understanding the Dictionary

  • Name is a column label.
  • Course is another column label.
  • Marks contains numerical values.
  • Each row represents one student's record.
  • The default index identifies rows with labels 0, 1, and 2.

5. Understanding Indexes, Rows and Columns

A DataFrame uses row indexes and column labels to identify its data. By default, a DataFrame created from a dictionary generally receives a zero-based integer index.

Term Meaning Example
Index Labels identifying rows. 0, 1, 2
Row A record across the columns. Samadhan, Python, 80
Column A named field containing values. Name, Course, Marks
Column labels The names used to identify columns. Name and Marks

Inspect Index and Columns

print(students.index)
print(students.columns)

Output:

RangeIndex(start=0, stop=3, step=1)
Index(['Name', 'Course', 'Marks'], dtype='object')

The index identifies the rows, while the columns property returns the column labels.

6. Shape, Size and Dimensions

Pandas provides attributes that describe the structure of a Series or DataFrame.

Attribute Purpose For the Example DataFrame
shape Returns a tuple representing the dimensions. (3, 3)
size Returns the total number of elements. 9
ndim Returns the number of dimensions. 2
print("Shape:", students.shape)
print("Size:", students.size)
print("Dimensions:", students.ndim)

Output:

Shape: (3, 3)
Size: 9
Dimensions: 2
Important distinction: shape is a tuple containing the number of rows and columns. size is the total number of cells. ndim tells you how many dimensions the object has.

7. Accessing a Column and Its Values

A column can be selected using its label inside square brackets. Selecting a single column usually returns a Series.

print(students["Name"])

print("All marks:")
print(students["Marks"])

Output:

0    Samadhan
1     Pradnya
2    Samiksha
Name: Name, dtype: object

All marks:
0    80
1    92
2    86
Name: Marks, dtype: int64

Notice that selecting one column returns its values together with the index. Later, we will explore more selection methods, including loc and iloc.

8. Choosing the Right Data Structure

Requirement Suggested Structure
Store one labelled set of marks. Series
Store student names, courses and marks together. DataFrame
Inspect row labels. df.index
Inspect column names. df.columns
Check the number of rows and columns. df.shape
Count all table elements. df.size

9. Practice Exercise

CIIT Training Institute Student Dataset

Create a DataFrame with three students: Samadhan, Pradnya, and Samiksha.

Complete these tasks:

  1. Create a Series containing their marks.
  2. Create a DataFrame containing their names and courses.
  3. Print the index and column labels.
  4. Display the shape, size, and dimensions.
  5. Select and display the Marks column.

Starter Code

import pandas as pd

marks = pd.Series(
    [80, 92, 86],
    index=["Samadhan", "Pradnya", "Samiksha"]
)

students = pd.DataFrame({
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Course": ["Python", "Pandas", "Data Analysis"],
    "Marks": [80, 92, 86]
})

print("Series:")
print(marks)

print("\nDataFrame:")
print(students)

print("\nIndex:", students.index)
print("Columns:", students.columns)
print("Shape:", students.shape)
print("Size:", students.size)
print("Dimensions:", students.ndim)

Quick Revision Questions

  1. What is the difference between a Series and a DataFrame?
  2. What does a DataFrame index identify?
  3. What is returned by the columns attribute?
  4. What is the difference between shape and size?
  5. What is the value of ndim for a DataFrame?

Summary

Pandas provides Series and DataFrames for labelled data. A Series represents one-dimensional data, while a DataFrame organizes data into rows and columns.

You learned how to create both structures, inspect indexes and column labels, access a column, and use the shape, size, and ndim attributes to understand a dataset's structure.