Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Pandas DataFrames

Learn how to create a DataFrame, organize data into rows and columns, inspect its structure, select columns, and perform basic calculations using practical datasets.

1. Learning Objectives

After completing this lesson, you will be able to:

  • Explain what a Pandas DataFrame is.
  • Create a DataFrame from dictionaries and lists.
  • Understand DataFrame rows, columns, and indexes.
  • Inspect dimensions, column names, and data types.
  • Select individual and multiple columns.
  • Understand the difference between a Series and a DataFrame.
  • Perform calculations using DataFrame columns.
  • Apply DataFrames to a practical student dataset.

2. What Is a Pandas DataFrame?

A DataFrame is a two-dimensional labelled data structure provided by Pandas. It organizes information into rows and columns, similar to a spreadsheet or a database table.

Each row can represent a record, while each column represents a field. For example, a student dataset may contain student names, marks, and cities.

Pandas DataFrame Rows + Columns + Index Index Name Marks City 0 Samadhan 85 Pune 1 Pradnya 92 Mumbai 2 Samiksha 78 Nashik

Figure 1: A DataFrame organizes records into labelled columns and indexed rows.

Remember: A DataFrame can contain multiple columns with different data types, such as text, integers, floating-point numbers, and dates.

3. Creating a DataFrame from a Dictionary

One of the most common ways to create a DataFrame is by passing a dictionary to pd.DataFrame(). Dictionary keys become column names, and their values provide the column data.

import pandas as pd

data = {
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Marks": [85, 92, 78],
    "City": ["Pune", "Mumbai", "Nashik"]
}

df = pd.DataFrame(data)

print(df)

Output:

       Name  Marks    City
0  Samadhan     85    Pune
1   Pradnya     92  Mumbai
2  Samiksha     78  Nashik

Understanding the Code

  • import pandas as pd imports Pandas.
  • data stores the source information in a dictionary.
  • pd.DataFrame(data) creates the DataFrame.
  • print(df) displays the table.
Tip: When using a dictionary of lists, each list should normally contain the same number of values so the columns align correctly.

4. Creating a DataFrame from a List of Dictionaries

A list of dictionaries is useful when each dictionary represents one record. Pandas combines the records into a single tabular structure.

import pandas as pd

students = [
    {"Name": "Samadhan", "Marks": 85},
    {"Name": "Pradnya", "Marks": 92},
    {"Name": "Samiksha", "Marks": 78}
]

df = pd.DataFrame(students)

print(df)

Output:

       Name  Marks
0  Samadhan     85
1   Pradnya     92
2  Samiksha     78

In this example, each dictionary describes one student. The keys Name and Marks become columns in the resulting DataFrame.

Important: If some dictionaries omit a key that appears in other records, Pandas can represent the missing value in that column with a missing-value marker.

5. Creating a DataFrame from a List of Lists

You can create a DataFrame from a list of lists by supplying the column names through the columns parameter.

import pandas as pd

data = [
    ["Samadhan", 85],
    ["Pradnya", 92],
    ["Samiksha", 78]
]

df = pd.DataFrame(
    data,
    columns=["Name", "Marks"]
)

print(df)

Output:

       Name  Marks
0  Samadhan     85
1   Pradnya     92
2  Samiksha     78

Here, the first value in each inner list goes into the Name column, and the second value goes into the Marks column.

Common mistake: Ensure the number of column labels matches the number of values in each row. An incompatible shape can raise an error.

6. Understanding DataFrame Dimensions

DataFrame properties help you understand the size and structure of your dataset.

import pandas as pd

data = {
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Marks": [85, 92, 78],
    "City": ["Pune", "Mumbai", "Nashik"]
}

df = pd.DataFrame(data)

print("Shape:", df.shape)
print("Rows:", len(df))
print("Columns:", len(df.columns))
print("Total cells:", df.size)
print("Dimensions:", df.ndim)

Output:

Shape: (3, 3)
Rows: 3
Columns: 3
Total cells: 9
Dimensions: 2
Property Meaning Example
df.shape Number of rows and columns. (3, 3)
len(df) Number of rows. 3
df.size Total number of cells. 9
df.ndim Number of dimensions. 2

7. Inspecting Columns and Data Types

Use columns to view column labels and dtypes to inspect the data type of each column.

print("Column names:")
print(df.columns)

print("Data types:")
print(df.dtypes)

Example output:

Column names:
Index(['Name', 'Marks', 'City'], dtype='object')

Data types:
Name     object
Marks     int64
City     object
dtype: object

Text columns may use the object dtype, while whole-number values may use an integer dtype such as int64. The exact dtype can depend on the values and environment.

Previewing DataFrame Rows

print(df.head(2))
print(df.tail(2))
  • head(2) displays the first two rows.
  • tail(2) displays the last two rows.
  • head() and tail() use five rows by default.

8. Selecting a Single Column

You can select a single column by placing its name inside square brackets.

print(df["Name"])

Output:

0    Samadhan
1     Pradnya
2    Samiksha
Name: Name, dtype: object

Selecting one column with df["Name"] returns a Pandas Series. The original row index labels are retained.

Remember: A Series is one-dimensional. A DataFrame is two-dimensional. This distinction is useful when selecting data for analysis.

9. Selecting Multiple Columns

To select multiple columns, provide a list of column names inside the square brackets.

selected = df[["Name", "Marks"]]

print(selected)

Output:

       Name  Marks
0  Samadhan     85
1   Pradnya     92
2  Samiksha     78

The result is a DataFrame containing only the requested columns. The row index is preserved.

Expression Result
df["Name"] A Series.
df[["Name"]] A one-column DataFrame.
df[["Name", "Marks"]] A DataFrame with two columns.
Important: Column names must match the actual labels in your DataFrame. Requesting a column that does not exist raises a KeyError.

10. Creating a Calculated Column

A DataFrame can store the results of calculations in a new column. This is a common operation in data analysis.

import pandas as pd

data = {
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Python": [80, 95, 88],
    "Pandas": [85, 92, 90]
}

df = pd.DataFrame(data)

df["Average"] = (
    df["Python"] + df["Pandas"]
) / 2

print(df)

Output:

       Name  Python  Pandas  Average
0  Samadhan      80      85     82.5
1   Pradnya      95      92     93.5
2  Samiksha      88      90     89.0

Pandas performs the addition element by element and divides each result by two. The calculated values are assigned to the new Average column.

Calculate the Average of the Class

print(df["Average"].mean())

Output:

88.33333333333333
Practical use: Calculated columns are useful for total marks, average marks, revenue, profit, discounts, and other derived measures.

11. Practical Exercise

CIIT TRAINING INSTITUTE: Student Data Analysis

Create a DataFrame containing information about Samadhan, Pradnya, and Samiksha.

Complete the following tasks:

  1. Create columns for Name, Python, and Pandas.
  2. Display the complete DataFrame.
  3. Display only the Name column.
  4. Select the Name and Python columns.
  5. Display the DataFrame shape and data types.
  6. Create an Average column.
  7. Calculate the class average.

Practice Code

import pandas as pd

data = {
    "Name": ["Samadhan", "Pradnya", "Samiksha"],
    "Python": [80, 95, 88],
    "Pandas": [85, 92, 90]
}

df = pd.DataFrame(data)

print(df)

print("\nName column:")
print(df["Name"])

print("\nSelected columns:")
print(df[["Name", "Python"]])

print("\nShape:", df.shape)
print("\nData types:")
print(df.dtypes)

df["Average"] = (
    df["Python"] + df["Pandas"]
) / 2

print("\nAverage marks:")
print(df[["Name", "Average"]])

print("\nClass average:", df["Average"].mean())

Revision Questions

  1. What is a Pandas DataFrame?
  2. How can you create a DataFrame from a dictionary?
  3. What is the purpose of the columns parameter?
  4. What information does df.shape return?
  5. What is the difference between df["Name"] and df[["Name"]]?
  6. How can you inspect the data types of DataFrame columns?
  7. How can you create a calculated column?

Summary

A Pandas DataFrame is a two-dimensional labelled data structure that organizes information into rows and columns. You learned to create DataFrames from dictionaries, lists of dictionaries, and lists of lists.

You also explored DataFrame dimensions, column labels, data types, and row previews. We used single and multiple column selection and created a calculated column for student marks analysis. DataFrames are fundamental to data analysis because they allow you to organize, inspect, transform, and calculate information in a structured format.