Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Scaling with PyArrow, Dask and Polars

Explore modern Python data-processing tools that help work with large datasets, efficient storage formats, parallel execution and optimized DataFrame operations.

1. Introduction to Scalable Data Processing

Pandas is widely used for data manipulation and analysis. However, some workloads become difficult to manage when datasets grow beyond available memory or require parallel processing.

Tools such as PyArrow, Dask and Polars provide additional capabilities for working with columnar data, large files and more demanding data-processing workloads.

Learning Objectives

  • Understand scalable data-processing concepts.
  • Learn the purpose of PyArrow.
  • Read and write Parquet files.
  • Understand Dask DataFrames.
  • Learn lazy execution and compute().
  • Work with Polars DataFrames and LazyFrames.
  • Compare different Python data-processing tools.
  • Select appropriate tools for different workloads.

2. Why Do We Need Alternative Data Processing Tools?

A traditional Pandas workflow often loads data into memory before performing operations. This works well for many datasets but can become challenging when datasets exceed available resources.

Challenge Possible Solution
Large file storage and interchange PyArrow and Parquet
Processing partitioned datasets Dask
Parallel and distributed computation Dask
Optimized DataFrame operations Polars
Columnar data representation Apache Arrow
Remember: These tools solve different problems. They are not direct replacements for one another in every situation.

3. Introduction to PyArrow

PyArrow is a Python library that provides access to Apache Arrow, a columnar in-memory data format designed for efficient data interchange and processing.

Apache Arrow provides a common representation for structured data that can be shared across different systems and libraries.

Key Features of PyArrow

  • Columnar memory representation.
  • Efficient data interchange between supported libraries.
  • Support for Parquet files.
  • Arrow tables and arrays.
  • Integration with Pandas and other data tools.

4. Installing PyArrow

Install PyArrow using pip from your terminal.

pip install pyarrow

Import the library and check its version.

import pyarrow as pa

print(pa.__version__)

The version number depends on the installed PyArrow release.

5. Creating an Arrow Table

PyArrow provides the Table structure for representing tabular data.

import pyarrow as pa

data = {
    "Name": ["Amit", "Neha", "Raj"],
    "Age": [25, 30, 22],
    "Salary": [35000, 50000, 28000]
}

table = pa.table(data)

print(table)

The dictionary is converted into an Arrow Table containing three columns.

Important: Arrow tables use a columnar representation. This makes them useful for data interchange and analytical workloads.

6. Converting Pandas DataFrame to PyArrow

PyArrow can convert a Pandas DataFrame into an Arrow Table using Table.from_pandas().

import pandas as pd
import pyarrow as pa

df = pd.DataFrame({
    "Name": ["Amit", "Neha", "Raj"],
    "Age": [25, 30, 22]
})

arrow_table = pa.Table.from_pandas(
    df,
    preserve_index=False
)

print(arrow_table)

The resulting Arrow Table can be used in workflows that support Apache Arrow.

Converting Arrow Table Back to Pandas

result = arrow_table.to_pandas()

print(result)

The to_pandas() method converts the Arrow Table back into a Pandas DataFrame.

7. Introduction to Parquet Files

Apache Parquet is a columnar storage file format designed for efficient storage and analytical queries.

Unlike CSV files, Parquet stores structured data in a column-oriented binary format and supports compression and schema information.

CSV Parquet
Text-based format. Columnar binary format.
Usually stores values as text. Stores typed column data.
Does not inherently store column types. Stores schema information.
Common for simple data exchange. Useful for analytical storage workflows.

8. Writing and Reading Parquet Files

Pandas supports Parquet file operations through to_parquet() and read_parquet().

import pandas as pd

df = pd.DataFrame({
    "Product": ["Laptop", "Mouse", "Keyboard"],
    "Price": [50000, 500, 1500],
    "Quantity": [2, 5, 3]
})

# Save as Parquet
df.to_parquet(
    "sales.parquet",
    engine="pyarrow",
    index=False
)

print("Parquet file created successfully!")

Reading the Parquet File

result = pd.read_parquet(
    "sales.parquet",
    engine="pyarrow"
)

print(result)

Output

    Product  Price  Quantity
0    Laptop  50000         2
1     Mouse    500         5
2  Keyboard   1500         3
Practical use: Parquet is useful for storing large analytical datasets when column selection and compression are important.

9. Introduction to Dask

Dask is a Python library for parallel and distributed computing. It provides familiar interfaces for working with collections of data that may be partitioned across multiple tasks.

Dask DataFrames divide data into partitions and organize operations into a task graph. Execution can happen later when a result is requested.

Key Features of Dask

  • Parallel computing.
  • Partitioned DataFrame processing.
  • Lazy task execution.
  • Integration with Pandas and NumPy.
  • Support for larger-than-memory workflows.
  • Distributed execution across multiple workers.

10. Installing Dask

Install Dask with its DataFrame dependencies.

pip install "dask[dataframe]"

Import Dask DataFrame in your Python program.

import dask.dataframe as dd

print(dd.__version__ if hasattr(dd, "__version__") else "Dask DataFrame imported successfully")

11. Reading CSV Files Using Dask

Dask provides read_csv() to read CSV datasets as partitioned DataFrames.

import dask.dataframe as dd

df = dd.read_csv("sales.csv")

print(df.head())

Dask creates a collection of partitions rather than immediately materializing the entire dataset as a single Pandas DataFrame.

Note: Dask may need to inspect parts of a CSV file to infer its data types. For inconsistent data, provide suitable dtype information explicitly.

12. Performing Operations with Dask DataFrame

Dask DataFrames support many familiar operations such as selecting columns, filtering rows and calculating aggregations.

import dask.dataframe as dd

df = dd.read_csv("sales.csv")

# Select required columns
selected = df[["Product", "Price"]]

# Filter records
filtered = selected[
    selected["Price"] > 1000
]

# Calculate total price
total_price = filtered["Price"].sum()

print(total_price.compute())

The computation is executed when compute() is called.

13. Understanding Lazy Execution in Dask

Lazy execution means that operations can be described first and executed later.

import dask.dataframe as dd

df = dd.read_csv("sales.csv")

filtered = df[df["Price"] > 1000]

total = filtered["Price"].sum()

# Execute the computation
result = total.compute()

print(result)

Dask builds a task graph for the operations. The graph is executed when a concrete result is requested.

Remember: You can combine multiple operations before calling compute(). This gives Dask an opportunity to optimize the execution graph.

14. Introduction to Polars

Polars is a fast DataFrame library written in Rust with a Python interface. It provides an expression-based API for data manipulation and supports eager and lazy execution.

Key Features of Polars

  • Rust-based implementation.
  • Column-oriented data processing.
  • Expression-based operations.
  • Lazy query execution.
  • Query optimization.
  • Parallel execution for supported operations.
  • CSV and Parquet file support.

15. Installing Polars

Install Polars using pip.

pip install polars

Import the library and check the installed version.

import polars as pl

print(pl.__version__)

16. Creating a Polars DataFrame

Polars provides DataFrame for storing and manipulating structured data.

import polars as pl

df = pl.DataFrame({
    "Name": ["Amit", "Neha", "Raj"],
    "Age": [25, 30, 22],
    "Salary": [35000, 50000, 28000]
})

print(df)

The resulting DataFrame contains three columns and three rows.

17. Performing Operations in Polars

Polars supports filtering, selecting columns and calculating new columns through expressions.

import polars as pl

df = pl.DataFrame({
    "Product": ["Laptop", "Mouse", "Keyboard"],
    "Price": [50000, 500, 1500],
    "Quantity": [2, 5, 3]
})

result = df.with_columns(
    (pl.col("Price") * pl.col("Quantity"))
    .alias("Total")
)

print(result)

The with_columns() method adds a calculated Total column to the DataFrame.

Filtering Records

filtered = df.filter(
    pl.col("Price") > 1000
)

print(filtered)

Selecting Columns

selected = df.select([
    "Product",
    "Price"
])

print(selected)

18. Understanding LazyFrame in Polars

Polars supports lazy execution through LazyFrame. Instead of immediately executing every operation, Polars builds a query plan that can be optimized before execution.

import polars as pl

query = (
    pl.scan_csv("sales.csv")
    .filter(pl.col("Price") > 1000)
    .select([
        "Product",
        "Price"
    ])
)

result = query.collect()

print(result)

Here, scan_csv() creates a lazy query. The filter and column selection are added to the query plan. Calling collect() executes the query and returns the result.

Best practice: Use lazy execution when several operations can be combined into one query plan. This allows Polars to apply optimizations before execution.

19. Pandas vs PyArrow vs Dask vs Polars

Each library has different strengths. The right choice depends on the type of data, workload, execution environment and application requirements.

Feature Pandas PyArrow Dask Polars
Main Purpose Data analysis Columnar data interchange Parallel computing DataFrame processing
Language Implementation Python and native components Python bindings for Arrow Python orchestration Rust with Python API
Lazy Execution Generally eager Not a general DataFrame query engine Yes Yes
Parallel Processing Depends on operation and libraries Supports efficient data processing Yes Yes, for supported operations
Large Data Workflows Limited by available memory in common workflows Efficient data interchange and storage Partitioned and distributed workflows Optimized DataFrame workflows
Common Use Exploration and analysis Arrow and Parquet Distributed processing Fast analytical transformations

20. When Should You Use Each Tool?

Use Pandas When

  • You need a familiar Python DataFrame API.
  • Your dataset fits comfortably in available memory.
  • You are performing exploratory data analysis.
  • You need integration with Python data science libraries.

Use PyArrow When

  • You need Apache Arrow data structures.
  • You work with Parquet files.
  • You need efficient columnar data interchange.
  • You integrate multiple analytical tools.

Use Dask When

  • You need partitioned DataFrame processing.
  • You want to distribute computations across workers.
  • Your workload can be represented as parallel tasks.
  • You need a familiar DataFrame-style API for larger workflows.

Use Polars When

  • You need efficient DataFrame transformations.
  • You want expression-based data processing.
  • You want lazy query optimization.
  • Your workflow benefits from its execution model.

21. Common Mistakes and Best Practices

Common Mistakes

  • Assuming PyArrow, Dask and Polars perform the same role.
  • Calling compute() repeatedly for the same Dask result.
  • Collecting a very large Polars LazyFrame without considering memory.
  • Converting large datasets between libraries unnecessarily.
  • Assuming Parquet will always be smaller than every CSV file.
  • Choosing a distributed system for a workload that does not need one.
  • Comparing library performance without equivalent test conditions.

Best Practices

  • Understand the strengths of each library.
  • Use Parquet for suitable analytical storage workflows.
  • Use Dask when partitioning and parallel execution are beneficial.
  • Use Polars lazy queries when they suit your processing pipeline.
  • Benchmark with representative datasets.
  • Check memory usage and output correctness.
  • Keep data conversions to a minimum.

22. Practice Exercises

Exercise 1: Create an Arrow Table

Create an Arrow Table containing Name, Age and Salary columns using PyArrow.

import pyarrow as pa

data = {
    "Name": ["Amit", "Neha", "Raj"],
    "Age": [25, 30, 22]
}

# Write your solution below

Exercise 2: Save Data as Parquet

Create a Pandas DataFrame and save it as a Parquet file using PyArrow.

import pandas as pd

df = pd.DataFrame({
    "Product": ["Laptop", "Mouse"],
    "Price": [50000, 500]
})

# Write your solution below

Exercise 3: Filter Using Dask

Read a CSV file with Dask and select records where Price is greater than 1000.

import dask.dataframe as dd

# Read the CSV file
# Apply the Price condition
# Execute using compute()

Exercise 4: Create a Polars Lazy Query

Read a CSV using Polars scan_csv(), filter records and select required columns. Execute the query using collect().

import polars as pl

# Write your solution below

Summary

PyArrow, Dask and Polars provide additional capabilities for scalable Python data processing. PyArrow supports efficient columnar data interchange and Parquet storage, while Dask provides partitioned and parallel computation.

Polars offers expression-based DataFrame operations, parallel execution and lazy query optimization. Each tool has different strengths and is suitable for different data-processing requirements.

Select the appropriate library based on dataset size, available memory, execution requirements and storage needs. Always benchmark performance and validate results before using an optimized workflow in production.