Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

Concatenation in Pandas

Learn how to combine multiple Series and DataFrames using Pandas concat(), align columns, manage indexes, and build larger datasets from smaller tables.

Learning objectives: Understand pd.concat(), axis, ignore_index, keys, join options, and common concatenation patterns.

1. What Is Concatenation?

Concatenation means combining two or more Pandas objects along a selected axis. You can stack DataFrames vertically, place them side by side, or combine Series objects.

Real-world example

Suppose Samadhan has student records for January, Pradnya has records for February, and Samiksha has records for March. If all three datasets contain compatible columns, you can concatenate them into one DataFrame.

Two Ways to Concatenate DataFrame A Rows 1–2 DataFrame B Rows 3–4 pd.concat() axis=0 Stack rows Combined rows A row 1 A row 2 B row 1, B row 2 DataFrame A Student DataFrame B Score axis=1 • Place columns side by side Indexes are aligned

Use axis=0 to append rows and axis=1 to align objects side by side.

2. Import Pandas

Start by importing the Pandas library.

import pandas as pd

The alias pd is the standard convention used in most Pandas examples.

3. Create Two DataFrames

Consider two DataFrames containing student information. Both have the same columns, but they contain different rows.

df1 = pd.DataFrame({
    "Student": ["Samadhan", "Pradnya"],
    "Score": [85, 92]
})

df2 = pd.DataFrame({
    "Student": ["Samiksha", "Aarav"],
    "Score": [88, 76]
})

print("DataFrame 1:")
print(df1)

print("\nDataFrame 2:")
print(df2)

Expected output

DataFrame 1:
    Student  Score
0  Samadhan     85
1   Pradnya     92

DataFrame 2:
    Student  Score
0  Samiksha     88
1     Aarav     76

Both DataFrames have the same column structure, making them suitable for row-wise concatenation.

4. Basic Concatenation with pd.concat()

The pd.concat() function combines Pandas objects into one result.

combined = pd.concat([df1, df2])

print(combined)

Expected output

    Student  Score
0  Samadhan     85
1   Pradnya     92
0  Samiksha     88
1     Aarav     76

Notice that the original index labels are preserved. Both input DataFrames have indexes 0 and 1, so the combined DataFrame contains duplicate index labels.

Key point: Concatenation does not automatically guarantee unique index labels. Use ignore_index=True when you want a fresh sequential index.

5. Use ignore_index=True

The ignore_index=True parameter tells Pandas to create a new sequential index for the concatenated result.

combined = pd.concat(
    [df1, df2],
    ignore_index=True
)

print(combined)

Expected output

    Student  Score
0  Samadhan     85
1   Pradnya     92
2  Samiksha     88
3     Aarav     76

Use this option when combining records from multiple files or batches and the original row indexes are not meaningful.

6. Concatenate Three or More DataFrames

The input to pd.concat() is a list or another iterable of Pandas objects. You can combine several DataFrames in one operation.

df_january = pd.DataFrame({
    "Student": ["Samadhan", "Pradnya"],
    "Score": [85, 92]
})

df_february = pd.DataFrame({
    "Student": ["Samiksha", "Aarav"],
    "Score": [88, 76]
})

df_march = pd.DataFrame({
    "Student": ["Meera", "Kabir"],
    "Score": [91, 82]
})

all_students = pd.concat(
    [df_january, df_february, df_march],
    ignore_index=True
)

print(all_students)

Expected output

    Student  Score
0  Samadhan     85
1   Pradnya     92
2  Samiksha     88
3     Aarav     76
4     Meera     91
5     Kabir     82

This pattern is especially useful when loading monthly reports, quarterly records, or multiple files with compatible columns.

7. Understand the axis Parameter

The axis parameter determines the direction in which Pandas combines the objects.

Value Direction Common use
axis=0 Row-wise Append records vertically.
axis=1 Column-wise Place columns side by side.

Example A: axis=0

result = pd.concat([df1, df2], axis=0, ignore_index=True)

print(result)

Expected output

    Student  Score
0  Samadhan     85
1   Pradnya     92
2  Samiksha     88
3     Aarav     76

Example B: axis=1

For a column-wise example, use DataFrames with matching indexes and different columns.

names = pd.DataFrame({
    "Student": ["Samadhan", "Pradnya", "Samiksha"]
})

scores = pd.DataFrame({
    "Score": [85, 92, 88]
})

result = pd.concat([names, scores], axis=1)

print(result)

Expected output

    Student  Score
0  Samadhan     85
1   Pradnya     92
2  Samiksha     88
Important: With axis=1, Pandas aligns objects by index labels, not simply by the visible order of their rows. If the indexes do not match, missing values may appear.

8. Concatenation with Different Columns

DataFrames do not need to have identical columns. By default, row-wise concatenation uses the union of column labels and fills missing entries with NaN.

df_a = pd.DataFrame({
    "Student": ["Samadhan", "Pradnya"],
    "Score": [85, 92]
})

df_b = pd.DataFrame({
    "Student": ["Samiksha", "Aarav"],
    "Attendance": [95, 87]
})

combined = pd.concat(
    [df_a, df_b],
    ignore_index=True
)

print(combined)

Expected output

    Student  Score  Attendance
0  Samadhan   85.0         NaN
1   Pradnya   92.0         NaN
2  Samiksha    NaN        95.0
3     Aarav    NaN        87.0

The combined DataFrame contains every column found in either input. Rows without a value for a particular column receive a missing value.

9. Use join="inner" and join="outer"

The join parameter controls how Pandas handles columns or other axis labels that do not appear in every input object during concatenation.

Example A: join="outer"

The default is join="outer", which keeps the union of the non-concatenation axis labels.

outer_result = pd.concat(
    [df_a, df_b],
    ignore_index=True,
    join="outer"
)

print(outer_result)

Expected output

    Student  Score  Attendance
0  Samadhan   85.0         NaN
1   Pradnya   92.0         NaN
2  Samiksha    NaN        95.0
3     Aarav    NaN        87.0

Example B: join="inner"

Inner concatenation keeps only column labels shared by both DataFrames.

inner_result = pd.concat(
    [df_a, df_b],
    ignore_index=True,
    join="inner"
)

print(inner_result)

Expected output

    Student
0  Samadhan
1   Pradnya
2  Samiksha
3     Aarav

Only the Student column is shared by both DataFrames, so the inner result contains that column.

10. Use keys to Identify Data Sources

The keys parameter adds an outer index level that identifies the source of each row. This is useful when combining regional reports, monthly datasets, or department records.

january = pd.DataFrame({
    "Student": ["Samadhan", "Pradnya"],
    "Score": [85, 92]
})

february = pd.DataFrame({
    "Student": ["Samiksha", "Aarav"],
    "Score": [88, 76]
})

combined = pd.concat(
    [january, february],
    keys=["January", "February"]
)

print(combined)

Expected output

                  Student  Score
January  0       Samadhan     85
         1        Pradnya     92
February 0       Samiksha     88
         1          Aarav     76

The result has a MultiIndex. The first index level identifies the source, and the second level preserves the original row index.

Access rows from a specific source

print(combined.loc["January"])

Expected output

    Student  Score
0  Samadhan     85
1   Pradnya     92

11. Concatenate Series Objects

Pandas can concatenate Series objects as well as DataFrames.

Example A: Concatenate Series vertically

series_a = pd.Series([10, 20, 30])
series_b = pd.Series([40, 50, 60])

result = pd.concat(
    [series_a, series_b],
    ignore_index=True
)

print(result)

Expected output

0    10
1    20
2    30
3    40
4    50
5    60
dtype: int64

Example B: Concatenate Series side by side

names = pd.Series(["Samadhan", "Pradnya", "Samiksha"], name="Student")
scores = pd.Series([85, 92, 88], name="Score")

result = pd.concat([names, scores], axis=1)

print(result)

Expected output

    Student  Score
0  Samadhan     85
1   Pradnya     92
2  Samiksha     88

12. Use verify_integrity=True

When concatenating along rows, you may want to ensure that the resulting index has no duplicate labels. Set verify_integrity=True to raise a ValueError if the concatenated axis contains duplicate labels.

try:
    result = pd.concat(
        [df1, df2],
        verify_integrity=True
    )
except ValueError as error:
    print(type(error).__name__)
    print(error)

Expected output

ValueError
Indexes have overlapping values: Index([0, 1], dtype='int64')

The exact wording of the error can vary by Pandas version. In this example, both DataFrames contain the same index labels, so the integrity check fails.

Tip: Use ignore_index=True if you want a fresh index rather than preserving the original labels. Use verify_integrity=True when existing labels must remain unique.

13. Concatenate DataFrames from Multiple CSV Files

A common data-engineering task is combining several CSV files with the same column structure.

import pandas as pd
from pathlib import Path

data_folder = Path("data")

files = sorted(data_folder.glob("*.csv"))

dataframes = [
    pd.read_csv(file)
    for file in files
]

if dataframes:
    combined = pd.concat(dataframes, ignore_index=True)
    print(combined.head())
else:
    print("No CSV files found.")

This example reads all CSV files directly inside the data directory, sorts their paths for predictable processing order, and combines them into one DataFrame.

Before combining files: Confirm that their column names and data types are compatible. If files contain different schemas, inspect the resulting columns and missing values after concatenation.

14. Common Mistakes and Best Practices

Mistake 1: Expecting automatic index renumbering

By default, concatenation preserves existing index labels. Set ignore_index=True when you need a new sequential index.

Mistake 2: Using axis=1 with mismatched indexes

Column-wise concatenation aligns rows using index labels. Inspect the indexes before combining datasets side by side.

Mistake 3: Ignoring missing columns

Outer concatenation can introduce missing values when the inputs have different columns. Check the resulting schema and missing-value counts.

print(combined.isna().sum())

Mistake 4: Concatenating inside a loop unnecessarily

Repeatedly concatenating a growing DataFrame can be inefficient. Collect the pieces in a list and concatenate them once.

parts = []

for values in batches:
    parts.append(pd.DataFrame(values))

if parts:
    result = pd.concat(parts, ignore_index=True)

Mistake 5: Confusing concatenation with merging

Concatenation combines objects along an axis. Merging combines tables using matching key values, similar to a database join. Use pd.merge() when records need to match through a key such as StudentID.

15. Concatenation vs Merge vs Join

Operation Main purpose Typical function
Concatenation Stack rows or align objects along columns. pd.concat()
Merge Combine records using one or more key columns. pd.merge()
Join Combine DataFrames, commonly using indexes. DataFrame.join()

For example, concatenate two monthly student files when you want all records in one table. Merge a student table with a marks table when you need to match records using StudentID.

16. Practice Exercises

Exercise 1: Basic concatenation

Create two DataFrames with Student and Score columns, then concatenate them vertically.

Hint: Use pd.concat([df1, df2]).

Exercise 2: Reset the index

Concatenate two DataFrames and create a continuous index starting from zero.

Hint: Use ignore_index=True.

Exercise 3: Column-wise concatenation

Combine a Student DataFrame with a Score DataFrame side by side.

Hint: Use axis=1 and inspect their indexes.

Exercise 4: Different columns

Concatenate one DataFrame with Student and Score columns and another with Student and Attendance columns. Observe the missing values.

Hint: Compare join="outer" and join="inner".

Exercise 5: Identify data sources

Concatenate January and February DataFrames using keys to identify each source.

Hint: Use keys=["January", "February"].

Exercise 6: Combine CSV files

Read all CSV files in a folder and combine them into one DataFrame.

Hint: Use Path.glob(), pd.read_csv(), and pd.concat().

17. Revision Questions

  1. What is concatenation in Pandas?
  2. What does pd.concat() do?
  3. What is the difference between axis=0 and axis=1?
  4. Why would you use ignore_index=True?
  5. What is the difference between join="inner" and join="outer"?
  6. How does the keys parameter help identify data sources?
  7. What happens when concatenated DataFrames have different columns?
  8. How can verify_integrity=True help detect duplicate index labels?
  9. Why is collecting DataFrames in a list often preferable to concatenating repeatedly in a loop?
  10. When should you use merge instead of concat?

Summary

The Pandas pd.concat() function combines multiple Series or DataFrames along a selected axis. Use axis=0 to stack rows and axis=1 to align objects side by side. The ignore_index parameter resets the result's index, keys adds source labels, and join controls how non-concatenation axis labels are combined. Concatenation is particularly useful for combining compatible files, monthly reports, and batches of records.

Key concepts: pd.concat(), axis, ignore_index, keys, join, verify_integrity, and index alignment.