Concatenation in Pandas
Learn how to combine multiple Series and DataFrames using Pandas concat(), align columns, manage indexes, and build larger datasets from smaller tables.
Learning objectives: Understand pd.concat(), axis, ignore_index, keys, join options, and common concatenation patterns.
1. What Is Concatenation?
Concatenation means combining two or more Pandas objects along a selected axis. You can stack DataFrames vertically, place them side by side, or combine Series objects.
Real-world example
Suppose Samadhan has student records for January, Pradnya has records for February, and Samiksha has records for March. If all three datasets contain compatible columns, you can concatenate them into one DataFrame.
Use axis=0 to append rows and axis=1 to align objects side by side.
2. Import Pandas
Start by importing the Pandas library.
import pandas as pd
The alias pd is the standard convention used in most Pandas examples.
3. Create Two DataFrames
Consider two DataFrames containing student information. Both have the same columns, but they contain different rows.
df1 = pd.DataFrame({
"Student": ["Samadhan", "Pradnya"],
"Score": [85, 92]
})
df2 = pd.DataFrame({
"Student": ["Samiksha", "Aarav"],
"Score": [88, 76]
})
print("DataFrame 1:")
print(df1)
print("\nDataFrame 2:")
print(df2)
Expected output
DataFrame 1:
Student Score
0 Samadhan 85
1 Pradnya 92
DataFrame 2:
Student Score
0 Samiksha 88
1 Aarav 76
Both DataFrames have the same column structure, making them suitable for row-wise concatenation.
4. Basic Concatenation with pd.concat()
The pd.concat() function combines Pandas objects into one result.
combined = pd.concat([df1, df2]) print(combined)
Expected output
Student Score 0 Samadhan 85 1 Pradnya 92 0 Samiksha 88 1 Aarav 76
Notice that the original index labels are preserved. Both input DataFrames have indexes 0 and 1, so the combined DataFrame contains duplicate index labels.
ignore_index=True when you want a fresh sequential index.
5. Use ignore_index=True
The ignore_index=True parameter tells Pandas to create a new sequential index for the concatenated result.
combined = pd.concat(
[df1, df2],
ignore_index=True
)
print(combined)
Expected output
Student Score 0 Samadhan 85 1 Pradnya 92 2 Samiksha 88 3 Aarav 76
Use this option when combining records from multiple files or batches and the original row indexes are not meaningful.
6. Concatenate Three or More DataFrames
The input to pd.concat() is a list or another iterable of Pandas objects. You can combine several DataFrames in one operation.
df_january = pd.DataFrame({
"Student": ["Samadhan", "Pradnya"],
"Score": [85, 92]
})
df_february = pd.DataFrame({
"Student": ["Samiksha", "Aarav"],
"Score": [88, 76]
})
df_march = pd.DataFrame({
"Student": ["Meera", "Kabir"],
"Score": [91, 82]
})
all_students = pd.concat(
[df_january, df_february, df_march],
ignore_index=True
)
print(all_students)
Expected output
Student Score 0 Samadhan 85 1 Pradnya 92 2 Samiksha 88 3 Aarav 76 4 Meera 91 5 Kabir 82
This pattern is especially useful when loading monthly reports, quarterly records, or multiple files with compatible columns.
7. Understand the axis Parameter
The axis parameter determines the direction in which Pandas combines the objects.
| Value | Direction | Common use |
|---|---|---|
axis=0 |
Row-wise | Append records vertically. |
axis=1 |
Column-wise | Place columns side by side. |
Example A: axis=0
result = pd.concat([df1, df2], axis=0, ignore_index=True) print(result)
Expected output
Student Score 0 Samadhan 85 1 Pradnya 92 2 Samiksha 88 3 Aarav 76
Example B: axis=1
For a column-wise example, use DataFrames with matching indexes and different columns.
names = pd.DataFrame({
"Student": ["Samadhan", "Pradnya", "Samiksha"]
})
scores = pd.DataFrame({
"Score": [85, 92, 88]
})
result = pd.concat([names, scores], axis=1)
print(result)
Expected output
Student Score 0 Samadhan 85 1 Pradnya 92 2 Samiksha 88
axis=1, Pandas aligns objects by index labels, not simply by the visible order of their rows. If the indexes do not match, missing values may appear.
8. Concatenation with Different Columns
DataFrames do not need to have identical columns. By default, row-wise concatenation uses the union of column labels and fills missing entries with NaN.
df_a = pd.DataFrame({
"Student": ["Samadhan", "Pradnya"],
"Score": [85, 92]
})
df_b = pd.DataFrame({
"Student": ["Samiksha", "Aarav"],
"Attendance": [95, 87]
})
combined = pd.concat(
[df_a, df_b],
ignore_index=True
)
print(combined)
Expected output
Student Score Attendance 0 Samadhan 85.0 NaN 1 Pradnya 92.0 NaN 2 Samiksha NaN 95.0 3 Aarav NaN 87.0
The combined DataFrame contains every column found in either input. Rows without a value for a particular column receive a missing value.
9. Use join="inner" and join="outer"
The join parameter controls how Pandas handles columns or other axis labels that do not appear in every input object during concatenation.
Example A: join="outer"
The default is join="outer", which keeps the union of the non-concatenation axis labels.
outer_result = pd.concat(
[df_a, df_b],
ignore_index=True,
join="outer"
)
print(outer_result)
Expected output
Student Score Attendance 0 Samadhan 85.0 NaN 1 Pradnya 92.0 NaN 2 Samiksha NaN 95.0 3 Aarav NaN 87.0
Example B: join="inner"
Inner concatenation keeps only column labels shared by both DataFrames.
inner_result = pd.concat(
[df_a, df_b],
ignore_index=True,
join="inner"
)
print(inner_result)
Expected output
Student 0 Samadhan 1 Pradnya 2 Samiksha 3 Aarav
Only the Student column is shared by both DataFrames, so the inner result contains that column.
10. Use keys to Identify Data Sources
The keys parameter adds an outer index level that identifies the source of each row. This is useful when combining regional reports, monthly datasets, or department records.
january = pd.DataFrame({
"Student": ["Samadhan", "Pradnya"],
"Score": [85, 92]
})
february = pd.DataFrame({
"Student": ["Samiksha", "Aarav"],
"Score": [88, 76]
})
combined = pd.concat(
[january, february],
keys=["January", "February"]
)
print(combined)
Expected output
Student Score
January 0 Samadhan 85
1 Pradnya 92
February 0 Samiksha 88
1 Aarav 76
The result has a MultiIndex. The first index level identifies the source, and the second level preserves the original row index.
Access rows from a specific source
print(combined.loc["January"])
Expected output
Student Score 0 Samadhan 85 1 Pradnya 92
11. Concatenate Series Objects
Pandas can concatenate Series objects as well as DataFrames.
Example A: Concatenate Series vertically
series_a = pd.Series([10, 20, 30])
series_b = pd.Series([40, 50, 60])
result = pd.concat(
[series_a, series_b],
ignore_index=True
)
print(result)
Expected output
0 10 1 20 2 30 3 40 4 50 5 60 dtype: int64
Example B: Concatenate Series side by side
names = pd.Series(["Samadhan", "Pradnya", "Samiksha"], name="Student") scores = pd.Series([85, 92, 88], name="Score") result = pd.concat([names, scores], axis=1) print(result)
Expected output
Student Score 0 Samadhan 85 1 Pradnya 92 2 Samiksha 88
12. Use verify_integrity=True
When concatenating along rows, you may want to ensure that the resulting index has no duplicate labels. Set verify_integrity=True to raise a ValueError if the concatenated axis contains duplicate labels.
try:
result = pd.concat(
[df1, df2],
verify_integrity=True
)
except ValueError as error:
print(type(error).__name__)
print(error)
Expected output
ValueError Indexes have overlapping values: Index([0, 1], dtype='int64')
The exact wording of the error can vary by Pandas version. In this example, both DataFrames contain the same index labels, so the integrity check fails.
ignore_index=True if you want a fresh index rather than preserving the original labels. Use verify_integrity=True when existing labels must remain unique.
13. Concatenate DataFrames from Multiple CSV Files
A common data-engineering task is combining several CSV files with the same column structure.
import pandas as pd
from pathlib import Path
data_folder = Path("data")
files = sorted(data_folder.glob("*.csv"))
dataframes = [
pd.read_csv(file)
for file in files
]
if dataframes:
combined = pd.concat(dataframes, ignore_index=True)
print(combined.head())
else:
print("No CSV files found.")
This example reads all CSV files directly inside the data directory, sorts their paths for predictable processing order, and combines them into one DataFrame.
14. Common Mistakes and Best Practices
Mistake 1: Expecting automatic index renumbering
By default, concatenation preserves existing index labels. Set ignore_index=True when you need a new sequential index.
Mistake 2: Using axis=1 with mismatched indexes
Column-wise concatenation aligns rows using index labels. Inspect the indexes before combining datasets side by side.
Mistake 3: Ignoring missing columns
Outer concatenation can introduce missing values when the inputs have different columns. Check the resulting schema and missing-value counts.
print(combined.isna().sum())
Mistake 4: Concatenating inside a loop unnecessarily
Repeatedly concatenating a growing DataFrame can be inefficient. Collect the pieces in a list and concatenate them once.
parts = []
for values in batches:
parts.append(pd.DataFrame(values))
if parts:
result = pd.concat(parts, ignore_index=True)
Mistake 5: Confusing concatenation with merging
Concatenation combines objects along an axis. Merging combines tables using matching key values, similar to a database join. Use pd.merge() when records need to match through a key such as StudentID.
15. Concatenation vs Merge vs Join
| Operation | Main purpose | Typical function |
|---|---|---|
| Concatenation | Stack rows or align objects along columns. | pd.concat() |
| Merge | Combine records using one or more key columns. | pd.merge() |
| Join | Combine DataFrames, commonly using indexes. | DataFrame.join() |
For example, concatenate two monthly student files when you want all records in one table. Merge a student table with a marks table when you need to match records using StudentID.
16. Practice Exercises
Exercise 1: Basic concatenation
Create two DataFrames with Student and Score columns, then concatenate them vertically.
Hint: Use pd.concat([df1, df2]).
Exercise 2: Reset the index
Concatenate two DataFrames and create a continuous index starting from zero.
Hint: Use ignore_index=True.
Exercise 3: Column-wise concatenation
Combine a Student DataFrame with a Score DataFrame side by side.
Hint: Use axis=1 and inspect their indexes.
Exercise 4: Different columns
Concatenate one DataFrame with Student and Score columns and another with Student and Attendance columns. Observe the missing values.
Hint: Compare join="outer" and join="inner".
Exercise 5: Identify data sources
Concatenate January and February DataFrames using keys to identify each source.
Hint: Use keys=["January", "February"].
Exercise 6: Combine CSV files
Read all CSV files in a folder and combine them into one DataFrame.
Hint: Use Path.glob(), pd.read_csv(), and pd.concat().
17. Revision Questions
- What is concatenation in Pandas?
- What does
pd.concat()do? - What is the difference between
axis=0andaxis=1? - Why would you use
ignore_index=True? - What is the difference between
join="inner"andjoin="outer"? - How does the
keysparameter help identify data sources? - What happens when concatenated DataFrames have different columns?
- How can
verify_integrity=Truehelp detect duplicate index labels? - Why is collecting DataFrames in a list often preferable to concatenating repeatedly in a loop?
- When should you use merge instead of concat?
Summary
The Pandas pd.concat() function combines multiple Series or DataFrames along a selected axis. Use axis=0 to stack rows and axis=1 to align objects side by side. The ignore_index parameter resets the result's index, keys adds source labels, and join controls how non-concatenation axis labels are combined. Concatenation is particularly useful for combining compatible files, monthly reports, and batches of records.
Key concepts: pd.concat(), axis, ignore_index, keys, join, verify_integrity, and index alignment.