Master Pandas From Scratch

Learn Pandas fundamentals, Series, DataFrames, data selection, filtering, cleaning, sorting, grouping, merging, reshaping, string operations, time series, data analysis, performance optimization, visualization, and practical projects through structured lessons.

COURSE SYLLABUS
1. Pandas Introduction and Environment Setup
Introduction to Pandas Installation and Environment Setup
2. Pandas Core Data Structures
Data Structures Overview Pandas Series DataFrames Indexes, Rows and Columns Shape, Size and Dimensions
3. Pandas Data Input and Output
Data Input and Output Overview Reading CSV, Excel and JSON Files Writing and Exporting Files
4. Initial Data Inspection
Data Inspection Overview Head and Tail Data Info and Types Descriptive Statistics
5. Selection and Indexing
Selection and Indexing Overview Column Selection loc and iloc Index Management
6. Filtering and Conditional Selection
Filtering Overview Boolean Filtering Multiple Conditions isin, between and str.contains
7. Basic Data Cleaning
Data Cleaning Overview Missing Values Duplicate Data Rename and Replace Data Type Conversion
8. Vectorized Operations and Sorting
Operations and Sorting Overview Column Operations Sorting Values and Indexes Sum, Mean, Max and Min
9. GroupBy and Aggregation
GroupBy Overview Grouping Data Aggregation Functions Group Filtering Group Transform
10. Merging, Joining and Concatenating
Merge, Join and Concat Overview Concatenation Merging DataFrames Joining by Index
11. Reshaping and Pivoting
Reshaping Overview Pivot Tables Melting Data Stack and Unstack Cross Tabulation
12. String and Categorical Data
String and Categorical Overview String Operations Categorical Data One-Hot Encoding
13. Time Series Analysis
Time Series Overview DateTime Operations Time-Based Indexes Resampling Data Shift and Rolling Windows
14. Custom Functions and Mapping
Apply and Mapping Overview Apply and Lambda Functions Map Operations Element-Wise Operations
15. Performance and Large Datasets
Performance Overview Memory Optimization Processing Files in Chunks PyArrow, Dask and Polars
16. Pandas Data Visualization
Visualization Overview Basic Plotting Histograms, Lines, Scatter and Box Plots Matplotlib and Seaborn

String Operations in Pandas

Learn how to manipulate, clean, search, replace, split, and validate text values in Pandas using the powerful string accessor. Explore practical examples for handling real-world textual data.

1. Introduction to String Operations

String operations are used to process and transform text values in a dataset. In data analysis, text columns often contain names, addresses, email addresses, product descriptions, categories, and other textual information that may require cleaning or transformation.

Pandas provides the .str accessor to apply string methods to an entire Series. These methods operate on individual values without requiring an explicit Python loop.

Common String Operations

  • Case conversion: Convert text to uppercase, lowercase, or title case.
  • Whitespace removal: Remove unnecessary spaces from text.
  • Searching: Check whether a value contains a particular pattern.
  • Replacement: Replace selected text with new text.
  • Splitting: Divide a string into multiple parts.
  • Validation: Check whether strings match a required pattern.

2. Creating a Sample DataFrame

Start by creating a DataFrame containing names, email addresses, and department names. These columns will be used throughout the lesson.

import pandas as pd

df = pd.DataFrame({
    "Name": ["  asha patil ", "RAVI SHARMA", "neha joshi"],
    "Email": [
        "asha@example.com",
        "ravi@example.com",
        "neha@example.com"
    ],
    "Department": ["IT", "Human Resources", "Finance"]
})

print(df)

Expected Output

           Name              Email       Department
0    asha patil   asha@example.com               IT
1    RAVI SHARMA  ravi@example.com   Human Resources
2     neha joshi  neha@example.com          Finance

The Name column contains inconsistent capitalization and extra spaces. String methods can standardize these values before further analysis.

3. Accessing String Methods with .str

Use the .str accessor to apply supported string methods to every value in a Series.

names = pd.Series(
    ["asha", "ravi", "neha"],
    dtype="string"
)

print(names.str.upper())

Expected Output

0    ASHA
1    RAVI
2    NEHA
dtype: string

The upper() method converts each string to uppercase. The accessor provides a convenient way to transform an entire column without writing a loop for every row.

4. Changing String Case

Pandas supports several methods for changing the capitalization of text values.

Method Purpose Example Result
str.lower() Converts text to lowercase. hello world
str.upper() Converts text to uppercase. HELLO WORLD
str.title() Capitalizes the beginning of each word. Hello World
str.capitalize() Capitalizes the first character and lowercases the remaining characters. Hello world
names = pd.Series(
    ["asha patil", "RAVI SHARMA", "neha joshi"],
    dtype="string"
)

print("Lowercase:")
print(names.str.lower())

print("\nUppercase:")
print(names.str.upper())

print("\nTitle case:")
print(names.str.title())

Expected Output

Lowercase:
0      asha patil
1     ravi sharma
2       neha joshi
dtype: string

Uppercase:
0      ASHA PATIL
1     RAVI SHARMA
2       NEHA JOSHI
dtype: string

Title case:
0      Asha Patil
1     Ravi Sharma
2       Neha Joshi
dtype: string
Tip: Use consistent capitalization when comparing text values. However, do not change capitalization when the original case is meaningful, such as in some identifiers or passwords.

5. Removing Extra Whitespace

Text imported from external sources may contain leading or trailing spaces. Pandas provides the following methods to handle whitespace.

  • str.strip() removes whitespace from both ends.
  • str.lstrip() removes whitespace from the left side.
  • str.rstrip() removes whitespace from the right side.
names = pd.Series(
    ["  Asha  ", " Ravi", "Neha   "],
    dtype="string"
)

clean_names = names.str.strip()

print(clean_names)

Expected Output

0    Asha
1    Ravi
2    Neha
dtype: string

Removing extra whitespace helps avoid duplicate-looking values that differ only because of spaces.

6. Combining Multiple String Operations

String methods can be chained together. For example, remove extra spaces first and then standardize capitalization.

df["Clean_Name"] = (
    df["Name"]
    .str.strip()
    .str.title()
)

print(df[["Name", "Clean_Name"]])

Expected Output

           Name      Clean_Name
0    asha patil      Asha Patil
1    RAVI SHARMA     Ravi Sharma
2     neha joshi       Neha Joshi

Chaining methods makes transformations concise and readable. Each operation is applied to the result of the previous operation.

7. Searching Text with contains()

The str.contains() method checks whether each string contains a specified pattern. By default, the pattern is interpreted as a regular expression.

names = pd.Series(
    ["Asha Patil", "Ravi Sharma", "Neha Joshi"],
    dtype="string"
)

result = names.str.contains(
    "sh",
    case=False,
    na=False
)

print(result)

Expected Output

0    False
1     True
2    False
dtype: boolean

The search is case-insensitive because case=False. Only the value Ravi Sharma contains the letters "sh" in this example.

Filtering a DataFrame

filtered_df = df[
    df["Name"].str.contains(
        "asha",
        case=False,
        na=False
    )
]

print(filtered_df)

Expected Output

         Name              Email Department
0  asha patil  asha@example.com          IT
Note: Because contains() uses regular expressions by default, special characters can have a special meaning. Use regex=False when you want a literal substring search.

8. Checking Prefixes and Suffixes

Use str.startswith() to check the beginning of a string and str.endswith() to check its ending.

emails = pd.Series([
    "asha@example.com",
    "ravi@company.org",
    "neha@example.com"
], dtype="string")

print(emails.str.endswith(".com"))
print(emails.str.startswith("asha"))

Expected Output

0     True
1    False
2     True
dtype: boolean

0     True
1    False
2    False
dtype: boolean

These methods can help identify file extensions, email domains, product codes, and other values with predictable prefixes or suffixes.

9. Replacing Text with replace()

Use str.replace() to replace a substring or pattern with another value.

departments = pd.Series(
    ["Information Technology", "Human Resources", "Information Technology"],
    dtype="string"
)

result = departments.str.replace(
    "Information Technology",
    "IT",
    regex=False
)

print(result)

Expected Output

0                  IT
1    Human Resources
2                  IT
dtype: string

The regex=False argument treats the old value as literal text rather than a regular expression.

10. Splitting Strings

The str.split() method divides text into parts based on a separator. With expand=True, Pandas returns the split parts as separate DataFrame columns.

names = pd.Series([
    "Asha Patil",
    "Ravi Sharma",
    "Neha Joshi"
], dtype="string")

result = names.str.split(" ", expand=True)

print(result)

Expected Output

      0       1
0  Asha   Patil
1  Ravi  Sharma
2  Neha   Joshi

The first output column contains the first names, and the second contains the surnames.

Assigning Meaningful Column Names

result.columns = ["First_Name", "Last_Name"]

print(result)

Expected Output

  First_Name Last_Name
0       Asha     Patil
1       Ravi    Sharma
2       Neha     Joshi

11. Joining Strings

The str.cat() method can concatenate strings from a Series. It can also combine values from two Series element by element.

Joining a Series into One String

names = pd.Series(["Asha", "Ravi", "Neha"])

result = names.str.cat(sep=", ")

print(result)

Expected Output

Asha, Ravi, Neha

Combining Two Series

first = pd.Series(["Asha", "Ravi", "Neha"])
last = pd.Series(["Patil", "Sharma", "Joshi"])

full_name = first.str.cat(last, sep=" ")

print(full_name)

Expected Output

0     Asha Patil
1    Ravi Sharma
2     Neha Joshi
dtype: object

12. Extracting Text with Regular Expressions

Regular expressions can identify and extract text patterns. The str.extract() method extracts capture groups from strings that match a regular expression.

emails = pd.Series([
    "asha@example.com",
    "ravi@company.org",
    "neha@example.com"
], dtype="string")

result = emails.str.extract(
    r"([^@]+)@([^@]+)"
)

result.columns = ["Username", "Domain"]

print(result)

Expected Output

  Username       Domain
0     asha  example.com
1     ravi  company.org
2     neha  example.com

The expression uses two capture groups: one for the text before the at sign and another for the text after it. The extracted groups become separate columns.

Important: This example demonstrates basic text extraction; it is not a complete email address validation rule.

13. Checking String Length

The str.len() method returns the length of each string. It can help identify unexpectedly short or long values.

codes = pd.Series(
    ["AB12", "XYZ789", "P5", "LMN456"],
    dtype="string"
)

print(codes.str.len())

Expected Output

0    4
1    6
2    2
3    6
dtype: Int64

The length values can be used to filter data or flag records that do not follow an expected format.

valid_codes = codes[codes.str.len() == 6]

print(valid_codes)

Expected Output

1    XYZ789
3    LMN456
dtype: string

14. Handling Missing Values in String Operations

Text columns can contain missing values. String operations may preserve missing values or allow you to specify how they should be treated, depending on the method and the dtype.

names = pd.Series(
    ["Asha", None, "Ravi"],
    dtype="string"
)

print(names.str.upper())

Expected Output

0    ASHA
1    <NA>
2    RAVI
dtype: string

If missing names should be excluded from a search, use the na parameter where the selected method supports it.

result = names.str.contains(
    "a",
    case=False,
    na=False
)

print(result)

Expected Output

0     True
1    False
2     True
dtype: boolean
Best practice: Decide how missing text should be handled before filtering or transforming data. Do not replace missing values with empty strings unless that behavior makes sense for your task.

15. Common String Methods Reference

Method Purpose
str.lower()Convert strings to lowercase.
str.upper()Convert strings to uppercase.
str.title()Convert text to title case.
str.strip()Remove whitespace from both ends.
str.contains()Test whether a pattern occurs.
str.startswith()Check the beginning of a string.
str.endswith()Check the ending of a string.
str.replace()Replace matching text.
str.split()Split text into separate parts.
str.cat()Concatenate strings.
str.extract()Extract capture groups from matching text.
str.len()Return the length of each string.

16. Common Mistakes and Best Practices

Common Mistakes

  • Forgetting to use the .str accessor for Series string operations.
  • Applying text methods to incompatible values without checking their data types.
  • Ignoring leading or trailing spaces in text comparisons.
  • Using regular expressions unintentionally when a literal search is required.
  • Ignoring missing values when searching or filtering text.
  • Assuming all text values follow the same format.

Best Practices

  • Inspect text columns and data types before transformation.
  • Clean text consistently before comparing or grouping values.
  • Use regex=False for literal replacement or search when appropriate.
  • Choose an explicit missing-value policy for filtering.
  • Use meaningful names when splitting text into new columns.
  • Validate transformed values before using them in further analysis.

17. Revision Questions

  1. What is the purpose of the Pandas .str accessor?
  2. What is the difference between str.lower() and str.upper()?
  3. How does str.strip() help clean text values?
  4. What does str.contains() return?
  5. How can you replace a substring in every value of a Series?
  6. What is the purpose of expand=True in str.split()?
  7. How can you extract parts of an email address using regular expressions?
  8. How should missing values be handled during string operations?

18. Practical Exercises

Exercise 1: Standardize Names

Create a Series containing names with inconsistent capitalization and extra spaces. Clean the names using strip() and title().

Goal: Produce consistent name values.

Exercise 2: Filter Email Addresses

Create a DataFrame containing email addresses from different domains. Filter the rows whose addresses end with @example.com.

Goal: Practice text searching and filtering.

Exercise 3: Split Full Names

Create a Series containing full names. Split each name into first-name and last-name columns.

Goal: Practice str.split() and column naming.

Exercise 4: Validate Product Codes

Create a Series containing product codes of different lengths. Use str.len() to select codes that match the required length.

Goal: Practice basic text validation.

Summary

Pandas string operations make it easy to process text stored in a Series or DataFrame column. The .str accessor supports capitalization, whitespace removal, searching, replacement, splitting, concatenation, extraction, and length checks. These operations are useful for cleaning inconsistent text, preparing data for filtering, and extracting structured information from textual values. Always consider missing values, data types, and regular-expression behavior when building reliable data-cleaning workflows.