Every cleaning job starts the same way: look. Profiling means computing the shape of your dataset — row counts, data types, null rates, value distributions, suspicious ranges — before changing a single cell. Ten minutes of profiling routinely saves hours of cleaning the wrong thing.

import pandas as pd

df = pd.read_csv("orders.csv")

df.info()          # dtypes + non-null counts per column
df.describe()      # min / max / mean — impossible values surface here
df["country"].value_counts()  # "US", "USA", "us ", "United States"…

1. Null rates per column — a 40%-empty column is a design problem, not a fillna problem.
2. Impossible ranges — age 250, negative prices, dates in the future.
3. Cardinality surprises — 47 variants of the same country name.
4. Format drift — dates stored as text, numbers with currency symbols.

Duplicates arrive from retried API calls, merged systems, and copy-paste exports. They inflate counts, skew averages, and make you doubt which record is authoritative. The fix is a decision: define what makes two rows the same, then keep one — usually the most recent.

# Same customer, latest record wins
df = (df.sort_values("updated_at")
        .drop_duplicates(subset=["email"], keep="last"))

Blanks stop calculations cold — or worse, silently skew them. There are three legitimate moves: drop the row or column, impute a value (median, forward-fill, model-based), or flag the gap and keep it visible. The wrong default is pretending the gap isn’t there.

df["age_was_missing"] = df["age"].isna()  # keep the evidence
df["age"] = df["age"].fillna(df["age"].median())

Structural errors are the tyranny of small differences: capitalization, abbreviations, date formats, units, trailing spaces. Each variant becomes its own group, and every grouped report quietly lies. Pick one canonical form per column and enforce it.

df["country"] = (df["country"].str.strip().str.lower()
    .replace({"usa": "united states", "us": "united states",
              "u.s.a.": "united states"}))
df["order_date"] = pd.to_datetime(df["order_date"], format="mixed")

1. Dates: 03/04/2026 — March 4th or April 3rd? Pick ISO and document it.
2. Units: kg vs lb, USD vs EUR — convert, don’t concatenate.
3. Categories: one canonical spelling per value, kept in a mapping table.
4. Whitespace: invisible, and the cause of half your failed joins.

A price of −4 is an error. A customer buying 10,000 units is a whale — or fraud. Outlier handling is not deletion; it is investigation. Statistical fences (like the 1.5×IQR rule) flag candidates; domain knowledge decides their fate.

q1, q3 = df["amount"].quantile([0.25, 0.75])
iqr = q3 - q1
fence = (q1 - 1.5 * iqr, q3 + 1.5 * iqr)
suspect = df[~df["amount"].between(*fence)]  # review, don't delete

The job ends not when the data looks clean, but when it fails loudly the next time it isn’t. Validation turns your assumptions into executable tests — frameworks like Great Expectations or pandera run them on every new batch, forever.

assert df["email"].notna().all(), "null emails found"
assert df["amount"].ge(0).all(), "negative amount"
assert df["order_date"].max() <= pd.Timestamp.today()
# loud failure now > quiet wrong dashboard for six months