Six lessons from building a small, tested Python library for data cleaning

The full library, tests, and worked example are on GitHub: eiva-orce/data-cleaning-best-practices.

Most data cleaning advice reads like a recipe; drop the nulls, fill the gaps, remove the duplicates and move on.

In practice, almost none of that is safe to do without considering the effects. Every one of those steps is a decision that depends on what the downstream analysis actually needs and getting it wrong doesn’t throw an error, it just quietly corrupts your dataset.

I put together a small open-source library, “data-cleaning-best-practices”, of tested diagnostic functions for the checks I keep reaching for (missing values, suspicious categoricals, outliers, duplicates…) and walked them through a real, messy 10k-row book dataset. Sharing a summary of the six principles that came out of it.

TipTL;DR

None of this is exotic. It’s the ordinary, unglamorous work of knowing your data well enough to trust it, and building small, tested tools that make each decision visible instead of hiding it inside a one-line pandas call.

Note1. Diagnose before changes

The function’s only job is to surface the problem clearly and a human still decides what to do about it.

from cleaning_utils import missing_value_report

missing_value_report(df)
# -> count + % missing per column, nothing is dropped or filled

Note2. Scope every drop and fill to the column that actually matters

df.dropna() with no arguments drops a row if any column is missing, including columns nobody downstream ever touches. On the book dataset this quietly deleted rows over a missing ISBN which is a field the analysis never used. Scoping the drop to the specific column that mattered left the rest of the dataset intact. The same discipline applies to fills: log how many rows changed, so a fill is a visible, intentional choice rather than something that happens in the background.

from cleaning_utils import scoped_dropna, fillna_with_log

df = scoped_dropna(df, subset=["original_publication_year"])
df = fillna_with_log(df, column="language_code", value="unknown")
# -> prints how many rows were dropped / filled, so nothing changes silently

Note3. A broader filter is not a safer filter

This is the lesson that actually cost me time, and it’s the most important one in the repo. I was filtering book titles for known junk entries with specific brand names and it caught exactly one bad row with zero false positives. Broadening the filter to catch more generic “maybe junk” words pushed matches from 1 to 55 (anything with “Guide” or “Notes” in the title got removed). 54 of those were real books!!!

from cleaning_utils import compare_filter_precision

compare_filter_precision(
    df, text_column="title",
    narrow_terms=["BookRags", "SparkNotes", "CliffsNotes"],
    broad_terms=["Guide", "Notes"],
)
# -> shows exactly which extra rows the broader filter catches, before you commit to it

Note4. A statistical outlier is not automatically a data error

Outlier detection flagged one of the oldest known works of literature by publication year as an extreme statistical outlier. It’s also completely correct data. Outlier detection tells you where to look but it doesn’t tell you what’s wrong.

from cleaning_utils import outlier_report

outlier_report(df, column="original_publication_year", method="iqr")
# -> flags the row; confirming it's an error vs. genuine extreme is still a human call

Note5. Categorical columns hide entities that don’t belong

A real author writes a handful of books, not dozens. So when one author name turned up far more often than any real author would, that disproportionate frequency was the signal (not the missing data, not an outlier, just a value appearing where it statistically shouldn’t). It turned out not to be a book at all!!!

from cleaning_utils import flag_suspicious_categorical

flag_suspicious_categorical(df, column="authors", max_share=0.02)

Note6. Duplicates require judgment, not automatic deduplication**

Two rows sharing a title might be different editions, different translations, or a genuine scrape error. There’s no rule that resolves that automatically. What’s “correct” depends entirely on what the downstream analysis needs. The right move is surfacing duplicates for review, not silently merging or dropping them.

from cleaning_utils import duplicate_report

duplicate_report(df, column="title")
# -> surfaces the pairs; you decide what "duplicate" means for your analysis

---

So what: the time this actually saves

TipImpact

Turning ad-hoc checks into a tested, reusable library means the second, third, and tenth dataset don’t cost the same hour the first one did.

Every one of these checks used to be a one-off snippet, rewritten slightly differently each time a new dataset landed on my working list, re-deriving the same groupby().value_counts() logic, re-remembering which column needed a scoped drop, re-arguing with myself about whether a filter was too broad. That’s cheap the first time and expensive the fifth time. Making the checks as tested functions with a clear output removes that cost. Compare the two paths on a new dataset:

# Ad-hoc, written fresh each time
missing = df.isna().sum()
missing_pct = (missing / len(df) * 100).round(2)
report = pd.concat([missing, missing_pct], axis=1, keys=["count", "pct"])
report = report[report["count"] > 0].sort_values("pct", ascending=False)

# vs. reusable, tested
missing_value_report(df)

Both give the same answer. The second is one call I’ve already trusted multiple times.