Key takeaways
- Clean one important source before connecting everything.
- Define each field and each total before building charts.
- Check duplicates and reconcile the cleaned data with a trusted source.
- Document the process so it can be repeated next month.
On this page
Start with one source
Do not connect every system at once. Pick the source that matters most for the first question, such as sales, invoices, or stock. Get one source right before adding the next.
A narrow, well-understood dataset is more useful than a broad, unreliable one. It also makes it easier to spot problems early.
Define the fields
Write down what each important field means. A date can be an invoice date, a payment date, or a delivery date. They are not interchangeable. A customer can be a billing entity, a location, or a contact.
Clear definitions prevent most reporting arguments. They also make the report easier to maintain when someone new takes over.
Remove duplicates
Duplicates usually appear when data is exported more than once or when several systems hold the same customer or order. Use a stable key and check the row count before and after cleaning.
Keep an exceptions list rather than deleting uncertain records silently. Someone may need to review them.
Standardise formats
Dates, currencies, product codes, and category names should use a consistent format. Mixed formats cause broken joins and misleading totals.
A simple mapping table is often enough. Document any manual mappings so they can be reused.
Reconcile the totals
Before building the report, reconcile the cleaned data with a trusted source: the bank, the ledger, or the source system. If the totals do not match, stop and fix the data.
Do not hide the difference in a chart or an “other” category. A reconciliation difference is a finding, not a design choice.
Document the process
Write a short note explaining where the data comes from, how it is cleaned, and who owns it. Include the refresh frequency and any known limitations.
The next person should be able to run the process without guessing. That documentation is part of the data product.
Frequently asked questions
How clean does the data need to be?+
Clean enough to reconcile with a trusted total and support the decision. Perfect data is rarely available; unexplained differences are the real problem.
Should I clean data in Excel or Python?+
Use Excel for small, one-off datasets with manual review. Use Python or SQL when the process repeats, files are large, or multiple sources need to be joined.
How often should data be cleaned?+
Clean at the source where possible. For recurring reports, run the same validation and reconciliation checks every refresh.
Need help applying this?
Improve your reporting automation.
Review the service scope or tell us what you are trying to improve. We will reply with a clear next step.
Free KPI definition templateRelated reading
A data quality checklist before starting a BI project
The checks that prevent a dashboard from failing after launch: ownership, definitions, completeness, duplicates, and reconciliation.
When to move from spreadsheets to a database
The signals that a spreadsheet has become a database, and the simplest path to a more reliable system.