← Back

Data Quality for AI: Six Checks and a Worked Example

TL;DR

Test the records needed for one decision. A six-row catalogue shows how to find duplicates, missing prices, unsupported currencies and expired records.

Conceptual illustration for Data Quality for AI: Six Checks and a Worked Example

Data is good enough for an AI application when it supports the specific decision the application must make. A complete spreadsheet can still contain outdated prices. A clean document library can still mix policies for different customers. A million records will not solve either problem if the system cannot identify which record is authoritative.

Start with one output, its source fields and the mistakes that would make it unusable. Then test a small, representative slice of the data against those requirements. This gives you a repair list before you spend money connecting a model to the full collection.

This guide includes a deliberately flawed, synthetic catalogue and a reproducible validation exercise. It is a test of data rules, not a benchmark of an AI model or a claim about a client's records.

Define what the AI must be able to answer

Consider an assistant that helps staff prepare product quotations. Before choosing a model, specify what a valid quotation needs: an unambiguous product identifier, a current price, a currency, the appropriate customer scope and a source record that an authorised employee can inspect.

Now ask what happens when one of those items is missing. Should the assistant ask a question, refer the quotation to a person or refuse to produce a price? Decide explicitly. Otherwise a model may produce a plausible number where the business needed an unresolved state.

The same principle applies to other workflows. A support assistant needs the policy that applies to this customer today. A document processor needs a field definition and evidence for the extracted value. A forecasting project needs a meaningful outcome label and a way to avoid learning from information that would not have been available at prediction time.

Six useful questions about the records

The Government Data Quality Framework distinguishes completeness, uniqueness, consistency, timeliness, validity and accuracy. Those dimensions help separate different problems. In particular, a populated field is not necessarily correct.

For the quotation example, translate them into concrete checks:

Question Check What a failure means
Are required values present? Price and currency are populated The quote cannot be completed from this record
Is the item uniquely identified? One current record per product and customer scope The system needs a rule for duplicates or versions
Do sources agree? Sales and catalogue records use the same effective terms Someone must identify the authoritative source
Is the record current? Effective and expiry dates cover the quote date An old price must not silently become today's price
Is the format allowed? Price is numeric; currency is supported Normalise or reject before downstream use
Is the value true? Compare with an approved price source A valid number may still be wrong

These are separate checks, not six percentages to average into an impressive overall score. A quotation with a perfect format and the wrong currency remains wrong.

Data quality checks: required fields, unique records, current versions and an authoritative source check.
Fictional sample and transparent assumptions; not a client result or model benchmark.

A six-row catalogue with four different problems

Our synthetic input contains the following rows. The complete fixture and expected findings are included in the downloadable test pack.

Row SKU Price Currency Current? Deliberate issue
1 A100 12.00 GBP Yes None in the tested fields
2 A100 12.00 GBP Yes Duplicate of row 1
3 B200 blank GBP Yes Missing price
4 C300 18.50 EUR Yes Currency outside this GBP-only workflow
5 D400 8.00 GBP No Expired price
6 E500 22.00 GBP Yes None in the tested fields

We ran deterministic validation against this fixture. It flags one repeated SKU, one missing price, one unsupported currency and one expired record. Those results show that the checks detect the deliberately inserted cases. They do not establish the accuracy of rows 1 and 6, prove that a real catalogue has the same error rate or show that an LLM can quote reliably.

Notice the distinction between a duplicate group and a duplicate row. A100 forms one group containing two records. Keeping one and removing the redundant copy leaves one usable candidate, provided the records really refer to the same product and scope. With conflicting prices, automatic deletion would hide the disagreement instead of resolving it.

Repair the source, or define the exception

Give every finding an owner and a decision. The missing B200 price might belong to the commercial team. The expired D400 record may require a current catalogue import. The EUR value may be legitimate for another market, in which case the correct response is to exclude it from this GBP workflow rather than overwrite it.

Record the original value, the corrected value, the reason and the applicable version. Avoid cleaning data only inside a prompt where the change cannot be traced. If multiple applications use the same flawed source, a prompt-level workaround also leaves the underlying problem in place for everyone else.

Some exceptions should remain exceptions. If the current price has not been approved, the system should ask for it. Turning every missing value into a guessed value can make a dataset look complete while making it less trustworthy.

Sample by the ways the work can fail

A random sample is useful, but it can miss a rare document type that causes serious errors. Start with the actual workload and deliberately cover different sources, record ages, languages, layouts and customer scopes where those differences matter.

Keep ordinary cases and challenging cases distinguishable. If you build a test pack mostly from known failures, its error rate is not an estimate of the entire production population. Conversely, a clean demonstration pack selected by hand will tell you little about messy daily work.

For a document workflow, retain the source page alongside the expected fields. For a retrieval system, retain the relevant passage and access scope. The worked RAG guide separates finding the right evidence from producing an answer; a data-quality check should preserve that distinction.

Prevent yesterday's answer leaking into tomorrow's test

If you tune prompts or train a model using a set of examples, reserve other examples for evaluation. Where the task has a time dimension, consider whether a chronological split is more representative than a random split. Records from the same customer, document or incident may also need to stay together to prevent near-duplicates appearing on both sides.

The important question is what information will really exist when the system makes its decision. A support ticket's final resolution cannot fairly be used as an input when testing how the system would classify the ticket at arrival. A later correction to an invoice cannot be treated as information the original processor already knew.

Document how the evaluation set was constructed. That lets a reviewer challenge the test rather than accepting a single accuracy number without context.

Separate access rules from content quality

A record can be accurate, current and forbidden to the person asking the question. Check customer boundaries, employee roles and source permissions independently of whether the text looks useful.

When access changes, test the entire route, including search indexes, cached results and downloadable exports. A successful permission change in the source application does not itself prove that every copied version has become inaccessible. The AI integration guide explains why the connection between systems needs explicit operating rules.

Data minimisation also makes debugging easier: the smaller the relevant input, the clearer it is which source supports a result. Do not export a whole customer database merely to test a narrow classification task.

Set a decision for the first pilot

Finish the assessment with a short decision record: which data may be used, which cases remain unsupported, which defects must be repaired and what happens when a new defect appears. Name the person who owns the authoritative source and the person who can pause the workflow.

The download includes the six-row fixture and a repair log template. The Python example below implements the four checks using only the standard library. Run it as a small example, then replace the rules with checks derived from your own task. A clean run means the selected rules passed; it is not a general certificate that the data is safe or accurate.

For an AI readiness audit, bring one representative export, the required output and a few examples that currently need manual judgement. That is enough to begin a useful discussion about the work needed before a pilot, without pretending the entire organisation needs to be perfectly cleaned first.

Run the four checks yourself

Copy this Python 3 example into a local file. It uses only the synthetic rows shown above and makes no network requests.

from collections import Counter
rows = [{'sku': 'A100', 'price': 12, 'currency': 'GBP', 'current': True}, {'sku': 'A100', 'price': 12, 'currency': 'GBP', 'current': True}, {'sku': 'B200', 'price': None, 'currency': 'GBP', 'current': True}, {'sku': 'C300', 'price': 18.5, 'currency': 'EUR', 'current': True}, {'sku': 'D400', 'price': 8, 'currency': 'GBP', 'current': False}, {'sku': 'E500', 'price': 22, 'currency': 'GBP', 'current': True}]
counts = Counter(r['sku'] for r in rows)
findings = {
 'repeated_sku_groups': sum(n > 1 for n in counts.values()),
 'missing_prices': sum(r['price'] is None for r in rows),
 'unsupported_currencies': sum(r['currency'] != 'GBP' for r in rows),
 'expired_records': sum(not r['current'] for r in rows)
}
assert findings == dict(repeated_sku_groups=1, missing_prices=1, unsupported_currencies=1, expired_records=1)
print(findings)
Michal Vávra
Written by
Head of AI & Development

Michal has been with Pixelfield since September 2012 and has held technical leadership roles since 2017. He is the technical force behind our most demanding builds, combining AI expertise with the infrastructure and software architecture needed to make it work in production. His work spans LLM applications, multi-agent orchestration and complex integrations. He sets the engineering standards behind reliability, performance and scale, and guides the team through the decisions that determine how a system runs long after launch.