AI Data Processing
Cleaning and matching data at a scale people cannot manage manually.
What is AI data processing?
AI data processing cleans and standardises your records, matches the same entity across different systems, fills gaps where a reliable source exists, and runs on a schedule so the data stays clean rather than drifting back within months.
Messy data blocks work that would otherwise be straightforward. The same customer recorded five ways, phone numbers in four formats, addresses that do not match between the sales system and the delivery system.
Cleaning it once is a project. Keeping it clean is the part most businesses skip, which is why the same cleanup gets commissioned again eighteen months later.
We clean, standardise and match, then run it as a scheduled job so new records are corrected as they arrive rather than accumulating into the next backlog.
How we treat an uncertain match
Duara handles data cleanup from Nairobi, frequently after a migration, a merger, or the discovery that two departments have been reporting different customer counts for years.
We treat uncertain matches as items for review rather than merging them, and we keep every change reversible. Merging two genuinely different customers is far more damaging than leaving a duplicate in place, and that asymmetry drives every threshold we set.
Who this is for
- The same customer appears several times
- Systems disagree about the same record
- A migration or merger left data in a mess
- Reports differ depending which system produced them
Industries we serve
- Retail and eCommerce
- Financial services
- Telecommunications
- Healthcare records
- Utilities billing
- Membership organisations
Problems this solves
These conditions usually follow a system change, and they compound quietly until somebody tries to report across the whole business.
- One entity recorded many times. The same customer exists under several spellings, so history is fragmented and totals are wrong in a way nobody can quantify.
- Formats that never agreed. Phone numbers, dates and addresses were entered differently by different people over years, so matching fails on technicalities.
- Systems that disagree. Sales and delivery hold different addresses for the same customer and neither is marked as authoritative.
- Cleanups that do not last. A cleanup project completes, nothing changes about how records are created, and the mess returns within two years.
What we deliver
A cleaning and matching process that standardises formats, links records across systems, holds uncertain matches for review, and runs on a schedule so quality is maintained rather than restored periodically.
Duplicates identified and merged with a full change record
Formats standardised across systems
Records matched between systems that previously disagreed
Gaps filled only where a reliable source exists
A scheduled job so data stays clean going forward
Quality reporting so drift is visible before it is a problem
Features
- Asymmetric caution on merges. Uncertain matches are held rather than merged, because combining two real customers is far worse than leaving a duplicate.
- Every change reversible. The original values are retained, so any merge or correction can be undone without reconstructing from backups.
- Kenyan format handling. Phone numbers, national identifiers and address conventions are handled as they are actually written here rather than generically.
- Scheduled rather than one off. The job runs continuously so new records are corrected on arrival, which is what stops the mess returning.
- Quality measured over time. Duplication and completeness are tracked, so decay is visible while it is still small.
Benefits
- Reports finally reconcile. Once entities are matched across systems, departments stop reporting different totals for the same thing.
- Marketing and billing improve immediately. Correct contact details mean messages arrive and invoices reach the right person first time.
- The cleanup is not repeated. A scheduled job prevents the accumulation that makes the next cleanup necessary.
- Migrations become safer. Clean, matched data is the difference between a system move that works and one that produces months of corrections.
Technologies we use
- Fuzzy matching. Identifying records that describe the same entity despite spelling, spacing and format differences.
- Format standardisation. Normalising phone numbers, dates, names and addresses to a consistent form across systems.
- Cross system linking. Establishing which record in one system corresponds to which in another, with confidence recorded.
- Change journaling. Retaining original values so every correction and merge can be reversed.
- Scheduled execution. Running as a recurring job so new and changed records are processed continuously.
Threshold setting is the whole job. Too aggressive and real customers get merged; too cautious and the review queue is unusable. We tune it on your data with you.
How the process works, step by step
Data processing usually takes four to six weeks for the initial clean, then continues as a scheduled job.
-
Step 1
Free data profile
We measure duplication, completeness and format inconsistency so the starting position is documented.
-
Step 2
Rule design with your team
Matching rules and thresholds are set with the people who know which differences matter and which do not.
-
Step 3
Dry run and review
Proposed merges and corrections are produced without applying anything, so you can see the consequences first.
-
Step 4
Supervised application
Changes apply in batches with the reversal path tested before volume increases.
-
Step 5
Scheduled maintenance
The job runs continuously, with quality reporting so drift is caught early.
Quality assurance and how we work
The commitments attached to changing your records at scale.
- Uncertain matches are never merged. They wait for a person. Merging two genuinely different customers is far more damaging than leaving a duplicate in place.
- Originals are always retained. Every change is reversible from the change record without needing to restore a backup.
- Dry run before anything applies. You see the proposed merges and corrections, with counts, before a single record is altered.
- Thresholds are set with you. Where the caution line sits is your decision, because your team knows which differences are real.
- We report decay, not just the clean. Quality tracking continues afterwards, because a cleanup without maintenance simply defers the problem.
Frequently asked questions about AI data processing
Could it merge two real customers by mistake?
Uncertain matches are held for review rather than merged. Combining two genuinely different customers is far more damaging than leaving a duplicate, and thresholds reflect that.
Is the original data kept?
Yes. Every change is journaled with the original value, so any merge or correction can be reversed without restoring a backup.
Do we see changes before they apply?
Yes. A dry run produces the proposed merges and corrections with counts, so you see the consequences before anything is altered.
Who sets the matching thresholds?
You do, with our recommendation. Your team knows which differences between records are real and which are typing variation.
Does it handle Kenyan phone formats?
Yes, including the several ways the same number gets stored, which is one of the most common causes of duplication here.
Why does the cleanup need to be scheduled?
Because a one off clean without ongoing maintenance simply defers the problem. The mess returns within about two years.
How long does the initial clean take?
Four to six weeks, after which it continues as a recurring job rather than ending.
What is a golden record?
The single agreed version of an entity, assembled from the best available fields across the duplicates being merged.
Can it match records across different systems?
Yes, and cross system linking is usually where the largest reporting discrepancies get resolved.
What if two systems disagree about a value?
Precedence rules are agreed with you, so one system is authoritative for each field rather than the conflict being resolved arbitrarily.
Will gaps be filled automatically?
Only where a reliable source exists, and the source is recorded against the value so nobody treats an inference as verified.
Should we start with part of the data?
Yes, and we recommend it. Running against one region or product line first proves the matching rules before they touch the whole set.
Does this help before a system migration?
Considerably. Clean matched data is the difference between a migration that works and one that produces months of corrections afterwards.
How do we tell whether processing is accurate?
Quality reporting tracks duplication and completeness over time, so both the improvement and any later drift are visible.
Where does our data actually go?
Processing happens within your environment, and records are never used to train public AI models.
What about records with no unique identifier?
Those are the hard cases and fuzzy matching handles them, though more of them land in review, which we report honestly.
Can it detect records that have gone stale?
Yes, such as contacts who have almost certainly changed employer, which is a common and rarely tracked source of wasted effort.
Who needs to be available while we do this?
Whoever knows the data best, usually operations rather than IT, since they know which differences are meaningful.
Does it process records written in Swahili?
Yes, including name and address conventions that generic matching handles poorly.
What does data processing cost by volume?
Record count and processing frequency set the cost, quoted against your own data volumes before work begins.
Glossary of terms
Data quality terms in plain language.
- Deduplication
- Finding records that describe the same thing and combining them into one.
- Fuzzy matching
- Matching records that are similar but not identical, allowing for spelling and format differences.
- Golden record
- The single agreed version of an entity, assembled from the best fields across duplicates.
- Normalisation
- Converting values into a consistent format, such as one standard way of writing a phone number.
- Data drift
- The gradual return of inconsistency as new records are created without controls.
- Change journal
- The record of what was altered and what the original value was, enabling reversal.
Related services in AI Implementation
Talk to us about AI data processing
Run a count of distinct customers in two of your systems and compare them. The gap between those numbers is usually the argument.
Start your project