Removing duplicate data

What deduplication actually is

Deduplication is the process of collapsing a dataset so that each real-world entity — a person, a company, an order — appears exactly once. It sounds trivial, but the hard part is defining identity. Before you can remove a duplicate, you have to decide which columns make two rows “the same”: the whole row, a single key like an email address, or a composite key such as first name plus last name plus phone. That decision is a business rule, not a technical one, and getting it wrong either deletes records you wanted to keep or leaves duplicates behind.

Why duplicate data matters

Duplicates are expensive in ways that are easy to miss. They inflate headline numbers — a list of 10,000 “contacts” that is really 8,200 unique people quietly overstates your reach by 22%. They break CRM and marketing-automation imports, which reject rows on a unique-email constraint or, worse, create two records for one person and then split that person’s history across both. They cause the same customer to be emailed or called twice, which erodes trust and can trigger spam complaints. And in analytics, double- counted rows skew every metric downstream, from conversion rate to revenue per lead. Clean, deduplicated data is the precondition for every other system that consumes it.

The edge cases that break it

Exact-match deduplication is the easy 80%. The remaining 20% is where most lists go wrong:

• Case and whitespace. “[email protected]”, “[email protected]”, and “ [email protected] ” are the same address to a human and to an email server, but three different strings to a naïve match. Normalizing case and trimming spaces before comparing is what turns those three rows into one.
• Partial and fuzzy matches. “Bob Smith” and “Robert Smith” at the same company are almost certainly one person, but no exact rule catches them. These need a chosen match key and human review rather than blind deletion.
• Which copy survives. When two rows match, the copies are rarely identical across every column — one may have a phone number the other lacks. Keeping the wrong copy silently discards good data, which is why a keep-first / keep-last choice and a preview matter.
• Composite identity. Email is a strong key; a name alone is a weak one. Deduplicating on the wrong column either merges distinct people who share a name or fails to catch the same person entered with two emails.

How to approach it

Work in a deliberate order. First, decide the match key — the column or combination of columns that defines a unique record. Second, normalize that key: lowercase it and trim spacing so cosmetic differences do not hide real matches. Third, preview the flagged duplicates before you delete anything, and choose whether to keep the first or last copy of each group. Finally, export the cleaned file and re-run any downstream import. Doing this in the browser keeps the whole sequence private: because Sigmera processes the file locally with the browser’s JavaScript engine, there is no server to upload to and nothing is retained, satisfying the data-minimization requirement of GDPR Article 5.