Back to App

How to Find and Remove Duplicate Rows in Excel

Last updated: 2026-05-22 · by xlstools team

Duplicates creep into spreadsheets through countless paths: re-running a database export, merging CRM systems, copy-paste mistakes, double-imports. This guide walks through both deduplication modes, how to pick the right key columns, and how to verify the result before discarding data.

Two ways to detect duplicates

The most important decision is what "duplicate" means for your data.

1. Full-row match

Two rows are duplicates only if every cell is identical. The strictest mode — even a single stray space in one column makes the rows count as distinct.

Use when: you want to remove only exact, byte-for-byte copies. Common after concatenating exports that may include the same row twice.

2. Key-column match

Pick one or more "key" columns; rows with identical values in those columns are treated as duplicates regardless of other fields.

Use when: you have a logical primary key (email, order_id, employee_id) and want one row per key, even if the other columns differ slightly (different update timestamps, formatting variations, etc.).

Step-by-step walkthrough

Step 1 — Upload your Excel file

.xlsx or .xls. The tool reads it in your browser and shows the column list.

Step 2 — Pick the dedup mode

Toggle between Full-row match and Key-column match. If you pick key-column, also select which columns are the key.

Step 3 — Choose which duplicate to keep

When duplicates are found, one row is kept and the others are removed. You decide:

  • Keep first occurrence (default) — preserves the earliest row in the original order
  • Keep last occurrence — preserves the latest row (useful when later rows have more complete data)

Step 4 — Review the preview

The tool highlights the duplicate rows so you can scan them before deciding to proceed. Sanity-check: do they actually look like duplicates, or did you pick too narrow a key?

Step 5 — Download

Two files come out of one run:

  1. Unique rows — the deduplicated result (this is your clean dataset)
  2. Duplicate rows — what was removed (keep this for audit)

Picking the right key columns

This is where most dedup jobs go wrong.

Key column choice Result
email ✅ Strong — emails are usually unique per person
order_id ✅ Strong — order IDs are unique by definition
employee_id + date ✅ Good for time-series (one row per employee per day)
first_name + last_name ⚠️ Weak — two different "John Smith"s collapse into one
phone_number ⚠️ Weak — formatting drift (+1-555-1234 vs 5551234) causes false negatives
timestamp_ms ❌ Useless — too granular, no duplicates ever detected

Rule of thumb: pick the columns that uniquely identify the entity you care about, not just columns that happen to be filled in.

Case sensitivity

String comparison is case-sensitive by default: "[email protected]" ≠ "[email protected]". If you need case-insensitive dedup:

  1. First run the Excel Cleaner to trim whitespace (often the real culprit)
  2. If case still varies, lowercase the key column in source (in Excel: =LOWER(A2)) before deduplicating

Common use cases

Cleaning customer lists from multiple sources

You imported customers from your CRM, your email marketing tool, and your billing system. Each has the same customer with slight variations. Dedup by email (after lowercasing) to get one unified contact per person.

Removing duplicate transactions

Bank exports occasionally include the same transaction twice if you re-pull an overlapping date range. Dedup by date + amount + description to get a clean ledger.

Finding double-punched timesheets

Hourly employees sometimes clock in twice. Dedup a payroll-period timesheet by employee_id + date and review the "duplicate rows" output before discarding — you'll catch fraud and honest mistakes alike.

Pitfalls and how to avoid them

Problem Cause Fix
"Duplicates" detected but rows look different Key column has whitespace or case differences Run the Cleaner first; consider lowercasing the key
Real duplicates not detected Picked too granular a key (e.g. includes timestamp) Drop the timestamp from the key; or run a Compare job first to understand the differences
Wrong row was kept "Keep first" vs "Keep last" mismatch Re-run with the other setting — the duplicate-rows file shows you what would have been kept the other way
File too large to dedup More than 500k rows in a slow browser Try Firefox or Chrome; close other tabs. SheetJS dedup is O(n) but final serialization is the bottleneck.

Always review before discarding

The two-file output (unique + duplicates) exists specifically so you can audit before deleting. Workflow:

  1. Run dedup
  2. Open the "duplicate rows" file
  3. Spot-check 10 random rows — do they look like real duplicates of something in "unique rows"?
  4. If yes, you can safely discard "duplicate rows"; if no, your key columns are wrong

Privacy & security

All detection and export happens in your browser via SheetJS. Personally-identifiable information (email, employee IDs, transaction data) never reaches any server.