Whitespace RemoverText Tools
Cleans text by trimming, collapsing, joining, or removing whitespace.
How to remove duplicates from a list without losing its order
A mailing list, a keyword list, or a column copied from a spreadsheet has the same entries several times. Clean the spacing, remove the repeats, and sort what is left.
Why duplicates survive a quick cleanup
Two lines that look the same to you can differ to a computer. "apple" and "apple " with a trailing space are different strings, and so are Apple and apple. Text copied from a website or a Word document can also carry non-breaking or zero-width spaces that you can't see at all. A plain dedupe leaves all of those in place.
The cost shows up later: the same person gets an email twice, a keyword list reports the wrong count, or an import fails on a unique column. Sorting before you dedupe causes a quieter problem. It throws away the original order, which matters when the latest entry is the one you want to keep.
Clean the spacing, remove the repeats, then sort
Paste the list into the Whitespace Remover first. Turn on Trim each line and Collapse repeated spaces, and remove blank lines if you don't need them. Special-space cleanup normalizes the non-breaking and zero-width spaces that come with copy and paste. The tool works on plain text and doesn't understand CSV columns.
Then remove the repeats with Remove Duplicate Lines. Choose Keep first or Keep last, depending on which copy you trust. Ignore case makes Apple and apple count as one line, and Trim lines compares lines without the spaces at their ends. The kept lines stay in their original order, so b, a, b, c becomes b, a, c. Matching is line by line, so two repeated phrases inside one long line aren't treated as duplicates.
If you need the list in order, finish with the Line Sorter. It sorts A to Z, Z to A, naturally, numerically, by length, at random, or in reverse. Natural sort puts file2 before file10, and numeric sort reads each line as a number, so 20, 3, 100 becomes 3, 20, 100. Sorting keeps every line, which is why it comes after the dedupe.
Remove duplicates from a list
- Paste the list into the Whitespace Remover, turn on Trim each line, and copy the result.
- Paste it into Remove Duplicate Lines and choose Keep first or Keep last.
- Turn on Ignore case if Apple and apple should count as the same entry.
- Copy the cleaned list and check how many lines were removed.
- If you need it sorted, paste it into the Line Sorter, choose a mode, and copy the result.
What these text tools won't do
- Matching works line by line. Put one item on each line before you start.
- Comma-separated values inside one line aren't split, deduplicated, or sorted.
- The tools read plain text. They don't understand CSV columns or quoted fields.
- Each tool accepts text files up to 10 MB.
- Random sort gives a different order on every run.
Frequently asked questions
How do I remove duplicates from a list but keep the order?
Use Remove Duplicate Lines. It keeps the first or last copy of each line and leaves the kept lines in their original order.
Why aren't some duplicate lines removed?
The lines differ in case or in spaces you can't see. Turn on Ignore case and Trim lines, or clean the text in the Whitespace Remover first.
Should I keep the first or the last duplicate?
Keep first when the earliest copy is the one you trust. Keep last when later lines hold the version you want.
Does removing duplicates also sort the list?
No. Sort it afterwards with the Line Sorter if you need an order.
Related tools
Tools used in this guide
Remove Duplicate LinesText Tools
Keeps one copy of each line and tells you how many lines were removed.
Line SorterText Tools
Sorts lines A to Z, Z to A, naturally, numerically, by length, or randomly.
