Skip to the tool

Remove duplicate lines

MDV Tools

Paste lines. Each distinct line comes back once, in the order it first appeared or sorted. Matching can ignore case and surrounding whitespace; the line that is kept is printed as you typed it.

Runs on this device Works offline
0 characters
Result
paste lines above
How this works

Lines are split at \n after \r\n is normalised; a trailing newline is not a line. Each line is reduced to a key — passed through String.prototype.trim when surrounding whitespace is ignored, through toLowerCase when case is ignored — and the key goes into a Map. The first line to produce a key is kept and every later line with the same key is removed; with "keep last occurrence" the later line replaces the earlier one and takes the later position. What is printed is the original line, not the key, so a trailing space or a capital survives on the copy that was kept. "Drop blank lines" removes lines that are empty or whitespace-only before any comparison, and counts them as removed.

Comparison is by exact UTF-16 code units of the key. No Unicode normalisation is applied, so é as one code point and e plus a combining acute are two different lines, and toLowerCase folds only what Unicode's language-neutral mapping does — İ does not match i and ß does not match ss. Sorting uses Intl.Collator with numeric: true in the browser's locale, so file2 sorts before file10 and accented letters fall next to their base letter rather than after z; Z–A is the same comparison reversed. The sort compares the printed line, so leading whitespace still counts even when it is ignored for matching.

Input above 2,000,000 characters is refused rather than attempted, so the tab never hangs. Nothing is uploaded: there is no server in this file.

The Dark button remembers its setting in this browser's local storage. Nothing you type or drop into the page is stored.