Remove Duplicate Lines

Paste a list into the left box and the right box shows it again with every repeat taken out. The order stays as it was, the first copy of each line stays where it was, and four counters tell you how many lines came in, how many were kept, how many duplicates were removed and how many different lines repeated. This is a quick way to remove duplicate lines when you have a list of names, emails, URLs, keywords, log entries or anything else that came from more than one source.

The settings sit under the two boxes. They decide what counts as the same line (case, spaces, punctuation, one column), which copy you keep (first or last), whether empty lines go, whether the result is sorted, and what separates the items (new lines, commas, semicolons, tabs, spaces or a character you type). Every change updates the result at once.

Below the settings you get two lists that most tools do not show: the most repeated lines with how many times each one appears, and every removed line with its line number and the line it was a copy of. So you can check the work instead of trusting it.

Most people who need this are in the middle of something else: a mailing list that was merged twice, a keyword export with the same phrase on several rows, a log with the same error a thousand times. Opening a spreadsheet or writing a script for that is slow. Here you paste, look and copy, and the job is done in seconds.

Because the tool runs as a web page, you can remove duplicate lines online from any computer with a modern browser, with nothing to install and no account. The work itself is done by JavaScript on your own device, which is why the result appears while you type and why your text does not travel to a server.

The page keeps no history of what you pasted. If you close the tab, the text is gone. Only your choices, such as Keep first or the separator, are stored in the browser so the page opens the way you left it.

Plain text is the simplest case. Every line break starts a new item, so a list copied from a document, a chat, an email or a web page works as it is. Paste it into Your text and the Result box shows the clean list straight away, with one copy of each line.

When you remove duplicate lines from text that was copied from different places, small differences often hide repeats. One line ends with a space, another starts with two, a third uses a capital letter. By default the tool ignores spaces at the start and the end of a line, so “apple” and “apple ” already count as one. The other differences are covered by the options in the next section.

The result keeps each line exactly as it was written in its first appearance (or last, if you choose Keep last). Nothing is lowercased, trimmed or changed in the output; the options only change how lines are compared, not how they look.

Four check boxes under What counts as the same decide when two lines are treated as one. You can combine them freely, and the counters update at once so you can see what each one changes.

If you need to remove duplicate lines case insensitive, tick Ignore upper / lower case. Then “Paris”, “paris” and “PARIS” are one line, and the copy you keep is shown with its original capital letters. This is the setting most people want for names, email addresses and tags, where capitals are an accident rather than a difference.

Ignore spaces at the start and end is on by default, because trailing spaces are invisible and almost never meant. Treat several spaces as one goes further and turns any run of spaces or tabs inside the line into a single space, so “New York” matches “New York”. Ignore punctuation removes everything that is not a letter, a digit or a space before comparing, so “Hello, world!” matches “Hello world”. Letters with accents and letters from any script count as letters.

Many quick methods sort the list first, because sorted repeats sit next to each other and are easy to drop. That changes the order of your data, which is often not acceptable: a ranked list, a playlist, a timeline or a log must stay in its original sequence.

This tool does not need to sort. With Keep first, the first time a line appears it stays in place and every later copy is dropped. With Keep last, the last copy stays where it is and the earlier ones go, which is useful when later rows are newer, such as an updated address further down an export. Either way you remove duplicate lines without sorting anything, and the Sort the result menu stays on No (keep order) unless you change it.

Which one should you choose? Keep first suits most lists, because the entry you saw first is usually the one you want and the order reads naturally. Keep last suits data that grows over time, such as a sign-up sheet where people corrected their details further down, or an export where every change adds a new row at the end. The position matters too: with Keep last the kept line stays where its last copy was, so the result follows the order of the latest entries.

If you do want an ordered list, Sort the result offers A → Z and Z → A, applied after the duplicates are removed. In the Every line with its count view there is also Most repeated first.

The Show buttons at the top switch between four results from the same input. You do not need to paste again; the text stays and only the right box changes.

Lines that appear once is handy for comparing two lists. Paste both lists one after the other: lines that are in both appear twice and drop out, and what remains is the set of lines found in only one of them, as long as neither list repeats lines on its own. For a line-by-line comparison that shows where two texts differ, the Diff Checker is the better tool.

Every line with its count puts the number first and the line after a tab, so you can paste the result into a spreadsheet and get two columns. Choose Most repeated first in Sort the result to turn it into a ranking.

Press Open a file above Your text to load a .txt, .csv, .tsv, .log or .md file from your device. The file name appears next to the label, the text fills the box, and the result appears as usual. When you press Download, the cleaned file is saved as the original name with “-deduped” added, as a .txt file.

To remove duplicate lines from a file of several megabytes, nothing special is needed. The file is read by your browser and processed in memory, so its size is limited by your device rather than by an upload limit. While the text is up to about 3 million characters (roughly 3 MB of plain text), the result updates live as you change settings. Above that, a red Remove duplicates button appears under the text box, so the page does not redo the whole job on every click; change your settings first, then press it.

Line endings from Windows (CR LF), macOS and Linux (LF) and old Mac files (CR) are all recognised as line breaks, so files from any system split correctly. The output uses plain line feeds.

A CSV row is one line, so whole-row duplicates are removed like any other line. Keep Ignore spaces at the start and end on and exact repeated rows disappear, with the header row kept because it appears only once.

Often the rows are not exact copies: the same customer appears twice with a different date or note. For that, set Compare only column to the number of the column that should be unique, counting from 1, and set split by to the separator of your file (a comma for CSV, a semicolon for many European exports). Now only that field is compared, and the first (or last) row for each value is kept whole. Column 0 means the whole line is compared.

This is the way to remove duplicate lines from csv data by email, ID, SKU or URL. One honest limit: the column is found by a plain split on the separator, so a field that itself contains the separator inside quotes, like “Smith, John”, shifts the columns. If your file has quoted commas, compare by a column that comes before them, or convert the file with CSV to JSON and back with JSON to CSV after cleaning.

Duplicates are not always on separate lines. Under Items are separated by you can choose New lines, Commas, Semicolons, Tabs, Spaces (words) or Other, where you type any separator such as a pipe. The same rules then apply to each item, and the result is joined back with the same kind of separator: a comma and a space for commas, a semicolon and a space for semicolons, and a single space for words.

Pick Spaces (words) to remove duplicate words from a sentence, a tag list or a list of keywords typed on one line. Any run of spaces, tabs or line breaks counts as one separator. Words with punctuation attached, such as “word,” and “word”, are different items unless you also tick Ignore punctuation, and Ignore upper / lower case makes “The” and “the” the same word.

With commas selected, a line such as “red, blue, red, green” becomes “red, blue, green”. The spaces after the commas do not matter, because spaces at the start and end of each item are ignored by default. To count how many words a text has before and after, use the Word Counter.

Most lists that need cleaning come from somewhere else: two exports joined together, contacts from several devices, keywords from different research tools, links collected over weeks. The table below shows settings that work well for common cases.

If you need to remove duplicates from list data before an import, check the Lines that repeat counter first. If it is higher than you expected, switch to Only duplicates to see which entries were doubled; if it is lower, one of the What counts as the same options is probably needed.

The page works the same on a phone, a tablet and a computer. On a small screen the two boxes stack, with Your text above the Result, and the settings follow below. On a wide screen they sit side by side.

On a phone, long-press inside Your text and choose Paste, then press Copy above the result and paste it where you need it. Open a file uses the phone's file picker, so you can load a text file from downloads or a cloud folder. This makes it easy to delete duplicate lines online from a list someone sent you in a chat, without waiting until you are at a computer.

The tool is free to use, with no sign-up, no limit on how many times you use it and no watermark or added text in the output. Because the work is done on your device, there is no server cost per job, so there is no daily quota to run into.

You can remove duplicate lines free from small lists and large files alike. The only practical limit is the memory of your device: a modern computer handles files of many megabytes, while an older phone may slow down on very large ones.

If you work in a terminal, the same jobs can be done with built-in commands. Each one has its own rules about order and case, which is worth knowing before you choose. The table below compares them with the options on this page.

sort -u sorts the input and keeps one copy of each line, so the original order is lost. uniq only removes repeats that sit next to each other, which is why it is almost always used after sort; uniq -c adds counts, uniq -d shows only the repeated lines and uniq -u only the lines that appear once. awk '!seen[$0]++' keeps the first copy of each line and keeps the order, and awk '!seen[tolower($0)]++' does the same while ignoring case.

In PowerShell, Get-Unique compares each line only with the one before it, so it needs sorted input, the same as uniq. Sort-Object -Unique sorts and removes repeats, ignoring case by default, and Select-Object -Unique keeps the original order. All of these are fine tools; this page is simply quicker when the text is already on your screen, and it shows you what was removed.

When at least one line repeats, two lists appear under the settings. Most repeated lines shows up to 100 lines that appear more than once, each with its count, the most frequent first. It is a quick way to see what dominates a log, a survey answer list or a keyword export.

Removed lines lists every copy that was dropped (the first 300 are shown), with its line number on the left and “same as line N” on the right, where N is the line that was kept. If you chose another separator, the numbers count items instead of lines. This list is the easiest way to check that the options did what you meant, for example that Ignore punctuation did not merge two lines you wanted to keep.

Empty lines are counted separately. With Remove empty lines ticked, the label shows how many blank lines were dropped, and they do not appear in the removed list. Untick it to keep blank lines, in which case all blank lines after the first count as duplicates of it.

The four counters above the boxes give the same story in numbers. Lines in is every item in your text, blank ones included. Kept is the length of the clean list in the Unique lines view. Duplicates removed is how many copies were dropped, and Lines that repeat is how many different lines had at least one copy. If Kept plus Duplicates removed is less than Lines in, the difference is the blank lines that were dropped.

For every line the tool builds a comparison key. It starts from the whole line, or from one column if you set Compare only column. Then, in this order, it trims spaces at the start and end, turns runs of spaces into one space, strips punctuation and lowercases the text, each step only if its option is ticked. Two lines are duplicates when their keys are equal.

In a first pass the tool counts how often each key appears and notes where it appears first and last. In a second pass it walks through the lines again: a line is kept if it sits at the remembered first position (Keep first) or last position (Keep last) for its key, and otherwise it goes to the Removed lines list together with the number of the line that was kept. Both passes look at each line once, so the time grows in step with the size of the text rather than exploding on long lists.

The key is only used for comparing. What you see in the result is always the original line, with its own capitals, spaces and punctuation. When several lines share a key, the one shown in Only duplicates, Lines that appear once and the count view is the first one that appeared.

Empty lines are handled before any of this. With Remove empty lines ticked, lines that are blank or contain only spaces are skipped completely, so they are neither counted nor listed as removed.

Spreadsheet apps have their own commands for repeated rows, but a quick paste is often faster when you only need one column cleaned. Select the column, copy it, paste it into Your text, and each cell arrives as one line. Press Copy above the result and paste it back into an empty column. Because Keep first keeps the order, the rows that remain line up with what you expect.

When you copy several columns from a spreadsheet, the cells in each row are separated by tabs. To compare by one of them, set Compare only column to its number and type a tab character in split by (copy one from your pasted text if your keyboard will not type it into the box). The whole row, with all its columns, is kept, so you can paste it straight back.

Text editors and word processors are handled the same way: copy the text, paste it in, copy the result back. If the text was written with soft line breaks or long paragraphs, each paragraph is one line here, so whole paragraphs are compared.

Start with the defaults and look at the four counters before ticking more options. Each option makes more lines match, so it is easier to add them one at a time and watch Duplicates removed grow.

Use the Removed lines list as a check. If a line you wanted to keep appears there, read the “same as line N” note, scroll to that line in your text and compare the two.

Keep a copy of the original until you are sure. Use as input replaces the text in the left box with the result, which is handy for a second pass, but the original is then gone from the page.

Clean before you count. If you want to know how many different entries a list has, the Kept counter is that number once the options match how you think about your data.

For CSV files with a header row, the header appears only once and is kept, so the file stays valid. If two files with the same header were joined, the second header is removed as a duplicate of the first.

The most common reason two lines that look identical are both kept is a character you cannot see. Text copied from web pages, PDFs and word processors often carries non-breaking spaces, tabs instead of spaces, zero-width spaces or a byte order mark at the start of a file.

Non-breaking spaces and tabs count as spaces here. At the start or end of a line they are ignored by default, and inside a line Treat several spaces as one turns any mix of them into one ordinary space. A byte order mark at the very start of a file is also treated as a space, so the first line of a file still matches its copies further down.

Zero-width spaces and similar invisible marks are not spaces, so they make two lines different. Ignore punctuation removes them along with commas and full stops, because it keeps only letters, digits and spaces. If you do not want to lose real punctuation from the comparison, check the Removed lines list to see whether the lines matched as you expected.

Typographic quotes and dashes are another trap: a curly apostrophe and a straight one are different characters, as are a hyphen and an en dash. Ignore punctuation treats them all the same, by removing them.

Everything happens inside your browser tab. The text you paste and the files you open are read by the page on your device and are not uploaded, which matters when the list holds customer emails, phone numbers or internal data.

The page stores only your settings, in your browser's local storage, so it opens with the same options next time. Clearing your browser data removes them. The text itself is never stored.

Once the list is clean, you may want to order it in more ways than A → Z: Sort Text sorts by natural order, numbers, length, last word or a column, and can remove repeats at the same time. To see what changed between the original and the cleaned version, paste both into the Diff Checker.

For data you will import elsewhere, JSON to CSV and CSV to JSON convert between the two formats, and the Word Counter counts words, characters and lines in any text.