Lesson 32: uniq Command
In this lesson, you'll learn how to use the uniq command to filter, count, and display duplicate lines in text files in Linux.
The uniq command is used to filter duplicate text efficiently. Moreover, it is a handy command that lists or removes duplicate lines that are present adjacent.
The uniq command also counts the duplicate entries. It is important to note that the uniq command works only when duplicate entries are adjacent.
Let's discuss the uniq command in-depth with practical examples.
uniq Command Syntax
The syntax of the uniq command is very easy to understand and is similar to other Linux commands:
$ uniq [OPTIONS] [INPUT] [OUTPUT]
It is important to note that all the options and parameters of the uniq command are optional.
uniq Command Options
| Option | Description |
|---|---|
-c |
Count the number of times each line is repeated |
-i |
Ignore case when comparing lines |
-d |
Print only one copy of each duplicated line |
-D |
Print all duplicate lines |
-u |
Print only unique (non-duplicated) lines |
--all-repeated=separate |
Print all duplicate lines, separated by a blank line between groups |
Create a Sample Text File
To begin, first let's create a simple text file with an vi editor and add the following duplicate contents located in the adjacent lines.
$ vi linux-distributions.txt
$ cat linux-distributions.txt
Ubuntu
Ubuntu
Debian
Debian
CentOS
CENTOS
Fedora
Alpine
UBUNTU
Now, let's use this file to understand the usage of the uniq command.
1. Remove Duplicate Lines from a File
One of the common uses of the uniq command is to remove the adjacent duplicate lines from the text file, as shown.
$ uniq linux-distributions.txt
Ubuntu
Debian
CentOS
CENTOS
Fedora
Alpine
UBUNTU
In the above output, we can see that the uniq command has successfully eliminated the duplicated lines.