Module 3: Text Processing & Search

Lesson 32: uniq Command

In this lesson, you'll learn how to use the uniq command to filter, count, and display duplicate lines in text files in Linux.

uniq Command: A simple tool for filtering out or reporting duplicate lines in sorted text input

The uniq command is used to filter duplicate text efficiently. Moreover, it is a handy command that lists or removes duplicate lines that are present adjacent.

The uniq command also counts the duplicate entries. It is important to note that the uniq command works only when duplicate entries are adjacent.

Let's discuss the uniq command in-depth with practical examples.

uniq Command Syntax

The syntax of the uniq command is very easy to understand and is similar to other Linux commands:

$ uniq [OPTIONS] [INPUT] [OUTPUT]

It is important to note that all the options and parameters of the uniq command are optional.

uniq Command Options

Option	Description
`-c`	Count the number of times each line is repeated
`-i`	Ignore case when comparing lines
`-d`	Print only one copy of each duplicated line
`-D`	Print all duplicate lines
`-u`	Print only unique (non-duplicated) lines
`--all-repeated=separate`	Print all duplicate lines, separated by a blank line between groups

Create a Sample Text File

To begin, first let's create a simple text file with an vi editor and add the following duplicate contents located in the adjacent lines.

$ vi linux-distributions.txt
$ cat linux-distributions.txt

Ubuntu
Ubuntu
Debian
Debian
CentOS
CENTOS
Fedora
Alpine
UBUNTU

Now, let's use this file to understand the usage of the uniq command.

1. Remove Duplicate Lines from a File

One of the common uses of the uniq command is to remove the adjacent duplicate lines from the text file, as shown.

$ uniq linux-distributions.txt

Ubuntu
Debian
CentOS
CENTOS
Fedora
Alpine
UBUNTU

In the above output, we can see that the uniq command has successfully eliminated the duplicated lines.

Module 1: File & Directory Management