/juːˈniːk/
noun or command … “filtering adjacent duplicates.”
uniq is a classic UNIX command-line utility used to detect, filter, or report repeated lines in a text stream. Its defining trait is subtle but crucial: it only works on adjacent duplicate lines. If identical lines are separated by other content, uniq will treat them as different unless the data is preprocessed.
Because of this behavior, uniq is almost always paired with sort. Sorting groups identical lines together, after which uniq can do its real work. This design reflects the old UNIX philosophy: small tools that do one thing well and compose cleanly through pipes.
At its simplest, uniq removes consecutive duplicate lines:
uniq file.txtIf file.txt contains repeated lines back-to-back, only the first occurrence is kept. Everything else is discarded.
uniq becomes more interesting with flags:
-c… prefixes each line with the number of times it occurs consecutively.-d… outputs only lines that are duplicated.-u… outputs only lines that are unique (appear once).-i… ignores case when comparing lines.
A common pattern looks like this:
sort access.log | uniq -c | sort -nrThis pipeline counts identical lines, then sorts them numerically in reverse order. The result is a frequency table … extremely useful for log analysis, debugging, and quick data exploration.
Conceptually, uniq is not about uniqueness in the mathematical sense. It is about runs of identical data. Think of it as a compression pass over a stream, collapsing repetition into a single representative (optionally annotated with a count). That makes it fast, simple, and perfectly suited to streaming text.
In short, uniq is a quiet powerhouse. It does not search globally, it does not build sets, and it does not remember the past beyond the previous line. That limitation is intentional … and when combined with other tools, it becomes a sharp instrument rather than a blunt one.