Build a pipeline one stage at a time

Most pipelines I write start as one command that prints too much. I add a stage, get something wrong, add another, and after five minutes I have a line nobody wants to read, including me a week later. The fix is dull: build the pipeline one stage at a time and look at the output after each stage. Here is the process, using a question that comes up on almost every web server: which client addresses hit it most?

$ awk '{print $1}' | sort | uniq -c | sort -rn | head -n 2 access.log

1. in< access.logthe file, 5 lines

   203.0.113.7 - - [07/Nov/2026:10:12:01] "GET / HTTP/1.1" 200
   198.51.100.4 - - [07/Nov/2026:10:12:03] "GET /a HTTP/1.1" 200
   203.0.113.7 - - [07/Nov/2026:10:12:09] "GET /b HTTP/1.1" 404
   192.0.2.88 - - [07/Nov/2026:10:12:15] "GET / HTTP/1.1" 200
   203.0.113.7 - - [07/Nov/2026:10:12:22] "GET /c HTTP/1.1" 200

2. 1awk '{print $1}'keep field 1, the client address

   203.0.113.7
   198.51.100.4
   203.0.113.7
   192.0.2.88
   203.0.113.7

3. 2sortput equal lines side by side

   192.0.2.88
   198.51.100.4
   203.0.113.7
   203.0.113.7
   203.0.113.7

4. 3uniq -ccollapse neighbours, prefix a count

   1 192.0.2.88
   1 198.51.100.4
   3 203.0.113.7

5. 4sort -rnbiggest count first

   3 203.0.113.7
   1 198.51.100.4
   1 192.0.2.88

6. 5head -n 2stop after two lines

   3 203.0.113.7
   1 198.51.100.4

Start with the raw data

I'll assume access.log uses nginx's default combined format. Look at a few lines first:

head -n 3 access.log

A typical line looks like this, trimmed:

203.0.113.7 - - [07/Nov/2026:10:12:01 +0000] "GET /index.html HTTP/1.1" 200 512 "-" "Mozilla/5.0"

The client address is the first field. Extract it and check before going further:

awk '{ print $1 }' access.log | head -n 5

If that prints addresses, the stage works. If it prints dashes or dates, you picked the wrong field, and you learned it in a second instead of after the whole pipeline ran.

Add a stage, then look again

Counting requires sorting, and this is the step people skip:

awk '{ print $1 }' access.log | sort | uniq -c | head -n 5

uniq -c only collapses adjacent duplicate lines. Compare these two:

printf 'a\nb\na\n' | uniq -c
printf 'a\nb\na\n' | sort | uniq -c

The first prints 1 a, 1 b, 1 a. The second prints 2 a and 1 b. Sorting puts identical lines next to each other, which is the only thing uniq knows how to count. So any time you reach for uniq -c, ask whether the input is sorted.

To rank the results, sort numerically in reverse and keep the top ten:

awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head -n 10

uniq -c pads counts with leading spaces, which sort -n handles without trouble.

Peek mid-pipeline with tee

When a stage is hard to reason about, tee lets you look without breaking the flow. Its standard output keeps going down the pipe, and a copy goes wherever you point it. Here I send the last three lines to stderr so they show on screen while the counts still run (this uses bash process substitution):

awk '{ print $1 }' access.log | tee >(tail -n 3 >&2) | sort | uniq -c | sort -rn | head -n 10

I use tail in the process substitution rather than head. A reader that exits early can make tee fail with a broken pipe partway through the data, while tail reads everything.

Pass filenames safely with find and xargs -0

Suppose the question is different: which file extensions take the most space under an upload directory? Filenames can contain spaces and newlines, and find's default output separates names with newlines, so xargs would split a name like my photo.jpg into two arguments. Use NUL separators on both ends:

find /srv/uploads -type f -print0 \
  | xargs -0 stat -c '%s %n' \
  | awk '{
      size = $1
      name = $0
      sub(/^[0-9]+ /, "", name)
      sub(/.*\//, "", name)
      ext = "(none)"
      if (name ~ /\./) { ext = tolower(name); sub(/.*\./, "", ext) }
      total[ext] += size
    }
    END { for (e in total) print total[e], e }' \
  | sort -rn | head -n 10

Test the awk block first on a directory with three or four files, since that is where mistakes hide. The stat -c format is GNU; on macOS it is stat -f '%z %N'.

Check exit codes, not just output

A pipeline's status is the status of its last command. Here the first command fails and the pipeline still reports success:

grep -c 'PATTERN' missing.log | wc -l
echo "${PIPESTATUS[@]}"

PIPESTATUS holds the status of each stage from the most recent pipeline, so this prints 2 0: grep failed because the file is missing, and wc succeeded. Read it on the very next line, since the next command overwrites it.

In scripts, set -o pipefail makes the whole pipeline fail when any stage does. It has two catches. A stage that exits early, such as head, can make the producer die of SIGPIPE, so yes | head -n 1 returns 141 under pipefail. And grep returns 1 when nothing matches, which a pipeline will treat as a failure. Know both cases before you turn it on everywhere.