Home/Learn/Linux/Text Processing — grep, awk, sed, cut, sort

Text Processing — grep, awk, sed, cut, sort

Intermediate

Shell

Linux text processing tools — grep, awk, sed, cut, sort, uniq — are the foundation of log analysis, data extraction, and pipeline automation. Combining them with pipes creates powerful one-liners.

Overview

Linux's "everything is text" philosophy means that every tool produces text output that can be piped into the next. grep filters lines, cut extracts columns, sort orders output, uniq deduplicates, awk processes fields, and sed transforms text. These tools are available on every Linux system without installing anything. Mastering them transforms log analysis from "I need to write a Python script" to "I can get the answer in 10 seconds at the terminal". The pipe | is the glue that chains them.

grep — Pattern Matching

grep searches for patterns (regex) in files or stdin. It is the first tool you reach for when looking at logs.

bash — grep for log analysis

# Basic grep
grep "error" /var/log/nginx/error.log           # lines containing "error"
grep -i "error" /var/log/syslog                 # case-insensitive
grep -v "DEBUG" app.log                         # lines NOT matching
grep -n "error" app.log                         # show line numbers
grep -c "error" app.log                         # count matching lines
grep -l "error" /var/log/*.log                  # just file names

# Context
grep -A 3 "OutOfMemoryError" app.log            # 3 lines AFTER match
grep -B 2 "OutOfMemoryError" app.log            # 2 lines BEFORE match
grep -C 5 "exception" app.log                   # 5 lines around match

# Extended regex (grep -E or egrep)
grep -E "ERROR|FATAL|CRITICAL" app.log          # OR
grep -E "^2024-01-1[5-8]" app.log               # lines starting with date range
grep -E "[0-9]{1,3}(.[0-9]{1,3}){3}" access.log # IP addresses

# Recursive search in files
grep -r "password" /etc/ 2>/dev/null            # search all files in /etc
grep -rl "TODO" /opt/myapp/src/                 # just file names

# Combined with other tools
cat /var/log/nginx/access.log | grep " 500 " | wc -l    # count 500 errors
grep "ERROR" app.log | grep "2024-01-15"        # errors on specific date

awk — Field Processing

awk processes text field by field. It is the go-to tool for extracting columns from structured text (logs, CSV, /etc/passwd). Fields are split by whitespace (or a custom delimiter with -F).

bash — awk field processing

# awk basics: $1=field1, $2=field2, $NF=last field, NR=line number
# Pattern: awk '/regex/ {action}' file

# Print specific fields from /etc/passwd
awk -F: '{print $1, $3}' /etc/passwd           # username and UID
awk -F: '$3 >= 1000 {print $1}' /etc/passwd    # users with UID >= 1000 (real users)

# Process access.log (IP - - [date] "GET /path HTTP/1.1" status bytes)
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
# â†‘ top 10 IPs by request count

awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
# â†‘ HTTP status code frequency

awk '$9 == "500"' /var/log/nginx/access.log    # only 500 errors

# Arithmetic in awk
awk '{sum += $10} END {print "Total bytes:", sum}' /var/log/nginx/access.log

# Conditional
awk '{if ($9 >= 400) print $0}' access.log     # response code >= 400

# Custom separator in output
awk -F: '{print $1 "," $3 "," $6}' /etc/passwd   # CSV output

# Filter and reformat
df -h | awk 'NR>1 {print $5, $6}' | sort -rn   # disk usage % and mount, sorted

sed, cut, sort, uniq — Transformation Pipeline

A chain of simple tools connected by pipes can answer complex questions about logs and data without writing a script.

bash — sed, cut, sort, uniq pipelines

# sed — stream editor (substitute, delete, print lines)
sed 's/ERROR/CRITICAL/g' app.log               # replace all ERROR with CRITICAL
sed 's/password=[^ ]*/password=REDACTED/g' log # redact passwords
sed '/DEBUG/d' app.log                          # delete DEBUG lines
sed -n '/ERROR/p' app.log                       # print only ERROR lines (-n = no default print)
sed -n '10,20p' file                            # print lines 10 to 20

# cut — extract columns by delimiter or byte position
cut -d: -f1,3 /etc/passwd                       # fields 1 and 3, colon-delimited
cut -d' ' -f1 /var/log/nginx/access.log         # first field (IP address)
echo "2024-01-15T10:30:45" | cut -dT -f1        # date part: 2024-01-15

# sort and uniq
sort -k3 -n /etc/passwd                         # sort by 3rd field numerically
sort -k5 -rh du_output.txt                      # reverse human-readable size sort
sort -u file.txt                                 # sort + deduplicate
uniq -c sorted.txt                              # count consecutive duplicates
uniq -d sorted.txt                              # show only duplicate lines

# Real-world pipeline examples:

# Top 10 most-requested URLs from nginx log
awk '{print $7}' /var/log/nginx/access.log | \
  sort | uniq -c | sort -rn | head -10

# Count unique IPs per hour
awk '{print substr($4,2,14), $1}' /var/log/nginx/access.log | \
  sort -u | awk '{print $1}' | uniq -c

# Find the most common error messages
grep "ERROR" app.log | \
  sed 's/[0-9]{4}-[0-9-T:]*//g' | \    # strip timestamps
  sort | uniq -c | sort -rn | head -20

Key Points to Remember

1grep filters lines; awk processes fields; sed transforms text; cut extracts columns.
2Pipe | is the glue: each tool's stdout becomes the next tool's stdin.
3sort | uniq -c | sort -rn gives you a frequency count for any text field.
4grep -E (extended regex) supports | (OR), + (one or more), ? (zero or one).
5awk's END block runs once after all lines — use for totals and summaries.
6Always use grep -v to exclude noise before grep-ing for the signal.

Interview Questions

How would you find the top 10 IP addresses in an nginx access log?

What is the difference between grep, awk, and sed?

Ask Aria about Text Processing — grep, awk, sed, cut, sort

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Linux Performance Monitoring

Cron, at & Automation