The Unix Text Processing Trinity
Three tools born in Bell Labs that still power every server, every pipeline,
and every sysadmin's toolkit — over 50 years later.
Learn their history and master them the right way.
$ grep -n "ERROR" /var/log/app.log
142:ERROR: Connection timeout to db-primary
287:ERROR: Out of memory in worker-3
$ sed -i 's/localhost/0.0.0.0/g' config.yaml
$ awk '{sum += $3} END {print sum}' sales.csv
1847293.50
$
01 — Overview
Three tools, three philosophies
Each tool follows the Unix philosophy of doing one thing well. Together, they form the most powerful text processing toolkit ever created — no installation required on any Unix system.
02 — History
Born in Bell Labs
All three tools emerged from AT&T Bell Labs during the golden age of Unix development in the 1970s — a decade that shaped modern computing.
1973
grep is born from ed
Ken Thompson wrote grep overnight as a standalone tool. The name comes from the ed editor command
g/re/p — "globally search for a regular expression and print matching lines."
Doug McIlroy had asked Thompson to add regex search to ed for large files, and Thompson's
solution was to extract the functionality into its own program. This was one of the first
examples of the Unix philosophy: small, composable tools connected by pipes.
1973–1979
grep evolves: egrep & fgrep
Alfred Aho wrote egrep (extended grep), adding support for the +, ?,
and | operators — full regular expression syntax. He also created fgrep (fixed grep),
which uses the Aho-Corasick algorithm for extremely fast multi-pattern matching without regex.
These variants were later unified as flags: grep -E and grep -F.
👤 Alfred Aho, Bell Labs
1974
sed brings editing to streams
Lee E. McMahon developed sed at Bell Labs as a non-interactive version of the ed editor. The key innovation was processing text as a stream — reading from standard input, applying transformations, and writing to standard output. This made it perfect for pipelines and automation. McMahon's sed could handle files too large to fit in memory, a critical capability for 1970s hardware.
👤 Lee E. McMahon, Bell Labs
1977
awk — a language, not just a command
Alfred Aho, Peter Weinberger, and Brian Kernighan created awk as a pattern-matching programming language. Named after their initials (A-W-K), it was designed to process structured data by automatically splitting lines into fields. awk introduced concepts like BEGIN/END blocks, associative arrays, and field-based processing that influenced later languages including Perl and Python.
👤 Aho, Weinberger & Kernighan, Bell Labs
1985
The AWK Programming Language
Aho, Kernighan, and Weinberger published "The AWK Programming Language" — the definitive reference. This coincided with "new awk" (nawk), a major revision adding user-defined functions, multiple input streams, and computed regex. It cemented awk's position as a serious programming tool, not just a command-line utility.
📖 Addison-Wesley, 1988
1999
GNU grep — the modern standard
GNU grep, maintained by Mike Haertel and the GNU project, became the de facto implementation
on Linux systems. It unified grep, egrep, and fgrep into a single binary with flags,
added --color highlighting, recursive search (-r), Perl-compatible
regex (-P), and significant performance optimizations using Boyer-Moore
and other algorithms.
👤 Mike Haertel & GNU Project
2024
Still evolving: second edition of TAWKPL
Brian Kernighan co-authored the second edition of "The AWK Programming Language" — nearly four decades after the original. Kernighan also continues to maintain the original "one true awk" from Bell Labs. Meanwhile, gawk (GNU awk), led by Arnold Robbins, keeps adding features like network I/O, loadable extensions, and namespace support — proof that awk remains a living, evolving tool on all fronts.
👤 Brian Kernighan & Arnold Robbins, 2024
03 — Examples
From basic to battle-tested
Real-world examples organized by tool and difficulty level.
grep -rn "import requests" --include="*.py" ./src/ grep -i "error" /var/log/syslog grep -v "^#" config.conf
-r recursive, -n line numbers, -i case-insensitive, -v invert match.
grep -C 3 "segfault" /var/log/kern.log grep -A 5 "Exception" app.log grep -B 2 "FATAL" app.log
-C context (before+after), -A after, -B before. Perfect for log analysis.
grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' contacts.txt grep -oE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' access.log grep -nE '^\s*(def|function|func|fn)\s+\w+' *.py *.js *.go
-o prints only the matching part. -E enables extended regex.
grep -E "error|warning|critical" /var/log/syslog | sort | uniq -c | sort -rn grep -P '(?<=password=)[^\s]+' config.env grep -rlZ "import os" *.py | xargs -0 grep -l "subprocess"
Combining grep with pipes and xargs for complex multi-stage filtering.
sed 's/foo/bar/' input.txt sed 's/foo/bar/g' input.txt sed -i 's/http:/https:/g' urls.txt sed -i.bak 's/old/new/g' config.yml
The s command is sed's bread and butter. g flag = all occurrences. -i = in-place edit.
sed '/^#/d' config.conf sed '/^$/d' file.txt sed -n '10,20p' bigfile.log sed '5i\# This is a new comment' script.sh
d deletes, p prints, i inserts. -n suppresses default output.
sed -E 's/^(\w+) (\w+)$/\2, \1/' names.txt sed -E 's/(\w+)/"\1"/g' words.txt sed -E 's|https?://([^/]+).*|\1|' urls.txt
\1, \2 reference captured groups. -E enables extended regex (no escaping parens).
sed '/BEGIN_BLOCK/,/END_BLOCK/d' config.txt sed '/error/s/level=info/level=error/' app.log sed -e 's/foo/bar/g' -e 's/baz/qux/g' -e '/^$/d' input.txt sed '/\[database\]/a host=db.production.local' config.ini
Range patterns (/start/,/end/) and chained operations make sed a surgical text editor.
awk '{print $1, $3}' data.txt awk -F',' '{print $2}' users.csv awk '{print $NF}' access.log awk -F':' 'BEGIN{OFS="\t"} {print $1, $3, $7}' /etc/passwd
$1, $2... are fields. $NF = last field. $0 = entire line. -F sets delimiter.
awk '$3 > 100' sales.txt awk '$2 ~ /error/' log.txt awk '$3 > 50 && $4 == "USD"' transactions.csv awk '/TODO/ {print NR": "$0}' source.py
awk processes each line and lets you filter by any condition — numeric, string, or regex.
awk '{sum += $3} END {print "Total:", sum}' sales.csv awk '{sum += $2; n++} END {print "Avg:", sum/n}' scores.txt awk 'NR==1{min=max=$3} $3>max{max=$3} $3<min{min=$3} END{print min,max}' data.txt awk '{count[$1]++} END {for (k in count) print k, count[k]}' access.log
awk's variables, arithmetic, and associative arrays make it a command-line spreadsheet.
awk -F',' ' BEGIN { printf "%-20s %10s %8s\n", "Product", "Revenue", "Units" printf "%-20s %10s %8s\n", "-------", "-------", "-----" } NR > 1 { rev[$1] += $3; units[$1] += $2 } END { for (p in rev) printf "%-20s %10.2f %8d\n", p, rev[p], units[p] } ' sales.csv
awk's printf and associative arrays can generate full reports directly on the command line.
04 — Playground
Try it live
Enter your input text and command to see results instantly. This simulates grep, sed, and awk behavior right in your browser — no server required.
Click "Run" or press Ctrl+Enter to execute...
05 — Cheat Sheet
Quick reference
The most useful flags and patterns at a glance. Bookmark this page.
-iCase-insensitive matching
-vInvert match (non-matching lines)
-r / -RRecursive search in directories
-nShow line numbers
-lList filenames only
-cCount matching lines
-oPrint only matching portion
-EExtended regex (egrep)
-PPerl-compatible regex
-wMatch whole words only
-A n / -B n / -C nContext: after / before / both
--include="*.ext"Filter files by extension
--color=autoHighlight matches in color
s/old/new/Substitute first occurrence
s/old/new/gSubstitute all occurrences
s/old/new/giCase-insensitive substitution
/pattern/dDelete matching lines
/pattern/pPrint matching lines
-n 'Np'Print line N
-iEdit file in place
-i.bakIn-place with backup
-e 'cmd1' -e 'cmd2'Multiple operations
/start/,/end/Range pattern (from/to)
\1, \2Backreferences
y/abc/xyz/Transliterate characters
$0Entire line
$1, $2, $NFFields (1st, 2nd, last)
NRCurrent line number
NFNumber of fields
-F','Set field separator
BEGIN { ... }Run before processing
END { ... }Run after processing
/regex/ { ... }Pattern-action block
$2 ~ /pat/Field matches regex
printf "fmt", argsFormatted output
array[key]++Associative arrays
length(), split(), substr()Built-in functions
06 — Tricks & Tips
Power user secrets
Techniques that separate beginners from professionals.
TRICK 01
grep + xargs for bulk operations
Find files containing a pattern and do something with each one — safely handling spaces in filenames.
grep -rlZ "TODO" src/ | xargs -0 code grep -rcl "TODO" src/ | sort -t: -k2 -rn
TRICK 02
sed for renaming files in bulk
Combine sed with shell loops to rename hundreds of files using regex patterns.
for f in *.jpeg; do mv "$f" "$(echo "$f" | sed 's/\.jpeg$/.jpg/')" done for f in *; do mv "$f" "$(echo "$f" | sed 's/.*/\L&/')" done
TRICK 03
awk as a calculator
Use awk for quick math right on the command line — no bc or dc needed.
echo | awk '{print 2^32}' ls -l | awk '{ if ($5 > 1073741824) printf "%s\t%.1fG\n", $NF, $5/1073741824 else if ($5 > 1048576) printf "%s\t%.1fM\n", $NF, $5/1048576 else printf "%s\t%.1fK\n", $NF, $5/1024 }'
TRICK 04
The pipe trinity
Combine all three tools in a single pipeline for maximum power.
grep " 404 " access.log \ | awk '{print $1}' \ | sort | uniq -c | sort -rn \ | head -10 find / -size +100M 2>/dev/null \ | xargs ls -lh \ | awk '{print $5, $NF}' \ | sed 's|/home/user|~|g' \ | sort -hr
TRICK 05
sed's hold space (multi-line magic)
sed has a hidden "hold space" buffer for complex multi-line transformations.
sed 'N;s/\n/ /' file.txt sed -n '1!G;h;$p' file.txt sed '$!N; /^\(.*\)\n\1$/!P; D' file.txt
TRICK 06
awk's getline for external commands
Execute shell commands from within awk and use their output.
awk '{ "date +%H:%M:%S" | getline ts close("date +%H:%M:%S") print ts, $0 }' input.txt awk '{ cmd = "dig +short -x " $1 cmd | getline hostname close(cmd) print $1, hostname }' ips.txt
TRICK 07
grep --color in pipelines
Force color output even when piping to keep matches highlighted.
grep --color=always "error" log.txt | less -R grep --color=always -E "ERROR|$" app.log grep --color=always "ERROR" log.txt \ | GREP_COLORS='mt=01;33' grep --color=always -E "WARN|$"
TRICK 08
awk for JSON-like output
Generate structured output formats directly from awk.
awk -F',' 'NR>1 { printf "%s{\"name\":\"%s\",\"age\":%s}", (NR>2?",":""), $1, $2 } BEGIN{print "["} END{print "]"}' data.csv awk 'BEGIN{print "<table>"} { print "<tr>" for(i=1;i<=NF;i++) print "<td>"$i"</td>" print "</tr>" } END{print "</table>"}' data.txt
07 — Deep Dive
awk is a programming language
Most people learn awk as a one-liner tool. But awk is a complete, Turing-complete programming language — and it was designed to replace the need for piping grep into sed.
Here's the key insight most tutorials miss: awk already contains grep and sed inside it.
The /pattern/ syntax is grep. The substitution functions sub() and gsub() are sed.
Field splitting, variables, arrays, functions, and control flow are what make it a language.
When you pipe grep | sed | awk, you're often doing in three processes what awk can do alone in one.
Alfred Aho, Peter Weinberger, and Brian Kernighan designed awk in 1977 as a pattern-action language: for each line of input, test patterns and execute corresponding actions. This simple paradigm turns out to be extraordinarily powerful.
if
Control flow
Full if/else, for, while, do-while loops, break, continue, and next (skip to next line). You can write any algorithm.
[ ]
Associative arrays
Hash maps built in — count["errors"]++ just works. No imports, no declarations. Delete with delete array[key]. Iterate with for (k in array).
f()
User-defined functions
Define reusable functions with function name(args) { ... }. Supports recursion, local variables, and returning values.
%d
Formatted output
C-style printf for precise formatting — align columns, control decimal places, pad strings. Generate reports, CSV, JSON, or HTML directly.
>
Multiple I/O streams
Read from files with getline, write to multiple files with print > "file", pipe to and from shell commands. One awk program, many outputs.
~
Built-in regex engine
Pattern matching with ~ and !~, match(), sub(), gsub(), split(). All the regex power of grep and sed, unified in one tool.
grep | sed → awk
These side-by-side examples show how awk absorbs the roles of grep and sed — doing searching, transformation, and computation in a single pass through the data.
Task: Find error lines and extract the message
grep + sed pipeline
grep "ERROR" app.log \ | sed 's/.*ERROR: //'
→
awk alone
awk '/ERROR/ { sub(/.*ERROR: /, "") print }' app.log
Task: Count 404 errors per IP address and show the top 10
grep + awk + sort pipeline
grep " 404 " access.log \ | awk '{print $1}' \ | sort | uniq -c \ | sort -rn | head -10
→
awk alone
awk '/ 404 / { ip[$1]++ } END { for (i in ip) print ip[i], i }' access.log | sort -rn | head -10
Task: Replace "localhost" with "0.0.0.0" only in lines containing "bind"
sed (conditional)
sed '/bind/s/localhost/0.0.0.0/g' \ config.yaml
→
awk alone
awk '/bind/ { gsub(/localhost/, "0.0.0.0") } {print}' config.yaml
Task: Find CSV rows where revenue > 10000, format as a report with totals
grep + awk + column pipeline
grep -v "^#" sales.csv \ | awk -F',' '$3 > 10000 { print $1, $3 }' \ | column -t
→
awk alone — full report
awk -F',' ' !/^#/ && $3 > 10000 { printf "%-20s %10.2f\n", $1, $3 total += $3; n++ } END { printf "%-20s %10.2f\n", "TOTAL ("n" rows)", total }' sales.csv
Anatomy of an awk program
A full awk program has three blocks. Every feature of the language — variables, arrays, functions, I/O — works in any of them.
function humanize(bytes) { USER-DEFINED FUNCTION if (bytes > 1073741824) return sprintf("%.1fG", bytes/1073741824) if (bytes > 1048576) return sprintf("%.1fM", bytes/1048576) return sprintf("%.1fK", bytes/1024) } BEGIN { RUNS ONCE BEFORE INPUT FS = "," # field separator (like -F) OFS = "\t" # output separator print "File", "Size", "Type" } /\.(log|tmp)$/ { PATTERN → grep's job gsub(/.*\//, "", $1) ACTION → sed's job size[$NF] += $2 # associative array print $1, humanize($2), $NF } END { RUNS ONCE AFTER INPUT print "---" for (ext in size) printf "Total %s: %s\n", ext, humanize(size[ext]) }
08 — Comparison
When to use which?
A side-by-side comparison to help you pick the right tool for the job.
| Feature | grep | sed | awk |
|---|---|---|---|
| Primary purpose | Search & filter lines | Transform text streams | Process structured data |
| Best for | Finding patterns in files | Find-and-replace, deletions | Columnar data, reports |
| Regex support | BRE, ERE, PCRE | BRE, ERE | ERE |
| Variables | No | Hold/pattern space only | Full variables & arrays |
| Arithmetic | No | No | Yes (full math) |
| Field splitting | No | Manual (regex) | Automatic (-F) |
| In-place editing | No | Yes (-i) |
Via gawk -i inplace |
| Programming constructs | None | Branches, labels | if/else, for, while, functions |
| Speed for simple search | Fastest | Fast | Good |
| Learning curve | Easy | Medium | Medium–Hard |
| Typical one-liner | grep -rn "bug" |
sed 's/old/new/g' |
awk '{print $2}' |
09 — FAQ
Frequently asked questions
Each tool has a distinct focus. grep is a search tool — it scans text and prints lines matching a pattern. sed is a stream editor — it reads text, applies transformations (substitutions, deletions, insertions), and outputs the result. awk is a programming language for structured text — it automatically splits lines into fields and supports variables, arrays, and arithmetic. Think of it as: grep finds, sed changes, awk computes.
Start with grep — it's the simplest and you'll use it constantly. Next, learn basic sed substitutions (s/old/new/g) and line deletions. Finally, tackle awk for field-based processing. You can be productive with grep in 10 minutes, sed in an hour, and awk in an afternoon. Mastery of each takes longer, but basic usage covers 90% of daily needs.
Absolutely. These tools are available on virtually every Unix/Linux/macOS system without installation. They process text faster than most alternatives for common tasks, they compose beautifully with pipes, and they're the foundation of shell scripting. While Python, Perl, and modern alternatives exist, grep/sed/awk remain the fastest path from "I have a text problem" to "it's solved" — especially on servers where you can't install additional software.
ripgrep (rg) is a modern alternative to grep written in Rust. It's faster for recursive searches, respects .gitignore by default, and uses PCRE2 regex. However, grep is universally available (no installation needed), supports the POSIX standard for portability, and is the tool referenced in virtually all documentation and tutorials. Learn grep first — then use ripgrep if you need speed for large codebase searches.
For one-liners and quick data extraction, awk is often faster to write and execute than Python. You don't need to import modules, open files, or write boilerplate. However, for complex logic, error handling, API calls, or anything beyond text processing, Python is the better choice. The sweet spot for awk is tasks you can express in 1–5 lines. If your awk script exceeds 20 lines, it's probably time to switch to Python.
There are several options. WSL (Windows Subsystem for Linux) gives you full native versions. Git Bash includes grep, sed, and awk (via MinGW). Cygwin provides a full POSIX environment. You can also install GnuWin32 for standalone Windows ports. WSL is the recommended approach as it provides the most compatible and performant experience.
BRE (Basic Regular Expressions) is the default for grep and sed. Characters like (, ), {, }, +, and ? are literal — you must escape them to use as metacharacters: \(, \+, etc. ERE (Extended Regular Expressions), enabled with grep -E or sed -E, treats these as metacharacters by default. ERE is what most people expect from regex. When in doubt, use -E.
Perl was born directly from these three tools. In 1987, Larry Wall created Perl specifically to replace the awkward combination of grep, sed, and awk in his workflows. Perl inherited regex syntax from grep, the s/// substitution operator from sed, and concepts like $_ (the default variable), split, field processing, and BEGIN/END blocks from awk. In a sense, Perl is what you get when you merge all three into a single general-purpose language. This heritage also influenced later languages — Python's re module and Ruby's built-in regex support trace their lineage through Perl back to grep. Even grep -P (Perl-compatible regex) acknowledges this relationship by bringing Perl's enhanced regex syntax back into grep itself.
gawk (GNU awk) is the most widely used implementation of awk on Linux systems — when you type awk on most distributions, you're actually running gawk. It extends the original awk with features like network I/O, loadable extensions, namespace support, and persistent memory. A major milestone came in February 2026 with gawk 5.4, which switched its default regular expression engine to MinRX — a new, fully POSIX-compliant, non-backtracking matcher with polynomial time guarantees, written by Mike Haertel (the original author of GNU grep). The previous GNU regex engine was not fully POSIX-compliant, particularly around longest leftmost submatch rules. On top of that, gawk 5.4 is also faster at reading disk files — roughly 9% faster on large files thanks to removing unnecessary timeout checks. The old regex engine remains available via the GAWK_GNU_MATCHERS environment variable but is scheduled for eventual removal. Other awk implementations include mawk (default on Debian/Ubuntu, optimized for speed), nawk (the "new awk" from Bell Labs), and the original one true awk maintained by Brian Kernighan himself.