Open navigation menu

Pipelines and Data Flow

Constructs multi-stage pipelines connecting commands with pipes for data transformation, filtering, and aggregation in a streaming fashion.

BashCli Tooling

Code

snippet
#!/usr/bin/env bash
set -euo pipefail

# Count unique IP addresses from a log
grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}" /var/log/syslog 2>/dev/null \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10

# Find largest files in a directory
find /var/log -type f -printf "%s %p\n" 2>/dev/null \
    | sort -rn \
    | head -5 \
    | while read -r size path; do
        printf "%10d  %s\n" "$size" "$path"
    done

# Pipeline with tee for branching output
seq 1 100 \
    | tee >(grep -c "even" > /dev/stderr) \
    | grep -v "0" \
    | wc -l

# Named pipes for inter-process communication
fifo="/tmp/myfifo-$$"
mkfifo "$fifo"

( seq 1 10 > "$fifo" ) &
( while read -r num; do
    echo "Received: $num"
  done < "$fifo"
) &

wait
rm -f "$fifo"

# Complex pipeline for log analysis
cat /var/log/syslog 2>/dev/null \
    | grep -E "ERROR|WARN" \
    | awk "{print \$1, \$2, \$NF}" \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -20 \
    | column -t

Related Examples