When dealing with rotated Apache logs in .gz format, the zgrep command is your first line of defense. This GNU zip-aware version of grep handles compressed files seamlessly:

# Basic zgrep usage
zgrep "/special-page" /var/log/apache2/access.log.*.gz

# Case-insensitive search with line numbers
zgrep -in "admin" /path/to/logs/*.gz

# Count occurrences (-c flag)
zgrep -c "404" /var/log/httpd/access_log.*.gz

Combine zgrep with regular expressions for powerful log analysis:

# Find POST requests to API endpoints
zgrep -E 'POST.*/api/v[1-3]/' *.gz

# Extract unique IPs hitting a specific URL
zgrep "/wp-login.php" access.log.2023*.gz | awk '{print $1}' | sort | uniq -c | sort -nr

For massive log collections, GNU parallel dramatically speeds up processing:

# Install parallel if needed
sudo apt-get install parallel

# Process files in parallel (4 threads)
find /log/directory -name "*.gz" | parallel -j4 "zgrep 'pattern' {}"

# With output control
find . -name "access*.gz" | parallel --eta -j8 "zgrep -H '500 Internal Server Error' {}"

Build powerful one-liners for log insights:

# Top 10 requested URLs
zcat *.gz | awk '{print $7}' | sort | uniq -c | sort -rn | head -10

# Hourly traffic breakdown
zgrep -h "" *.gz | awk '{print $4}' | cut -d: -f2 | sort | uniq -c

For large-scale analysis:

• Use --binary-files=text flag with zgrep for some binary-like logs
• Consider pigz (parallel gzip) for faster decompression
• Cache frequent searches by extracting subsets to temporary files

# Using pigz with zgrep
ZGREP=$(which zgrep)
alias zgrep="$ZGREP --use-compress-program=pigz"

As a sysadmin or developer working with Apache servers, you've likely encountered rotated log files in .gz format. The challenge arises when you need to quickly search through these compressed logs for specific patterns without the overhead of extracting them first.

Unix pipes offer an elegant solution for this common task. Here's the basic command structure:

zcat *.gz | grep "pattern"

This command does three things:
1. zcat decompresses and concatenates all .gz files
2. The pipe (|) sends the output to grep
3. grep searches for your pattern in the uncompressed stream

For more complex searches, you can chain additional commands:

zcat access.log*.gz | grep -E "/api/v1/users|/admin" | awk '{print $1}' | sort | uniq -c | sort -nr

This example:
- Searches for two URL patterns
- Extracts client IPs ($1 in Apache logs)
- Counts unique IPs
- Sorts by frequency

When dealing with large log sets, consider these optimizations:

# Process files sequentially to reduce memory usage
for f in *.gz; do zcat "$f" | grep "pattern"; done

# Use zgrep for simpler syntax (though slightly less flexible)
zgrep "pattern" *.gz

To count how often a URI appears:

zcat *.gz | grep -c "/specific/uri"

Apache logs include timestamps, which you can filter:

zcat access.log-2023*.gz | awk '$4 >= "[01/Jan/2023:00:00:00" && $4 <= "[31/Jan/2023:23:59:59"'

For very large log sets, consider:

# Use pigz for parallel decompression
pigz -dc *.gz | grep "pattern"

# Use agrep for approximate matching
zcat *.gz | agrep -3 "possible_misspelled_pattern"