Does cURL Cache HTTP Responses? How to Disable Caching for Accurate Benchmarking


1 views

When running performance tests against an Akamai server from an AWS instance, I noticed dramatically different response times between initial and subsequent requests for a 3MB video file:

# First request (slow)
curl -v https://example.com/video.mp4 -o /dev/null
# Took ~5 seconds

# Second request (fast)
curl -v https://example.com/video.mp4 -o /dev/null  
# Took ~200ms

The short answer is no, cURL doesn't have built-in response caching. The speed difference you're observing comes from other layers in the stack:

  • DNS Caching: Both system-level and cURL's DNS cache
  • TCP Connection Reuse: Keep-alive connections being reused
  • CDN-Level Caching: Akamai's edge servers caching content
  • OS-Level Caching: Filesystem caching of downloaded content

To ensure each request is truly fresh, use these cURL options:

curl -v \
  -H "Cache-Control: no-cache" \
  -H "Pragma: no-cache" \
  --dns-servers 8.8.8.8 \
  --no-keepalive \
  https://example.com/video.mp4 -o /dev/null

For more accurate HTTP benchmarking:

# wrk (modern HTTP benchmarking tool)
wrk -t4 -c100 -d30s https://example.com/video.mp4

# vegeta (golang-based load tester)
echo "GET https://example.com/video.mp4" | vegeta attack -duration=30s | vegeta report

Add these headers to check caching behavior:

curl -v \
  -H "X-Cache: DEBUG" \
  -H "X-Cache-Key: DEBUG" \
  https://example.com/video.mp4

This will often return headers showing whether the response came from cache (HIT) or origin (MISS).


During recent performance testing of an Akamai server from an AWS instance, I observed puzzling behavior with cURL requests:


# First request (slow)
curl -v https://example.com/video.mp4 -o /dev/null
# Time: ~5000ms

# Subsequent request (fast)
curl -v https://example.com/video.mp4 -o /dev/null  
# Time: ~200ms

Contrary to common assumption, cURL itself doesn't maintain any caching mechanism. The speed difference points to caching occurring at different layers:

  • HTTP-level caching (via Cache-Control headers)
  • DNS caching at OS level
  • TCP connection reuse
  • CDN edge caching (Akamai in this case)

For accurate benchmarking, disable all potential caching layers:


# Disable HTTP caching
curl -H "Cache-Control: no-cache" -H "Pragma: no-cache" https://example.com/video.mp4

# Force DNS refresh and new TCP connection
curl --resolve example.com:443: --connect-timeout 5 https://example.com/video.mp4

# Comprehensive non-cached request
curl -v \
  -H "Cache-Control: no-cache, no-store, must-revalidate" \
  -H "Pragma: no-cache" \
  -H "Expires: 0" \
  --dns-servers 8.8.8.8 \
  --connect-timeout 5 \
  

When cURL isn't sufficient, consider these specialized tools:


# wrk - modern HTTP benchmarking
wrk -t12 -c400 -d30s https://example.com/video.mp4

# siege - configurable load testing
siege -c 100 -t 1M -b https://example.com/video.mp4

# vegeta - versatile load tester
echo "GET https://example.com/video.mp4" | vegeta attack -duration=30s -rate=100 | vegeta report

When testing Akamai or other CDNs, additional headers may be needed:


# Bypass Akamai cache
curl -H "Pragma: akamai-x-get-cache-key" -H "Pragma: akamai-x-cache-on" https://example.com/video.mp4

# Cloudflare cache bypass
curl -H "CF-Cache-Status: BYPASS" https://example.com/video.mp4

Remember that many CDNs have their own cache control mechanisms that may override standard HTTP headers in their edge configurations.