Troubleshooting Varnish “FetchError no backend connection”: Comprehensive Debugging Guide for Backend Connectivity Issues


2 views

When examining the varnishlog output, we see a critical pattern:

12 VCL_call     c miss fetch
12 FetchError   c no backend connection
12 VCL_call     c error deliver
12 TxStatus     c 503

This indicates Varnish can't establish a connection to the defined backend (127.0.0.2:8080). Let's analyze the infrastructure components:

First, confirm the backend service is actually listening:

netstat -tlnp | grep 8080
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      3086/nginx
tcp6       0      0 :::8080                 :::*                    LISTEN      3086/nginx

The default.vcl shows this backend definition:

backend default {
    .host = "127.0.0.2";
    .port = "8080";
}

Notice the potential IP mismatch - while nginx listens on all interfaces (0.0.0.0), Varnish tries to connect to 127.0.0.2 specifically.

Verify direct backend access:

curl -v http://127.0.0.2:8080
telnet 127.0.0.2 8080

Solution 1: Adjust Varnish backend configuration:

backend default {
    .host = "127.0.0.1";  # Or the actual server IP
    .port = "8080";
    .connect_timeout = 5s;
    .first_byte_timeout = 30s;
    .between_bytes_timeout = 60s;
}

Solution 2: Ensure proper host resolution in /etc/hosts:

127.0.0.1 localhost
127.0.0.2 varnish-backend

Enable detailed logging in VCL:

sub vcl_backend_error {
    set beresp.http.Content-Type = "text/html; charset=utf-8";
    synthetic( {"Error: "} + beresp.status + " " + beresp.reason );
    return (deliver);
}

Check system limits and Varnish parameters:

sysctl net.ipv4.tcp_tw_reuse
ulimit -n

Ensure nginx accepts connections from Varnish:

server {
    listen 127.0.0.2:8080;
    server_name www.mysite.com;
    # ... rest of config
}
  • Verify IP binding and routing tables
  • Check firewall rules (iptables/nftables)
  • Test with both IPv4 and IPv6 addresses
  • Monitor connection attempts with tcpdump

When examining the varnishlog output, we see the critical error sequence:

12 VCL_call     c miss fetch
12 FetchError   c no backend connection
12 VCL_call     c error deliver
12 TxStatus     c 503

This indicates Varnish cannot establish a connection to the defined backend (127.0.0.2:8080) despite the configuration appearing correct.

The backend definition in default.vcl:

backend default {
    .host = "127.0.0.2";
    .port = "8080";
}

Checking the running services shows nginx is indeed listening:

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      3086/nginx
tcp6       0      0 :::8080                 :::*                    LISTEN      3086/nginx

1. Backend Connectivity Test

First verify basic connectivity:

curl -v http://127.0.0.2:8080
telnet 127.0.0.2 8080

2. IP Binding Verification

The configuration shows a potential mismatch:

  • Varnish backend points to 127.0.0.2
  • Nginx shows listening on 0.0.0.0 (all interfaces)

Try modifying the backend definition:

backend default {
    .host = "127.0.0.1"; // Try loopback
    .port = "8080";
    .probe = {
        .url = "/";
        .timeout = 1s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
    }
}

Varnish Backend Health Checks

Implement proactive monitoring:

import directors;

probe health_check {
    .url = "/";
    .interval = 5s;
    .timeout = 1s;
    .window = 5;
    .threshold = 3;
}

backend server1 {
    .host = "127.0.0.1";
    .port = "8080";
    .probe = health_check;
}

sub vcl_init {
    new cluster = directors.round_robin();
    cluster.add_backend(server1);
}

Network Namespace Considerations

If using network namespaces or complex routing:

# Check routing tables
ip route show table all
# Verify interface binding
ss -tulnp | grep 8080

1. Standardize IP addresses (use either 127.0.0.1 or 0.0.0.0 consistently)
2. Implement proper health checks
3. Verify SELinux/AppArmor permissions:

# For SELinux
setsebool -P httpd_can_network_connect 1
# Check denials
ausearch -m avc -ts recent

4. Consider connection pooling parameters:

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .max_connections = 100;
    .connect_timeout = 5s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 10s;
}