Troubleshooting ERR_SSL_PROTOCOL_ERROR in Kubernetes Ingress with Cert-Manager and Let’s Encrypt


7 views

You've deployed an nginx service in Kubernetes with a Let's Encrypt certificate managed by cert-manager. HTTP works fine, but HTTPS connections fail with various SSL errors:


ERR_SSL_PROTOCOL_ERROR (Chrome)
SSL_ERROR_RX_RECORD_TOO_LONG (Firefox)
Assessment failed: No secure protocols supported (SSL Labs)

Your setup appears correct at first glance:


apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: example
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80

And the Ingress:


apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
    - hosts:
      - 'example.com'
      secretName: example-production-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 443

Several problems stand out:

  1. SSL Passthrough Misconfiguration: The ssl-passthrough annotation requires special handling
  2. Port Mismatch: Your service forwards 443 to port 80, but the Ingress expects HTTPS on 443
  3. Backend Protocol: Your nginx pod might not be configured for HTTPS

First, modify your Service definition:


apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: example
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 443  # Changed from 80 to 443

Then update your Ingress:


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    kubernetes.io/ingress.class: nginx
    # Remove ssl-passthrough unless specifically needed
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
  - hosts:
    - example.com
    secretName: example-production-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 443

After applying these changes:


kubectl get ingress -n example
kubectl describe certificate -n example
curl -v https://example.com

Check these key points:

  • The certificate is properly mounted in the ingress controller pod
  • Your nginx pod is listening on port 443 with SSL enabled
  • The ingress controller logs show successful TLS handshake

If you don't need SSL passthrough, this simpler configuration often works better:


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    kubernetes.io/ingress.class: nginx
spec:
  tls:
  - hosts:
    - example.com
    secretName: example-production-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80  # Ingress handles SSL termination

This approach lets the ingress controller handle SSL termination while communicating with your service over HTTP.


When setting up HTTPS for a Kubernetes service with Nginx Ingress and Cert-Manager, I encountered a puzzling scenario where HTTP worked perfectly but HTTPS connections failed with various SSL-related errors:

Chrome: ERR_SSL_PROTOCOL_ERROR
Firefox: SSL_ERROR_RX_RECORD_TOO_LONG
SSL Labs: Assessment failed: No secure protocols supported

Here's the original setup that wasn't working as expected:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: example
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80
  selector:
    app: example

The key issue here was the service configuration mapping HTTPS (443) to the same targetPort as HTTP (80). This means SSL termination wasn't properly handled.

The ingress had several potential issues:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
    - hosts:
      - 'example.com'
      secretName: example-production-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 443

After troubleshooting, these changes resolved the issue:

  1. Removed the ssl-passthrough annotation unless specifically needed
  2. Corrected the service port mapping
  3. Verified the certificate chain

Here's the corrected service definition:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: example
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
  selector:
    app: example

And the updated ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    kubernetes.io/ingress.class: nginx
spec:
  tls:
  - hosts:
    - example.com
    secretName: example-production-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

To confirm the fix worked:

kubectl get certificate -n example
kubectl describe ingress -n example
curl -v https://example.com
openssl s_client -connect example.com:443 -servername example.com
  • Check ingress controller logs: kubectl logs -n ingress-nginx [controller-pod]
  • Verify certificate status: kubectl describe certificate
  • Test with different SSL protocols: curl --tlsv1.2 https://example.com

Remember that SSL/TLS configuration depends on your specific Nginx Ingress Controller version and Kubernetes cluster setup. Always check the documentation for your particular environment.