How to Resize Persistent Volumes in a StatefulSet Without Downtime: A Kubernetes Elasticsearch Case Study


2 views

When managing Elasticsearch clusters in Kubernetes, we often encounter storage constraints as data grows. The Kubernetes error message clearly states the limitation:

StatefulSet.apps "es-cluster" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.

This restriction exists because StatefulSets maintain stable persistent identities for their pods. Direct modification of volumeClaimTemplates would break this guarantee.

Here's how to safely expand storage without recreating the StatefulSet:

# First, check your current PVCs
kubectl get pvc -n your-namespace

# Identify the PVCs associated with your Elasticsearch pods
# They typically follow the pattern: datadir-es-cluster-0, datadir-es-cluster-1, etc.

For each Elasticsearch pod (one at a time to maintain availability):

# 1. Scale down the replica to remove a pod
kubectl scale statefulset es-cluster --replicas=X-1

# 2. Edit the PVC to request more storage
kubectl edit pvc datadir-es-cluster-0
# Change spec.resources.requests.storage to your new size (e.g., 20Gi)

# 3. Verify the storage provider supports expansion
kubectl get storageclass
# Check the allowVolumeExpansion field

# 4. Wait for the resize to complete
kubectl get pvc datadir-es-cluster-0 -w

# 5. Scale back up
kubectl scale statefulset es-cluster --replicas=X

For larger clusters, consider scripting this operation:

#!/bin/bash
NAMESPACE="elasticsearch"
STATEFULSET="es-cluster"
NEW_SIZE="20Gi"
REPLICAS=$(kubectl get statefulset $STATEFULSET -n $NAMESPACE -o jsonpath='{.spec.replicas}')

for ((i=0; i<$REPLICAS; i++)); do
    PVC="datadir-${STATEFULSET}-$i"
    
    # Scale down one pod
    kubectl scale statefulset $STATEFULSET -n $NAMESPACE --replicas=$i
    
    # Patch PVC
    kubectl patch pvc $PVC -n $NAMESPACE --type merge -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"$NEW_SIZE\"}}}}"
    
    # Wait for resize completion
    while [[ $(kubectl get pvc $PVC -n $NAMESPACE -o jsonpath='{.status.capacity.storage}') != $NEW_SIZE ]]; do
        sleep 5
    done
    
    # Scale back up
    kubectl scale statefulset $STATEFULSET -n $NAMESPACE --replicas=$((i+1))
done

Before proceeding:

  • Verify your storage class supports volume expansion (allowVolumeExpansion: true)
  • Ensure your cloud provider supports online volume expansion
  • Take a snapshot before making changes
  • Monitor Elasticsearch shard rebalancing during the process

For storage classes that don't support expansion, consider:

# 1. Create a VolumeSnapshot
kubectl apply -f - <

When working with stateful applications like Elasticsearch in Kubernetes, storage requirements often grow over time. The core limitation we face is that StatefulSet specifications are mostly immutable after creation - particularly the volumeClaimTemplates section which defines our PersistentVolumeClaims (PVCs).

The Kubernetes API server explicitly prevents modifications to StatefulSet specs except for:

- spec.replicas
- spec.template
- spec.updateStrategy

This immutability is by design to maintain the integrity of stateful applications. The error message you encountered:

StatefulSet.apps "es-cluster" is invalid: spec: Forbidden: updates to statefulset 
spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

is Kubernetes enforcing this protection.

Here's how to safely expand storage for your Elasticsearch cluster:

# Step 1: Patch individual PVCs (not the StatefulSet)
kubectl patch pvc elasticsearch-data-es-cluster-0 -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

# Step 2: Verify the PVC status
kubectl get pvc elasticsearch-data-es-cluster-0 -o yaml | grep -A5 resources

# Step 3: Expand the underlying volume (cloud provider specific)
# AWS EBS example:
aws ec2 modify-volume --volume-id vol-123456 --size 20

# GCP PD example:
gcloud compute disks resize pd-name --size 20GB --zone us-central1-a

For clusters with multiple data pods, use this script to resize sequentially:

#!/bin/bash
NAMESPACE=elasticsearch
STATEFULSET=es-cluster
NEW_SIZE=20Gi
REPLICAS=$(kubectl get statefulset $STATEFULSET -n $NAMESPACE -o jsonpath='{.spec.replicas}')

for ((i=0; i<$REPLICAS; i++)); do
  PVC="${STATEFULSET}-data-${STATEFULSET}-${i}"
  
  # Scale down the replica to detach PVC
  kubectl scale statefulset $STATEFULSET -n $NAMESPACE --replicas=$i
  
  # Resize PVC
  kubectl patch pvc $PVC -n $NAMESPACE -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"$NEW_SIZE\"}}}}"
  
  # Verify resize completed
  while [[ $(kubectl get pvc $PVC -n $NAMESPACE -o jsonpath='{.status.capacity.storage}') != $NEW_SIZE ]]; do
    sleep 5
  done
  
  # Bring pod back online
  kubectl scale statefulset $STATEFULSET -n $NAMESPACE --replicas=$((i+1))
  
  # Wait for pod to be ready
  kubectl rollout status statefulset $STATEFULSET -n $NAMESPACE
done

Ensure your StorageClass supports volume expansion:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: elasticsearch-storage
provisioner: kubernetes.io/aws-ebs
allowVolumeExpansion: true
parameters:
  type: gp2

After storage expansion:

# Reallocate shards if needed
POST _cluster/reroute?retry_failed=true

# Verify disk thresholds
GET _cat/allocation?v

For production environments, consider using Elasticsearch's hot-warm architecture with separate StatefulSets for different data tiers.