How to Check Zookeeper Cluster Status and Identify Leader Node via CLI


2 views

When working with Apache Zookeeper in production environments, checking cluster status is a fundamental administrative task. The most straightforward method is using the stat command via the Zookeeper CLI.

$ echo stat | nc localhost 2181
Zookeeper version: 3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
Clients:
 /192.168.1.100:44824[0](queued=0,recved=1,sent=0)
 /192.168.1.101:34218[1](queued=0,recved=152,sent=152)

Latency min/avg/max: 0/0.5/23
Received: 153
Sent: 152
Connections: 2
Outstanding: 0
Zxid: 0x200000002
Mode: follower
Node count: 487

The Mode field in the output shows whether the current node is a leader or follower. To find all cluster members and their roles:

$ echo mntr | nc localhost 2181
zk_version  3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a
zk_avg_latency  0
zk_max_latency  23
zk_min_latency  0
zk_packets_received 154
zk_packets_sent 153
zk_num_alive_connections 2
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count   487
zk_watch_count  0
zk_ephemerals_count 0
zk_approximate_data_size 47895
zk_open_file_descriptor_count 32
zk_max_file_descriptor_count 1048576

For a complete view of the ensemble configuration:

$ echo conf | nc localhost 2181
clientPort=2181
secureClientPort=-1
dataDir=/var/lib/zookeeper/version-2
dataLogDir=/var/lib/zookeeper/version-2
tickTime=2000
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
serverId=2
initLimit=10
syncLimit=5
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0

For frequent monitoring, create a simple bash script:

#!/bin/bash
ZK_NODES=("zk1.example.com" "zk2.example.com" "zk3.example.com")

for node in "${ZK_NODES[@]}"; do
  echo "Checking $node:"
  echo -e "stat\nmntr" | nc $node 2181 | grep -E 'Mode|zk_server_state'
  echo "------------------------"
done

This script will output each node's status in a readable format, making it easy to identify the leader at a glance.

Zookeeper supports several four-letter commands for diagnostics:

  • ruok - Checks if server is running in non-error state
  • srvr - Gives full server stats
  • cons - Lists all active connections
  • crst - Resets connection stats counters
$ echo srvr | nc localhost 2181
Zookeeper version: 3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
Latency min/avg/max: 0/0.5/23
Received: 157
Sent: 156
Connections: 2
Outstanding: 0
Zxid: 0x200000002
Mode: follower
Node count: 487

To inspect your ZooKeeper cluster's status, you'll need to use the zkCli.sh or zkCli.cmd utility that comes with ZooKeeper installation. First connect to any cluster node:

./zkCli.sh -server host:port
# Example:
./zkCli.sh -server zk1.example.com:2181

Once connected, these commands reveal cluster information:

# Get basic server stats
stat /

# Detailed server status
srvr

# List all cluster members (requires ZooKeeper 3.5+)
config | grep "^server\\."

# Get current leader information
stat / | grep -i leader

For automation or when zkCli isn't available:

# Using telnet/nc (replace 2181 with your client port)
echo stat | nc localhost 2181 | grep Mode
echo mntr | nc localhost 2181 | grep zk_server_state

# Four-letter commands via HTTP (if admin server enabled)
curl http://localhost:8080/commands/stat

A leader node will show:

Mode: leader
zk_server_state: leader

Followers will display:

Mode: follower
zk_server_state: follower

The mntr command provides comprehensive metrics:

echo mntr | nc localhost 2181

# Key metrics to monitor:
zk_avg_latency
zk_outstanding_requests
zk_znode_count
zk_watch_count
zk_ephemerals_count

Here's a bash script to check all cluster members:

#!/bin/bash
ZK_NODES=("zk1:2181" "zk2:2181" "zk3:2181")

for node in "${ZK_NODES[@]}"; do
  echo -n "$node: "
  echo stat | nc ${node%:*} ${node#*:} | grep "Mode:"
done

Newer versions provide RESTful endpoints:

curl http://localhost:8080/commands/mntr
curl http://localhost:8080/cluster