When managing Prometheus servers, tracking the number of time series in your LevelDB storage is crucial for capacity planning and performance tuning. Excessive time series can lead to high memory usage and slower queries.

The most straightforward way is through Prometheus's HTTP API:

# Query all series count
curl -s "http://localhost:9090/api/v1/series?match[]={__name__=~'.+'}" \
  | jq '.data | length'

# Breakdown by metric name
curl -s "http://localhost:9090/api/v1/series?match[]={__name__=~'.+'}" \
  | jq -r '.data[].__name__' | sort | uniq -c | sort -nr

For deeper inspection of the LevelDB files:

# Install LevelDB tools
go get github.com/google/leveldb

# Analyze the storage directory
leveldb_analyze -db=/path/to/prometheus/data \
  -print_series_count=true \
  -show_metric_names=false

Here's a script to track time series growth over time:

#!/bin/bash
TS_COUNT=$(curl -s "http://localhost:9090/api/v1/series?match[]={__name__=~'.+'}" | jq '.data | length')
DATE=$(date +%Y-%m-%d)
echo "$DATE,$TS_COUNT" >> /var/log/prometheus_series_count.csv

Based on your findings, consider these adjustments in prometheus.yml:

storage:
  tsdb:
    retention: 15d  # Reduce retention if needed
    max_samples_per_send: 1000000
    max_series_per_metric: 100000

For visualization, try these PromQL queries:

# Show highest cardinality metrics
topk(10, count by (__name__)({__name__=~".+"}))

# Track cardinality changes
count(count by (__name__)({__name__=~".+"}))

As a Prometheus maintainer, you'll often need to monitor the cardinality of time series stored in LevelDB. The number of active time series directly impacts memory usage and query performance. Here's how to get this critical metric:

The simplest method is using Prometheus's built-in HTTP API:

# Get the total number of time series
curl -s 'http://localhost:9090/api/v1/series?match[]={__name__=~".+"}' | jq '.data | length'

# Get count by metric name
curl -s 'http://localhost:9090/api/v1/series?match[]={__name__=~".+"}' | jq '.data | group_by(.__name__) | map({metric: .[0].__name__, count: length})'

For deeper analysis, you can examine LevelDB files directly. First, locate your storage directory (typically --storage.tsdb.path in Prometheus config):

# Install leveldb tools if needed
sudo apt-get install libleveldb-dev

# Use leveldb command-line tool to inspect
cd /path/to/prometheus/data
leveldb dump --hex 2>/dev/null | grep -c '^[0-9a-f]\+:.*__name__'

Track time series growth using these PromQL queries:

# Current active series count
count({__name__=~".+"})

# Series count by job
count by (job)({__name__=~".+"})

# Track growth rate
rate(prometheus_tsdb_head_series_created_total[1h])

When series count exceeds these thresholds, consider tuning:

• > 100K series: Review retention policies
• > 500K series: Check for high-cardinality labels
• > 1M series: Evaluate sharding or remote storage

Create a monitoring rule to alert on sudden spikes:

groups:
- name: cardinality-alerts
  rules:
  - alert: HighSeriesCardinality
    expr: count({__name__=~".+"}) > 1000000
    for: 30m
    labels:
      severity: critical
    annotations:
      summary: "High time series cardinality ({{ $value }})"