How to Permanently Delete and Migrate Graphite Whisper Counter Metrics


3 views

When working with Graphite's whisper database, many developers encounter a frustrating behavior: deleted .wsp files mysteriously reappear within seconds. This occurs because Graphite's carbon-cache process maintains an active cache of recently received metrics.

  1. Stop carbon-cache:
    sudo service carbon-cache stop  # Systemd
    # OR
    sudo /etc/init.d/carbon-cache stop  # SysVinit
  2. Delete the whisper file:
    rm /opt/graphite/storage/whisper/stats/message/foo.wsp
  3. Clean carbon's cache:
    rm -rf /opt/graphite/storage/whisper/carbon-cache/*
  4. Restart services:
    sudo service carbon-cache start
    sudo service apache2 restart  # If using graphite-web

For large-scale migrations, consider using whisper-merge.py:

whisper-merge.py \
  --from=stats.message.foo \
  --to=stats.messages.foo \
  --xFilesFactor=0.1 \
  --aggregationMethod=sum

Add these to your carbon.conf:

[cache]
MAX_CREATES_PER_MINUTE = 0  # Disables automatic creation
WHISPER_AUTOFLUSH = True

Check active metrics with:

find /opt/graphite/storage/whisper -name "*.wsp" | grep "message"

When working with Graphite's whisper database, many developers encounter this frustrating scenario: you delete a .wsp file to remove outdated metrics like stats.message.foo, only to find it regenerates automatically within seconds - just empty this time.

Graphite's carbon-cache process maintains an in-memory index of all metrics it has seen. When you delete a whisper file but carbon-cache still has the metric in its index, it will recreate the file structure during its next write cycle.

Here's the proper sequence to permanently remove or rename metrics:

Step 1: Stop Carbon Cache Writing

First, prevent new writes to the whisper files:

# For systemd systems
sudo systemctl stop carbon-cache

# For init.d systems
sudo /etc/init.d/carbon-cache stop

Step 2: Clear Carbon's Internal Cache

Delete the carbon-cache's internal database:

sudo rm /var/lib/carbon/carbon-cache-*/whisper/*.wsp
sudo rm /var/lib/carbon/carbon-cache-*/*.pid

Step 3: Remove Target Metric Files

Now safely delete your old metric:

sudo rm /opt/graphite/storage/whisper/stats/message/foo.wsp

Step 4: Restart Carbon Cache

# systemd
sudo systemctl start carbon-cache

# init.d  
sudo /etc/init.d/carbon-cache start

For ongoing prevention, add this to /etc/carbon/carbon.conf:

[blacklist]
pattern = ^stats\\.message\\..*

This will make carbon-cache permanently ignore the old metric pattern.

After 24 hours, check your metrics retention:

find /opt/graphite/storage/whisper -name "*.wsp" | grep "message"

Should return no results if successful.

For batch renaming operations, use this Python script with whisper-tools:

import whisper
from whisper_tools import migrate

src = "/opt/graphite/storage/whisper/stats/message/foo.wsp"
dest = "/opt/graphite/storage/whisper/stats/messages/foo.wsp"

whisper.migrate(src, dest, move=True)
print(f"Migrated {src} to {dest}")