When working with Graphite's whisper database, many developers encounter a frustrating behavior: deleted .wsp
files mysteriously reappear within seconds. This occurs because Graphite's carbon-cache process maintains an active cache of recently received metrics.
- Stop carbon-cache:
sudo service carbon-cache stop # Systemd # OR sudo /etc/init.d/carbon-cache stop # SysVinit
- Delete the whisper file:
rm /opt/graphite/storage/whisper/stats/message/foo.wsp
- Clean carbon's cache:
rm -rf /opt/graphite/storage/whisper/carbon-cache/*
- Restart services:
sudo service carbon-cache start sudo service apache2 restart # If using graphite-web
For large-scale migrations, consider using whisper-merge.py
:
whisper-merge.py \
--from=stats.message.foo \
--to=stats.messages.foo \
--xFilesFactor=0.1 \
--aggregationMethod=sum
Add these to your carbon.conf
:
[cache]
MAX_CREATES_PER_MINUTE = 0 # Disables automatic creation
WHISPER_AUTOFLUSH = True
Check active metrics with:
find /opt/graphite/storage/whisper -name "*.wsp" | grep "message"
When working with Graphite's whisper database, many developers encounter this frustrating scenario: you delete a .wsp
file to remove outdated metrics like stats.message.foo
, only to find it regenerates automatically within seconds - just empty this time.
Graphite's carbon-cache process maintains an in-memory index of all metrics it has seen. When you delete a whisper file but carbon-cache still has the metric in its index, it will recreate the file structure during its next write cycle.
Here's the proper sequence to permanently remove or rename metrics:
Step 1: Stop Carbon Cache Writing
First, prevent new writes to the whisper files:
# For systemd systems
sudo systemctl stop carbon-cache
# For init.d systems
sudo /etc/init.d/carbon-cache stop
Step 2: Clear Carbon's Internal Cache
Delete the carbon-cache's internal database:
sudo rm /var/lib/carbon/carbon-cache-*/whisper/*.wsp
sudo rm /var/lib/carbon/carbon-cache-*/*.pid
Step 3: Remove Target Metric Files
Now safely delete your old metric:
sudo rm /opt/graphite/storage/whisper/stats/message/foo.wsp
Step 4: Restart Carbon Cache
# systemd
sudo systemctl start carbon-cache
# init.d
sudo /etc/init.d/carbon-cache start
For ongoing prevention, add this to /etc/carbon/carbon.conf
:
[blacklist]
pattern = ^stats\\.message\\..*
This will make carbon-cache permanently ignore the old metric pattern.
After 24 hours, check your metrics retention:
find /opt/graphite/storage/whisper -name "*.wsp" | grep "message"
Should return no results if successful.
For batch renaming operations, use this Python script with whisper-tools:
import whisper
from whisper_tools import migrate
src = "/opt/graphite/storage/whisper/stats/message/foo.wsp"
dest = "/opt/graphite/storage/whisper/stats/messages/foo.wsp"
whisper.migrate(src, dest, move=True)
print(f"Migrated {src} to {dest}")