Implementing Automatic Tiered Storage in Linux: Memory→SSD→HDD→Remote Archiving Solutions

Modern systems deal with data that has varying access patterns and importance levels. A well-designed tiered storage solution can significantly improve performance while reducing costs. Here's how to implement this on Linux using open-source tools.

Btrfs offers built-in support for tiering when combined with LVM or device mapper:


# Create SSD and HDD partitions
sudo mkfs.btrfs /dev/ssd1
sudo mkfs.btrfs /dev/hdd1

# Mount with mixed devices
sudo mount -o device=/dev/ssd1,device=/dev/hdd1 /mnt/tiered

ZFS provides more advanced tiering capabilities:


# Create ZFS pool with SSD cache
zpool create datapool mirror /dev/hdd1 /dev/hdd2
zpool add datapool cache /dev/ssd1
zpool add datapool log /dev/nvme0n1

bcache lets SSDs act as cache for HDDs:


# Make backing device (HDD)
sudo make-bcache -B /dev/sdb

# Make caching device (SSD)
sudo make-bcache -C /dev/nvme0n1

# Attach cache to backing device
echo [SSD-UUID] > /sys/block/bcache0/bcache/attach

dm-cache provides more granular control:


# Create cache setup
lvcreate -L 100G -n cache_meta vg /dev/ssd1
lvcreate -L 1T -n cache vg /dev/ssd1
lvcreate -L 10T -n slow vg /dev/hdd1

# Setup dm-cache
lvconvert --type cache-pool --poolmetadata vg/cache_meta vg/cache
lvconvert --type cache --cachepool vg/cache vg/slow

Flashcache (Facebook's solution):


# Load module
modprobe flashcache

# Create cache
flashcache_create -p back cachedisk /dev/ssd1 /dev/hdd1

# Mount
mount /dev/mapper/cachedisk /mnt/cache

EnhanceIO provides dynamic tiering:


# Install
git clone https://github.com/stec-inc/EnhanceIO
cd EnhanceIO/Driver/enhanceio

# Configure
eio_cli create -d /dev/hdd1 -s /dev/ssd1 -m wb -c enhancd

For memory→SSD→HDD→remote setups, combine multiple technologies:


# RAM tier (tmpfs)
mount -t tmpfs -o size=20G tmpfs /mnt/ram

# SSD tier (bcache)
make-bcache -B /dev/hdd1
make-bcache -C /dev/ssd1

# Remote tier (autofs)
echo "/mnt/remote /etc/auto.remote" >> /etc/auto.master
echo "archive -fstype=nfs,rw storage.example.com:/archive" > /etc/auto.remote

Essential commands for managing tiered storage:


# ZFS stats
zpool iostat -v 2

# bcache status
cat /sys/block/bcache0/bcache/stats_total/*

# dm-cache info
dmsetup status

For automated tiering based on access patterns, consider implementing custom policies using inotify or fanotify to track file access patterns.

Tiered storage architectures fundamentally work by moving data between different storage media based on access patterns and performance requirements. In Linux environments, we can implement this through several complementary approaches:

Btrfs offers built-in tiering capabilities when combined with LVM or device mapper:


# Create a Btrfs filesystem with SSD and HDD tiers
mkfs.btrfs -d single -m dup /dev/ssd1 /dev/hdd1
mount -o ssd,discard,compress=zstd /dev/ssd1 /mnt/tiered

ZFS provides sophisticated caching layers through its ARC (memory), L2ARC (SSD), and ZIL configurations:


# ZFS pool with cache devices
zpool create tank mirror /dev/sda /dev/sdb
zpool add tank cache /dev/nvme0n1
zpool add tank log /dev/nvme1n1

The device mapper caching (dm-cache) subsystem enables transparent block-level tiering:


# Create cached device (SSD as cache, HDD as origin)
pvcreate /dev/slow_hdd /dev/fast_ssd
vgcreate storage_vg /dev/slow_hdd
lvcreate -L 1T -n origin storage_vg /dev/slow_hdd
lvcreate -L 100G -n cache_meta storage_vg /dev/fast_ssd
lvcreate -L 900G -n cache_block storage_vg /dev/fast_ssd
lvconvert --type cache-pool --poolmetadata storage_vg/cache_meta \
          --cachemode writeback storage_vg/cache_block
lvconvert --type cache --cachepool storage_vg/cache_block \
          storage_vg/origin

bcache provides another approach for SSD caching of HDDs:


# Setup bcache backing and caching devices
make-bcache -B /dev/sdX
make-bcache -C /dev/nvme0n1
echo /dev/nvme0n1 > /sys/fs/bcache/register
echo /dev/sdX > /sys/fs/bcache/register

For the final tier (archive/remote storage), consider these options:

rclone mount with crypt for encrypted cloud storage
GlusterFS or Ceph for distributed storage
Tahoe-LAFS for secure decentralized storage

Implement custom tiering policies using tools like:


# Sample script to move cold files to archive
find /mnt/tiered -atime +30 -type f -exec mv {} /mnt/archive \;

Combine with systemd timers or cron jobs for automated tier transitions.

Key metrics to monitor when implementing tiered storage:

Cache hit ratios
Latency percentiles
IOPS distribution across tiers

Tools like iostat -xmt 1 and bcc-tools can provide real-time insights into tier performance.