Linux storage can be a little confusing at first, but once you see the full storage stack, I believe it becomes much easier to understand. Luckily I found a helpful diagram, thanks to Mr. Werner Fischer.

Simplified Linux Storage Stack

There’s a lot going on here, so I made a condensed version.

+---------------------------+
| Application / User Space  |  ← rsync, cp, syscalls, open(), close()
+---------------------------+
             ↓
+---------------------------+
| Virtual Filesystem (VFS)  |  ← Kernel abstraction for ext4, xfs, etc.
+---------------------------+
             ↓
+---------------------------+
|     Filesystem Layer      |  ← ext4, xfs, Btrfs, ZFS (Btrfs/ZFS may replace LVM)
+---------------------------+
             ↓
+---------------------------+
| Optional Mapper Layers    |
|---------------------------|
| - LVM                     |  ← /dev/mapper/vg1-lv_data
| - LUKS / dm-crypt         |  ← /dev/mapper/cryptroot          
| - RAID (mdadm)            |  ← /dev/md0
+---------------------------+
             ↓
+---------------------------+
|     Block Device Layer    |  ← /dev/sda, /dev/nvme0n1, disks, partitions
+---------------------------+
             ↓
+---------------------------+
|     Physical Disk Layer   |  ← SATA, NVMe, SSD, HDD
+---------------------------+

The “Optional Mapper Layers” are usually configured as follows:

• RAID → LUKS → LVM → ext4/XFS
• /dev/mapper/* is a result of the device-mapper abstraction used by both LUKS and LVM
• Btrfs and ZFS provide features like snapshots, volume management, and checksumming, making them capable of replacing LVM and RAID in many use cases.

I hope to make a series of posts going through the storage stack, showing practical examples at each layer. Topics I plan to cover include

• Configuring RAID with mdadm
• Encrypting disks with LUKS
• LVM

Example Tools at the Different Layers

To further solidify our understanding of the storage stack, let’s take a look at different tools, and see which layer they operate on.

💡 Note: Some commands such as dd work at multiple layers, as it can write to a file, or directly to disk. This should not be treated as a definitive list. It is meant to give you a rough idea of the different tools and the layer they operate at.

Physical Disk Layer

# display hardware info for disks
lshw -class disk

# show SMART health data
smartctl -a /dev/sdX

# list NVMe devices
nvme list

# benchmark drive read performance
hdparm -tT /dev/sdX

# show kernel messages for disk
dmesg | grep -i sdX

Block Device Layer

# list block devices
lsblk -f

# show UUIDs
blkid

# partition tools
parted /dev/sdX
fdisk /dev/sdX
gdisk /dev/sdX

# show block device performance
iostat -xz 1

# show per-process I/O
iotop -ao

# Clone a disk
sudo dd if=/dev/sda of=/dev/sdb bs=64K conv=noerror,sync status=progress

Optional Mapper Layers (LVM, LUKS, RAID)

LVM

# display physical volumes
pvs
pvdisplay

# display logical volumes
lvs
lvdisplay

# display volume groups
vgs
vgdisplay

# extend logical volume and resize the filesystem
lvextend -L +5G -r /dev/vg1/lv_data

LUKS

# encrypt a device
cryptsetup luksFormat /dev/sdX3

# open encrypted device
cryptsetup open /dev/sdX3 cryptroot

# close encrypted device
cryptsetup close cryptroot

Software RAID (mdadm)

# create RAID1 array
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sd[ab]1

# assemble RAID array
mdadm --assemble --scan

# show RAID status
cat /proc/mdstat

# detail info about RAID array
mdadm --detail /dev/md0

Filesystem Layer

# create ext4 filesystem
mkfs.ext4 /dev/sdX1

# create xfs filesystem
mkfs.xfs /dev/sdX1

# check and repair ext4 filesystem
fsck /dev/sdX1

# list Btrfs subvolumes
btrfs subvolume list /mnt

# check and repair XFS
xfs_repair /dev/sdX1

Virtual Filesystem (VFS) Layer

# show mounted filesystems
findmnt
mount
cat /proc/mounts

# mount a filesystem
mount /dev/sdX1 /mnt

# unmount a filesystem
umount /mnt

Application / User Space

# syscalls: open, read, write, close, etc.
# copy files
cp source dest

# sync files and directories
rsync -avh /src /dest

# create an archive
tar -czf archive.tar.gz folder/

# show file space usage
du -hx --max-depth=1 ./

# show filesystem usage
df -hT