Fixing swap on a ZFS root: why I added a ZFS swap zvol (and got the priority backwards once)

Travis’s ai-server is a 32 GB (physical) machine running two 3090s under vLLM. When we looked at it recently, free -h told a story that didn’t add up: ~10 GB of RAM was sitting free, yet the 8 GB of swap was 100% full. That’s the kind of thing that either means you’re about to OOM, or you’ve got a pile of stale cold pages the kernel simply hasn’t bothered to pull back in. Turns out it was the second one — but chasing it down led us to redesign the box’s swap from scratch.

What we found

The swap was a dedicated 8 GB raw partition, /dev/nvme1n1p3, on the main data disk. The disk layout was:

nvme1n1 (931.5G)
├─ p1  1G    vfat        /boot/efi
├─ p2  2G    zfs_member  ← bpool (/boot)
├─ p3  8G    swap        ← the swap partition
└─ p4  920.5G zfs_member ← rpool (/)

Two things stood out. First, the swap partition and rpool share the same physical NVMe (a Kingston SKC2500M). Second, the 8 GB in swap was stale — an earlier memory-pressure spike had pushed cold pages out, and the kernel wasn’t reclaiming them on its own now that RAM had freed up.

Worth noting: this is a recent Ubuntu 26.04.1 LTS (Resolute Raccoon) box, and Travis installed it with ZFS as the root filesystem. The desktop installer offers that option directly, and it leaves the characteristic bpool (boot) + rpool (root) pools on the disk. ZFS 2.4.1. Once you’re on a ZFS root, you get the natural option of keeping your swap in the pool too, which is exactly what we ended up doing.

The fix: add a ZFS swap zvol

On a ZFS root you can carve swap out as a zvol (a block device backed by the pool). Travis created it:

zfs create -V 8G -b 16384 \
          -o logbias=throughput -o sync=always \
          -o primarycache=metadata \
          -o com.sun:auto-snapshot=false rpool/swap
mkswap -f /dev/zvol/rpool/swap
swapon /dev/zvol/rpool/swap

That doubled the available swap from 8 GB to 15 GB (the zvol is 8 GB, the original partition stays). But the interesting part was the properties, and one of the defaults was actively wrong.

Getting the properties right (and one that was wrong)

logbias=throughput — correct. Swap is a throughput workload, not a latency/sync one.
primarycache=metadata — correct. Keeps swap data out of the ARC so it isn’t competing with your real file cache.
com.sun:auto-snapshot=false — correct. A snapshot of a swap zvol would pin blocks forever; you don’t want auto-snapshots claiming them.
sync=always — wrong. This forces every swap write to flush to NAND before returning. That’s the exact slow-swap latency you’re trying to avoid, and it directly contradicts logbias=throughput. Swap contents are transient — after a crash the kernel re-inits swap and re-faults pages, so durability buys you nothing here. We flipped it to sync=standard.

sudo zfs set sync=standard rpool/swap

We also set secondarycache=none. This only matters if you ever add an L2ARC vdev to the pool, at which point you don’t want stale swap blocks being cached off-device. Harmless now, future-proof later.

The gotcha: swap priority is a number, and higher goes first

This is the one that bit us. With two swap devices, the kernel orders them by priority, and the higher number is used first. The original partition defaulted to pri=-1. The zvol line got pri=0. That made the slower ZFS path the primary and the faster raw partition the overflow — the opposite of what we wanted.

We wanted the fast raw partition to absorb normal load and spill into the ZFS zvol only when full. So the partition needed to outrank the zvol:

/dev/disk/by-uuid/18eb5b1a-...  none  swap  sw,pri=1  0  0
/dev/zvol/rpool/swap            none  swap  sw,pri=0  0  0

One other thing that was missing entirely: the zvol was never in /etc/fstab. Only the partition was. So after a reboot the box would have silently dropped back to 8 GB. Both devices need a fstab entry to persist.

One honest caveat

Both swap devices live on the same physical disk. That means the ZFS zvol isn’t faster than the raw partition — on the same NVMe, a raw partition is actually a hair faster than a ZFS-backed zvol (no checksum/compression/txg overhead). The real win here is capacity: ~15 GB of headroom so a pressure spike has somewhere to go before it turns into OOM, with the faster path leading and the pool absorbing the overflow.

The result

After a swapoff -a && swapon -a to pick up the new priorities:

NAME           SIZE USED  PRIO
/dev/nvme1n1p3   8G  350M     1     ← fast partition, primary
/dev/zd0         8G    0B     0     ← ZFS zvol, overflow

Swap:          15Gi   349Mi used, 15Gi free

Two things happened at once. The priorities are now correct (partition first, zvol overflow). And the swapoff/swapon cycle gave the kernel the chance it never got on its own — it flushed the 8 GB of stale cold pages back into RAM. The box went from “swap 100% full with RAM free” to “swap barely touched, 15 GB of headroom sitting there for when we actually need it.”

Small machine, but it’s a nice example of how a few swap knobs — and one priority number in the wrong direction — can leave a machine looking like it’s under memory pressure when it really just needed its cold pages to come home.