First, a brief description of this PVE server’s disk setup: there are two 2T mechanical hard drives. One is used to install the PVE system and allocate virtual disks to VMs. The other 2T drive is dedicated to periodic automatic VM backups (stored with ZSTD compression, keeping the last 3 backups). One VM, vm-100 (KVM mode, though the issue should be independent of virtualization type), has two virtual disks: a 200G /dev/sda1 and a 1T /dev/sdb1. The 200G disk is used to install the OS (Debian 10), and the 1T disk is used for service data storage.
Initially, one service’s data was stored on the main system partition on the 200G virtual disk. Since I did not anticipate how fast the service data would grow, the 200G system disk quickly filled up. So I scheduled some time to migrate that service’s data directory to the 1T disk mount point. After a series of operations (I even messed up once by using cp without preserving file permissions, causing the service to fail to start after migration, and had to redo it with cp -a), the service resumed normal operation.
However, during subsequent PVE automatic backup tasks, backup failures started to occur. Checking the PVE logs, I saw errors like this (from the automatic backup notification email):
Continue reading…