Move Proxmox VM from old to new cluster FAST

We recently encountered a bit of a problem regarding moving VMs from our old Proxmox Cluster (v6) to our new Proxmox Cluster (v7). There are plenty of guides out there to explain how to use incremental backup and then restore with Proxmox Backup (PBS), along with other guides on how to backup and restore from the command line using vzdump and qmrestore. All of these guides make use of the Proxmox backup and restore functionality, which in our experience/setup tends to be extremely slow. To restore a VM with 381GB of used disk space took 8h 53min. This is not to mention the time it took to backup the VM (even incremental to PBS was slow).

We scratched our heads for quite some time thinking about how to reduce the amount of downtime for clients. The solution in our case was to make use of ZFS sync. This method allows you to copy a ZFS snapshot directly to the destination storage of your new cluster without battling with slow backup and restore speeds. This is obviously dependent on your setup using ZFS storage.

Just before we begin, it is worth mentioning a utility called pve-zsync from Proxmox which does the heavy lifting for you. The trouble with this tool is that you cannot specify the name of VM drive to be stored on the destination. In other words, if your VM on the old cluster is 123 and your disk is vm-123-disk-0, then the exact same name will be created on the new cluster when transferring. This is obviously undesired if you already have VM’s running on the new cluster with hard drive names which would conflict with those coming from the old cluster.

I would advise doing this late at night, ensuring you have a backup of the VM, and then shutting it down (Should you not wish to shut the VM down while the snapshot is transferring, you can use the following guide to do things incrementally). Although shutting down the VM to be moved to the new cluster is not entirely necessary, I think its best just to be sure. Assuming VM 123 is running on node1 from the old cluster, start by shutting down VM123 through the Proxmox UI. Next, run the following commands on node1 from the old cluster:

zfs snapshot YOUR-OLD-CLUSTER-ZFS-POOL-NAME/vm-123-disk-1@your-vm-snapshot-name

Replace YOUR-OLD-CLUSTER-ZFS-POOL-NAME with the name of the ZFS pool on your old cluster. Replace your-vm-snapshot-name with any name you desire.

zfs list -t snapshot

You’ll now see a list of snapshots, including the one you just made.

zfs send -v -- YOUR-OLD-CLUSER-ZFS-POOL-NAME/vm-123-disk-1@your-vm-snapshot-name | cstream -t 312000000 | ssh -o 'BatchMode=yes' root@node-on-new-cluster -- zfs recv -F -- YOUR-NEW-CLUSTER-ZFS-POOL-NAME/your-desired-vm-id-disk-0

A couple of things to note:

  • Replace YOUR-OLD-CLUSTER-ZFS-POOL-NAME with the name of your ZFS pool on your old cluster. Replace YOUR-NEW-CLUSTER-ZFS-POOL-NAME with the name of the your ZFS pool on your new cluster.
  • Replace your-vm-snapshot-name with the name of the snapshot created above.
  • The cstream -t command specifies the bandwidth limit in bytes. In this case it is 312 MB per second. Change as needed.
  • Replace node-on-new-cluster with the IP address of the node in your new cluster.
  • Replace your-desired-vm-id with the ID of the VM you’d like to create on the new cluster. For example vm-140-disk-0. This doesn’t have to be the same as the VM ID, but for clarity’s sake this is highly recommended.

Once the copy is complete, you need to copy the config of the VM from the old cluster to the new cluster. Assuming the VM ID on the new cluster is 140, you’ll run the following command on the node from the old cluster:

scp /etc/pve/qemu-server/123.conf root@node-on-new-cluster:/etc/pve/qemu-server/140.conf

The final step is to edit 140.conf on the new node in the new cluster and ensure the storage path is the same:

nano /etc/pve/qemu-server/140.conf

Change the following line:

scsi0: YOUR-OLD-CLUSER-ZFS-POOL-NAME:vm-123-disk-0

to

scsi0: YOUR-NEW-CLUSTER-ZFS-POOL-NAME:vm-140-disk-0

Start up your VM on the new cluster through the Proxmox UI, and Bob’s your uncle 🙂 Should you not be concerned about using the same VM/Disk names when transferring between the old and new clusters, then you can make use of the Proxmox tool pve-zsync. This needs to be installed first as follows:

sudo apt install pve-zsync

Next, perform the sync of VM 123 from the old cluster to the new cluster at a bandwidth of 500 MBps:

pve-zsync sync --source 123 --dest node-on-new-cluster:YOUR-NEW-CLUSTER-ZFS-POOL-NAME --verbose --name vm-123-disk-0 --limit 500000

A couple of things to note:

  • Replace 123 with the ID of the VM you’d like to move from the old to the new cluster.
  • Replace node-on-new-cluster with the IP address of the node on the new cluster.
  • Replace YOUR-NEW-CLUSTER-ZFS-POOL-NAME with the name of the ZFS pool on the new cluster
  • Replace vm-123-disk-0 with a friendly name to help you identify what this transfer was.
  • Replace 500000 with the bandwidth limit you’d like to set for the transfer.

Once the transfer is finished, you can copy the VM conf files as above and start up your VM on the new node.