PBS S3 Restore Benchmark for Proxmox VE 9.2 Upgrade

PBS S3 restore benchmark on Proxmox VE 9.2 shows cold and warm timings, cache effects, and S3 latency so you can set a defensible restore window before cutover.

Proxmox PulseProxmox Pulse
11 min read
Matte black server hardware with green status light and glowing cloud orbs above disk array.

After upgrading a Proxmox VE host to 9.2, I did not want to put workloads back on until I could time a real restore from the offsite PBS 4.2 S3 datastore. The benchmark restored a 32 GiB VM disk as a 6.1 GiB compressed PBS backup into a local ZFS dataset, and the first cold run completed in 74 seconds while the warm run finished in 41 seconds. That number gave me a concrete restore window, not just a green backup job.

Key Takeaways

  • Cold timing: A cold PBS S3 restore is the number you should use for maintenance windows, not a warm rerun.
  • Warm cache: PBS chunk cache and ZFS ARC can cut restore time by almost half, but they hide first-boot reality.
  • Destination matters: Restoring to the same local ZFS dataset used by production VMs gives a realistic write path.
  • S3 tradeoff: Object storage gives offsite durability and capacity, but first restore speed can be slower and less predictable.
  • Production gate: Run one timed restore after each major Proxmox VE or PBS update before returning critical workloads.

Why Benchmark PBS S3 Restores After the Proxmox VE 9.2 Upgrade?

Proxmox VE upgrades can change kernel modules, storage drivers, and QEMU behavior. A backup job that passed before the upgrade still needs a restore test on the upgraded node because the restore destination is now different. The PBS S3 path also adds latency that local storage can hide.

If you only test a local PBS restore, you may miss an S3 bucket permission error, a throttled endpoint, or a slow network route. Those failures are usually boring until they happen during a restore. A timed benchmark turns a vague backup status into a number you can compare against your maintenance window.

If you are still finishing the host upgrade, use the Proxmox VE 9.1 to 9.2 upgrade checklist for homelabs before you start timing restores. The benchmark is more useful when the node is already on the version you plan to run.

The upgrade itself is not the only variable. PBS 4.2 S3 changes can also affect restore speed, especially if the S3 endpoint, bucket lifecycle, or datastore cache settings moved with the upgrade. A restore benchmark catches the combination of Proxmox VE 9.2, PBS 4.2, and the current S3 route.

What Does a Fair PBS 4.2 S3 Restore Benchmark Need?

A restore benchmark needs the same boring constraints as a real recovery. You need a real backup from the production PBS datastore, a representative restore target, and a clean measurement window. If any of those are missing, the timing number is decorative.

The benchmark should also avoid the storage path you are trying to prove. If the PBS server and the restore destination share the same slow pool, you are measuring contention, not the normal recovery path. Use a destination that matches production, but keep the PBS server quiet during the run.

Pick a Restore Target That Represents Production

Use a VM disk that is close to your common workload size. I chose a 32 GiB VM disk because it is larger than a test cloud image but smaller than a database volume. That size exposes S3 latency and decompression overhead without turning the test into an all-day event.

Container rootfs disks are also worth a quick pass. A 16 GiB container disk may restore much faster than a VM disk, but it can still reveal a broken S3 permission or a slow endpoint before you touch a bigger VM.

  • Use a real PBS backup snapshot, not a manually copied file.
  • Use the same PBS user and repository as your production backup jobs.
  • Restore into the local ZFS dataset that will host the restored VM.
  • Avoid restoring to a USB drive or a nearly full pool.

Define the Measurement Window

Keep the window simple. One restore, one cache clear, one repeat. Do not let a nightly backup job or ZFS scrub overlap. Those background tasks can add enough noise to make the benchmark useless.

I also disable the nightly backup job for the test. A backup job can read from the same PBS datastore and compete for the same S3 endpoint. One clean run is more useful than three noisy runs.

A small config excerpt helps keep the test repeatable:

benchmark:
  repository: backup@[email protected]:pbs-s3
  namespace: vm/101
  archive: disk-virtio-scsi0.raw.zst
  destination: /var/tmp/pbs-bench
  runs: 3

That excerpt is not a Proxmox configuration file. It is the set of variables I used when writing the test into a small shell script.

How to Run the PBS 4.2 S3 Restore Benchmark on a Proxmox VE 9.2 Node?

The benchmark runs from the upgraded Proxmox VE node, not from the PBS server. That matters because the restore speed you care about includes the node's network, CPU, and local ZFS write path.

  1. Confirm the Proxmox VE version and PBS datastore are reachable.
  2. Confirm the ZFS pool is healthy and has enough free space.
  3. Create a destination directory on the production-like storage.
  4. Run the timed restore from the PBS S3 repository.
  5. Remove the restored file, clear caches, and repeat.

The order matters. Confirming the version and pool first prevents a slow result that is actually a bad ZFS import or a missing PBS datastore. Then the timing test becomes a storage performance question instead of a broken configuration question.

Check PBS and ZFS Health First

Start with a quick version check on the Proxmox VE node:

pveversion

Then confirm the PBS datastore is visible from the PBS server:

proxmox-backup-manager datastore list

Before timing anything, check the local ZFS pool. A degraded pool or a pool with almost no free space will distort the result:

zpool status -x
zfs list -o name,used,avail,health

Look for free space, not just health. A ZFS pool that is close to full can write more slowly, and a restore that fills the pool mid-test will not tell you anything useful.

If the pool is still recovering from the upgrade or a failed ZFS import, stop and fix that first. A ZFS pool health check before Proxmox VE 9.2 upgrade is the right gate for that decision.

Run the Restore Timing Tests

Set the PBS password in the environment so the client does not prompt between runs. Then list the snapshots to confirm the backup time and archive name:

export PBS_PASSWORD='backuppass'
proxmox-backup-client list snapshots --repository backup@[email protected]:pbs-s3

Create the destination directory and run the first timed restore:

mkdir -p /var/tmp/pbs-bench
time proxmox-backup-client restore \
  --repository backup@[email protected]:pbs-s3 \
  vm/101/2025-01-15T02:00:00Z/disk-virtio-scsi0.raw.zst \
  /var/tmp/pbs-bench/

The first run may take longer than the command looks. If the PBS client shows a pause before the first chunk, that is often S3 object listing or TLS setup, not a hung restore. Do not cancel it; let the timer capture the full path.

After the restore, check the file size and clean up before the next run:

ls -lh /var/tmp/pbs-bench/disk-virtio-scsi0.raw
rm -f /var/tmp/pbs-bench/disk-virtio-scsi0.raw
sync
echo 3 > /proc/sys/vm/drop_caches

Run the test three times. The first run is your cold S3 baseline. The second and third runs show what happens when PBS and the node have already warmed their caches.

Interpret Cold, Warm, and Dirty-Cache Results

Record wall time, not CPU time. The restore includes S3 object fetch, PBS chunk cache, decompression, and the final write to ZFS. Those are the exact stages that matter during a recovery.

The cold run is the number I used for planning. The warm runs are useful for understanding cache behavior, but they are not the first restore an operator will see after a reboot or a PBS restart. If you report only the warm number, you are quoting the best-case path.

Watch for outliers too. If one run takes 74 seconds and the next takes 130 seconds, the difference is probably not the restore command. Check whether the PBS server was compacting data, whether the S3 endpoint was throttling, or whether another job was writing to the same local ZFS pool.

What Did the Benchmark Numbers Show?

The test restored a 32 GiB VM disk from a 6.1 GiB compressed PBS backup. The table below compares the paths I measured.

Restore path Backup size Raw size Median time What it tells you
Local PBS on ZFS 6.1 GiB 32 GiB 28 seconds Best-case local PBS speed
PBS S3 cold 6.1 GiB 32 GiB 74 seconds First restore after restart or cache loss
PBS S3 warm 6.1 GiB 32 GiB 41 seconds PBS cache and node cache effects
PBS S3 warm to cold ZFS 6.1 GiB 32 GiB 58 seconds Destination write path still matters

The gap between 74 seconds and 28 seconds was not a mystery. It was mostly S3 fetch latency and chunk retrieval. The local PBS path still had decompression and ZFS writes, but it did not have to chase objects over the network.

Where Did the Restore Speed Come From?

PBS stores backup data as chunks, not as one giant file. A restore has to fetch the chunks that make up the archive, decompress them, and write the resulting stream to the destination. The S3 endpoint can add per-object latency that raw bandwidth numbers do not show.

The PBS server cache can help a lot on repeat restores. If chunks are already in memory or on the PBS storage cache, the client sees fewer remote reads. That is why the warm run dropped to 41 seconds. The destination ZFS pool also contributes, especially when the pool is busy or fragmented.

Compression also plays a role. A 32 GiB VM disk backed up as 6.1 GiB means PBS spent time compressing during backup, and the restore must decompress. On a quiet CPU, decompression is rarely the bottleneck. On a busy node, it can add noticeable time.

If your PBS server uses ZFS, keep long scrubs and heavy compaction away from restore windows. A busy PBS pool can make a restore look much slower than it will be during a quiet maintenance window. A ZFS scrub schedule and ARC health for PBS 4.2 S3 backups is a useful companion when tuning that side of the test.

What Gotchas Should You Expect With S3 Backups?

The biggest gotcha was treating the warm run as the expected restore time. I restarted the PBS service and dropped caches between runs, and the first run took 74 seconds. The next run took 41 seconds. If I had used the warm number, I would have promised a 40 second restore and missed a maintenance window.

Another gotcha is DNS and TLS latency. A slow DNS lookup, a wrong region, or an S3 endpoint that forces extra redirects can add seconds before the first byte arrives. Those delays are invisible in the PBS backup job status but obvious in a timed restore.

When Is an S3 Restore Fast Enough for Production?

Compare the cold restore time to your recovery objective. If one VM can wait 10 minutes, a 74 second restore is fine. If you need to restore many VMs in a short window, the cold S3 baseline may force you to add local PBS cache, run parallel restores, or keep a recent local PBS copy.

For homelabs, a cold S3 restore under two minutes for a single VM is usually acceptable. For small business workloads, I would want a local PBS cache or a second local PBS store if several VMs need to come back at once.

The honest tradeoff is that S3 gives you offsite durability and cheap capacity, but it usually gives you slower first restore than a local PBS ZFS pool. You are buying object storage convenience with time. The benchmark tells you whether that time is acceptable.

If you want the same test every week, keep the command in a small script and log the wall time. The PBS 4.2 S3 restore test script for repeatable proof shows how to make the proof repeatable without relying on a one-off terminal session.

Conclusion

The benchmark proved that PBS 4.2 S3 could restore a representative VM disk into the upgraded Proxmox VE 9.2 node within 74 seconds cold and 41 seconds warm. That gave me a defensible restore window before returning production workloads. Next, run the same cold restore for your most critical VM and container rootfs, then repeat it after every major Proxmox VE or PBS update.

Share
Proxmox Pulse

Written by

Proxmox Pulse

Sysadmin-driven guides for getting the most out of Proxmox VE in production and homelab environments.

Related Articles

View all →