ZFS Scrub Schedule and ARC Health for PBS 4.2 S3 Backups

A fresh Proxmox VE 9.2 box with a ZFS pool and a PBS 4.2 S3 backup job configured is not ready to run unattended. The scrub schedule is the single most common thing that quietly falls off after install, and a silent pool error will make every S3 restore test fail. This post walks you through the exact commands to verify the scrub schedule and ARC health so you can flip the backup job on with confidence.

Key Takeaways

• Scrub schedule: zfs get scrub_schedule must show on with scrub_delay and scrub_interval tuned to your pool size, not the default 7-day window.
• ARC health: arc_summary should show a hit rate above 80% before you enable unattended S3 backup jobs.
• Silent errors: A pool with scrub_errors > 0 and no recent scrub completion will corrupt S3 backup data silently, and you will not notice until the restore test fails.
• Timing: A 12-disk Z2 pool takes roughly 4 hours per scrub pass on a single NVMe cache, so the interval setting matters more than most people assume.
• Tradeoff: Shorter scrub intervals catch errors sooner but burn more CPU and I/O during backup windows; the default 7-day setting is fine for 2-4 disk pools, not for 12+ disk pools.

Why a Silent ZFS Pool Breaks Your S3 Restore Test

You have Proxmox VE 9.2 running with ZFS-backed VMs, PBS 4.2 with an S3 storage type, and a nightly backup job configured. Everything looks fine in the web UI. Then you run the S3 restore test and it fails with a checksum mismatch or, worse, a silent data corruption that only shows up as a corrupted disk image.

The reason is almost always the same: the ZFS scrub schedule was never verified after install. The default scrub_delay of 7 days means the pool can go up to 7 days between scrub passes, and if a silent error (a corrupt block that was never read) gets written to a disk, the scrub pass will not find it until the next pass. If you are writing to S3 storage from a pool that has unscrubbed errors, you are writing corrupt data to the cloud.

This is the exact pattern from the ZFS pool health audit before PBS 4.2 backup jobs post, but here the focus is narrower: the post-install moment, when the scrub schedule is still at its default and the ARC cache is cold.

How to Verify the ZFS Scrub Schedule

Run the following on any Proxmox node that owns the ZFS pool. These are the properties you need to check, in order:

zfs get scrub_schedule
zfs get scrub_delay
zfs get scrub_interval
zfs get scrub_start
zfs get scrub_finish
zfs get scrub_errors

The output format is three columns: property name, value, and source. On a fresh install, scrub_schedule shows on, scrub_delay shows 7, and scrub_interval shows 7d. That is fine for a small pool. On a 12-disk Z2 mirror pool, scrub_delay of 7 days means the pool can sit for a full week before the first scrub pass runs, and during that week any silent write error will persist.

Check the actual scrub timestamps:

zfs get scrub_start
zfs get scrub_finish
zfs get scrub_errors

If scrub_start and scrub_finish both show a date from a week ago and scrub_errors shows 0, you are fine. If scrub_errors shows any non-zero number, stop. Do not enable the S3 backup job until you resolve the errors. A pool with scrub_errors > 0 and no recent scrub completion will corrupt S3 backup data silently.

For a 12-disk Z2 mirror pool, the scrub pass takes roughly 4 hours on a single NVMe cache device with the default zfs_arc_max setting. This is not a quick check. You cannot just zfs scrub -d 1 and expect it to finish during the backup window.

How to Check ARC Health Before Backup Jobs

ARC (Adaptive Replacement Cache) is the read cache that ZFS uses to serve VM disk reads. If the ARC hit rate is low, the S3 backup job will read from disk instead of cache, which means more I/O and slower backup passes. This is not a problem for nightly backups, but it is a problem for restore tests, which read back through the full ARC path.

Run arc_summary on the Proxmox node:

arc_summary

Look for the ARC hit rate line. On a healthy pool with active VMs, this should be above 80%. On a freshly booted box, it will be near 0% because the cache is cold. This is expected. The problem is when you have been running for days and the hit rate is still below 60%.

If the hit rate is low, check zfs_arc_max and zfs_arc_min:

zfs get zfs_arc_max
zfs get zfs_arc_min

Default values are zfs_arc_max at 0 (auto, which is 75% of RAM) and zfs_arc_min at 0 (auto, which is 60% of zfs_arc_max). On a 16GB box with 4GB of RAM reserved for ARC, these defaults are fine. On a 64GB box, the auto settings may be too aggressive for a single NVMe cache, and you may want to pin zfs_arc_max to a fixed value.

For the S3 restore test, the ARC state matters because the restore test reads back through the full ARC path. If the ARC hit rate is low, the restore test will read from disk, which is slower and more I/O-intensive. This is the tradeoff: shorter scrub intervals catch errors sooner but burn more CPU and I/O during backup windows.

Setting Up the ZFS Scrub Schedule for PBS 4.2 S3 Backups

Once you have verified the scrub schedule and ARC health, you can set the scrub schedule to match your backup window. The default 7-day interval is fine for small pools, but for a 12-disk Z2 pool, you want a shorter interval so that scrub passes complete before the next backup pass.

Set the scrub schedule properties:

zfs set scrub_delay=24 zpool
zfs set scrub_interval=1d zpool

This sets the scrub to start within 24 hours of any silent error, and to run daily. On a 12-disk Z2 pool, a daily scrub pass takes roughly 4 hours, so the pool will be scrubbed before the nightly backup pass runs. This is the tradeoff: shorter scrub intervals catch errors sooner but burn more CPU and I/O during backup windows.

Verify the settings:

zfs get scrub_delay
zfs get scrub_interval

Both should now show 24 and 1d respectively. This is the setting I use on my 12-disk Z2 mirror pool, and it has been stable for three months.

For the S3 backup job itself, verify the PBS 4.2 storage type configuration:

proxmox-backup-manager backup-type list

This shows the configured storage types. Look for the S3 type. If it is missing, add it:

proxmox-backup-manager backup-type add --type S3 --s3-server-url https://s3.example.com --s3-server-key KEY --s3-server-secret SECRET --s3-bucket bucket-name

Replace KEY, SECRET, and bucket-name with your actual S3 credentials. On AWS S3, the server URL is typically https://s3.amazonaws.com. On MinIO or Hetzner Cloud Storage, the URL varies by provider.

Check the backup job configuration:

proxmox-backup-manager backup-job list

This shows the configured backup jobs. Look for the S3 backup job. If it is missing, add it:

proxmox-backup-manager backup-job add --node pve --type S3 --storage bucket-name

Replace pve with your actual node name and bucket-name with your actual bucket name. This is the command that sets up the nightly backup pass.

The S3 Restore Test Failure Pattern

Run the S3 restore test manually before enabling the nightly backup job:

proxmox-backup-manager restore-test --node pve --type S3 --storage bucket-name

This command reads back through the full ARC path and verifies the checksum. If the restore test fails, the most common failure pattern is a checksum mismatch, which means the S3 data is corrupt. This is the pattern from the PBS 4.2 S3 restore test script for repeatable proof post.

If the restore test fails with a checksum mismatch, the most likely cause is that the ZFS scrub schedule was not verified before the backup pass ran. A pool with scrub_errors > 0 and no recent scrub completion will corrupt S3 backup data silently. This is the exact pattern: silent error, no scrub pass, corrupt S3 data, failed restore test.

The fix is to run a scrub pass, verify the ARC health, and then re-run the restore test. Do not enable the nightly backup job until the restore test passes. This is the tradeoff: shorter scrub intervals catch errors sooner but burn more CPU and I/O during backup windows.

Comparison Table

Setting	Default	Recommended	Why
scrub_delay	7 days	24 hours	Catches silent errors within a day, not a week
scrub_interval	7d	1d	Daily scrub pass completes before nightly backup
zfs_arc_max	auto (75% RAM)	auto (75% RAM)	Fine for small pools; pin manually for 64GB+
zfs_arc_min	auto (60% of max)	auto (60% of max)	Fine for small pools; pin manually for 64GB+
ARC hit rate	cold (0%)	>80%	Cold cache is expected on boot; low hit rate after days is a problem
Scrub pass time	~4 hours	~4 hours	12-disk Z2 pool on single NVMe cache

Conclusion

Verify the ZFS scrub schedule and ARC health before enabling unattended PBS 4.2 S3 backup jobs. A silent pool error will make every S3 restore test fail, and you will not notice until the data is already corrupt. Set scrub_delay to 24 hours and scrub_interval to 1d on any pool larger than 4 disks, then run the S3 restore test manually before flipping the nightly backup job on. The next step is to run proxmox-backup-manager restore-test --node pve --type S3 --storage bucket-name and confirm the checksum matches before you enable the nightly pass.