Schedule PBS S3 Sync Around ZFS Scrub Time Windows

Schedule PBS S3 sync around ZFS scrubs to avoid read contention, keep Proxmox Backup Server offsite copies fresh, and make backup windows predictable.

Proxmox PulseProxmox Pulse
12 min read
Matte black storage array and backup node in a quiet server room with soft circular light.

If your Proxmox Backup Server 4.2 datastore lives on ZFS, the quietest backup failure is a scheduled S3 sync that starts while a monthly or weekly ZFS scrub is still running. The sync job does not fail dramatically; it just crawls, the S3 upload window stretches, and the offsite copy is still missing hours of data when you need it. The fix is to give the scrub and the sync job separate time windows, then confirm the pool is quiet before the next upload.

Key Takeaways

  • Separate windows: schedule PBS S3 sync jobs so they do not start until the ZFS scrub has finished or is far from completion.
  • Reads collide: scrubs and PBS sync jobs both read heavily, and they compete for pool bandwidth, ARC, and CPU.
  • Predictability beats speed: a slower sync at a known time is easier to monitor than a sync that hides behind a scrub.
  • Measure first: confirm pool size, scrub duration, and actual sync runtime before moving schedules.
  • Keep offsite fresh: if scrub delay forces a later sync, make the window explicit instead of letting jobs overlap by accident.

Why ZFS Scrubs and PBS S3 Syncs Collide

A ZFS scrub reads the pool to verify checksums, while a Proxmox Backup Server sync job reads local chunks to send them to a remote S3-compatible store. If both run at once, they are not asking for different resources. They ask for the same read bandwidth, the same ARC, the same CPU for verification, and the same disk queue depth.

That collision is worse on a Proxmox VE 9.2 host that also has live VMs or containers. The scrub may already be running at a low priority, but PBS sync jobs are not free. They can read hundreds of gigabytes of backup chunks, verify data, and push network traffic. The pool sees a long read storm, and the S3 upload sees slower local reads.

You can see the effect before changing any schedule. Start with pool status and current scan state:

zpool status -t tank

Then watch real-time pool I/O while a sync or scrub is active:

zpool iostat -v tank 5 3

If the pool is 8 TiB and a scrub routinely takes four hours, a daily S3 sync scheduled at the same hour will not behave like the same sync on an idle pool. A 400 GiB sync window that normally finishes in 90 minutes can stretch to three or four hours once the scrub steals read bandwidth.

How to Check the Current PBS Sync Window

On the PBS node, list the sync jobs for the datastore that holds your Proxmox VE 9.2 backups. Replace the datastore name if yours is not called local.

proxmox-backup-manager sync-job list --datastore local

Look for the schedule field. A common default or hand-written schedule may be as simple as:

daily

or a more specific systemd-style calendar expression:

Mon-Sat *-*-* 03:00:00

The exact value matters less than the behavior it creates. If the sync job starts at 03:00 and your ZFS scrub starts at 02:00 on the first Sunday of the month, you have built a collision into the calendar. The job may still finish, but it will be slower, less predictable, and harder to alert on.

If you have multiple PBS datastores, check each one. A sync job for a small namespace may be harmless, while a sync job for the main S3 offsite datastore can saturate the pool. The sync job list output helps you find jobs that are enabled but forgotten.

How to Find a Safe ZFS Scrub Window

A ZFS scrub is not a backup. It is a health check that reads the pool to find latent errors. That makes it a good candidate for a low-traffic maintenance window, but only if the window is actually low-traffic. A scrub at 02:00 is often low-traffic for VM workloads, but it can be terrible for PBS sync jobs.

Start by checking whether a scrub is currently running:

zpool status tank

If you need to test the impact, start a scrub manually and watch the sync job behavior:

zpool scrub tank

If the timing test goes badly, stop the scrub:

zpool scrub -s tank

For a scheduled scrub, many administrators use cron or a systemd timer. A simple weekly Sunday scrub at 02:00 looks like this:

# /etc/cron.d/zfs-scrub-tank
0 2 * * 0 root /sbin/zpool scrub tank

The pool name must match your real ZFS pool. On Proxmox VE 9.2, the pool may be named something other than tank, and PBS may use a separate pool for its datastore. If PBS lives on its own pool, the collision may be smaller, but ARC pressure and CPU contention can still slow the sync.

A safe window usually has three properties. The scrub starts when backup jobs are not running. The PBS sync job starts only after the scrub is finished, or it starts many hours before the scrub. The window is documented so the next person does not add a verify job or prune job into the same hour. If you also manage ZFS snapshots for VM disks, keep snapshot cleanup out of the same window when possible.

Example: Moving a Daily S3 Sync Away From a Weekly Scrub

Suppose your Proxmox VE 9.2 environment backs up nightly to PBS, and PBS syncs to S3 daily. The ZFS pool scrub runs every Saturday at 00:15. The old sync schedule is:

daily

That is simple, but on Saturday it can overlap with the scrub. A better schedule splits the weekday and weekend behavior.

  1. Create or edit a weekday sync job with a schedule like this:
Mon-Fri *-*-* 04:00:00
  1. Create a Saturday sync job that waits until after the scrub has had time to finish:
Sat *-*-* 09:00:00
  1. Create a Sunday sync job if you still want a weekend offsite copy:
Sun *-*-* 04:00:00
  1. Leave the scrub schedule unchanged, but monitor the first two weeks to confirm the scrub finishes before 09:00.

Do not stack the weekday and weekend jobs; give each job its own day. This pattern keeps most sync jobs early enough to produce a fresh offsite copy, while giving the weekly scrub a protected window. If the scrub sometimes takes six hours, move the Saturday sync later, not earlier. If the scrub finishes in three hours most of the time, you can keep 09:00 and alert when the scrub is still running at 08:30.

The schedule expressions are not magic. They are a way to make the operator's intent visible. If you cannot explain when the sync runs and when the scrub runs, the schedule is too clever.

What to Watch During the Sync Window

After moving the schedule, watch the first few runs. You do not need exotic tooling. Pool status, PBS task logs, and a little patience are enough.

Check the scan state before the sync starts:

zpool status -t tank

Check pool read activity while the sync runs:

zpool iostat -v tank 5 3

Check pool sizes and health:

zfs list -t pool

On the PBS node, review recent service logs for sync job messages:

journalctl -u proxmox-backup-server -n 100 --no-pager

The PBS web interface also shows sync job status, but the command line is useful when you want to compare runs after hours. Look for three signals: the sync starts when expected, the pool is not in a scrub state, and the sync duration matches the baseline. If a sync that usually takes 80 minutes suddenly takes 240 minutes, do not assume S3 got slow first. Check the pool.

If you are preparing a Proxmox VE 9.2 upgrade, this is also a good time to review pool health before the node is under upgrade pressure. A pool that is quiet, scrubbed, and synced is much easier to reason about during an upgrade. The related ZFS pool health check before Proxmox VE 9.2 upgrade covers the pre-upgrade checks that pair well with this sync window.

Comparison: Sync During Scrub vs Outside Scrub

Situation S3 sync during scrub S3 sync outside scrub
Pool read load High and sustained Moderate and predictable
Sync duration Often 2x to 3x longer Close to baseline
Offsite freshness Delayed by hours Close to schedule
Restore confidence Harder to test during contention Easier to benchmark
Alert quality Noisy and ambiguous Cleaner and actionable

The table assumes the same pool, same PBS datastore, and same amount of pending data. Real environments vary, but the direction is usually the same: separating the jobs improves predictability.

What Are the Tradeoffs?

The honest tradeoff is freshness. If you delay the Saturday S3 sync until after a long scrub, your offsite copy may be several hours older than it would have been with a 04:00 sync. That may be fine for a homelab, but it matters more in a small business environment where a Saturday morning failure needs the most recent offsite restore point.

There is also a scheduling tradeoff. A simple daily sync is easy to remember. A split weekday/weekend schedule is more precise but needs documentation. If nobody knows why the Saturday sync starts at 09:00, the next maintainer may move it back to 04:00 and recreate the collision.

A third tradeoff is bandwidth. If you compress the sync window after scrub, you may need more S3 bandwidth or a shorter sync window. If your remote endpoint is a slow S3-compatible service, a later start may not buy you much. In that case, the better fix may be to reduce the amount of data synced daily by adjusting retention, namespaces, or lifecycle rules. The related PBS 4.2 S3 lifecycle rules discussion can help keep the remote store from growing into a slow, awkward sync target.

Gotchas That Make the Window Slip

The most common gotcha I have seen is a monthly scrub that starts at 02:00 and a daily sync that starts at 03:00. On a small pool, that may pass for months. On a 12 TiB pool, the scrub can still be running when the sync starts. The sync does not stop, but it slows down, and the operator sees a backup window that drifts every month.

Another gotcha is assuming PBS only reads the latest chunks. Sync jobs may need to read more than the obvious daily backup set, especially after failed syncs, changed retention, or namespace operations. If a sync job has been failing for a few days, the next successful run can be much larger than the normal daily delta.

A third gotcha is mixing PBS verify jobs, prune jobs, and sync jobs in the same maintenance window. Verify jobs can be read-heavy too. If you schedule scrub, verify, prune, and sync all between 02:00 and 06:00, the pool may never get a quiet moment. Keep the S3 sync window narrow and clean.

If your PBS datastore is on a different ZFS pool from the Proxmox VE 9.2 workload pool, you may still see ARC pressure. The ZFS ARC checks after PBS 4.2 S3 sync post is useful when the sync is slow but the workload pool looks healthy. ARC hits, misses, and reclaim behavior can explain why two separate pools still feel related.

How to Make the Window Stick

A schedule that works once is not enough. Make it visible and testable.

  1. Record the scrub pool, scrub start time, and expected duration.
  2. Record the PBS sync job schedule and normal duration.
  3. Run one manual scrub and one manual sync outside the normal window to measure the baseline.
  4. Add a simple alert if the scrub is still running when the sync window opens.
  5. Test a restore from S3 after the first adjusted sync, not just a restore from the local PBS datastore.

The restore test matters because the whole point of the S3 sync is to make offsite data usable. A sync job can report success while the remote store is still awkward to browse or restore. If you want a repeatable proof that the offsite copy is usable, the PBS 4.2 S3 backup window discussion has useful context for defining that window before you need to rely on it.

Keep the scope tight. This is not a replacement for ZFS pool health checks, PBS retention tuning, or Proxmox VE upgrade planning. It is a small scheduling fix that makes backup operations easier to monitor. The next step is to check your current PBS sync schedule, check your ZFS scrub schedule, and move one of them by a few hours.

Conclusion

Proxmox VE 9.2 backups become more predictable when PBS 4.2 S3 sync jobs are scheduled around ZFS scrubs instead of accidentally overlapping them. The practical outcome is a cleaner sync window, steadier offsite freshness, and fewer ambiguous slow-backup alerts. Your next step is to list the sync jobs, check the scrub state, and choose one fixed time for each job before the next scrub.

Share
Proxmox Pulse

Written by

Proxmox Pulse

Sysadmin-driven guides for getting the most out of Proxmox VE in production and homelab environments.

Related Articles

View all →