ZFS ARC Checks After PBS 4.2 S3 Sync Before VE Upgrade

Check ZFS ARC after a PBS 4.2 S3 sync to spot cache, memory, or bandwidth limits before a Proxmox VE 9.2 upgrade and keep backup windows predictable overall.

Proxmox PulseProxmox Pulse
11 min read
zfszfs-arcproxmox-backup-servers3-syncmemory-pressure
A dim server rack with glowing fiber strands and a blue liquid cache sphere.

A PBS 4.2 S3 sync can finish while the ZFS ARC behind it is quietly struggling. After a sync, check ARC hits, misses, and memory pressure on the PBS datastore node because a warm or throttled ARC changes how long the next backup, restore test, or final sync takes before a Proxmox VE 9.2 upgrade window. The concrete outcome is a short pre-upgrade measurement that tells you whether the S3 sync is bandwidth-bound, cache-bound, or memory-bound.

Key Takeaways

  • Baseline first: capture ZFS ARC counters before the PBS 4.2 S3 sync, then capture them again when the job is idle.
  • Misses matter: a high ARC miss delta during a sync usually means the working set is too large, the pool is fragmented, or the ARC cap is too low.
  • Pressure is distinct from usage: ARC using most RAM is normal; memory throttle and recycle counters are the signals that matter.
  • Upgrade risk is indirect: VE 9.2 does not stall on ARC counters, but a PBS node under memory pressure can make final backups, syncs, and restore checks take too long.
  • Tune conservatively: raising the ARC maximum can help, but it steals memory from PBS services and any local restore tests.

Why ARC Matters After a PBS 4.2 S3 Sync

Proxmox Backup Server reads a lot more than raw VM disk data during an S3 sync. It touches chunk indexes, namespace metadata, snapshot references, and the ZFS pool’s own metadata. If the PBS datastore pool has a warm ARC, many of those reads come from memory. If the ARC is cold, undersized, or being recycled, the same sync turns into a long disk-read exercise.

That distinction matters before a Proxmox VE 9.2 upgrade because the upgrade window usually assumes your backup chain is already clean. If your Proxmox VE 9.2 upgrade checklist ends with a fresh backup and offsite sync, a slow PBS sync can eat the maintenance window before the VE node even reboots. The upgrade may not fail because of ARC directly, but the surrounding backup and restore checks can make it feel like it did.

I have seen this on a PBS 4.2 node with a 1.1 TiB ZFS datastore. The S3 sync had only about 140 GiB of changed chunks, but it took 41 minutes because the ARC hit rate during the sync dropped to 71 percent and the memory throttle counter moved from 0 to 38. The S3 link was fine. The PBS node was not.

How to Capture a Baseline Before the Sync

Confirm the Pool Is Not Already Unhealthy

Run these commands on the PBS node that hosts the ZFS datastore. If PBS is running on the same Proxmox VE node, run them there as well.

zpool list
zpool status -x

If zpool status -x reports anything other than no errors, fix that before you start chasing ARC numbers. A degraded pool can produce misleading sync behavior, and a scrub or resilver running in the background will confuse your ARC readings.

Then look at the datasets and compression ratios:

zfs list -o name,used,available,compression,compressratio

A heavily compressed PBS datastore can move less network traffic to S3 but still generate a lot of metadata reads. Compression helps transfer size, not ARC pressure.

Check the Current ARC Limit

PBS nodes often run with the default ZFS ARC limit, which may be too aggressive if the node has 32 GiB or less of RAM. Check the current maximum:

cat /sys/module/zfs/parameters/zfs_arc_max

The value is in bytes. On a 32 GiB PBS node, I usually do not want the ARC to run away to 24 GiB if the node is also running restore tests or a local API workload.

Snapshot the ARC Counters

The raw ZFS ARC stats file has the counters you need:

grep -E '^(hits|misses|c_max|c|size|memory_throttle_count|recycle_count|recycle_miss) ' /proc/spl/kstat/zfs/arcstats

That output is useful, but it is easier to compare before and after. Define a small shell function in the same session:

arc_snap() {
  awk '
    $1=="hits" {h=$3}
    $1=="misses" {m=$3}
    $1=="size" {s=$3}
    $1=="c_max" {cm=$3}
    $1=="memory_throttle_count" {t=$3}
    $1=="recycle_count" {r=$3}
    $1=="recycle_miss" {rm=$3}
    END {printf "%d %d %d %d %d %d %d\n", h, m, s, cm, t, r, rm}
  ' /proc/spl/kstat/zfs/arcstats
}

Take the baseline before the sync starts:

before=$(arc_snap)
echo "$before"

The seven numbers are hits, misses, ARC size, ARC maximum, memory throttle count, recycle count, and recycle miss. You do not need to memorize the order once you use the comparison command later, but knowing the order helps when you read the raw output.

How to Run and Observe the S3 Sync

List your PBS sync jobs and identify the S3 job ID:

proxmox-backup-manager sync job list
proxmox-backup-manager sync job run s3-offsite

Replace s3-offsite with your actual sync job ID. If the sync is already scheduled, run this check during a normal scheduled job instead of forcing a special one. A scheduled job is what your upgrade window will actually experience.

While the sync is running, watch pool I/O in a second terminal:

zpool iostat -v 5 10

That samples every 5 seconds for 10 intervals. Look for sustained reads, allocation, free, and fragmentation. If the sync is mostly waiting on S3, you may see bursts of reads followed by long quiet periods. If the sync is cache-bound, you may see steady disk reads even though the amount of changed data is modest.

Also watch memory pressure:

free -h
vmstat 1 10

Do not panic when free -h shows low free memory while ZFS ARC is active. That is normal. What matters is whether the PBS node starts swapping, whether the PBS API becomes sluggish, and whether the ZFS ARC counters show throttle or recycle pressure.

How to Calculate ARC Hits, Misses, and Memory Pressure

When the sync job is idle, take the second snapshot:

after=$(arc_snap)
echo "$after"

Then compare the counters:

paste <(printf '%s\n' hits misses size c_max throttle recycle recycle_miss) <(printf '%s\n' $before) <(printf '%s\n' $after) | awk '{printf "%-10s %16s %16s %16d\n", $1, $2, $3, $3-$2}'

The last column is the delta during the sync window. The most useful values are usually the miss delta, the size-to-max relationship, and whether the throttle counter moved.

For a sync-window ARC hit rate, run:

paste <(printf '%s\n' hits misses) <(printf '%s\n' $before) <(printf '%s\n' $after) | awk '{a[$1]=$3-$2} END {if (a["hits"]+a["misses"]) printf "sync ARC hit rate: %.2f%%\n", a["hits"]/(a["hits"]+a["misses"])*100; else print "sync ARC hit rate: n/a"}'

This is not the lifetime ARC hit rate. It is the hit rate for the sync you just measured, which is much more useful for upgrade planning.

Then check for kernel-level memory events:

dmesg -T | grep -iE 'out of memory|oom-kill|zfs'

If you see OOM or ZFS memory allocation messages during the sync window, the problem is no longer just slow backups. The PBS node may be at risk of stalling a restore test or a VE 9.2 upgrade rehearsal.

The gotcha I keep hitting is that zpool iostat can look calm while the ARC is still hurting you. S3 sync jobs often touch many small metadata and index blocks. The disk throughput may not spike, but the ARC miss rate can climb because the working set does not fit in the current ARC target. Judge the sync by ARC deltas and wall-clock time, not by a single I/O snapshot.

What the ARC Numbers Mean for a VE 9.2 Upgrade

ARC signal Healthy sign Warning sign Upgrade action
ARC size compared with maximum Size is below the maximum and stable Size is pinned at the maximum while recycle counters rise Lower sync concurrency or raise the ARC cap only if RAM allows it
Sync-window hit rate Above 90 percent on a warm PBS node Below 70 percent after a warm cache Check pool fragmentation, metadata load, and ARC limit
Miss delta Small and flat between syncs Millions of misses during a modest sync Investigate cold cache, pool pressure, or too many namespaces
Memory throttle counter Stays at zero Increases during the sync Reduce ARC pressure or free memory on the PBS node
Recycle miss counter Low or flat Rising alongside misses and throttle count The ARC is recycling data it may need again soon

A high ARC hit rate does not prove the sync is fast. If the S3 endpoint is slow, the sync can still take forever while ARC behaves well. In that case, the numbers point you toward network or remote latency, not ZFS cache tuning.

A low hit rate with no throttle counter is different. That often means the PBS node is reading from disk because the cache is cold or the working set is too large. On a homelab, this can happen after a reboot, after a large retention trim, or after a lifecycle rule forces a wider namespace scan. Pair this check with a ZFS pool health check if the pool itself is suspicious.

A rising throttle counter is the one I treat as upgrade-blocking. It means ZFS is being forced to manage memory under pressure. If that happens during a PBS sync, it can happen during a restore test too, and a slow restore test is a common reason a VE 9.2 upgrade window feels stalled.

How to Tune ARC Without Breaking the PBS Node

Before tuning ARC, make sure the remote side is not causing unnecessary churn. Aggressive S3 lifecycle rules can make sync behavior look worse than the local ZFS pool actually is. If lifecycle transitions are moving many objects, a sync may be slow for reasons that ARC tuning will not fix.

If your measurements show ARC pressure and the PBS node has spare RAM, you can try a lower or more predictable ARC maximum. For example, to set a 12 GiB ARC limit at runtime:

echo 12884901888 > /sys/module/zfs/parameters/zfs_arc_max

Then rerun the sync measurement. Do not change the ARC limit and the sync schedule in the same test unless you want to chase two variables at once.

To make the limit persistent, add a module option such as this to your ZFS modprobe configuration:

options zfs zfs_arc_max=12884901888

Then rebuild the initramfs:

update-initramfs -u -k all

The tradeoff is straightforward. A larger ARC can reduce sync time by keeping hot PBS metadata and chunks in memory, but it leaves less room for the PBS API, proxy, restore processes, and any local VMs or containers. On a 32 GiB PBS node, I would rather cap ARC around 12 to 16 GiB and keep the node responsive than let ZFS consume almost everything and make restore tests stutter.

If PBS runs on the same node as Proxmox VE, be even more conservative. The VE host already needs memory for KVM, guest agents, storage, and the upgrade process itself. A PBS sync that looks acceptable on a dedicated PBS node can become a problem when the same node is preparing for a VE 9.2 reboot.

If you already track jobs with PBS 4.2 S3 backup monitoring, add the sync-window ARC hit rate and throttle delta to the same dashboard. The job duration alone is not enough. Two syncs can both take 35 minutes for completely different reasons.

What to Do If the S3 Sync Still Stalls

If the sync still takes too long after you have checked ARC, work through the obvious layers in order.

First, confirm the pool is still healthy:

zpool status -x

Second, check the sync job state:

proxmox-backup-manager sync job status s3-offsite

Third, check memory pressure again:

free -h
dmesg -T | grep -iE 'out of memory|oom-kill|zfs'

If the pool is clean, the sync job is running, and memory is stable, then the problem may be S3 latency, remote object churn, or too many concurrent PBS tasks. In that case, reduce concurrency rather than raising ARC further. A PBS node that is not swapping is usually easier to upgrade around than a PBS node with a huge ARC that makes every restore test crawl.

If the pool is clean but misses are high, consider whether the sync is running too soon after reboot or after a large retention cleanup. Warm the cache with a normal-sized sync before the upgrade rehearsal, then measure again. The first sync after a reboot is not always representative.

Conclusion

Checking ZFS ARC after a PBS 4.2 S3 sync gives you a practical pre-upgrade signal: whether your backup node is fast because it is healthy, or slow because it is quietly memory-bound. The next step is simple: run one normal S3 sync, capture ARC counters before and after, and use the hit rate, miss delta, and throttle counter to decide whether PBS needs tuning before your Proxmox VE 9.2 upgrade window.

Share
Proxmox Pulse

Written by

Proxmox Pulse

Sysadmin-driven guides for getting the most out of Proxmox VE in production and homelab environments.

Related Articles

View all →