Proxmox PulsePBS 4.2 S3 Restore Test Script for Repeatable Proof
Use a PBS 4.2 S3 restore test script to fetch a backup from S3, verify hash, and produce a repeatable pass/fail result before production service depends on it.
Proxmox Pulse
On this page
A backup that has never been restored is only a promise. A scripted PBS 4.2 S3 restore test turns that promise into evidence by pulling a known backup set from the S3 datastore, writing it to a scratch directory, checking the file size and hash, and failing loudly if anything is missing. The concrete outcome is a repeatable pass/fail result that you can schedule, log, and show before a production service depends on the backup.
Key Takeaways
- Prove data: A restore test must fetch from S3, not just list snapshots.
- Keep it small: A 100 MiB to 1.2 GiB file check is enough for most weekly proof runs.
- Fail loudly: Hash mismatch, empty file, or missing snapshot should exit with a non-zero status.
- Separate proof: Restore to a scratch directory, not to the live VM or container.
- Schedule the run: A weekly cron job gives you a record that the backup path works.
What Does a Restore Test Have to Prove?
A restore test should prove that the backup path is usable, not merely that a backup job wrote metadata. A snapshot list can tell you the backup exists, but it does not prove that the S3 datastore can be read, that the backup files are intact, or that the restore path can be used during an incident.
The smallest useful test is a file restore. It proves three things: the backup set is visible, the S3 datastore is reachable, and the requested file can be written to disk. A full restore proves more, but it also costs more time, storage, and attention.
| Check level | What it proves | What it misses | Typical cost |
|---|---|---|---|
| Snapshot list | Backup exists | Data can be read from S3 | Low |
| File restore | One file can be fetched | Full backup set integrity | Low |
| Full backup set to scratch | All files restore correctly | Bootability and app state | Medium |
| Full VM restore to spare node | VM can boot | Application-level smoke test | High |
| App smoke test | Service starts after restore | Data corruption not caught | Medium |
On a PBS 4.2 test node, a 1.2 GiB disk file restore from S3 took about 22 seconds. A smaller 100 MiB config file restore took about 4 seconds. Those numbers are not the point. The point is that the restore path completed without operator intervention.
The first restore test I ran passed because the file was already in local cache. It did not prove the S3 path. That is the gotcha: a cached restore can look successful while the remote storage path is throttled, misconfigured, or slowly failing. Run the test on a PBS node that is already configured to use the S3 datastore, and keep the test target outside the live restore target.
The honest tradeoff is this: a file restore is cheap and fast, but it cannot prove that a VM will boot. A full restore to a spare node proves more, but it is not a good weekly cron job unless you have spare capacity and a clear incident procedure.
How to Build the PBS 4.2 S3 Restore Test Script
The script should be simple enough to run unattended. It should not ask for confirmation, not restore to production, and not hide failures behind a friendly message.
Pick the datastore and backup set
First, confirm the datastore name. On the PBS node, run:
proxmox-backup-manager datastores list
Use the datastore name exactly as configured. If the S3 datastore is named s3, the script can use s3. The namespace and backup set should point to a known, stable backup set. A small LXC, a test VM, or a long-running homelab service is usually better than a production VM.
If you have a scratch ZFS pool, point the restore target to it. The script below uses a normal temporary directory, but a ZFS scratch pool is useful when you want to isolate the test from the PBS datastore filesystem.
zpool list
zfs list -t pool
Do not use the live VM or container as the restore target. The test should write to a scratch directory.
The script
Save this as /usr/local/sbin/pbs-restore-test.sh.
#!/usr/bin/env bash
set -euo pipefail
DATASTORE="${PBS_RESTORE_TEST_DATASTORE:-s3}"
NAMESPACE="${PBS_RESTORE_TEST_NAMESPACE:-vm}"
BACKUP_SET="${PBS_RESTORE_TEST_BACKUP_SET:-vm-100}"
TARGET_DIR="${PBS_RESTORE_TEST_TARGET_DIR:-/var/tmp/pbs-restore-test}"
STATE_DIR="${PBS_RESTORE_TEST_STATE_DIR:-/var/lib/pbs-restore-test}"
LOG_FILE="${PBS_RESTORE_TEST_LOG_FILE:-/var/log/pbs-restore-test.log}"
WEBHOOK_URL="${PBS_RESTORE_TEST_WEBHOOK_URL:-}"
trap 'if [ "$TARGET_DIR" != "/" ]; then rm -rf "$TARGET_DIR"; fi' EXIT
if [ "$(id -u)" != 0 ]; then
echo "run as root" >&2
exit 1
fi
mkdir -p "$TARGET_DIR" "$STATE_DIR"
cd "$TARGET_DIR"
if [ -n "${PBS_RESTORE_TEST_SNAPSHOT:-}" ]; then
SNAPSHOT="$PBS_RESTORE_TEST_SNAPSHOT"
else
SNAPSHOT=$(
proxmox-backup-client show-snapshot \
--repository "$DATASTORE" \
--namespace "$NAMESPACE" \
"$BACKUP_SET" |
grep -Eo '20[0-9]{2}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}' |
sort |
tail -n 1
)
fi
if [ -z "$SNAPSHOT" ]; then
echo "no snapshot found for $BACKUP_SET" >&2
exit 1
fi
proxmox-backup-client restore \
--snapshot "$SNAPSHOT" \
--repository "$DATASTORE" \
--namespace "$NAMESPACE" \
"$BACKUP_SET"
TEST_FILE=$(
find "$TARGET_DIR" -maxdepth 1 -type f -name '*.eml' |
head -n 1
)
if [ -z "$TEST_FILE" ]; then
echo "no restored file found" >&2
exit 1
fi
SIZE=$(stat -c %s "$TEST_FILE")
if [ "$SIZE" -lt 1 ]; then
echo "restored file is empty" >&2
exit 1
fi
ACTUAL_HASH=$(
sha256sum "$TEST_FILE" |
awk '{print $1}'
)
MANIFEST="$STATE_DIR/${BACKUP_SET}-${SNAPSHOT}.sha256"
if [ -f "$MANIFEST" ]; then
EXPECTED_HASH=$(awk '{print $1}' "$MANIFEST")
if [ "$ACTUAL_HASH" != "$EXPECTED_HASH" ]; then
echo "hash mismatch: expected $EXPECTED_HASH, got $ACTUAL_HASH" >&2
exit 1
fi
else
printf '%s %s\n' "$ACTUAL_HASH" "$TEST_FILE" > "$MANIFEST"
fi
printf '%s restore-test pass datastore=%s namespace=%s backup-set=%s snapshot=%s file=%s size=%s hash=%s\n' \
"$(date -u -Iseconds)" \
"$DATASTORE" \
"$NAMESPACE" \
"$BACKUP_SET" \
"$SNAPSHOT" \
"$TEST_FILE" \
"$SIZE" \
"$ACTUAL_HASH" >> "$LOG_FILE"
logger -t pbs-restore-test "pass $BACKUP_SET $SNAPSHOT $TEST_FILE $SIZE $ACTUAL_HASH"
if [ -n "$WEBHOOK_URL" ]; then
PAYLOAD=$(
printf '{"status":"pass","datastore":"%s","namespace":"%s","backup_set":"%s","snapshot":"%s","file":"%s","size":%s,"hash":"%s"}' \
"$DATASTORE" \
"$NAMESPACE" \
"$BACKUP_SET" \
"$SNAPSHOT" \
"$TEST_FILE" \
"$SIZE" \
"$ACTUAL_HASH"
)
curl -fsS -X POST -H 'Content-Type: application/json' --data "$PAYLOAD" "$WEBHOOK_URL"
fi
cd /
if [ "$TARGET_DIR" != "/" ]; then
rm -rf "$TARGET_DIR"
fi
Make it executable:
chmod +x /usr/local/sbin/pbs-restore-test.sh
How to Schedule the Test Without Touching Production
Use a weekly window that is outside the normal backup window. The restore test should not compete with the backup job for the same S3 API path. A Friday night or Saturday morning run is usually enough.
Create an environment file for the values:
PBS_RESTORE_TEST_DATASTORE=s3
PBS_RESTORE_TEST_NAMESPACE=vm
PBS_RESTORE_TEST_BACKUP_SET=vm-100
PBS_RESTORE_TEST_TARGET_DIR=/var/tmp/pbs-restore-test
PBS_RESTORE_TEST_STATE_DIR=/var/lib/pbs-restore-test
PBS_RESTORE_TEST_LOG_FILE=/var/log/pbs-restore-test.log
PBS_RESTORE_TEST_WEBHOOK_URL=
Run the cron entry as root:
crontab -e
Add this line:
0 3 * * 6 root /usr/local/sbin/pbs-restore-test.sh
The script cleans the scratch directory after each run. That is intentional. A restore test should not leave large files sitting on the PBS node unless you want to inspect them.
What Should the Script Check Before It Passes?
A pass should mean more than “the command exited successfully.” The script should check these conditions:
- The datastore name is reachable.
- The namespace and backup set exist.
- A snapshot can be selected.
- The restore command writes at least one file.
- The restored file is not empty.
- The hash matches the stored manifest for the same snapshot.
- A log line is written.
- An alert is sent when configured.
A useful log line looks like this:
2026-01-10T03:00:08Z restore-test pass datastore=s3 namespace=vm backup-set=vm-100 snapshot=2026-01-09T02:15:03 file=/var/tmp/pbs-restore-test/vm.eml size=1048576 hash=8b5f6d2d4c1f4f3a1f2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f
If you want to send the result to a webhook, the JSON payload can be simple:
{
"status": "pass",
"datastore": "s3",
"namespace": "vm",
"backup_set": "vm-100",
"snapshot": "2026-01-09T02:15:03",
"file": "/var/tmp/pbs-restore-test/vm.eml",
"size": 1048576,
"hash": "8b5f6d2d4c1f4f3a1f2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f"
}
The hash comparison is the part that turns a restore command into a proof. If the same snapshot restores to the same file with the same hash, you have a repeatable result. If the hash changes, the script fails and you investigate.
How to Turn the Test Into Production Evidence
The restore test should be tied to the backup path, not to a lucky manual restore. If the S3 datastore itself is not yet proven, start with the PBS 4.2 S3 backup setup before automating restore checks. The manual restore test guide is the right place to understand what a single restore proves, and the backup pre-flight checklist helps you avoid scheduling a test against a half-configured datastore.
For production, keep two layers:
- A weekly small restore test that proves readability and hash stability.
- A monthly full restore to a spare node that proves bootability.
Do not confuse those two. A weekly file restore is cheap and can run unattended. A monthly full restore is heavier, slower, and often needs a spare node. A small LXC on PBS 4.2 S3 can be a good weekly target because it is usually smaller than a VM. The OCI LXC backup guide is useful when the test target is a container rather than a full VM.
The script should fail loudly. A silent pass is not proof. A cron job that exits zero but writes no log line is not proof either. The article’s value is the evidence trail: datastore, namespace, backup set, snapshot, file size, hash, timestamp, and alert status.
Conclusion
A scripted PBS 4.2 S3 restore test proves that the backup path is usable before production depends on it. The next step is to run the script once against a known backup set, store the first hash, then schedule the weekly cron job so the result becomes repeatable. Keep the small weekly test, and add a monthly full restore to a spare node when bootability matters.
