Deduplication and Encryption in Proxmox Backup Server

Proxmox Backup Server uses an advanced block-level deduplication system (4 MB chunks), which lets you store dozens of backups while taking up barely more space than a single one. This fundamentally sets PBS apart from traditional backup solutions.

How PBS Deduplication Works

The PBS backup process:

The client reads the VM data and splits it into 4 MB blocks
It computes the SHA-256 hash of each block
It sends PBS only the blocks that are not yet in the storage
PBS stores each unique block only once

A real-world example of space savings:

Scenario	Without deduplication	With PBS deduplication
10 identical VMs (50 GB each)	500 GB	~60 GB
30 daily backups of one VM	1500 GB	~80 GB (incremental)
10 clones from a single template	500 GB	~55 GB

Viewing Deduplication Statistics

# Via CLI
proxmox-backup-manager datastore show backup-store

# Example output:
# Total Size: 2.5 TiB
# Used: 450 GiB
# Dedup Factor: 5.8x   <- 5.8x savings
# Disk I/O Stats: ...

# Via API
proxmox-backup-client snapshot list --repository pbs:backup-store

Configuring Backup Encryption

PBS supports end-to-end AES-256 GCM encryption. Encryption happens on the client side, and PBS stores only encrypted blocks:

# Create an encryption key
proxmox-backup-client key create /etc/pve/priv/backup.key

# View the key fingerprint
proxmox-backup-client key show /etc/pve/priv/backup.key

# Register the key in the storage configuration
# Storage → my-pbs → Edit → Encryption Key → select the key file

Exporting and Backing Up the Encryption Key

# Export the key to a protected format (password-protected)
proxmox-backup-client key export-config   /etc/pve/priv/backup.key   /tmp/backup-key-export.json

# Print the key as a QR code for physical storage
proxmox-backup-client key paperkey /etc/pve/priv/backup.key

Key backup rule: Store the encryption key in at least three places: on PBS itself (in /etc/pve/priv/), in a protected password vault (Bitwarden, 1Password), and physically (print the paperkey QR code). Losing the key means losing all backups.

Garbage Collection

To free up space from deleted backups, run garbage collection:

# Run it immediately
proxmox-backup-manager garbage-collect backup-store

# Automatically — via the web interface
# PBS → Datastore → backup-store → GC Schedule → set up a cron schedule

Checking Backup Integrity

# Verify a specific backup
proxmox-backup-client verify pbs:backup-store:vm/100/...

# Run a full datastore verification
proxmox-backup-manager verify backup-store --verify-outdated 1

# Set up a verification schedule
# PBS → Datastore → Verify Schedule

Deduplication and encryption errors

Three errors you may hit when verifying or restoring encrypted, deduplicated backups.

verify backup-store:vm/100/2025-03-15T01:00:00Z failed: chunk 3a5f9c... missing
TASK ERROR: unable to load encryption key: wrong password or corrupted key file
TASK ERROR: garbage collection: not enough space to complete phase1

chunk ... missing: a block in the chunk store is corrupted or was deleted manually — because blocks are shared, this can affect several backups at once. Restore the datastore from a backup of the PBS disk; never delete files from the chunk store directly.
wrong password or corrupted key file: the wrong encryption key file was specified, or the key got corrupted during a copy. Compare the key's fingerprint with proxmox-backup-client key show against the one used when the backup was created.
not enough space to complete phase1: garbage collection needs temporary free space to rebuild its index — if the datastore is completely full, GC cannot finish. Free up space by manually removing old backups before rerunning GC.

When the space savings are lower than expected

PBS deduplication is not equally effective for every kind of data.

Data that is already compressed or encrypted inside the VM (video, archives, encrypted disks) barely deduplicates — such blocks look like random data, so the savings ratio is close to 1.
VMs with heavy rewriting of large files across their entire length (busy databases) — each backup creates many new unique blocks instead of reusing old ones.
Client-side encryption only improves savings between backups sharing the same key — if different VMs use different keys, identical content encrypts differently and the blocks no longer match.

Verifying the result

# Deduplication ratio and space used
proxmox-backup-manager datastore show backup-store

# Full verification of a specific backup
proxmox-backup-client verify pbs:backup-store:vm/100/2025-03-15T01:00:00Z

# Confirm the encryption key matches the backup
proxmox-backup-client key show /etc/pve/priv/backup.key

If verify completes without errors and the key's fingerprint matches what is expected, the backup is intact and restorable. For installing and structuring PBS, see Proxmox Backup Server: installation.

Frequently asked questions

What happens if the PBS backup encryption key is lost?

The encryption key exists only on the client side — Proxmox Backup Server has no way to recover it. Without the key, encrypted blocks look like random data, and restoring a virtual machine from such a backup becomes impossible. Keep the key in several independent places: on the server, in a password manager, and printed on paper.

Why doesn't identical data across different VMs save space through deduplication?

When separate VMs use their own encryption keys, identical content turns into different byte sequences and no longer matches between backups. PBS compares blocks only after encryption, so real space savings appear only among backups encrypted with the same key. Sharing one key across a group of similar VMs restores the deduplication effect you would expect.

How often should Garbage Collection be run manually?

Running Garbage Collection manually usually isn't necessary — it's easier to set up a schedule directly on the datastore so the system frees space from deleted backups on its own. A manual run comes in handy before checking free space, or right after removing a batch of old backups, when waiting for the next scheduled pass would take too long.

Is it safe to free up space by deleting backup files directly from storage?

No, deleting files from the chunk store directly is risky — the same block can be shared by several backups because of deduplication, and removing a file can damage other snapshots. The right approach is to delete unwanted backups through the PBS client or web interface and then run Garbage Collection, which safely clears out blocks that nothing references anymore.