Skip to main content

Migrating data to S3 / MinIO Object Storage

Migration · 05.02.2025 · 4 min read
Illustration for “Migrating data to S3 / MinIO Object Storage”

The main mistake when moving to S3 or MinIO is switching the application to the new storage first and only then copying files. Some visitors get 404s on images and attachments in the meantime. The correct order is the reverse.

  1. Copy all existing files into the bucket while the application keeps writing to local disk or the old storage.
  2. Configure the plugin or application code for the new storage in test mode, without disabling the old one.
  3. Check that new files land in both the old location and the bucket at the same time.
  4. Run a final delta sync for files added during the migration.
  5. Switch the application to read only from the bucket and keep the old storage as a fallback for a few days.

Honestly about downtime: the file transfer itself runs without stopping the site, and a downtime window is only needed for the config switch and the final sync — usually 1-10 minutes depending on how many files changed, not the hours a full upload takes.

rclone: a universal tool for S3

# Install rclone
curl https://rclone.org/install.sh | bash

# Configure the S3 profile
rclone config
# Choose: n (new remote) → s3 → AWS/MinIO → enter your keys

Sample rclone configuration for MinIO

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
endpoint = https://minio.example.com
location_constraint =
server_side_encryption =

Syncing data

# Transfer a local folder to the bucket
rclone sync /var/www/uploads/ minio:mybucket/uploads/ --progress

# Sync between two S3-compatible storages
rclone sync s3:source-bucket minio:dest-bucket --progress

# Dry run without real changes
rclone sync /data minio:backup --dry-run
💡 Use rclone mount to mount an S3 bucket as a local drive: rclone mount minio:mybucket /mnt/s3 --daemon

Final sync before switching over

After the first full upload, the site keeps running and creating new files. Before switching, run rclone sync once more — it copies only what changed, shrinking the downtime window to a couple of minutes instead of the hours a full upload takes.

WordPress + S3 Offload Media

PluginProvidersCost
WP Offload Media LiteAWS S3, DigitalOcean, LinodeFree
WP Offload Media ProAll S3-compatible$99/year
Media CloudS3, MinIO, BackblazeFree + Pro

Checking before switching the application

Check that files are reachable straight from the bucket, bypassing the application code:

curl -I https://minio.example.com/mybucket/uploads/photo.jpg

A 200 OK response with the correct Content-Type means the file is genuinely reachable. If the bucket is private, check instead with a signed URL generated by the application.

Comparing bucket access schemes

SchemePublic accessWhen to use
Public bucketyes, via a direct linkimages, static site assets
Private bucket + signed URLsonly via a temporary linkdocuments, user files
Private bucket + CDN proxythrough a separate CDN domainhigh load, needs an edge cache

Common errors

ErrorCause and fix
SignatureDoesNotMatchWrong access/secret key, or clock drift on the server — sync the clock with NTP
403 Forbidden when reading a fileThe bucket or object has no public-read policy — check the bucket policy
NoSuchBucketThe bucket was created in a different region or with a typo in the name — check the endpoint and bucket name
Connection timeout during rclone syncOutbound port 443 is blocked on the server's firewall — allow outbound connections to the storage endpoint

Rollback in case of problems

  1. Do not delete local file copies right after the migration — keep them for at least a week.
  2. If file links stop opening en masse, point the application code back at local storage — the data there was never removed.
  3. Fix the access rights or bucket configuration and repeat the switch.

When this does not fit

For sites where users write files in real time (for example, uploading video while watching), a full sync does not work without a short write-lock window — otherwise files added during the final sync are lost. Such cases need an upload queue that writes straight to the bucket, bypassing local disk.

💡 After moving media files to S3, don't delete the local copies right away — make sure all URLs updated correctly. The general scheme for testing a site before a DNS change is described in syncing servers with rsync, and moving the site's own files is covered in file migration with rsync and scp.

Was this article helpful?
← Back to Knowledge Base Ask Support