Stable Diffusion on VPS: Image Generation on Your Server

Stable Diffusion on a GPU VPS

Stable Diffusion is an open-source image generation model. For comfortable work you need a GPU node with at least 6 GB of NVIDIA VRAM. CPU generation works, but it takes 5-15 minutes per image.

Server Requirements

• GPU VPS: NVIDIA RTX 3060/3080 (8+ GB VRAM) or RunPod/Vast.ai
• RAM: 16 GB minimum, Disk: 50+ GB
• OS: Ubuntu 22.04

Installing AUTOMATIC1111

apt update && apt install -y wget git python3 python3-pip python3-venv libgl1 libglib2.0-0
git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui
cd stable-diffusion-webui
# Download the SDXL model
cd models/Stable-diffusion
wget https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors
cd ../..
./webui.sh --api --listen --port 7860

Systemd Service

[Unit]
Description=Stable Diffusion WebUI
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/opt/stable-diffusion-webui
ExecStart=/opt/stable-diffusion-webui/webui.sh --api --listen --port 7860
Restart=on-failure

[Install]
WantedBy=multi-user.target

API: Generating from Python

import requests, base64

def generate_image(prompt: str, steps: int = 20) -> bytes:
    payload = {
        "prompt": prompt,
        "negative_prompt": "blurry, low quality",
        "steps": steps,
        "width": 512,
        "height": 512,
        "cfg_scale": 7,
        "sampler_name": "DPM++ 2M Karras"
    }
    r = requests.post("http://localhost:7860/sdapi/v1/txt2img", json=payload)
    return base64.b64decode(r.json()["images"][0])

img = generate_image("professional server photo, dark background, 4k")
with open("output.png", "wb") as f:
    f.write(img)

Common Errors on Launch

Almost all problems with a local Stable Diffusion setup come down to insufficient VRAM or incorrectly installed CUDA drivers — the model itself is rarely the source of the failure.

CUDA out of memory. Tried to allocate ... GiB

This error means there was not enough VRAM for the chosen resolution, batch size, or specific model. Reduce width/height, add the --medvram or --lowvram flag when launching webui.sh, or close other processes using the GPU (check with nvidia-smi).

RuntimeError: Torch is not able to use GPU

The web interface starts but cannot see the graphics card. This is usually caused by a mismatch between the CUDA version PyTorch was built for and the NVIDIA driver version on the server. Check the output of nvidia-smi and confirm the driver version supports the CUDA toolkit version listed in the project's requirements.

AUTOMATIC1111, ComfyUI, and InvokeAI: Which to Choose

When Stable Diffusion on VPS Is Not a Good Fit

• You only need occasional generations now and then — renting hourly GPU capacity from a cloud provider is cheaper than a permanently rented server.
• You need guaranteed single-digit-second latency under high load — peak load needs a pool of GPUs, not a single card.
• Your team is not ready to maintain CUDA drivers and track PyTorch version compatibility — managed services take that burden off your plate.
• The budget does not cover a GPU server even with hourly rental — on CPU, generating one image takes minutes, which does not work for interactive scenarios.

Verifying That Generation Works

curl http://localhost:7860/sdapi/v1/options
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

The first command should return JSON with the current WebUI settings — a connection error means the service is not running or the port is closed. The second shows how much VRAM is actually in use: a value close to the total right after startup, with no requests made, usually means the model loaded incorrectly or is stuck from a previous generation.

It is also worth budgeting time for the first run: on the first request after WebUI starts, the model loads into memory from disk, which can take noticeably longer than the image generation itself. If the service restarts on a schedule or after a crash, the first user request after the restart will be slow — that is normal and not a sign of breakage, but it is worth warning interface users or triggering a warm-up request right after the service starts.