LocalAI on a VDS: an OpenAI-compatible server

What is LocalAI?

LocalAI is an open-source REST API that's fully compatible with the OpenAI API but runs on local models. It supports GGUF and GPTQ models, image generation (Stable Diffusion), speech synthesis (TTS), and speech recognition (Whisper).

Advantages over Ollama

• Full OpenAI API compatibility (drop-in replacement without code changes)
• Image generation support
• Built-in TTS and STT
• More model formats

Installing via Docker

>docker run -d \
  --name localai \
  -p 8080:8080 \
  -v localai_models:/build/models \
  --restart always \
  localai/localai:latest-aio-cpu

Downloading a model

# Via the API
curl http://localhost:8080/models/apply -H "Content-Type: application/json" \
  -d '{"id": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf"}'

Using it as an OpenAI replacement

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="mistral-7b-instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Common setup errors

The model takes forever to load and the container gets killed on memory

Killed
exit code 137

Code 137 is the OOM Killer signal: the process ran out of RAM. LocalAI keeps the model itself and its context work buffers in RAM, so a GGUF model needs headroom noticeably larger than the model file's size on disk. If the server hits OOM, pick a more aggressively quantized model (say, Q4 instead of Q8) or a VDS with more memory.

The API returns 404 for a model name you know is right

{"error":{"message":"could not load model - all backends returned error","code":404}}

LocalAI matches a model name against a config file in the models directory. If a model was added manually without a YAML descriptor, or the name in the request does not match the file name, the backend cannot find a match. Check the model list via GET /v1/models and use the exact name from there.

Whisper and image-generation modes fail to start on a weak CPU

Error: failed to load model: context deadline exceeded

The AIO image bundles several heavy modes at once, and their initial startup needs noticeably more resources than a plain text chat. On a weak VDS without a GPU, loading may simply not fit inside the default timeout — check the logs for CPU and memory pressure rather than treating it as a hang. Running these modes reliably usually needs either a more powerful CPU with more cores or a CUDA-capable GPU.

LocalAI, Ollama, and vLLM: which to choose

All three give you local inference, but they target different scenarios — from a single terminal command to a production GPU cluster.

When LocalAI is not a good fit

LocalAI's versatility comes at a price: more moving parts than a tool built for a single narrow job.

• You need maximum GPU inference throughput under many concurrent users — vLLM's request batching handles that better
• Simplicity matters most: one command, one model, minimal configuration — it is easier to start with Ollama
• The server has no GPU at all and limited RAM — heavy modes (Stable Diffusion, Whisper) will be extremely slow or fail to start under those conditions
• You need a rare model format that is not in LocalAI's registry — a dedicated inference server for that specific format is simpler
• The team values minimal configuration over feature breadth — the AIO image is noticeably heavier on resources than a single-purpose server

Verifying the result

Before switching production code to LocalAI, check both API compatibility and real-world performance.

1. Request the list of loaded models and confirm the one you need is there
2. Send a test chat completions request and check that the response comes back without errors
3. Measure response time on a typical prompt — if it is unacceptably high, check whether inference is running without the expected GPU acceleration
4. Check the container logs for memory-pressure warnings
5. If you need an image or speech mode, test that specific mode separately — the basic text chat can work fine while a heavy mode fails

curl http://localhost:8080/v1/models

Related: a basic local model server — Ollama on a VDS, working out the VRAM you need — how much VRAM a model needs, model quantization — llama.cpp and GGUF.

Frequently asked questions

How much RAM does a model with several billion parameters in GGUF actually need?

The rule of thumb: RAM headroom should sit noticeably above the model file's size on disk, because LocalAI keeps the model itself plus the request's context buffers in memory. If the server hits an out-of-memory kill, pick a more aggressive quantization such as Q4 instead of Q8, or move to a VDS with more memory available.

Why can't LocalAI see a model that was uploaded to the server manually over SFTP?

Matching a model name to its file happens through a YAML descriptor in the models directory, not simply by the file being present. Without that descriptor, or when the name in a request doesn't match the file name, the backend returns an error. Check the exact name through GET /v1/models and use it unchanged in requests.

Does LocalAI need a GPU, or is a plain VDS without a graphics card enough?

For basic text chat, the CPU build of the AIO image is usually enough, especially with a small quantized model. Image generation and speech recognition modes are noticeably heavier on resources though: on a weak CPU without CUDA, their initial startup can easily hit the timeout. Running those modes reliably all but requires a GPU.

Is LocalAI better or worse than Ollama for a small project running a single model?

Ollama is quicker to launch and configure when all that's needed is plain text chat with no extra features. LocalAI adds drop-in OpenAI API compatibility plus image generation and speech recognition, but asks for more resources and configuration. For one simple job that doesn't need those extras, Ollama is usually the more convenient pick.