Installing Ollama on a VDS: Running Local LLMs

What is Ollama?

Ollama is an open-source tool for running large language models locally. It automatically manages model downloads, provides a REST API compatible with OpenAI, and runs on Linux, macOS, and Windows.

System Requirements

Installing Ollama

# One-command install
curl -fsSL https://ollama.com/install.sh | sh

# Check status
systemctl status ollama
systemctl enable ollama

Downloading and Running a Model

# Llama 3.2 (2 GB)
ollama run llama3.2

# Mistral 7B (4.1 GB)
ollama run mistral

# Phi-3 Mini for low-power VDS (1.7 GB)
ollama run phi3:mini

# List of models
ollama list

Configuring API Access from Outside

# Edit the service
systemctl edit ollama

# Add to [Service]:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Testing via the API

curl http://localhost:11434/api/generate \
  -d '{"model":"llama3.2","prompt":"Hello!","stream":false}'

Common installation errors

«model requires more system memory than is available»

Error: model requires more system memory (5.2 GiB) than is available (3.8 GiB)

Ollama estimates the memory needed for the model weights and context, and refuses to start the model up front rather than crashing mid-generation. The fix is to pick a more lightly quantized model (say, the q4_K_M variant instead of q8_0) or a model with fewer parameters.

Port 11434 is unreachable from outside after changing OLLAMA_HOST

curl: (7) Failed to connect to server.example.com port 11434: Connection refused

If the service did not reload its configuration after systemctl edit ollama, the environment variable never takes effect. Be sure to run systemctl daemon-reload and systemctl restart ollama, and check that the firewall (ufw, iptables) allows the port through.

«no such model» right after pulling a tag

Error: pull model manifest: file does not exist

The model name has a typo, or the version tag does not exist in the Ollama library. Check the list of available tags on the model's page and copy the name together with its quantization tag exactly.

«could not connect to ollama app, is it running?»

Error: could not connect to ollama app, is it running?

The CLI talks to the service over a local socket and does not start it itself. This error is typical after a server reboot if the service was never enabled at boot: check its state with systemctl status ollama and, if it is inactive, run systemctl enable --now ollama.

Comparing ways to run local LLMs

The right tool depends on what matters most: a fast start, full OpenAI API compatibility, or control over the build for specific hardware.

When Ollama is not a good fit

Ollama deliberately trades some flexibility for simplicity, and in a number of scenarios that simplicity starts to get in the way.

• You need maximum throughput with many concurrent users on a GPU — dedicated inference servers like vLLM do better here through request batching
• You need fine-grained control over quantization and build parameters for specific hardware — that means working with llama.cpp directly
• The task involves image generation or speech synthesis — Ollama works with text-only and partly multimodal LLMs only
• The VDS has no RAM headroom even for the lightest model — an external API is the more sensible choice over local inference
• You need full control over the model file's format and metadata — Ollama hides those details behind its own Modelfile format

Verifying the result

After installation, confirm not just that the service is running, but that it actually responds to requests.

1. Confirm the service is running: systemctl status ollama should show active (running)
2. Check the list of loaded models: ollama list
3. Send a test request via the API from the local machine and, if you opened the port externally, from an outside host too
4. Measure the response time on a short prompt to gauge real speed on this VDS
5. Check that the service survives a server reboot: is autostart enabled via systemctl enable

Related: a friendly chat interface on top of Ollama — Open WebUI, working out the VRAM a model needs — how much VRAM a model needs, running without a GPU via quantization — llama.cpp and GGUF.