Ollama: Local LLMs on a VPS Without API Keys

AI Agents with Ollama: Local LLMs Without API Keys

Ollama lets you run large language models (LLaMA 3, Mistral, Gemma, Qwen) directly on your VPS without external APIs. Data never leaves the server, there are no token fees, and you get full control over the models.

Installing Ollama

curl -fsSL https://ollama.com/install.sh | sh
systemctl enable ollama
systemctl start ollama

# Download and run a model
ollama pull llama3.2:3b       # 2 GB — for servers with 4 GB RAM
ollama pull llama3.1:8b       # 5 GB — for 8 GB RAM
ollama pull mistral:7b        # 4 GB — good quality
ollama pull qwen2.5-coder:7b  # specializes in code

# Start a chat
ollama run llama3.2:3b

REST API

# Chat (OpenAI-compatible format)
curl http://localhost:11434/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "llama3.2:3b",
    "messages": [{"role": "user", "content": "Write a bash script for MySQL backups"}]
  }'

# Generate (native API)
curl http://localhost:11434/api/generate   -d '{"model": "mistral", "prompt": "Write a haiku about servers", "stream": false}'

Python Integration

pip install ollama

import ollama

response = ollama.chat(
    model='llama3.2:3b',
    messages=[
        {'role': 'system', 'content': 'You are a Linux administration expert.'},
        {'role': 'user', 'content': 'How do I optimize nginx for high load?'}
    ]
)
print(response['message']['content'])

# Streaming
for chunk in ollama.chat(model='llama3.2:3b',
    messages=[{'role': 'user', 'content': 'Tell me about Docker'}], stream=True):
    print(chunk['message']['content'], end='', flush=True)

Exposing Ollama for External Access

# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

systemctl daemon-reload && systemctl restart ollama

Model Selection Recommendations

Common Errors

Problems with Ollama on a VPS usually come from a port already in use or from not enough memory for a specific model — both are easy to mistake for "the model just does not work."

Error: listen tcp 127.0.0.1:11434: bind: address already in use

Port 11434 is already taken by another Ollama process — for example, the service is running both as a systemd unit and manually via ollama serve in a separate terminal. Check which process holds the port with ss -tlnp | grep 11434 and keep only one running instance.

Error: model requires more system memory than is available

The model does not fit in the server's available RAM. Options: pick a more compact quantization of the same model, reduce the number of models loaded at once with the OLLAMA_MAX_LOADED_MODELS variable, or move to a VPS with more RAM.

Quantization Levels: What the Letters in a Model Name Mean

When Ollama Is Not a Good Fit

• You need the best possible answer quality for hard tasks — large proprietary models via API are usually more accurate than anything that fits on a typical VPS.
• Load is unstable and spiky — keeping a server with RAM permanently reserved for a model, just for occasional bursts, can cost more than paying per token with an external provider, see the AI API comparison.
• You need a specific capability like image generation or high-quality speech recognition — specialized models such as Whisper for transcription fit those tasks better.
• The team is not ready to tune quantization for specific hardware — a managed API removes that work entirely.

Verifying the Model Responds Correctly

curl -s http://localhost:11434/api/generate -d '{"model": "llama3.2:3b", "prompt": "2+2=", "stream": false}'
ollama ps

The first command should return JSON with a response field containing coherent text. The second shows which models are currently loaded in memory and how much space that takes — useful if the server has become slower than usual and you need to check whether Ollama is keeping an unused model loaded.

It is also worth considering resource contention if other services run on the same server. By default, Ollama uses all available CPU cores for inference on the processor, and without limits set through cgroups or environment variables, generating a response can noticeably slow down other processes on the VDS, including the web server and the database.

Frequently asked questions

Do you need a GPU to run Ollama on a VPS?

No, Ollama runs fine on a plain CPU — the model loads into system memory and generation happens on the processor, just slower than with a GPU. A graphics card speeds up inference but is not required: when picking a model for a GPU-less VPS, the main constraint is available RAM, not whether a video accelerator is present.

How can you tell which models are currently loaded into memory on an Ollama server?

A command that lists loaded models shows the active inference processes and how much memory each one currently occupies. That's useful once a server starts answering slower than usual: it lets you see whether Ollama is keeping an unused model sitting in memory and unload just that one without restarting the whole service.

Do you have to expose the Ollama port externally to reach it from another computer?

Yes, by default the service only listens on the server's local address, and reaching it from outside needs the port opened via an environment variable and a restart. Doing so is unsafe, since an open API port lets anyone reach the model — add a reverse proxy with authentication, or restrict access by IP with a firewall.

How does a local model in Ollama fundamentally differ from using an AI API from a cloud provider?

The main difference is where data stays and what you pay for: a local model never sends requests outside and isn't billed per token, but its quality is capped by what fits in server memory. A cloud API is often more accurate on hard tasks, but it costs money per request and the data leaves the infrastructure.