Monitoring AI Agents on a VPS: Langfuse and Metrics

Why do you need AI monitoring?

AI agents in production require monitoring: how many tokens are spent, how long the model takes to respond, which requests cause errors, how accurate the responses are. Without monitoring, you do not learn about problems until users complain.

Langfuse — open-source AI observability

>docker run -d --name langfuse \
  -p 3000:3000 \
  -e DATABASE_URL=postgresql://postgres:password@host:5432/langfuse \
  -e NEXTAUTH_SECRET=random-secret \
  --restart always \
  langfuse/langfuse:latest

Integration with LangChain

from langfuse.callback import CallbackHandler
from langchain_ollama import OllamaLLM

handler = CallbackHandler(
    public_key="pk-...",
    secret_key="sk-...",
    host="http://localhost:3000"
)
llm = OllamaLLM(model="llama3.2", callbacks=[handler])
result = llm.invoke("Test request")

Key metrics for AI agents

System metrics via Prometheus

# prometheus.yml
scrape_configs:
  - job_name: "ollama"
    static_configs:
      - targets: ["localhost:11434"]
    metrics_path: "/metrics"

Common setup errors

Langfuse fails to start because of the database

Error: P1001: Can't reach database server at `host`:5432

Langfuse expects a separately running PostgreSQL instance and does not spin one up inside its own image. Check that DATABASE_URL points to a genuinely reachable host and port, and that the database itself already exists — the container only applies migrations, it does not create a database server.

Traces from LangChain never show up in the interface

A common cause is that the Langfuse callback handler was created in code but never actually passed into the model call. Make sure the handler object is explicitly passed in the callbacks parameter on every chain invocation, not just when the LLM is initialized.

The Prometheus dashboard shows empty graphs for Ollama

up{job="ollama"} == 0

The up metric is zero when Prometheus cannot reach the scrape endpoint. Check that Ollama's port is reachable from the machine running Prometheus, and that the /metrics path actually returns data rather than 404 — some Ollama versions require metrics export to be enabled separately.

Langfuse builds broken links behind a reverse proxy

Warning: NEXTAUTH_URL is not set, using fallback

Without an explicit NEXTAUTH_URL pointing at the real public address behind Nginx, the service can generate redirect links to the container's internal address. Set this environment variable to the external HTTPS domain before the first run. The easiest check is opening the login page through the public address and confirming that after logging in the browser is not redirected to the container's internal port.

Options for AI observability

The right tool depends on what you actually need to observe: the content of LLM calls, system-level metrics, or both at once.

When this level of monitoring is overkill

A full observability stack is not justified at every stage of a project — sometimes it just adds infrastructure without adding value.

• A prototype with occasional manual requests — tracing every call gives no useful data at that volume
• A one-developer team with no external users — plain logging to a file is enough
• The VDS budget cannot support another service with its own database — Langfuse needs a separate PostgreSQL and a noticeable amount of RAM
• You only need an uptime check, not a look inside response content — a simple healthcheck covers that
• All LLM calls are already logged through the application's existing logging system — duplicating it with a second service is not always worth it

Verifying the result

After setup, confirm that data actually flows all the way from a model call to the dashboard.

1. Open the Langfuse interface and confirm a test model call shows up in the trace list
2. Check that the trace shows the prompt, the response, and the execution time
3. Open /metrics on Ollama directly and confirm the endpoint returns data
4. Check in Prometheus that the ollama target is in the UP state
5. Open Langfuse through the public domain and confirm redirects do not lead to the container's internal address

curl http://localhost:11434/metrics | head

Related: the agents worth monitoring — what an AI agent is, a framework with ready-made callbacks — LangChain on a VDS.