Consul: Service Discovery and Distributed Config

When the number of services across your servers grows past a dozen, /etc/hosts and hardcoded IP addresses in config files stop working: addresses change every time a VPS is recreated, and updating them by hand on every server is a guaranteed mistake sooner or later. Consul from HashiCorp solves this with service discovery: services register themselves, and everyone else finds them by name through DNS or an HTTP API, with no single file of addresses.

What Consul does and when you need it

Consul stores three things: a catalog of services with their addresses and ports, the health check status of every instance, and a distributed key-value store for configuration. An application does not ask "what is the IP of the billing service" — it asks "give me a healthy address for the billing service" — and only gets the instances that pass their health check.

Consul is worth deploying once you have more than 5-6 servers that keep changing: replicas are added, VPS instances get recreated, IPs change after migrations. On two permanent servers with one service each, Consul is unnecessary complexity — a DNS record or a config line is enough.

Architecture: server, agent, service catalog

Consul server and quorum

Server nodes store cluster state through the Raft protocol and agree on quorum. Fault tolerance requires an odd number of servers: 3 servers survive losing one, 5 survive losing two. A single server is only good for testing — in production it is a single point of failure for the whole cluster.

Consul agent on every node

An agent in client mode runs on every other server — it does not store cluster state, only registers local services and polls their health checks, passing data to the server nodes through the gossip protocol.

Installing and starting a Consul cluster

The consul package installs from HashiCorp's official repository, separately for server nodes and for agents.

curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
apt update && apt install -y consul
consul version

A minimal server node config sets server mode, the number of servers in the quorum, and the address for the gossip protocol between nodes.

{
  "server": true,
  "bootstrap_expect": 3,
  "datacenter": "dc1",
  "data_dir": "/var/lib/consul",
  "bind_addr": "10.10.0.1",
  "retry_join": ["10.10.0.1", "10.10.0.2", "10.10.0.3"]
}

Registering a service and health checks

A service is registered with a definition file in the /etc/consul.d directory — the agent picks it up on startup or on the consul reload command.

{
  "service": {
    "name": "billing-api",
    "port": 8080,
    "check": {
      "http": "http://localhost:8080/health",
      "interval": "10s",
      "timeout": "2s"
    }
  }
}

If the health check fails three times in a row, Consul marks the instance as critical and stops returning its address in DNS and API responses — a load balancer or a neighboring service finds out about the outage within 30 seconds, instead of noticing a rise in 5xx errors from clients.

Service discovery through DNS and distributed config through KV

Any node with a running agent resolves the name billing-api.service.consul through the built-in DNS interface on port 8600 — it is enough to set Consul as the resolver for the .consul domain in systemd-resolved or dnsmasq. The consul kv put/get key-value store replaces a separate config server: feature flags and parameters like the database connection limit are read by the application at startup and updated without a deploy.

Consul is usually deployed on top of an existing private network — for example, a WireGuard mesh between nodes, so gossip traffic and the API never touch the public internet. Together with Nomad, Consul covers both service discovery and workload scheduling across the cluster with the same set of agents.

Checklist before production

• An odd number of server nodes (3 or 5), gossip and RPC ports closed off from the public internet.
• ACLs enabled at least in minimal mode — Consul without ACLs trusts anyone who can reach the API by default.
• A health check configured for every service, not just registration — without it a dead instance keeps receiving traffic.
• Scheduled backups with consul snapshot save — cluster state cannot be rebuilt out of nowhere.