Consul: Service Discovery and Distributed Config

When the number of services across your servers grows past a dozen, /etc/hosts and hardcoded IP addresses in config files stop working: addresses change every time a VPS is recreated, and updating them by hand on every server is a guaranteed mistake sooner or later. Consul from HashiCorp solves this with service discovery: services register themselves, and everyone else finds them by name through DNS or an HTTP API, with no single file of addresses.

What Consul does and when you need it

Consul stores three things: a catalog of services with their addresses and ports, the health check status of every instance, and a distributed key-value store for configuration. An application does not ask "what is the IP of the billing service" — it asks "give me a healthy address for the billing service" — and only gets the instances that pass their health check.

Consul is worth deploying once you have more than 5-6 servers that keep changing: replicas are added, VPS instances get recreated, IPs change after migrations. On two permanent servers with one service each, Consul is unnecessary complexity — a DNS record or a config line is enough.

Architecture: server, agent, service catalog

Consul server and quorum

Server nodes store cluster state through the Raft protocol and agree on quorum. Fault tolerance requires an odd number of servers: 3 servers survive losing one, 5 survive losing two. A single server is only good for testing — in production it is a single point of failure for the whole cluster.

Consul agent on every node

An agent in client mode runs on every other server — it does not store cluster state, only registers local services and polls their health checks, passing data to the server nodes through the gossip protocol.

Installing and starting a Consul cluster

The consul package installs from HashiCorp's official repository, separately for server nodes and for agents.

curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
apt update && apt install -y consul
consul version

A minimal server node config sets server mode, the number of servers in the quorum, and the address for the gossip protocol between nodes.

{
  "server": true,
  "bootstrap_expect": 3,
  "datacenter": "dc1",
  "data_dir": "/var/lib/consul",
  "bind_addr": "10.10.0.1",
  "retry_join": ["10.10.0.1", "10.10.0.2", "10.10.0.3"]
}

Registering a service and health checks

A service is registered with a definition file in the /etc/consul.d directory — the agent picks it up on startup or on the consul reload command.

{
  "service": {
    "name": "billing-api",
    "port": 8080,
    "check": {
      "http": "http://localhost:8080/health",
      "interval": "10s",
      "timeout": "2s"
    }
  }
}

If the health check fails three times in a row, Consul marks the instance as critical and stops returning its address in DNS and API responses — a load balancer or a neighboring service finds out about the outage within 30 seconds, instead of noticing a rise in 5xx errors from clients.

Service discovery through DNS and distributed config through KV

Any node with a running agent resolves the name billing-api.service.consul through the built-in DNS interface on port 8600 — it is enough to set Consul as the resolver for the .consul domain in systemd-resolved or dnsmasq. The consul kv put/get key-value store replaces a separate config server: feature flags and parameters like the database connection limit are read by the application at startup and updated without a deploy.

Consul is usually deployed on top of an existing private network — for example, a WireGuard mesh between nodes, so gossip traffic and the API never touch the public internet. Together with Nomad, Consul covers both service discovery and workload scheduling across the cluster with the same set of agents.

Checklist before production

• An odd number of server nodes (3 or 5), gossip and RPC ports closed off from the public internet.
• ACLs enabled at least in minimal mode — Consul without ACLs trusts anyone who can reach the API by default.
• A health check configured for every service, not just registration — without it a dead instance keeps receiving traffic.
• Scheduled backups with consul snapshot save — cluster state cannot be rebuilt out of nowhere.

Frequently asked questions

Why must the number of Consul server nodes be odd rather than even?

Raft needs a strict majority to agree on cluster state, and an even number of servers can split into two equal groups during a network split, leaving neither side with a majority to operate safely. An odd count always produces a clear majority on one side, so the cluster keeps functioning and stays consistent even after losing a node.

Does Consul replace a load balancer, or is a separate proxy still needed in front of services?

No, Consul tracks what is healthy and where it lives, it does not route or balance traffic itself. A reverse proxy, API gateway, or client library still has to read that catalog and decide which instance handles a given request; Consul supplies the up-to-date list, the actual routing decision stays outside it.

After a service starts failing its health check, how quickly does Consul stop sending traffic to it?

Once a check reports failure repeatedly rather than just once, Consul marks that instance critical and removes its address from DNS and API answers right away. Anything relying on a fresh lookup — a proxy, a balancer, another service — simply stops being handed that instance, so the switch happens automatically, without a human noticing the outage through rising errors.

Is a private network like a VPN mesh mandatory for running Consul, or can the agents talk over the public internet directly?

It is not mandatory, but doing it over the open internet is risky unless access lists and encryption are configured carefully, since gossip and API traffic were never meant to face the public internet. Running agents over an already private network, such as a mesh VPN between nodes, is the standard safe approach and keeps that traffic off the internet.