Model Provider Failover & Fallback Chains ✓ VERIFIED

2026-08-19 · 16 min read · Hermes · Docker · sun-port · Git · PostgreSQL · im-bot

Every AI agent you run — a Feishu support bot, a WeChat sales assistant, a cron job that publishes articles — is a thin shell around one thing: a call to a model provider. When that provider is healthy, everything works. When it isn't, everything stops. No amount of clever agent code survives a 503 from the one API you depend on. The fix is a fallback chain: an ordered list of providers the agent walks down, first to last, until one answers. This tutorial builds a production fallback chain around Hermes — with a circuit breaker that stops hammering dead providers, a PostgreSQL ledger that records every failover, a cron job that flips the active provider automatically, Docker healthchecks that restart wedged gateways, sun-port routing that drains degraded replicas, and an im-bot alert the moment a provider goes dark. All versioned in Git.

1. Why a Single Provider Is a Single Point of Failure

Think about what actually breaks when a model provider has an incident. It isn't just "responses are slow." The failure modes are broader than people expect:

• Hard outage — the endpoint returns 5xx for minutes to hours. Every turn fails immediately.
• Rate limiting — you exceeded a tier, or the provider is throttling everyone under load. Turns fail with 429.
• Silent degradation — responses still come back, but they're truncated, malformed JSON, or wrong. This is the worst one, because nothing throws an error you can catch.
• Billing lockout — a card expired or a prepaid balance hit zero at 3am. The API returns 402 or 401.

A single-provider agent has exactly one answer to any of these: be down. A fallback chain gives it a second, third, and fourth answer. The moment the primary fails, the agent retries on the next provider in line, and the user never sees a difference — or, at worst, sees a slightly different model name in the metadata.

2. What a Fallback Chain Actually Is

A fallback chain is an ordered list of provider/model pairs. It is not load balancing, and it is not random selection. When the agent needs a completion:

1. Try provider 1. If it succeeds, stop.
2. If it fails with a transient error (timeout, 429, 5xx), try provider 2.
3. Repeat down the list until one succeeds or the list is exhausted.

The order matters, and it should encode your priorities: cost, latency, and capability. A typical chain puts the cheapest capable model first and a reliable-but-expensive one last as the safety net:

1. deepseek-chat          # cheapest, primary
2. claude-sonnet-4-5      # mid-tier, stronger reasoning
3. gpt-4o-mini            # safety net, widely available

3. Prerequisites

$ docker --version
Docker version 27.3.1, build ce12230
$ git --version
git version 2.46.0
$ psql --version
psql (PostgreSQL) 16.4
$ hermes config get model.provider
deepseek

You need a working Hermes gateway (see the Ubuntu walkthrough), a running sun-port (see the reverse-proxy tutorial), and PostgreSQL (the Prisma backend tutorial or the Docker patterns cover bringing one up). The failover logic here also pairs with the observability pipeline we built earlier.

4. Configure the Fallback Chain in Hermes

Hermes reads its model config from ~/.hermes/config.yaml. A fallback chain is expressed as an ordered groups list under model, with the primary first and each fallback after it:

# ~/.hermes/config.yaml
model:
  default: deepseek-chat
  provider: deepseek
  groups:
    primary:
      - provider: deepseek
        model: deepseek-chat
        api_key_env: DEEPSEEK_API_KEY
      - provider: anthropic
        model: claude-sonnet-4-5
        api_key_env: ANTHROPIC_API_KEY
      - provider: openai
        model: gpt-4o-mini
        api_key_env: OPENAI_API_KEY

Each entry carries its own provider, model, and the environment variable that holds its API key — so keys never sit in the YAML, they live in .env. This is the same key-isolation discipline from our secrets tutorial.

$ hermes config get model.groups
[{'provider': 'deepseek', 'model': 'deepseek-chat', 'api_key_env': 'DEEPSEEK_API_KEY'},
 {'provider': 'anthropic', 'model': 'claude-sonnet-4-5', 'api_key_env': 'ANTHROPIC_API_KEY'},
 {'provider': 'openai', 'model': 'gpt-4o-mini', 'api_key_env': 'OPENAI_API_KEY'}]

5. Detect Failure — Distinguish Transient From Fatal

The first half of failover is knowing when to fall through. The agent must classify each error:

• Transient → retry, then fall through: timeout, 429 rate limit, 5xx server error, connection reset.
• Fatal → do NOT fall through: 401/403 bad credentials, 402 billing, invalid request body, missing API key. Retrying another provider with the same bad input will fail identically — and can trigger a rate limit on a provider that was fine.

Here's the classifier in practice, against a live endpoint:

$ curl -s -o /dev/null -w '%{http_code}\n' \
    -H "Authorization: Bearer $DEEPSEEK_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}' \
    https://api.deepseek.com/chat/completions
200

A 200 means the provider is up. The moment you see 429, 5xx, or a timeout, that provider should be marked suspect — and a circuit breaker takes over before the retries pile up.

6. Circuit Breaker — Stop Hammering a Dead Provider

Naive fallback has a nasty failure mode: if the primary is down, every request still hits it first, waits for a timeout, then falls through. Under load, that's thousands of wasted calls and a latency spike on every turn. A circuit breaker fixes it by tracking consecutive failures and, past a threshold, skipping the primary entirely for a cooldown window:

$ cat ~/.hermes/circuit_breaker.json
{
  "deepseek":  {"failures": 7, "opened_at": "2026-08-19T02:14:33Z", "cooldown_s": 120},
  "anthropic": {"failures": 0, "opened_at": null, "cooldown_s": 120},
  "openai":    {"failures": 0, "opened_at": null, "cooldown_s": 120}
}

The rules are simple and worth stating precisely:

• Failure threshold: 5 consecutive failures opens the circuit.
• Open state: requests skip this provider for cooldown_s seconds and go straight to the next one.
• Half-open probe: after the cooldown, one test request goes through. Success closes the circuit; failure reopens it with a longer cooldown.
• Only transient errors count. A 401 is fatal and must not trip the breaker — it's a config bug, not an outage.

This is why the classifier from section 5 matters: feed 401s into the breaker and you'll "fail over" away from a provider that's actually fine, straight into a second 401 on the fallback.

7. Record Every Failover in PostgreSQL

A failover that happens silently is a failover you can't debug later. Log every transition to a PostgreSQL table so you can answer "was DeepSeek down on Tuesday?" without guessing:

$ psql -U agent -d fleet -c "
CREATE TABLE IF NOT EXISTS provider_events (
  id           BIGSERIAL PRIMARY KEY,
  provider     TEXT NOT NULL,
  event        TEXT NOT NULL,          -- 'down', 'up', 'rate_limited', 'failover'
  target       TEXT,                   -- provider we fell through to
  http_status  INT,
  latency_ms   INT,
  occurred_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_provider_events_ts
  ON provider_events (occurred_at DESC);"
CREATE TABLE
CREATE INDEX

$ psql -U agent -d fleet -c "
INSERT INTO provider_events (provider, event, target, http_status)
VALUES ('deepseek', 'failover', 'anthropic', 503);"
INSERT 0 1

Now every circuit-open and every fall-through lands in provider_events, queryable by provider and time window:

$ psql -U agent -d fleet -c "
SELECT provider, event, target, http_status, occurred_at
FROM provider_events
WHERE occurred_at > now() - interval '24 hours'
ORDER BY occurred_at DESC LIMIT 5;"
 provider |  event   |  target   | http_status |        occurred_at
----------+----------+-----------+-------------+----------------------------
 deepseek | failover | anthropic |         503 | 2026-08-19 02:14:33+00
 deepseek | down     |           |         503 | 2026-08-19 02:14:30+00
 deepseek | up       |           |         200 | 2026-08-18 23:51:02+00

8. Automatic Failover With a Cron Job

The breaker degrades gracefully, but it's reactive — it only kicks in after a request fails. A cron job makes failover proactive: it probes each provider on a schedule, and the instant one is down, it flips the active provider and logs the event before a single user ever notices:

# /root/failover-check.sh — run every minute via cron
#!/bin/bash
set -euo pipefail

probe() {
  local name="$1" url="$2" model="$3"
  local var; var="$(echo "$name" | tr '[:lower:]' '[:upper:]')_API_KEY"
  local code
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 \
    -H "Authorization: Bearer ${!var}" \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
    "$url" 2>/dev/null || echo "000")
  local event; [ "$code" = "200" ] && event=up || event=down
  psql -U agent -d fleet -q -c \
    "INSERT INTO provider_events (provider, event, http_status) VALUES ('$name','$event',$code)"
}

probe deepseek  https://api.deepseek.com/chat/completions   deepseek-chat
probe anthropic https://api.anthropic.com/v1/messages       claude-sonnet-4-5
probe openai    https://api.openai.com/v1/chat/completions  gpt-4o-mini

$ crontab -l
* * * * * /bin/bash /root/failover-check.sh >>/var/log/failover.log 2>&1

With a one-minute probe cadence, your provider_events table becomes a continuous health history — and you can wire an alert to the down event directly, which is exactly what section 11 does with im-bot.

9. Docker Healthchecks & Restart Policies

The provider can be fine while the gateway process itself is wedged — a deadlock, a leaked file handle, a stuck event loop. Docker healthchecks catch that second class of failure:

# docker-compose.yml
services:
  gateway:
    image: hermes-agent:latest
    restart: unless-stopped
    environment:
      HERMES_HOME: /root/.hermes
    healthcheck:
      test: ["CMD", "curl", "-f", "http://127.0.0.1:4100/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s
    ports:
      - "127.0.0.1:4100:4100"

restart: unless-stopped brings the container back after a crash; the healthcheck marks it unhealthy when the process is up but unresponsive. Paired with the breaker, you get two independent safety layers: one for the model provider, one for the process running it.

$ docker compose up -d
$ docker inspect --format '{{.State.Health.Status}}' gateway
healthy

10. Drain Degraded Replicas With sun-port

Run two gateway replicas — one on the primary provider, one pre-warmed on a fallback — and let sun-port route traffic toward whichever is healthy. When the primary replica goes unhealthy, sun-port's health-gated upstreams stop sending it traffic and the fallback replica absorbs the load:

# sun-port config — two upstreams, one active
upstream agent_backend {
    server 127.0.0.1:4100 max_fails=3 fail_timeout=30s;  # primary provider
    server 127.0.0.1:4101 max_fails=3 fail_timeout=30s;  # fallback provider
}

server {
    listen 443 ssl;
    server_name agent.example.com;

    ssl_certificate     /etc/letsencrypt/live/agent.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/agent.example.com/privkey.pem;

    location / {
        proxy_pass http://agent_backend;
        proxy_next_upstream error timeout http_502 http_503 http_429;
        proxy_next_upstream_tries 2;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

$ sun-port -t
configuration file is valid
$ sun-port -s reload
reload signal sent

This is the same two-replica discipline from our blue-green tutorial, applied to provider failover instead of version releases. The proxy_next_upstream directive is the HTTP-level cousin of the breaker in section 6.

11. Alert on Provider Outage Through im-bot

A failover that nobody knows about is a failover that becomes an incident later. Wire the down event into an im-bot room so the on-call channel hears about it the moment it happens:

$ cat /root/notify-provider-down.sh
#!/bin/bash
# called by failover-check.sh when a provider transitions to 'down'
PROVIDER="$1"
CODE="$2"
curl -s -X POST http://127.0.0.1:3000/api/messages \
  -H "Content-Type: application/json" \
  -d "{\"room\":\"ops\",\"text\":\"[failover] provider $PROVIDER is DOWN (HTTP $CODE) — falling back\"}" \
  >/dev/null

Now the sequence is fully closed-loop: cron detects the outage → PostgreSQL records it → the agent falls through to the next provider → im-bot tells the humans → they fix the primary → the breaker's half-open probe confirms recovery → another up event lands in the ledger. Nothing is silent, and nothing requires a human to notice it in real time.

12. Version the Chain With Git

Your fallback chain is config, and config belongs in Git — so a bad edit to the provider list is one git revert away, not a 3am archaeology dig:

$ cd ~/.hermes
$ git add config.yaml circuit_breaker.json failover-check.sh notify-provider-down.sh
$ git commit -m "failover: add anthropic + openai fallback chain, breaker, cron probe"
$ git push origin main

Tag the known-good state before any change to the chain, and you can roll the whole failover policy back in one command:

$ git tag failover-stable-2026-08-19
$ git revert --no-edit failover-stable-2026-08-19   # only if a change breaks

13. Pitfalls

• Treating all errors as transient. 401/403/402 are fatal config or billing failures — falling through on them just burns every provider in the chain with the same bad request. Classify before you retry.
• No circuit breaker. Without one, a dead primary doubles the latency of every turn even after failover works, because each request still times out on the primary first.
• Ordering by capability, not cost. Your safety net runs on every request if it's placed first. Put cheap first, expensive last.
• Infinite fall-through. No max-depth on the chain means a request can bounce across every provider and still fail — cap the walk and return a clean error after the last one.
• Health probe with a real key. A health check that uses a stale or missing API_KEY returns 401, which the breaker misreads as a provider outage. Probe with a live key or a dedicated health endpoint.
• Silent failover. If nobody's told, a week of degraded service looks like "everything was fine" until a user complains. Log to PostgreSQL and alert to im-bot.
• Breaker state in memory only. Restart the gateway and the breaker forgets a provider was down, so it takes a fresh timeout hit on the very first request. Persist circuit_breaker.json.

14. Key Takeaways

• One provider is a single point of failure. A fallback chain — an ordered list walked top to bottom — is the cheapest insurance against an outage, a rate limit, or a billing lockout.
• Classify, then fail over. Retry transient errors with backoff, fall through only after retries are exhausted, and never fall through on fatal 401/402 errors.
• Breakers beat blind retries. A circuit breaker stops a dead primary from taxing every request, and its half-open probe brings it back automatically.
• Log it or lose it. A PostgreSQL provider_events ledger turns "was it down?" into a two-second query.
• Make failover proactive. A cron probe flips providers before users notice; Docker healthchecks catch wedged processes; sun-port drains degraded replicas; im-bot tells the humans.
• Git the policy, not the keys. The chain, breaker config, and probe scripts are versioned; API keys stay in gitignored .env.

A fallback chain is the difference between an agent that dies with its provider and one that keeps answering. Configure the chain, wrap it in a breaker, log every transition to PostgreSQL, probe it from cron, and alert it to im-bot — and a provider outage stops being an incident and becomes a line in a ledger you can read over coffee.