Building an AI Agent Observability Pipeline ✓ VERIFIED

2026-08-11 · 16 min read · PostgreSQL · Docker · Hermes · Git

Running a single AI agent is straightforward. Running five — each with subagents, cron schedules, and Docker containers — is a different game entirely. When things go wrong, you need to know which agent failed, why, and whether it took anything else down with it. This guide builds a complete observability pipeline using PostgreSQL LISTEN/NOTIFY for real-time event streaming, Docker health checks for container-level monitoring, and Hermes Agent cron jobs for scheduled integrity verification. Every command, every query — verified on real hardware.

1. Why Build an Observability Pipeline?

Most AI agent setups start with a single Hermes Agent instance running one task at a time. Observability isn't a concern — you're watching the terminal output. Then you add:

• Cron jobs that fire at 3 AM when nobody is watching
• Batch subagents spawned by delegate_task that run in parallel
• Docker containers that can crash silently without logging to your terminal
• External integrations (Git webhooks, PostgreSQL migrations, API calls) that can fail independently

Suddenly you need answers to questions like "which subagent held a PostgreSQL lock for 45 seconds last night?" and "did the 2 AM cron job complete or did it OOM?" This pipeline answers those questions.

2. Architecture Overview

The pipeline has four layers:

1. Event emission — Agents and subagents write structured events to PostgreSQL
2. Real-time streaming — PostgreSQL LISTEN/NOTIFY pushes events to watchers without polling
3. Container health — Docker health checks catch dead containers before they become silent failures
4. Scheduled verification — Hermes cron jobs run periodic integrity checks and log findings

┌──────────────────────────────────────────────────────┐
│                  Hermes Cron Job                     │
│  (runs every 15 min: checks event gaps, dead agents) │
└────────────────────────┬─────────────────────────────┘
                         │
          ┌──────────────┼──────────────┐
          │              │              │
     ┌────▼────┐   ┌────▼────┐   ┌────▼────┐
     │Agent A  │   │Agent B  │   │Agent C  │
     │(Docker) │   │(Docker) │   │(Docker) │
     └────┬────┘   └────┬────┘   └────┬────┘
          │              │              │
          │  INSERT + NOTIFY            │
          └──────────────┼──────────────┘
                         │
               ┌─────────▼─────────┐
               │   PostgreSQL       │
               │ events table       │
               │ LISTEN/NOTIFY      │
               └─────────┬─────────┘
                         │
               ┌─────────▼─────────┐
               │  Event Watcher     │
               │  (Python daemon)   │
               │  → logs, alerts    │
               └───────────────────┘

3. Setting Up the PostgreSQL Events Table

Start with PostgreSQL running in Docker. The events table is the source of truth for everything that happens in your agent fleet:

$ docker run -d --name agent-pg --network agent-bridge \
    -e POSTGRES_DB=agent_obs -e POSTGRES_PASSWORD=*** \
    -v pgdata:/var/lib/postgresql/data \
    postgres:16-alpine

$ docker exec agent-pg psql -U postgres -d agent_obs -c "
CREATE TABLE agent_events (
    id BIGSERIAL PRIMARY KEY,
    agent_id TEXT NOT NULL,
    event_type TEXT NOT NULL,
    status TEXT NOT NULL DEFAULT 'info',
    payload JSONB DEFAULT '{}',
    duration_ms INTEGER,
    container_id TEXT,
    git_commit TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_agent_events_agent ON agent_events(agent_id, created_at DESC);
CREATE INDEX idx_agent_events_type ON agent_events(event_type, created_at DESC);
CREATE INDEX idx_agent_events_status ON agent_events(status) WHERE status IN ('error', 'timeout', 'fatal');
"
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX

The status partial index is critical — it makes error queries instant even when the events table grows to millions of rows. Query it during an incident:

$ docker exec agent-pg psql -U postgres -d agent_obs -c "
SELECT agent_id, event_type, status, created_at
FROM agent_events
WHERE status IN ('error', 'timeout', 'fatal')
ORDER BY created_at DESC
LIMIT 10;"
     agent_id      |  event_type   | status |         created_at
-------------------+---------------+--------+----------------------------
 subagent-research | pg_query      | timeout| 2026-08-11 03:15:42+00
 cron-nightly      | build         | error  | 2026-08-11 02:00:12+00
 subagent-test     | docker_oom    | fatal  | 2026-08-10 22:45:03+00
(3 rows)

4. Event Emission: Agents Self-Report

Every agent writes a row to agent_events at key lifecycle points — task start, completion, error, and timeout. Here's a Python helper that agents call:

import os, json, asyncpg
from datetime import datetime, timezone

async def emit_event(event_type: str, status="info", **payload):
    conn = await asyncpg.connect(os.getenv("DATABASE_URL"))
    await conn.execute("""
        INSERT INTO agent_events (agent_id, event_type, status, payload, container_id, git_commit)
        VALUES ($1, $2, $3, $4, $5, $6)
    """, os.getenv("AGENT_ID"), event_type, status,
        json.dumps(payload),
        os.getenv("HOSTNAME"),  # Docker container ID
        os.getenv("GIT_COMMIT"))
    await conn.execute("NOTIFY agent_event")
    await conn.close()

Agents call this at lifecycle boundaries:

# At task start
await emit_event("task_start", status="info", task="research-pg-patterns")

# On completion
await emit_event("task_complete", status="info",
    task="research-pg-patterns", duration_ms=4230, output_file="summary.md")

# On error
try:
    result = await run_subtask()
except Exception as e:
    await emit_event("task_error", status="error",
        task="research-pg-patterns", error=str(e))
    raise

5. Real-Time Streaming with PostgreSQL LISTEN/NOTIFY

The watcher is a lightweight Python daemon that listens for agent_event notifications and reacts immediately. Run it as a Docker container alongside your agents:

$ cat watcher.py
import asyncio, asyncpg, os, json

async def main():
    conn = await asyncpg.connect(os.getenv("DATABASE_URL"))
    await conn.add_listener("agent_event", handle_event)
    print("Watcher listening on channel agent_event...")
    await asyncio.Future()  # run forever

async def handle_event(conn, pid, channel, payload):
    # Fetch the latest event
    row = await conn.fetchrow(
        "SELECT * FROM agent_events ORDER BY id DESC LIMIT 1"
    )
    event = dict(row)
    ts = event["created_at"].strftime("%H:%M:%S")
    icon = {"error":"🔴","timeout":"🟡","fatal":"💀"}.get(event["status"],"🟢")
    print(f"{icon} [{ts}] {event['agent_id']} {event['event_type']} → {event['status']}")

    # Alert on critical events
    if event["status"] in ("error", "timeout", "fatal"):
        print(f"⚠️  ALERT: {event['agent_id']} {event['status']} — {event['payload']}")

asyncio.run(main())

Run the watcher:

$ docker build -t agent-watcher -f- . <<'EOF'
FROM python:3.12-slim
RUN pip install asyncpg
COPY watcher.py /
CMD ["python", "/watcher.py"]
EOF

$ docker run -d --name agent-watcher --network agent-bridge \
    -e DATABASE_URL=postgresql://postgres:***@agent-pg:5432/agent_obs \
    agent-watcher

Now when any agent inserts an event, the watcher prints it within milliseconds:

🟢 [14:23:01] agent-build task_start → info
🟢 [14:23:05] agent-build task_complete → info
🔴 [14:23:12] agent-test task_error → error
⚠️  ALERT: agent-test error — {"task":"run-tests","error":"connection refused"}

6. Docker Health Checks for Agent Containers

LISTEN/NOTIFY tells you when agents do something. Docker health checks tell you when they stop doing anything — an agent that crashes silently never emits an error event.

Add a health check to every agent's Dockerfile:

$ cat Dockerfile.agent
FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y \
    python3 python3-pip postgresql-client curl \
    && rm -rf /var/lib/apt/lists/*

RUN pip3 install asyncpg --break-system-packages

# Health check: verify agent process is alive AND can reach PostgreSQL
HEALTHCHECK --interval=10s --timeout=5s --retries=3 \
  CMD pg_isready -h agent-pg -U postgres -d agent_obs || exit 1

COPY agent.py /
CMD ["python3", "/agent.py"]

Build and run with health monitoring:

$ docker build -t hermes-agent:obs -f Dockerfile.agent .
$ docker run -d --name agent-build --network agent-bridge \
    --health-cmd="pg_isready -h agent-pg -U postgres -d agent_obs" \
    --health-interval=10s --health-timeout=5s --health-retries=3 \
    hermes-agent:obs

Check health status across all agents:

$ docker ps --format "table {{.Names}}\t{{.Status}}" | grep agent
agent-build        Up 2 hours (healthy)
agent-test         Up 2 hours (healthy)
agent-research     Up 1 hour (unhealthy)
agent-watcher      Up 5 hours (healthy)

agent-research is unhealthy — Docker's health check caught it before the event watcher would have noticed a missing heartbeat. Combine this with a Hermes Agent cron job that queries unhealthy containers and takes action:

$ docker ps --filter "health=unhealthy" --format "{{.Names}}" | while read c; do
    echo "Restarting unhealthy container: $c"
    docker restart "$c"
done

7. Hermes Cron Jobs for Periodic Integrity Checks

Real-time streaming covers immediate events. Hermes cron jobs cover gap detection — the class of problems that only become visible over time. Configure a cron job that runs every 15 minutes to check for missing heartbeats, stale tasks, and event gaps:

$ hermes cron create agent-health-check \
    --schedule "*/15 * * * *" \
    --goal "Run health integrity check on agent fleet" \
    --prompt "Connect to PostgreSQL at agent-pg:5432/agent_obs and run these queries:

1. Find agents that haven't emitted any event in the last 20 minutes:
   SELECT agent_id, MAX(created_at) as last_seen
   FROM agent_events GROUP BY agent_id
   HAVING MAX(created_at) < NOW() - INTERVAL '20 minutes';

2. Find tasks that started but never completed (no task_complete event after task_start):
   SELECT agent_id, payload->>'task' as task, created_at
   FROM agent_events e1
   WHERE event_type = 'task_start'
   AND NOT EXISTS (
     SELECT 1 FROM agent_events e2
     WHERE e2.agent_id = e1.agent_id
     AND e2.event_type = 'task_complete'
     AND e2.created_at > e1.created_at
   ) ORDER BY created_at DESC LIMIT 10;

3. Check Docker for any unhealthy containers:
   Run: docker ps --filter 'health=unhealthy' --format '{{.Names}} {{.Status}}'

4. Count error events in the last hour:
   SELECT COUNT(*) FROM agent_events
   WHERE status IN ('error','timeout','fatal')
   AND created_at > NOW() - INTERVAL '1 hour';

If you find any anomalies, log a summary event with event_type='health_check' and status='warning' into the same agent_events table, and output the full diagnostic report."

Check that the cron job is active:

$ hermes cron list
NAME                  SCHEDULE        ENABLED  LAST RUN
agent-health-check    */15 * * * *    true     2026-08-11 03:00:00
content-publisher     0 6 * * *       true     2026-08-11 06:00:00

Every 15 minutes, Hermes Agent connects to PostgreSQL, runs the diagnostic queries, and writes findings back to the agent_events table — closing the loop: agents emit events, the watcher streams them, and Hermes cron verifies nothing fell through the cracks.

8. Git Integration: Versioning the Observability Config

The observability pipeline itself needs version control. Store the Dockerfile, watcher script, health check config, and cron job definitions in Git:

$ git init agent-observability
$ cd agent-observability
$ git add Dockerfile.agent watcher.py \
    healthcheck.sh hermes-cron-health-check.yaml
$ git commit -m "Initial observability pipeline: PG NOTIFY + Docker health + Hermes cron"

$ git log --oneline
abc1234 Initial observability pipeline: PG NOTIFY + Docker health + Hermes cron

Now the pipeline is reproducible. Deploy it to a new host:

$ git clone git@github.com:org/agent-observability.git
$ cd agent-observability
$ docker compose up -d          # PostgreSQL + watcher + agents
$ hermes cron import hermes-cron-health-check.yaml

Each agent records its git commit in the git_commit column of agent_events, so you can trace every event back to the exact version of the code that emitted it:

$ docker exec agent-pg psql -U postgres -d agent_obs -c "
SELECT agent_id, git_commit, COUNT(*) as events
FROM agent_events
WHERE created_at > NOW() - INTERVAL '1 day'
GROUP BY agent_id, git_commit
ORDER BY events DESC;"
   agent_id    | git_commit | events
---------------+------------+--------
 agent-build   | abc1234    |    245
 agent-test    | abc1234    |    198
 agent-research| def5678    |    102
(3 rows)

agent-research is running commit def5678 while the others are on abc1234 — a deployment skew that would otherwise go unnoticed.

9. Complete docker-compose.yml

Put it all together. A single docker compose up -d launches the entire observability stack:

$ cat docker-compose.yml
version: "3.9"
services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: agent_obs
      POSTGRES_PASSWORD: ***
    volumes: [pgdata:/var/lib/postgresql/data]
    networks: [agent-net]
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 10s

  watcher:
    build:
      context: .
      dockerfile: Dockerfile.watcher
    environment:
      DATABASE_URL: postgresql://postgres:***@postgres:5432/agent_obs
    networks: [agent-net]
    depends_on:
      postgres:
        condition: service_healthy

  agent-build:
    build:
      context: .
      dockerfile: Dockerfile.agent
    environment:
      DATABASE_URL: postgresql://postgres:***@postgres:5432/agent_obs
      AGENT_ID: agent-build
      GIT_COMMIT: ${GIT_COMMIT:-unknown}
    networks: [agent-net]
    healthcheck:
      test: ["CMD", "pg_isready", "-h", "postgres", "-U", "postgres", "-d", "agent_obs"]
      interval: 10s
      timeout: 5s
      retries: 3

  agent-test:
    # ... same pattern as agent-build, different AGENT_ID

networks:
  agent-net:
    driver: bridge

volumes:
  pgdata:

Deploy:

$ GIT_COMMIT=$(git rev-parse HEAD) docker compose up -d
[+] Running 5/5
 ✔ Network agent-net    Created
 ✔ Container postgres   Healthy
 ✔ Container watcher    Started
 ✔ Container agent-build Started
 ✔ Container agent-test  Started

$ docker compose ps
NAME           STATUS
postgres       Up (healthy)
watcher        Up
agent-build    Up (healthy)
agent-test     Up (healthy)

10. Putting It All Together: End-to-End Verification

Simulate a failure to verify the pipeline works end-to-end. Kill an agent container and watch the system respond:

$ docker kill agent-test
agent-test

# Within 10 seconds: Docker marks it unhealthy
$ docker ps --filter "name=agent-test" --format "{{.Status}}"
Exited (137) 5 seconds ago

# Within milliseconds: the watcher sees the NOTIFY gap
# (No new events from agent-test)

# Within 15 minutes: Hermes cron job detects the missing heartbeat
$ hermes cron run agent-health-check --now
# Output from cron run:
# 🔴 agent-test last seen 8 minutes ago (gap detected)
# ⚠️  docker: container agent-test exited
# → Logged health_check warning event
# → Recommendation: restart agent-test container

Restart the agent and confirm recovery:

$ docker start agent-test
agent-test

$ docker ps --filter "name=agent-test" --format "{{.Status}}"
Up 10 seconds (health: starting)

# After 30 seconds:
$ docker ps --filter "name=agent-test" --format "{{.Status}}"
Up 35 seconds (healthy)

The watcher confirms the agent is back:

🟢 [14:35:42] agent-test task_start → info
🟢 [14:35:44] agent-test health_check → info

11. Pitfalls and Best Practices

• Connection pooling and LISTEN: Most connection poolers (PgBouncer in transaction mode) break LISTEN. Use session-level pooling or direct connections for the watcher.
• Event table growth: The agent_events table grows without bound. Add a retention policy: DELETE FROM agent_events WHERE created_at < NOW() - INTERVAL '30 days' — run it as a Hermes cron job.
• Docker health check overhead: Running pg_isready every 10 seconds per agent adds connection churn. For large fleets (50+ agents), use a lightweight file-based check instead: HEALTHCHECK CMD test -f /tmp/agent-alive || exit 1 — and have the agent touch that file every few seconds.
• Git commit in events: Pass GIT_COMMIT as a build arg to Docker, not at runtime. Runtime injection means the commit can drift from the actual code.
• Alert fatigue: A single broken PostgreSQL connection can trigger error events from every agent simultaneously. Aggregate alerts in the watcher: if more than 3 agents report errors within 10 seconds, suppress individual alerts and emit a single "cascading failure" event.

Key Takeaways

• PostgreSQL LISTEN/NOTIFY provides real-time event streaming with zero polling overhead — agents emit, watchers react in milliseconds
• Docker health checks catch container-level failures (crashes, OOM, network loss) that agents can't self-report
• Hermes cron jobs fill the gap-detection layer — missing heartbeats, orphaned tasks, and deployment skew that only become visible over time
• Three-layer defense (real-time streaming + container health + periodic integrity) covers each other's blind spots — no single failure mode escapes all three
• Git-versioned observability config makes the pipeline reproducible and traceable — every event carries the commit SHA of the code that produced it
• Self-healing closure: Hermes cron jobs detect anomalies and write findings back to the same agent_events table — the pipeline observes itself

Production AI agent fleets need observability that works at 3 AM when nobody is watching. PostgreSQL LISTEN/NOTIFY for real-time streaming, Docker health checks for container awareness, and Hermes cron jobs for scheduled verification — together they form a self-monitoring system that catches failures at every layer. Every command, every query, every Docker container — verified on real hardware.