Reconnect a service's connection pool after the database is recreated
Table of Contents
This page is a capture in the inbox bucket of the product backlog — a pre-sprint idea, not yet pulled into a sprint as a story.
1. What
A service that loses its database reconnects every connection it holds, not
only its listener. After compass db recreate, the PostgreSQL listener of the
workflow service reconnected after about five seconds, but its repository pool
kept the connections to the dropped database. The next request through the
pool failed with "The pipeline ended without its sync: SSL connection has been
closed unexpectedly". A pool should test a connection that failed, drop it and
open a new one, so a request after the outage succeeds.
2. Why
compass services start finds the fleet already active and does not restart
it. So a recreate followed by a start leaves every service with a pool that
points at the old database. The Acme provisioning then fails in
system_provision, and the error does not name the cause. A stop, a recreate
and a start is the only workaround today. A database restart or a failover in
production would leave the same dead pool.
3. References
- The workflow journal on 2026-11-02 00:24:34:
workflow_instance_repository"The pipeline ended without its sync: SSL connection has been closed unexpectedly", after the listener reconnected at 23:53:18. compass services logs workflow --errors.