diff --git a/docs-source.json b/docs-source.json index 71ff457..0058e7f 100644 --- a/docs-source.json +++ b/docs-source.json @@ -1,6 +1,6 @@ { "repository": "getsphericon/sphericon", - "ref": "194f7aa7b5b435b9ad3f9795b32aa97fd578f98b", + "ref": "7d9dea4352f82afaee1ad814212c951125ea42be", "specs": [ "external", "telemetry" diff --git a/src/content/docs/docs/operations/monitoring.mdx b/src/content/docs/docs/operations/monitoring.mdx index ba8915f..dea590a 100644 --- a/src/content/docs/docs/operations/monitoring.mdx +++ b/src/content/docs/docs/operations/monitoring.mdx @@ -83,16 +83,19 @@ an event to the work of its consumers. ### Database pools -| Metric | Type | Labels | Meaning | -| ----------------------------- | ----- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `db_pool_open` | gauge | `pool` | Connections open (in use plus idle). | -| `db_pool_in_use` | gauge | `pool` | Connections in use. | -| `db_pool_idle` | gauge | `pool` | Idle connections. | -| `db_pool_max` | gauge | `pool` | Configured maximum (`SPHERICON_DB_MAX_OPEN_CONNS`, `SPHERICON_RIVER_DB_MAX_CONNS`). | -| `db_pool_wait_count` | gauge | `pool` | Cumulative requests that waited for a connection (`sql.DBStats.WaitCount`); use `increase()`. | +| Metric | Type | Labels | Meaning | +| ---------------------------------------------------- | ------- | ---------------- | ----------------------------------------------------------------------------------- | +| `db_sql_connection_open` | gauge | `pool`, `status` | Connections open: `status` is `inuse` or `idle`. | +| `db_sql_connection_max_open` | gauge | `pool` | Configured maximum (`SPHERICON_DB_MAX_OPEN_CONNS`, `SPHERICON_RIVER_DB_MAX_CONNS`). | +| `db_sql_connection_wait_total` | counter | `pool` | Requests that waited for a free connection (`sql.DBStats.WaitCount`). | +| `db_sql_connection_wait_duration_milliseconds_total` | counter | `pool` | Time spent waiting for a free connection. | +| `db_sql_connection_closed_max_idle_total` | counter | `pool` | Connections closed because of the idle limit (`SPHERICON_DB_MAX_IDLE_CONNS`). | +| `db_sql_connection_closed_max_idle_time_total` | counter | `pool` | Connections closed because they stayed idle too long. | +| `db_sql_connection_closed_max_lifetime_total` | counter | `pool` | Connections closed because they reached their maximum lifetime. | `pool` is `sql` (requests, ent, job workers) or `river` (the job queue's own fetch, -completion and leader election). Both are `database/sql` pools, so their counters mean the same. +completion and leader election). Both are `database/sql` pools whose statistics are exported by +[otelsql](https://github.com/XSAM/otelsql), so their metrics mean the same. ### Email diff --git a/src/content/docs/docs/operations/runbook.md b/src/content/docs/docs/operations/runbook.md index 11da21b..68fe570 100644 --- a/src/content/docs/docs/operations/runbook.md +++ b/src/content/docs/docs/operations/runbook.md @@ -123,7 +123,8 @@ Broadcast unsubscribe attribution follow their report jobs and may lag. 3. Are the jobs failing and retrying? See [`SphericonJobFailureRatioHigh`](#sphericonjobfailureratiohigh). 4. The queue fetches and completes jobs through its own `river` pool, and the workers do their work - through the `sql` pool: check `db_pool_in_use` of both against `db_pool_max` + through the `sql` pool: check `db_sql_connection_open{status="inuse"}` of both against + `db_sql_connection_max_open` ([`SphericonDBPoolSaturated`](#sphericondbpoolsaturated)). 5. Check that the river schema is applied. It is created by `sphericon migrate`; an instance that skipped migration has no job tables and the job metrics and workers fail. @@ -254,11 +255,12 @@ exceed `SPHERICON_DB_MAX_OPEN_CONNS`. ## SphericonDBPoolWaits **Means.** More than 100 times in 15 minutes a request or worker had to wait for a free -connection (`db_pool_wait_count`, either pool). Unlike saturation this catches short, repeated +connection (`db_sql_connection_wait_total`, either pool). Unlike saturation this catches short, repeated stalls. **Check and fix.** The same as [`SphericonDBPoolSaturated`](#sphericondbpoolsaturated). Waiting -that continues while `db_pool_in_use` is well below `db_pool_max` points at slow connection +that continues while `db_sql_connection_open{status="inuse"}` is well below +`db_sql_connection_max_open` points at slow connection establishment (network, TLS, a pooler) instead of a too-small pool. ## SphericonEmailSendErrorsHigh