Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs-source.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"repository": "getsphericon/sphericon",
"ref": "194f7aa7b5b435b9ad3f9795b32aa97fd578f98b",
"ref": "7d9dea4352f82afaee1ad814212c951125ea42be",
"specs": [
"external",
"telemetry"
Expand Down
19 changes: 11 additions & 8 deletions src/content/docs/docs/operations/monitoring.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,16 +83,19 @@ an event to the work of its consumers.

### Database pools

| Metric | Type | Labels | Meaning |
| ----------------------------- | ----- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `db_pool_open` | gauge | `pool` | Connections open (in use plus idle). |
| `db_pool_in_use` | gauge | `pool` | Connections in use. |
| `db_pool_idle` | gauge | `pool` | Idle connections. |
| `db_pool_max` | gauge | `pool` | Configured maximum (`SPHERICON_DB_MAX_OPEN_CONNS`, `SPHERICON_RIVER_DB_MAX_CONNS`). |
| `db_pool_wait_count` | gauge | `pool` | Cumulative requests that waited for a connection (`sql.DBStats.WaitCount`); use `increase()`. |
| Metric | Type | Labels | Meaning |
| ---------------------------------------------------- | ------- | ---------------- | ----------------------------------------------------------------------------------- |
| `db_sql_connection_open` | gauge | `pool`, `status` | Connections open: `status` is `inuse` or `idle`. |
| `db_sql_connection_max_open` | gauge | `pool` | Configured maximum (`SPHERICON_DB_MAX_OPEN_CONNS`, `SPHERICON_RIVER_DB_MAX_CONNS`). |
| `db_sql_connection_wait_total` | counter | `pool` | Requests that waited for a free connection (`sql.DBStats.WaitCount`). |
| `db_sql_connection_wait_duration_milliseconds_total` | counter | `pool` | Time spent waiting for a free connection. |
| `db_sql_connection_closed_max_idle_total` | counter | `pool` | Connections closed because of the idle limit (`SPHERICON_DB_MAX_IDLE_CONNS`). |
| `db_sql_connection_closed_max_idle_time_total` | counter | `pool` | Connections closed because they stayed idle too long. |
| `db_sql_connection_closed_max_lifetime_total` | counter | `pool` | Connections closed because they reached their maximum lifetime. |

`pool` is `sql` (requests, ent, job workers) or `river` (the job queue's own fetch,
completion and leader election). Both are `database/sql` pools, so their counters mean the same.
completion and leader election). Both are `database/sql` pools whose statistics are exported by
[otelsql](https://github.com/XSAM/otelsql), so their metrics mean the same.

### Email

Expand Down
8 changes: 5 additions & 3 deletions src/content/docs/docs/operations/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,8 @@ Broadcast unsubscribe attribution follow their report jobs and may lag.
3. Are the jobs failing and retrying? See
[`SphericonJobFailureRatioHigh`](#sphericonjobfailureratiohigh).
4. The queue fetches and completes jobs through its own `river` pool, and the workers do their work
through the `sql` pool: check `db_pool_in_use` of both against `db_pool_max`
through the `sql` pool: check `db_sql_connection_open{status="inuse"}` of both against
`db_sql_connection_max_open`
([`SphericonDBPoolSaturated`](#sphericondbpoolsaturated)).
5. Check that the river schema is applied. It is created by `sphericon migrate`; an instance that
skipped migration has no job tables and the job metrics and workers fail.
Expand Down Expand Up @@ -254,11 +255,12 @@ exceed `SPHERICON_DB_MAX_OPEN_CONNS`.
## SphericonDBPoolWaits

**Means.** More than 100 times in 15 minutes a request or worker had to wait for a free
connection (`db_pool_wait_count`, either pool). Unlike saturation this catches short, repeated
connection (`db_sql_connection_wait_total`, either pool). Unlike saturation this catches short, repeated
stalls.

**Check and fix.** The same as [`SphericonDBPoolSaturated`](#sphericondbpoolsaturated). Waiting
that continues while `db_pool_in_use` is well below `db_pool_max` points at slow connection
that continues while `db_sql_connection_open{status="inuse"}` is well below
`db_sql_connection_max_open` points at slow connection
establishment (network, TLS, a pooler) instead of a too-small pool.

## SphericonEmailSendErrorsHigh
Expand Down
Loading