Skip to content

Commit 5303cc2

Browse files
gabriele-wolfoxgabriele-wolfox
committed
docs(samples): add multi-server/multi-cluster OpenTelemetry sample
Adds operator/config/samples/opentelemetry-multi, deploying two Klio servers and three CNPG clusters (one in a different namespace, backed cross-namespace by one of the servers) against the same OTel/Prometheus/ Grafana stack the single-server sample uses. The existing sample only ever exercises one server and one cluster, so it can't validate the Grafana dashboard's namespace/server/cluster template variables or its per-tier and per-cluster aggregations. Validated end to end on Kind: all three clusters reconcile, stream WAL and complete backups, including the cross-namespace client whose certificates are bridged via copy-cross-namespace-secrets.sh (cert-manager Issuers are namespace scoped, so that certificate has to be generated in the server's namespace and copied over rather than issued locally). Closes #26 Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
1 parent 176676e commit 5303cc2

31 files changed

Lines changed: 1119 additions & 0 deletions

‎operator/config/samples/opentelemetry-multi/README.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# README
2+
3+
This directory contains a sample environment with **two Klio servers** and
4+
**three CloudNativePG clusters** distributed across them, one of which lives
5+
in a different namespace than the servers. It exercises the same
6+
OpenTelemetry + Prometheus + Grafana stack as
7+
`operator/config/samples/opentelemetry/`, but is meant to validate the
8+
Grafana dashboard's `$namespace`, `$server` and `$cluster` template
9+
variables and its per-tier/per-cluster aggregations against more than one
10+
server or cluster, something the single-server sample can't exercise.
11+
12+
## Topology
13+
14+
| Cluster | Namespace | Backed by |
15+
|-------------|-----------|-----------|
16+
| cluster-a | default | klio-a |
17+
| cluster-b | default | klio-b |
18+
| cluster-c | team-c | klio-b |
19+
20+
`klio-b` intentionally backs clusters in two different namespaces, since a
21+
single shared backup server serving multiple application namespaces is a
22+
realistic multi-tenant deployment and the case most likely to expose
23+
dashboard attribution bugs.
24+
25+
## Prerequisites
26+
27+
A running Kubernetes cluster with the following operators installed:
28+
29+
- CloudNativePG
30+
- Klio
31+
- cert-manager
32+
- OpenTelemetry
33+
- Prometheus
34+
35+
`jq` must also be available locally (used by `copy-cross-namespace-secrets.sh`).
36+
37+
## Deploying a Kubernetes cluster with the required operators
38+
39+
Assuming an environment with CloudNativePG, Klio and cert-manager
40+
created through the CloudNativePG `hack/setup-cluster.sh` script and
41+
the klio task
42+
43+
```shell
44+
KIND_CLUSTER_NAME=$(kind get clusters | grep pg-operator-e2e) task integration:deploy-to-kind
45+
```
46+
47+
you can install the OpenTelemetry operator by running:
48+
49+
```shell
50+
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
51+
```
52+
53+
You can install Prometheus using the Prometheus community Helm chart and
54+
the CloudNativePG example configuration:
55+
56+
```shell
57+
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
58+
```
59+
60+
```shell
61+
helm upgrade --install \
62+
-f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/docs/src/samples/monitoring/kube-stack-config.yaml \
63+
prometheus-community prometheus-community/kube-prometheus-stack
64+
```
65+
66+
## Deploying the sample configuration
67+
68+
1. Deploy the two servers, the two same-namespace clusters (cluster-a,
69+
cluster-b) and the shared OTel/Jaeger stack, all in `default`:
70+
71+
```shell
72+
kubectl apply -k operator/config/samples/opentelemetry-multi
73+
```
74+
75+
Wait for `klio-a`, `klio-b`, `cluster-a` and `cluster-b` to become ready
76+
before continuing.
77+
78+
1. cert-manager `Issuer`s are namespace-scoped, so the client certificate
79+
cluster-c needs (to authenticate to klio-b) can only be generated in
80+
`default`, where the `klio-server-ca` Issuer lives. Copy that
81+
certificate, plus the (pinned, not CA-validated — see the script's
82+
comments) server and OTel collector certificates, into the `team-c`
83+
namespace:
84+
85+
```shell
86+
./operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh
87+
```
88+
89+
1. Deploy cluster-c into `team-c`:
90+
91+
```shell
92+
kubectl apply -k operator/config/samples/opentelemetry-multi/team-c
93+
```
94+
95+
## Validating the Grafana dashboard
96+
97+
Trigger one base backup per cluster (needed before any backup/snapshot panel
98+
has data to show):
99+
100+
```shell
101+
kubectl apply -f operator/config/samples/opentelemetry-multi/backups-example.yaml
102+
```
103+
104+
Once all three backups complete and Prometheus has scraped a
105+
metrics-collection cycle, open the Klio Grafana dashboard and confirm:
106+
107+
- The `$namespace` variable offers both `default` and `team-c`.
108+
- The `$server` variable offers both `klio-a-klio-0` and `klio-b-klio-0`
109+
(the value is each server's pod hostname).
110+
- The `$cluster` variable offers `cluster-a`, `cluster-b` and `cluster-c`,
111+
and narrows correctly when `$namespace`/`$server` are filtered (e.g.
112+
selecting `$namespace=team-c` should only ever offer `cluster-c`).
113+
- Per-cluster and per-server panels correctly attribute data instead of
114+
aggregating everything together.
115+
116+
Any panel that fails to distinguish between clusters/servers/namespaces
117+
here is a dashboard bug to file separately; this sample's job is only to
118+
make that determination possible.

‎operator/config/samples/opentelemetry-multi/backups-example.yaml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Triggers one base backup on each of the three clusters, so the Grafana
2+
# dashboard has backup/snapshot data to show for all three (namespace,
3+
# server, cluster) combinations. Apply after cluster-a, cluster-b and
4+
# cluster-c are all healthy:
5+
#
6+
# kubectl apply -f operator/config/samples/opentelemetry-multi/backups-example.yaml
7+
apiVersion: postgresql.cnpg.io/v1
8+
kind: Backup
9+
metadata:
10+
name: cluster-a-backup-1
11+
namespace: default
12+
spec:
13+
method: plugin
14+
target: primary
15+
cluster:
16+
name: cluster-a
17+
pluginConfiguration:
18+
name: klio.cnpg.io
19+
---
20+
apiVersion: postgresql.cnpg.io/v1
21+
kind: Backup
22+
metadata:
23+
name: cluster-b-backup-1
24+
namespace: default
25+
spec:
26+
method: plugin
27+
target: primary
28+
cluster:
29+
name: cluster-b
30+
pluginConfiguration:
31+
name: klio.cnpg.io
32+
---
33+
apiVersion: postgresql.cnpg.io/v1
34+
kind: Backup
35+
metadata:
36+
name: cluster-c-backup-1
37+
namespace: team-c
38+
spec:
39+
method: plugin
40+
target: primary
41+
cluster:
42+
name: cluster-c
43+
pluginConfiguration:
44+
name: klio.cnpg.io

‎operator/config/samples/opentelemetry-multi/cluster_a.yaml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
apiVersion: postgresql.cnpg.io/v1
2+
kind: Cluster
3+
metadata:
4+
name: cluster-a
5+
spec:
6+
instances: 3
7+
imagePullPolicy: Always
8+
9+
projectedVolumeTemplate:
10+
sources:
11+
- secret:
12+
name: otel-collector-tls
13+
items:
14+
- key: ca.crt
15+
path: ca.crt
16+
- secret:
17+
name: klio-server-otel-client-tls
18+
items:
19+
- key: tls.crt
20+
path: tls.crt
21+
- key: tls.key
22+
path: tls.key
23+
24+
postgresql:
25+
pg_hba:
26+
- local replication all peer map=local
27+
28+
plugins:
29+
- name: klio.cnpg.io
30+
enabled: true
31+
parameters:
32+
pluginConfigurationRef: client-config-cluster-a
33+
34+
storage:
35+
size: 1Gi
36+
---
37+
apiVersion: klio.cnpg.io/v1alpha1
38+
kind: PluginConfiguration
39+
metadata:
40+
name: client-config-cluster-a
41+
spec:
42+
serverAddress: klio-a.default
43+
clientSecretName: cluster-a-klio-user
44+
serverSecretName: klio-a-tls
45+
clusterName: cluster-a
46+
containers:
47+
- name: klio-plugin
48+
env:
49+
- name: OTEL_SERVICE_NAME
50+
value: "klio-plugin"
51+
envFrom:
52+
- configMapRef:
53+
name: cluster-klio-otel-config
54+
- name: klio-restore
55+
env:
56+
- name: OTEL_SERVICE_NAME
57+
value: "klio-restore"
58+
envFrom:
59+
- configMapRef:
60+
name: cluster-klio-otel-config

‎operator/config/samples/opentelemetry-multi/cluster_a_klio_client_auth.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
apiVersion: cert-manager.io/v1
2+
kind: Certificate
3+
metadata:
4+
name: cluster-a-klio-user
5+
spec:
6+
commonName: klio@cluster-a
7+
secretName: cluster-a-klio-user
8+
9+
duration: 2160h # 90d
10+
renewBefore: 360h # 15d
11+
12+
isCA: false
13+
usages:
14+
- client auth
15+
16+
issuerRef:
17+
name: klio-server-ca
18+
kind: Issuer
19+
group: cert-manager.io

‎operator/config/samples/opentelemetry-multi/cluster_b.yaml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
apiVersion: postgresql.cnpg.io/v1
2+
kind: Cluster
3+
metadata:
4+
name: cluster-b
5+
spec:
6+
instances: 3
7+
imagePullPolicy: Always
8+
9+
projectedVolumeTemplate:
10+
sources:
11+
- secret:
12+
name: otel-collector-tls
13+
items:
14+
- key: ca.crt
15+
path: ca.crt
16+
- secret:
17+
name: klio-server-otel-client-tls
18+
items:
19+
- key: tls.crt
20+
path: tls.crt
21+
- key: tls.key
22+
path: tls.key
23+
24+
postgresql:
25+
pg_hba:
26+
- local replication all peer map=local
27+
28+
plugins:
29+
- name: klio.cnpg.io
30+
enabled: true
31+
parameters:
32+
pluginConfigurationRef: client-config-cluster-b
33+
34+
storage:
35+
size: 1Gi
36+
---
37+
apiVersion: klio.cnpg.io/v1alpha1
38+
kind: PluginConfiguration
39+
metadata:
40+
name: client-config-cluster-b
41+
spec:
42+
serverAddress: klio-b.default
43+
clientSecretName: cluster-b-klio-user
44+
serverSecretName: klio-b-tls
45+
clusterName: cluster-b
46+
containers:
47+
- name: klio-plugin
48+
env:
49+
- name: OTEL_SERVICE_NAME
50+
value: "klio-plugin"
51+
envFrom:
52+
- configMapRef:
53+
name: cluster-klio-otel-config
54+
- name: klio-restore
55+
env:
56+
- name: OTEL_SERVICE_NAME
57+
value: "klio-restore"
58+
envFrom:
59+
- configMapRef:
60+
name: cluster-klio-otel-config

‎operator/config/samples/opentelemetry-multi/cluster_b_klio_client_auth.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
apiVersion: cert-manager.io/v1
2+
kind: Certificate
3+
metadata:
4+
name: cluster-b-klio-user
5+
spec:
6+
commonName: klio@cluster-b
7+
secretName: cluster-b-klio-user
8+
9+
duration: 2160h # 90d
10+
renewBefore: 360h # 15d
11+
12+
isCA: false
13+
usages:
14+
- client auth
15+
16+
issuerRef:
17+
name: klio-server-ca
18+
kind: Issuer
19+
group: cert-manager.io

‎operator/config/samples/opentelemetry-multi/cluster_c_klio_client_auth.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# cluster-c lives in the "team-c" namespace (see ../team-c), but the
2+
# klio-server-ca Issuer is namespace-scoped and only exists here in
3+
# "default". This Certificate is therefore generated here and its resulting
4+
# secret is copied into "team-c" by ../copy-cross-namespace-secrets.sh.
5+
apiVersion: cert-manager.io/v1
6+
kind: Certificate
7+
metadata:
8+
name: cluster-c-klio-user
9+
spec:
10+
commonName: klio@cluster-c
11+
secretName: cluster-c-klio-user
12+
13+
duration: 2160h # 90d
14+
renewBefore: 360h # 15d
15+
16+
isCA: false
17+
usages:
18+
- client auth
19+
20+
issuerRef:
21+
name: klio-server-ca
22+
kind: Issuer
23+
group: cert-manager.io

‎operator/config/samples/opentelemetry-multi/cluster_klio_otel_config.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: cluster-klio-otel-config
5+
data:
6+
OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name"
7+
# Because traces use the gRPC protocol, Klio also enables Kopia's own
8+
# OTLP/gRPC trace exporter during backup operations: Kopia spans
9+
# (service.name=kopia) are sent to the same collector, reusing the
10+
# OTEL_EXPORTER_OTLP_* settings below (endpoint and TLS). Kopia can only
11+
# export traces over gRPC.
12+
OTEL_TRACES_EXPORTER: "otlp"
13+
OTEL_METRICS_EXPORTER: "otlp"
14+
OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"
15+
OTEL_EXPORTER_OTLP_ENDPOINT: "https://otel-collector:4317"
16+
OTEL_EXPORTER_OTLP_COMPRESSION: "gzip"
17+
OTEL_EXPORTER_OTLP_TIMEOUT: "10000"
18+
OTEL_EXPORTER_OTLP_INSECURE: "false"
19+
OTEL_EXPORTER_OTLP_CERTIFICATE: "/projected/ca.crt"
20+
OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE: "/projected/tls.crt"
21+
OTEL_EXPORTER_OTLP_CLIENT_KEY: "/projected/tls.key"

0 commit comments

Comments
 (0)