CVE remediation, LMDB sizing, replication hardening, and validation fixes - #2
Open
skreisigTSI wants to merge 8 commits into
Open
Conversation
…E scan The blocking CVE-scan CI job on main was failing due to new HIGH severity Go stdlib CVEs (2026-33818, 39821, 46600, 56853, 56858, 56859, 56860, 56862) present in both the powerdns-operator and lightningstream images. - Trigger a rebuild of powerdns-operator from the deployed feature branch (feat/watch-namespace-env, commit b6cc86a) against a patched Go toolchain. New image ghcr.io/telekom/powerdns-operator:sha-b6cc86a scans clean (0 HIGH/CRITICAL). Tag archive/watch-namespace-deployed-1a1bf0c created upstream to preserve the superseded commit per repo convention. - Bump lightningstream 1.0.2 -> 1.0.3, which clears the stdlib CVEs but introduces CVE-2026-46600 (x/net) and four util-linux TOCTOU local-privesc CVEs (2026-53612..53615). Neither is exploitable for this workload (no arbitrary DNS parsing, single-process container, no local multi-user access), so they are suppressed. - Update .trivyignore.yaml with time-boxed suppressions (expired-at: 2026-09-05) for the five remaining CVEs, each with justification per existing file convention. - Update deploy manifests (base + KRO), OCM component descriptor, and docs (ARCHITECTURE.md provenance section, AIR-GAP-DEPLOYMENT.md image table) to reference the new image versions/digests. Verified locally: trivy image scan (CI's exact flags) on both new images exits 0; yamllint, shellcheck, actionlint, validate-observability-manifests.sh, validate-github-actions.sh, kustomize build + kubeconform -strict on deploy/deploy-kro/monitoring overlays, validate-image-inventory.sh, validate-air-gap-localization.sh, and make ocm-build + validate-ocm-package.sh all pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Zone/RRset describe assertions ran grep -q on a preceding line and then called check "<desc>" true, so under set -euo pipefail the check was structurally guaranteed to report PASS. Wire check() directly to the grep command via bash -c so PASS/FAIL reflects the real assertion. Fixes #47 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
EXPECTED_RESOURCES in validate-ocm-package.sh listed the garage S3-store image but omitted garage-bootstrap, the init container that provisions the Garage bucket/access key. Both are declared as core resources in ocm/component-descriptor.yaml, so the completeness check left a gap in Specification Sec.4 Criterion 1 (Complete OCM Package). Verified: rebuilding the OCM package with garage-bootstrap removed from the descriptor now makes the check FAIL as expected; with it present, the check PASSes. Fixes #51 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…dns-auth The Authoritative Server Deployment/pod used app.kubernetes.io/name: pdns-auth, but its Service used dns-authoritative. The Service's own spec.selector (targeting the pods) already used pdns-auth, so this was not a runtime failure, but the inconsistency made label-based queries, monitoring selectors, and automation harder to reason about. Standardize on pdns-auth (matching the workload name and the existing Service selector) across: - deploy/base/authoritative/service.yaml - deploy/kro/powerdns-instance-rgd.yaml (serviceAuth block) - deploy/overlays/monitoring/service-monitors.yaml (pdns-authoritative ServiceMonitor selector, which targets the Service's own label) app.kubernetes.io/component is unchanged (dns-authoritative) as it was not part of the reported inconsistency. NetworkPolicy selectors and all hack/validate-*.sh pod selectors already used pdns-auth and required no changes. Verified: yamllint (CI config), kustomize build + kubeconform -strict on deploy, deploy/kro, deploy/overlays/monitoring all pass; hack/validate-observability-manifests.sh reports 16/16 passed; rendered manifests confirm the ServiceMonitor selector now matches the Service label. Fixes #57 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CI's CVE scan on main started failing again due to a new HIGH severity finding introduced into the powerdns/lightningstream:1.0.3 image (same digest already deployed): CVE-2026-14456, an OpenSSL denial of service via unbounded memory growth in the QUIC server implementation, affecting the libssl3t64/openssl/openssl-provider-legacy Debian packages (3.5.6-1~deb13u2, fixed in 3.5.7-1~deb13u2). No newer lightningstream tag has been published upstream (latest remains 1.0.3, unchanged since 2026-08-18), so a version bump isn't possible yet. Lightning Stream only uses OpenSSL as an HTTP/TLS client for S3 replication traffic and never runs a QUIC server, so the vulnerable server-side code path is not exercised. Added a time-boxed suppression (expired-at: 2026-09-05) to .trivyignore.yaml following the file's existing justification convention, and updated the header narrative. Verified locally: trivy image scan with CI's exact flags (--severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 --ignorefile .trivyignore.yaml) against powerdns/lightningstream:1.0.3@sha256:b8698c930fe347e55c307adeb7950d5c249410ec0f7460f96d955a8bf706852d now exits 0 with 0 vulnerabilities reported. yamllint (CI's exact scope/config) also passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PowerDNS (lmdb-map-size) and the Lightning Stream sidecar (map_size) must agree on the LMDB map size, but the value was duplicated verbatim across two base ConfigMaps and the KRO templates with no shared source. Any edit that missed one location would desynchronise the two writers on the shared environment. The Auth PVC was also fixed at 1Gi while the two LMDB environments alone map 1000 MB each, leaving no room for snapshots. Introduce a single lmdbMapSizeMB schema field and derive every dependent value from it: both pdns.conf and Lightning Stream map sizes, the Auth PVC (4 MiB per map-size MB, covering both environments plus snapshot and filesystem headroom), and the single-node Garage layout capacity. The static base manifests keep literal values but are cross-checked numerically by a new hack/validate-lmdb-config.sh contract wired into CI, which also rejects stray hardcoded values that would bypass the schema field. Enable Lightning Stream's storage.cleanup for both the filesystem and S3 backends. Cleanup is disabled by default upstream, so snapshots previously accumulated without bound on the very volume this change resizes. Add a pod-template revision annotation so existing deployments reload the changed configuration, and reconcile the Garage layout when the derived capacity changes rather than only on first bootstrap. Two KRO details are encoded here because neither is catchable locally (kubeconform skips the ResourceGraphDefinition): SimpleSchema markers take a single pipe with space-separated constraints, and integer schema fields interpolated into string templates must be wrapped in string(). Document the 4x sizing policy, the cleanup retention, the StorageClass allowVolumeExpansion prerequisite, and the fact that PVC expansion cannot be rolled back. Fixes #48 Fixes #49 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Follow-up to the review of the #48/#49 fix, covering the sync layer around the newly tunable lmdbMapSizeMB. Verified against the Lightning Stream 1.0.3 sources (config/config.go, status/httpd.go), which parse the config with yaml.UnmarshalStrict, so every key used here is accepted upstream. Scale sidecar memory with the map size. Lightning Stream buffers snapshots in memory per LMDB, and this bundle configures two (main + shard). With a fixed 128Mi limit, raising lmdbMapSizeMB scaled the PVC and the Garage layout but not the sidecar, leaving an OOMKill cliff that stops replication silently. The limit is now derived (lmdbMapSizeMB / 2 MiB), and the decompressed snapshot buffers are bounded to one per LMDB, since those are 3-10x the compressed size and dominate the memory profile. Make the sync layer observable. The sidecar serves /metrics, /healthz and a status page, but only when http.address is set, which it was not: the component whose failure modes motivated #49 was the only one without metrics. Enable the listener on :8500, expose it on the Auth Service and NetworkPolicy, scrape it from the existing ServiceMonitor, and extend the live validation to it. Probes deliberately test the listener rather than /healthz: /healthz returns 503 after a sustained storage outage, so gating readiness on it would evict a still-serving authoritative server and turn degraded replication into a DNS outage. Stabilise the instance name. Lightning Stream requires unique instance names, and a pod-derived name satisfied that but changed on every rollout, registering a new instance whose final snapshot lingered until remove_old_instances_interval elapsed. The Auth Deployment is pinned to a single replica (Recreate + RWO PVC), so the namespace is a safe stable identity. Also correct the stale claim that Lightning Stream has no *_filename credential options (simpleblob v1.0.0 does support them, documented as an alternative), derive the validator's PVC/revision/memory expectations from the observed map size instead of hardcoded literals, and key the Prometheus target check by (service, port) so the second pdns-auth target cannot mask the first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Configuration and documentation hardening follow-ups from the delivery review. No behavioural change: every manifest edit is comment-only, so the rendered resources are byte-identical apart from the comments. - Mark every `changeme` API key occurrence as a development placeholder (base Secret, Auth ConfigMap, KRO examples, KRO RGD, INSTALLATION.md). Document the key-generation command, the overlay override pattern, and the fact that the Auth ConfigMap value and the Operator Secret are two halves of one credential — a mismatch fails reconciliation with 401. The literal values are kept as-is because hack/validate-*.sh default PDNS_API_KEY to them. - Correct the misleading ACL comments. The recursor comment claimed `allow-from=0.0.0.0/0` limited queries to cluster-internal sources; it does not. The effective boundary is the NetworkPolicy set plus the ClusterIP Services. Mirrored into the KRO RGD, which carried no comments on these lines at all. - Add INSTALLATION.md §7, a 12-item production preflight checklist covering key rotation, CNI NetworkPolicy enforcement, forwarders, LoadBalancer source ranges, rate limiting, map sizing, the monitoring namespace label, replication alerting, the single-replica Auth availability ceiling, and the residual risk register. - Add OPERATIONS.md §6.4 with reference Kustomize patches for loadBalancerSourceRanges, externalTrafficPolicy Local plus per-client QPS, narrowed recursor and operator egress, and tighter webserver-allow-from. Shipped as documented snippets rather than an overlay because the correct values are environment-specific. - Cross-link residual risks 1, 2, 5, 6 and 7 to the new sections. Validation: kustomize build of deploy and both overlays, YAML parse of all KRO manifests, validate-lmdb-config (22/22), validate-observability- manifests (21/21), validate-image-inventory, validate-air-gap-localization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bundles eight changes developed and validated downstream in
telekom/odc-powerdns-bundle-dev: CVE scan remediation, validation-script fixes, LMDB/storage sizing unification, Lightning Stream replication hardening, and production operations documentation.Each commit is kept separate to preserve authorship and rationale.
Commits
fix(cve)fixvalidate-crds.shfixgarage-bootstrapto OCM package completeness checkfixapp.kubernetes.io/namelabel topdns-authfix(cve)fixfixdocsDetails
CVE remediation
Rebuilds the operator image, bumps lightningstream, and updates
.trivyignore.yamlwith justified suppressions for the OpenSSL QUIC DoS finding in lightningstream 1.0.3. Component references inocm/component-descriptor.yamlare updated accordingly.Validation script fixes
validate-crds.sh: describe checks previously ignored the actualgrepresult, so failures could pass silently.validate-ocm-package.sh:garage-bootstrapwas missing from the package completeness check.Label normalization
Aligns the Authoritative Server
app.kubernetes.io/namelabel topdns-authacross the Service, the KRO ResourceGraphDefinition, and the monitoring ServiceMonitors so selectors match consistently.LMDB and storage sizing
Unifies LMDB map size and PVC sizing across base manifests and KRO definitions, exposes them as instance parameters, and adds
hack/validate-lmdb-config.shto enforce consistency. Documentation inARCHITECTURE.md,INSTALLATION.md,OPERATIONS.md, andUPGRADE.mdis updated to match.Lightning Stream replication hardening
Hardens the replication setup across the deployment, service, network policies, and KRO definitions, with corresponding observability wiring and expanded validation coverage.
Operations documentation
Adds a production preflight checklist plus hardening guidance, and annotates the operator secret and authoritative/recursor ConfigMaps with production-relevant defaults.
Scope
internal/directory) is deliberately excludedValidation
Verified that the net diff of this branch is byte-identical to the corresponding downstream state, excluding downstream-only paths. All commits applied without conflicts.