Skip to content

CI-CD.md

Latest commit

 

History

History
160 lines (114 loc) · 12.1 KB

CI-CD.md

File metadata and controls

160 lines (114 loc) · 12.1 KB
Raw

CI/CD Pipelines

This repository uses GitHub Actions to validate changes, build the OCM package, scan delivered images, and run deployment smoke tests when a Kubernetes test cluster is configured.

Workflow

Stage Purpose Trigger
Quality checks YAML linting, ShellCheck, workflow linting, action allowlist validation, secret scanning, Kustomize rendering, and Kubernetes schema validation Pull requests, pushes to main, manual dispatch
OCM package Builds ocm/ctf.tar, validates the component archive, optionally creates ocm/ctf-bundled.tar, generates checksums, and creates a CycloneDX SBOM Pull requests, pushes to main, manual dispatch
CVE scans Scans every container image declared in ocm/component-descriptor.yaml, uploads SARIF reports, and generates a per-image CycloneDX SBOM After the OCM package build succeeds
Deployment smoke tests Applies deploy/ to a test cluster and runs the repository smoke-test scripts After build and scans succeed; skipped when no cluster secret is configured

The CVE scan runs after the OCM package stage so the validated component descriptor is the source of truth for the image list.

Artifacts

The OCM build job uploads these files to the GitHub Actions workflow run:

Artifact Description
ocm/ctf.tar OCM component archive built from the component descriptor
ocm/ctf.tar.sha256 Checksum for the component archive
ocm/ctf-bundled.tar Optional archive with copied image resources, created for manual runs with bundling enabled
ocm/ctf-bundled.tar.sha256 Optional checksum for the bundled archive
source-sbom.cdx.json CycloneDX SBOM of the repository source tree generated by Syft
sbom-image-*.cdx.json CycloneDX SBOM generated by Syft for each delivered container image
trivy-*.sarif CVE and configuration scan reports

Artifacts are retained for 30 days by default.

Registry publishing is intentionally not configured in the pull-request CI workflow. If the package should be published later, add a separate guarded workflow or job.

The publishing workflow should run only on trusted events such as pushes to main, release tags, or manual dispatch.

Provide the target OCI registry and credentials through repository variables or secrets.

Required repository settings

The workflow can run without custom secrets for pull-request validation. Deployment smoke tests require a Kubernetes test cluster and one configured cluster authentication mode.

Name Type Required Purpose
KUBE_AUTH_MODE Variable No Cluster authentication strategy: disabled, kubeconfig (default), or eks-oidc
KUBECONFIG Secret For kubeconfig mode Raw kubeconfig content for the test cluster
AWS_REGION Variable For eks-oidc mode AWS region containing the EKS cluster
EKS_CLUSTER_NAME Variable For eks-oidc mode EKS cluster name used to generate kubeconfig at runtime
AWS_ROLE_TO_ASSUME Variable For eks-oidc mode IAM role assumed through GitHub OIDC before generating kubeconfig
KUBE_AUTH_ALLOW_PR_DEPLOY Variable No Set to true only when the selected auth strategy and trust policy intentionally allow pull-request deployments; defaults to false
CI_NAMESPACE Variable No Namespace used by deployment tests; defaults to dns
RUN_NETWORK_SMOKE Variable No Set to true when the test cluster has reachable LoadBalancer DNS access
RUN_CVE_SCAN Variable No true (default) runs the image and configuration CVE scans and per-image SBOM generation; set to false to skip them, for example during fast iteration on deployment-only changes
VALIDATION_SUITES Variable No Comma-separated subset of cluster validation suites to run (operator, lmdb, replication, observability, health-probes, rolling-update, multi-instance); empty (default) runs all suites
TRIVY_SEVERITY Variable No CVE severities to scan; defaults to HIGH,CRITICAL
TRIVY_EXIT_CODE Variable No Image scan exit code on findings; defaults to 1 (blocking). Reviewed upstream findings are time-boxed in .trivyignore.yaml
TRIVY_CONFIG_EXIT_CODE Variable No Configuration scan exit code on findings; defaults to 0

Image CVE scans fail the workflow on HIGH/CRITICAL findings. Fix-available findings in upstream base images and third-party modules that have no rebuilt upstream image yet are suppressed through a reviewed, time-boxed allowlist in .trivyignore.yaml; each entry carries a justification and an expired-at date so it is re-triaged on expiry. The plain .trivyignore file holds misconfiguration suppressions for the configuration scan.

Branch protection

The main branch requires a pull request with at least one approving review and successful GitHub Actions checks before merge. The enforced required checks are the stable, single-instance jobs:

Required check Reason
1. Quality checks Blocks malformed manifests, scripts, workflows, and potential secrets
2. Build OCM package Ensures the OCM archive remains buildable and valid
3c. Config security scan Reports Kubernetes and workflow configuration risks
3e. CVE scan gate Aggregates the per-image CVE scan matrix into one stable gate

The image CVE scans run as a per-image matrix with dynamic check names, so they are required indirectly through the 3e. CVE scan gate job, which fails if any image scan fails. The image scans are blocking (TRIVY_EXIT_CODE=1 by default), so any new HIGH/CRITICAL finding outside the time-boxed .trivyignore.yaml allowlist turns the gate red. The per-image SBOM jobs and the cluster-conditional deployment and upgrade smoke tests are reported on each pull request but are not enforced merge gates.

Protection also requires stale-review dismissal, conversation resolution, blocks force pushes and branch deletion, and disallows direct pushes to main. Administrators are not forced through the gate, so a maintainer can still merge over a flaky conditional smoke run.

The policy is encoded as code in hack/setup-branch-protection.sh and can be reapplied or reproduced in another environment with the GitHub CLI:

REPO=owner/name BRANCH=main APPROVALS=1 ./hack/setup-branch-protection.sh

Handling CVE findings and renewing the allowlist

The image CVE scan gate turns red in two cases: a new HIGH/CRITICAL, fix-available finding appears in one of the scanned images, or an existing .trivyignore.yaml entry reaches its expired-at date and is re-triaged. Work through the following procedure; always prefer fixing over suppressing.

1. Reproduce the finding locally. The image list is the set of imageReference entries in ocm/component-descriptor.yaml. Scan each image with the same settings the pipeline uses and the current allowlist applied:

trivy image --severity HIGH,CRITICAL --ignore-unfixed --scanners vuln \
  --ignorefile .trivyignore.yaml --exit-code 1 <imageReference>

Drop --ignorefile to see the full, unsuppressed list of findings for an image.

2. Prefer a fix — bump to a clean image. Check whether the pinned tag has been rebuilt, or whether a newer patch tag scans clean:

  • Re-pull the pinned tag; if its digest changed, the upstream image was rebuilt — re-pin to the new digest.
  • Otherwise test the next patch tag (for example pdns-auth 4.9.15 → 4.9.16) and keep it only if it scans with zero findings.

When bumping an image, update the pin (tag and @sha256: digest) in every place it appears, so the manifests stay consistent with the component descriptor:

  • ocm/component-descriptor.yaml (both the version field and the imageReference),
  • the base manifest under deploy/base/<component>/deployment.yaml,
  • the matching line(s) in deploy/kro/powerdns-instance-rgd.yaml (must stay byte-identical to the base on the image line),
  • any tag references in the docs image table (docs/AIR-GAP-DEPLOYMENT.md) and version mentions (docs/UPGRADE.md, ADRs),
  • for the Authoritative image, the tag variables in hack/build-upgrade-packages.sh (REPO_AUTH_TAG, CANDIDATE_AUTH_TAG), otherwise the upgrade smoke job's version assertion breaks.

3. Suppress only the irreducible remainder. If no rebuilt or newer clean image exists yet — typically findings compiled into a third-party binary (statically-linked Go modules) — add or renew a time-boxed entry in .trivyignore.yaml. Each entry needs an id, a statement justifying why the finding is not exploitable in this deployment, and a short-dated expired-at:

vulnerabilities:
  - id: CVE-YYYY-NNNNN
    statement: >-
      Why this finding is not exploitable here (which module/base image it
      lives in, and why the vulnerable code path is not reached). Fixed by
      the next upstream rebuild.
    expired-at: 2026-08-05

Keep the window short (a few weeks) so every suppression is re-checked regularly, and update the header comment's review date. The plain .trivyignore file is separate — it holds misconfiguration suppressions for the configuration scan, not image CVEs.

4. Verify before committing. Re-run the step-1 command for all images with the updated allowlist and confirm each returns exit code 0. Then open a pull request; the 3e. CVE scan gate re-runs the full matrix and must be green before merge.

Deployment smoke tests

Until a test cluster is available, set KUBE_AUTH_MODE=disabled or leave the default kubeconfig mode without a KUBECONFIG secret; the deployment job then completes successfully with a summary explaining that it was skipped.

The deployment job supports interchangeable authentication strategies:

Mode How kubeconfig is prepared Typical use
disabled No kubeconfig is created; deployment smoke tests are skipped Repositories without a test cluster
kubeconfig Writes the KUBECONFIG secret to ~/.kube/config Generic Kubernetes clusters or legacy setup
eks-oidc Assumes AWS_ROLE_TO_ASSUME via GitHub OIDC and runs aws eks update-kubeconfig EKS clusters without static AWS credentials

After authentication, the workflow uses only kubectl, so the deployment and smoke-test steps remain independent of the selected cluster provider. When cluster authentication is configured, the job applies the manifests and runs:

bash hack/validate-crds.sh --cleanup
bash hack/validate-operator.sh --cleanup
bash hack/validate-status.sh --cleanup

The quality job always validates the metrics manifest contract and renders the optional Prometheus Operator overlay. When a cluster is configured, the observability validation suite installs or reuses Prometheus Operator, deploys the three packaged ServiceMonitors and a CI-only Prometheus instance, checks the live Recursor, Authoritative Server, and Operator /metrics endpoints, and requires all three Prometheus targets to report health=up. Selecting another explicit suite list without observability skips these additional monitoring resources.

During deployment, the workflow annotates pod templates with checksums of the mounted configuration and secret manifests. This triggers Kubernetes rollouts only when those inputs change, instead of restarting every workload on every run.

When external DNS access is available through the LoadBalancer service, set RUN_NETWORK_SMOKE=true to also run:

bash hack/validate-network-access.sh dns

When the multi-instance suite is selected (VALIDATION_SUITES includes multi-instance, or the variable is empty so all suites run), KRO creates additional per-instance namespaces such as pdns-mi-a and pdns-mi-b alongside the default dns namespace. Make sure the Kubernetes identity used by the CI workflow (the EKS Access Entry bound to the AWS_ROLE_TO_ASSUME role in eks-oidc mode, or the user inside the KUBECONFIG secret in kubeconfig mode) can read and write resources in every namespace that the bundle may create, not just dns. The simplest dev-environment setup binds the role to a cluster-admin policy; tighten this for production deployments to a namespace prefix wildcard or an aggregated ClusterRole that covers the bundle's resource groups.