feat!: replace v6 broker deployment with OIDC and no Parameter Store access

.github/workflows/security-tests.yml

@@ -0,0 +1,22 @@
+name: Deployment security tests
+on:
+  pull_request:
+  push:
+    branches: [master]
+permissions:
+  contents: read
+jobs:
+  offline-tests:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        with:
+          persist-credentials: false
+      - run: python3 -m unittest discover -s tests -v
+      - name: Check shell syntax
+        run: |
+          python3 - <<'PY'
+          import pathlib, subprocess
+          for path in pathlib.Path('.').rglob('*.sh'):
+              subprocess.run(['bash', '-n', str(path)], check=True)
+          PY

.gitignore

@@ -8,4 +8,12 @@ target/*
 *.iml
 /target/
 .settings
-*.swp
+*.swp
+__pycache__/
+*.pyc
+awsenvconf
+*_env
+.env
+.env.*
+.npmrc
+*.tar

README.md

@@ -1 +1,55 @@
-#Build Script - TC
+# Topcoder deployment scripts v2
+
+Deploy prebuilt images to an **existing** ECS/Fargate service without giving the deployment role access to Parameter Store or Secrets Manager. CircleCI OIDC v3 exchanges a short-lived job identity for an AWS STS session; the Auth0 client-secret broker is no longer used.
+
+**This is a breaking migration, not a drop-in v1 patch.** Do not repoint existing v1 tags. Pipelines that source `awsenvconf`, `deployvar_env`, or `buildvar_env` must migrate with their IAM roles. Old interfaces fail before network calls. EBS, S3, Lambda and secret-provisioning modes are not supported in v2. Historical `direct/` assets are outside the v2 deployer.
+
+Read [the migration guide](docs/MIGRATION.md), [the Auth0/AWS explanation](docs/AUTHENTICATION.md), and [the 19-service adoption checklist](docs/SERVICES.md) before testing.
+
+## Requirements
+
+- A reviewed, digest-pinned Linux executor containing Python >=3.10 with the `tarfile` extraction-filter security backport, AWS CLI v2, Docker/Buildx, Git and system CA certificates. No package installation in the credential-bearing job. This deployer has no external Python dependencies.
+- CircleCI Cloud with `CIRCLE_OIDC_TOKEN_V3`, protected exact development (`develop` or `dev`) and production (`master` or legacy `prod`) refs, and restricted deployment contexts containing no broker/static AWS credentials.
+- Separate per-service/per-environment deploy, task and execution roles, and a reviewed non-secret manifest.
+- Existing immutable ECR repositories with scan-on-push, ECS circuit-breaker rollback, positive desired capacity, explicit task health checks, and pre-provisioned runtime secret references.
+
+## Usage
+
+Copy `examples/deploy.dev.json` / `examples/deploy.prod.json` into the service repository and replace the example identifiers with administrator-verified values. These are placeholders, not guessed Topcoder targets. Generate IAM documents offline:
+
+```sh
+python3 -I tcdeploy.py policies --manifest deploy/dev.json > policies.json
+```
+
+An administrator installs the generated `trust` and `permissions` documents through reviewed infrastructure changes. Secret/KMS read permissions belong on the narrowly scoped execution role, never the deployment role.
+
+From the service checkout, in a build job without deployment context:
+
+```sh
+python3 -I /tmp/tc-deploy-scripts/build_image.py --image example-v6:candidate
+docker save --output /tmp/image.tar example-v6:candidate
+```
+
+The helper builds `git archive HEAD` in a temporary context, rejects credential/secret-file inputs and symlinks, accepts PEM files only when they contain public certificates, strips credentials from subprocess environments, and provides no secret/build-argument options. For nested Dockerfiles use `--dockerfile docker/Dockerfile`; for a sidecar use `--dockerfile clamav/Dockerfile --context clamav`. `--target migrate` can build a migration image; executing migrations remains a separately controlled operation.
+
+In a separately authorized deploy job, load the workflow-specific image archive, then:
+
+```sh
+/tmp/tc-deploy-scripts/awsconfiguration.sh --manifest deploy/dev.json
+/tmp/tc-deploy-scripts/master_deploy.sh --manifest deploy/dev.json \
+  --image example-v6=example-v6:candidate
+```
+
+`awsconfiguration.sh` is an optional identity check, not a credential exporter. `master_deploy.sh` authenticates internally. Neither writes AWS credentials to the checkout/default profile, logs, or shell exports. Use the entire SHA-pinned suite, not the old four copied shell files. See `examples/circleci-v2.yml`; replace its explicit executor/commit placeholders before use.
+
+The deployer verifies account/role, live service configuration, source label, ECR immutability, local/remote configuration digest, a scan completed within 24 hours with no MEDIUM/HIGH/CRITICAL/UNDEFINED findings, and the exact healthy running revision/digests. Only selected images change; roles, secret references, sidecars, resources, volumes and health checks remain intact. Non-root runtime images are required and `RESET_DB` cannot be enabled. The source label is a consistency check, not cryptographic build provenance; protect the pipeline and artifact writers.
+
+Use a unique CircleCI build number when retrying publication to immutable tags. A failed rollout remains a failure even if ECS rolls back. The script never blindly reverts over another deployment.
+
+## Tests
+
+```sh
+python3 -m unittest discover -s tests -v
+```
+
+Tests mock AWS/Docker and require no credentials, account, network or external Python packages. Passing them does not replace IAM simulation and a real dev rollout. GitHub CI runs them with read-only repository permissions.

awsconfiguration.sh

@@ -1,47 +1,10 @@
-#!/bin/bash
-AWSENV=$1
-AWS_REGION=$2
-BASE64_DECODER="base64 -d" # option -d for Linux base64 tool
-echo AAAA | base64 -d > /dev/null 2>&1 || BASE64_DECODER="base64 -D" # option -D on MacOS
-decode_base64_url() {
-  local len=$((${#1} % 4))
-  local result="$1"
-  if [ $len -eq 2 ]; then result="$1"'=='
-  elif [ $len -eq 3 ]; then result="$1"'='
-  fi
-  echo "$result" | tr '_-' '/+' | $BASE64_DECODER
-}
-
-if [ -z "$AWS_REGION" ];
-then
-    AWS_REGION="us-east-1"
+#!/usr/bin/env bash
+# Authentication check only. Credentials are never exported to the caller.
+set +x
+set -euo pipefail
+script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+if [[ "${1:-}" != --manifest ]]; then
+  echo 'The Auth0 broker/awsenvconf interface has been removed. Use --manifest FILE to check OIDC identity; master_deploy.sh authenticates internally. See docs/MIGRATION.md.' >&2
+  exit 2
 fi
-auth0cmd=$(echo "curl -k -X POST $CI_AUTH0_URL -H 'Content-Type: application/json' -d '{ \"client_id\": \"$CI_AUTH0_CLIENTID\", \"client_secret\": \"$CI_AUTH0_CLIENTSECRET\", \"audience\": \"$CI_AUTH0_AUDIENCE\", \"grant_type\": \"client_credentials\" , \"environment\" : \"$AWSENV\" , \"username\" : \"$CIRCLE_PROJECT_USERNAME\" , \"reponame\" : \"$CIRCLE_PROJECT_REPONAME\", \"build_num\": \"$CIRCLE_BUILD_NUM\", \"branch\": \"$CIRCLE_BRANCH\"}'")
-token=$( eval $auth0cmd | jq -r .access_token )
-tokenjsonformat=$( decode_base64_url $(echo -n $token | cut -d "." -f 2) )
-AWS_ACCESS_KEY_ID=$(echo $tokenjsonformat | jq -r . | grep AWS_ACCESS_KEY | cut -d '"' -f 4)
-AWS_SECRET_ACCESS_KEY=$(echo $tokenjsonformat | jq -r . | grep AWS_SECRET_KEY | cut -d '"' -f 4)
-AWS_ENVIRONMENT=$(echo $tokenjsonformat | jq -r . | grep AWS_ENVIRONMENT | cut -d '"' -f 4)
-AWS_SESSION_TOKEN=$(echo $tokenjsonformat | jq -r . | grep AWS_SESSION_TOKEN | cut -d '"' -f 4)
-AWS_ACCOUNT_ID=$(echo $tokenjsonformat | jq -r . | grep AWS_ACCOUNT_ID | cut -d '"' -f 4)
-aws configure set default.region $AWS_REGION
-aws configure set default.output json
-aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
-aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
-aws configure set aws_session_token $AWS_SESSION_TOKEN
-
-echo "">awsenvconf
-echo "export AWS_REGION=\"$AWS_REGION\"">>awsenvconf
-echo "export AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\"">>awsenvconf
-echo "export AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\"">>awsenvconf
-echo "export AWS_ENVIRONMENT=\"$AWS_ENVIRONMENT\"">>awsenvconf
-echo "export AWS_SESSION_TOKEN=\"$AWS_SESSION_TOKEN\"">>awsenvconf
-echo "export AWS_ACCOUNT_ID=\"$AWS_ACCOUNT_ID\"">>awsenvconf
-
-if grep -Fxq "awsenvconf" .dockerignore
-then
-    echo "awsenvconf exist in docker ignore file list"
-else
-    echo "awsenvconf" >> .dockerignore
-fi
-
+exec python3 -I "$script_dir/tcdeploy.py" check-auth "$@"

build_image.py

@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""Build committed source in a clean context without deployment credentials.
+
+Run in a separate build job, before attaching any deployment context. No package
+installation or repository-defined build command executes on the credential host.
+"""
+import argparse
+import os
+from pathlib import Path
+import re
+import subprocess
+import sys
+import tarfile
+import tempfile
+
+
+def run(command, env):
+    return subprocess.run(command, env=env, capture_output=True, check=True)
+
+
+def forbidden(path):
+    parts = Path(path).parts
+    name = parts[-1]
+    return (any(p in ('.aws', '.ssh', '.git') for p in parts)
+            or name in ('awsenvconf', 'buildenvvar', '.npmrc', '.netrc', 'fetched_parameters.json',
+                        'paramnames.json', 'paramwithvalue.json', 'cli-input.json', 'upload_object.json')
+            or name.endswith(('_env', '.key', '.p12', '.pfx'))
+            or name == '.env' or (name.startswith('.env.') and name not in ('.env.example', '.env.sample')))
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--image', required=True)
+    parser.add_argument('--dockerfile', default='Dockerfile')
+    parser.add_argument('--context', default='.')
+    parser.add_argument('--target')
+    args = parser.parse_args(argv)
+    if not re.fullmatch(r'[a-z0-9][a-z0-9_.:/-]+', args.image):
+        raise ValueError('Invalid local image tag')
+    for value in (args.dockerfile, args.context):
+        if Path(value).is_absolute() or '..' in Path(value).parts:
+            raise ValueError('Build paths must stay within the checkout')
+    if args.target and not re.fullmatch(r'[A-Za-z0-9_-]+', args.target):
+        raise ValueError('Invalid build target')
+    env = {'PATH': '/usr/local/bin:/usr/bin:/bin', 'LANG': 'C.UTF-8'}
+    for key in ('DOCKER_HOST', 'DOCKER_TLS_VERIFY', 'DOCKER_CERT_PATH'):
+        if key in os.environ:
+            env[key] = os.environ[key]
+    sha = run(['git', 'rev-parse', 'HEAD'], env).stdout.decode().strip()
+    if os.environ.get('CIRCLE_SHA1') and os.environ['CIRCLE_SHA1'] != sha:
+        raise ValueError('Checkout differs from CircleCI source SHA')
+    with tempfile.TemporaryDirectory(prefix='tc-build-') as temp:
+        archive = Path(temp) / 'source.tar'
+        run(['git', 'archive', '--format=tar', '--output=' + str(archive), 'HEAD'], env)
+        context = Path(temp) / 'context'
+        context.mkdir()
+        with tarfile.open(archive) as source:
+            members = source.getmembers()
+            if any(forbidden(m.name) or not (m.isfile() or m.isdir()) for m in members):
+                raise ValueError('Committed source includes credentials, secret-file names, symlinks or unsupported archive entries; sanitize before building')
+            for member in members:
+                if member.isfile() and member.name.endswith('.pem'):
+                    content = source.extractfile(member).read()
+                    if not re.fullmatch(rb'(?:\s*-----BEGIN CERTIFICATE-----[A-Za-z0-9+/=\s]+-----END CERTIFICATE-----\s*)+', content):
+                        raise ValueError('PEM build inputs must contain public certificates only')
+            source.extractall(context, filter='data')
+        env['DOCKER_CONFIG'] = str(Path(temp) / 'docker-config')
+        Path(env['DOCKER_CONFIG']).mkdir(mode=0o700)
+        command = ['docker', 'buildx', 'build', '--load', '--pull', '--no-cache',
+                   '--provenance=false', '--platform', 'linux/amd64',
+                   '--label', 'org.opencontainers.image.revision=' + sha,
+                   '--tag', args.image, '--file', str(context / args.dockerfile)]
+        if args.target:
+            command += ['--target', args.target]
+        command += [str(context / args.context)]
+        # No build-arg or secret option exists: fetch runtime values only in ECS.
+        # Build logs are suppressed on failure to avoid dumping accidental secrets.
+        run(command, env)
+    print('Built committed source ' + sha + ' as ' + args.image)
+
+
+if __name__ == '__main__':
+    try:
+        main()
+    except (ValueError, OSError, subprocess.SubprocessError, tarfile.TarError):
+        print('Build failed; check committed build inputs and secret-file exclusions. No deployment credentials were passed to Docker.', file=sys.stderr)
+        sys.exit(1)