Split bootstrap Terraform per environment to isolate staging and production IAM #5077

Summary

The bootstrap phase runs on every staging and production deploy, but a single Terraform root (infrastructure/bootstrap/) manages both environments in one state file via for_each = local.environments (default: ["staging", "production"]).

A staging deploy therefore updates production IAM roles and policies (nest-production-terraform, associated policies), and vice versa. The workflow environment input is used only for GitHub Environment approval — it is not passed to Terraform.

We should split bootstrap into per-environment Terraform roots with separate state, matching the pattern used by infrastructure/live/, so each pipeline only applies IAM bootstrap for its own environment.

Problem

Current behavior

ci-cd-staging.yaml  ──┐
                      ├──► run-ci-cd.yaml
ci-cd-production.yaml ┘         │
                                ▼
                    run-infrastructure-bootstrap.yaml
                                │
                                ▼
                    infrastructure/bootstrap/   (single root)
                    environments = ["staging", "production"]
                    state key: bootstrap/terraform.tfstate
                                │
                                ▼
                    Updates BOTH nest-staging-terraform
                            AND nest-production-terraform

Root causes

1. infrastructure/bootstrap/variables.tf defaults environments to ["staging", "production"].
2. All bootstrap resources use for_each = local.environments (main.tf, iam_policy_size_checks.tf).
3. run-infrastructure-bootstrap.yaml does not pass environment to tfvars — only aws_region, project_name, aws_role_external_id.
4. Single state file (bootstrap/terraform.tfstate in BOOTSTRAP_TF_STATE_BUCKET_NAME) holds both environments.
5. Shared aws_role_external_id is applied to every role's trust policy in one apply — if staging and production GitHub Environments use different ExternalIds, a staging bootstrap run can overwrite production's trust policy.

Impact

• Staging nightly deploy modifies production IAM without a production release.
• IAM policy changes intended for one environment affect the other.
• Larger blast radius and harder auditing ("who changed production IAM?").
• Violates environment isolation expected from separate GitHub Environments and deploy pipelines.

What works correctly today (for comparison)

• infrastructure/live/ is scoped per environment: tf_state_key: staging/terraform.tfstate vs production/terraform.tfstate, and nest-${{ inputs.environment }}-terraform role assumption.
• Bootstrap should follow the same model.

Proposed solution

Split bootstrap into separate Terraform roots per environment with isolated state and environment-scoped CI/CD applies.

Target architecture

infrastructure/modules/bootstrap-iam/     # shared module (single environment)
infrastructure/bootstrap-staging/         # root: staging IAM only
infrastructure/bootstrap-production/      # root: production IAM only

State (same bootstrap bucket, separate keys):
  staging/bootstrap/terraform.tfstate
  production/bootstrap/terraform.tfstate

CI/CD:
  staging pipeline    → apply bootstrap-staging only
  production pipeline → apply bootstrap-production only

Module extraction

Refactor current infrastructure/bootstrap/main.tf into a reusable module (e.g. infrastructure/modules/bootstrap-iam/):

• Remove for_each = local.environments — module takes a single environment input.
• Keep existing IAM policy documents, size checks, role, and policy attachments — parameterized by environment.
• Each environment deploy calls the module with its environment name.

This avoids duplicating large IAM policy documents while achieving state isolation.

Per-environment roots

Each root contains:

• backend.tf — distinct state key
• main.tf — module invocation with environment = "staging" or "production"
• variables.tf, providers.tf, outputs.tf
• terraform.tfbackend.example, terraform.tfvars.example
• tests/ — env-specific tftest (policy size checks for that environment only)

Documentation updates

• infrastructure/bootstrap/README.md → split or replace with per-env docs
• infrastructure/README.md — manual bootstrap setup steps
• infrastructure/live/README.md — remove implication that any deploy updates both roles
• infrastructure/state/README.md — clarify bootstrap state key layout if bucket/key conventions change

Acceptance criteria

• Staging CI/CD bootstrap apply touches only nest-staging-terraform and its policies.
• Production CI/CD bootstrap apply touches only nest-production-terraform and its policies.
• Separate Terraform state per environment (no shared for_each bootstrap state).
• aws_role_external_id is scoped per environment (no cross-env trust policy overwrite).
• Infrastructure tests pass for both bootstrap roots.
• Documentation reflects per-environment bootstrap lifecycle.

Non-goals

• Changing infrastructure/live/ deploy scoping (already correct).
• Splitting infrastructure/state/ (separate concern; state buckets already per environment).
• Migrating nest-bootstrap IAM user to OIDC (related, separate issue).
• Removing bootstrap from every deploy.