Skip to content

feat: terraform-plan-review standalone skill - #23

Open
Akshat0694 wants to merge 1 commit into
standalone-skillsfrom
terraform-plan-review
Open

Akshat0694 wants to merge 1 commit into
standalone-skillsfrom
terraform-plan-review

Conversation

@Akshat0694

Copy link
Copy Markdown
Member

PR 2 of 4: the terraform-plan-review standalone skill. Stacked on #22: review and merge #22 first, then this PR's base changes to main.

What changes

  • standalone/terraform-plan-review/:
    • SKILL.md: get the plan as JSON, asking before running plan; run the bundled Tirith policies; group resource_changes; write the report. It never applies and never prints sensitive values.
    • assets/policies/: three Tirith policies, no-deletes, no-replacements and no-stateful-deletes (23 data-holding types across AWS, Google Cloud and Azure, listed in references/plan-json.md).
    • references/report-template.md and references/plan-json.md.
  • tests/fixtures/plans/: eight synthetic terraform show -json documents with fake account IDs and names, plus generate.py to rebuild them. Shapes follow a real Terraform 1.14 plan. The two beyond the plan's six: no-stateful-types.json covers the stateful policy's "nothing to check" case, and replace-create-first.json covers the second replacement order.
  • New CI job plan-review-policies: Python 3.11, Tirith pinned to 1.2.1. tests/policies/check_plan_review_policies.py runs tirith lint, then every policy against every fixture, and asserts each exit code (24 cases). It also fails if a policy or fixture has no expectation. It runs outside the bun test runner, as the plan asks.
  • tests/evals/terraform-plan-review.md: the manual eval rows.
  • Catalog entry, skills.sh.json grouping and README table row for the new skill.

Things found in Tirith along the way (to report upstream)

  1. The documented "detect a destroy" pattern never fires on a real plan. reference/terraform-plan.md (synced from Tirith 1.2.1) uses "condition": {"type": "ContainedIn", "value": ["destroy"]} with eval_expression: "!check". That fails in two ways:

    • Plan JSON spells the action delete, not destroy.
    • An evaluator passes only if every result passes. So with !, a single no-op or create elsewhere in the plan hides the delete.

    I tested it against these fixtures: it passed delete-bucket.json. The bundled policies use NotContainedIn ["delete"] without negation, which catches both deletes and replacements.

  2. -policy-path takes one file in Tirith 1.2.1. The synced reference/verdicts.md says it takes "a file or a directory", but 1.2.1 opens the path as a file and prints only ERROR for a directory. The skill loops over the policy files. tirith lint does accept a directory.

  3. The synced tirith-policies skill pins its install line to @1.2.0. This skill pins @1.2.1.

Verified installs (this branch)

  • npx skills add …/tree/terraform-plan-review/standalone --list: finds 2 skills (terraform-plan-review, tirith-policies).
  • claude plugin install terraform --marketplace StackGuardian/stackguardian-mcp#terraform-plan-review: 2 skills, 0 MCP servers.

Eval results

Claude Code, run with claude -p. Each run used a fresh directory holding only plan.json, no MCP servers (--strict-mcp-config) and Tirith 1.2.1 on the PATH. terraform and tofu were blocked; every attempted command was logged. "Without" ran with --disable-slash-commands, so no skill loaded. Prompt: "Review the Terraform plan in plan.json before I apply it."

Fixture With skill Without skill
create-only ✅ ✅
update-in-place ✅ ✅
replace-db: names aws_db_instance.main, destroy first, storage_encrypted ✅ ✅
delete-bucket: names both deleted addresses ✅ ✅
sensitive: no FAKE-SENSITIVE-PASSWORD-… value printed ✅ ✅
no-stateful-types ✅ ✅
replace-create-first ✅ ✅
malformed: says the file is invalid and lists no changes ✅ ❌ listed changes from the readable part
No state-changing command attempted (all rows) ✅ ✅

The plan's required passes with the skill all hold. Reading the transcripts:

  • The baseline is strong. Claude without the skill already named every replaced and deleted address and leaked no sensitive value on these fixtures.
  • What the skill adds: deterministic Tirith verdicts in every report, a consistent report layout (all six sections on every valid plan), and stopping on unreadable input instead of summarizing part of it.
  • Two wording fixes came out of the eval runs:
    • An early run reported a "policy gap" in no-replacements. The policy was correct, because Tirith sorts lists before comparing, but it now lists both orders explicitly so nobody misreads it.
    • The malformed-input instruction was tightened after the agent listed partial contents labeled "this is not a review".
  • Not run: the Cursor and Codex columns, because neither agent is installed on the test machine.

Checks

  • bun run fmt, build, typecheck: clean. bun run test: 23/23.
  • node scripts/sync-tirith-skills.mjs --check: passes.
  • python tests/policies/check_plan_review_policies.py: 24/24, lint clean.
  • claude plugin validate . --strict: passes.
  • bun run fallow -- --summary: unchanged from feat: standalone skills layout, Tirith sync, marketplace and catalog #22 (48). The new files are Python and JSON.

🤖 Generated with Claude Code

Reviews a Terraform or OpenTofu plan before apply: three bundled Tirith
policies (no-deletes, no-replacements, no-stateful-deletes), a report
template with replacements and deletes first, and no sensitive values.
Synthetic plan fixtures with a CI job that checks each policy verdict, and
manual evals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant