[wmobley]

Why

On 2026-11-03 two composite runs were held by the Tapis Workflows engine for 4.6 h and 29 h before their model task launched. By then the 4 h per-run token in the run args had expired, and the task died on its first Tapis call with MODEL_PROVIDER_AUTH_FAILED. The task's own renewal (_renew_if_due) cannot help: it runs inside the task and needs a live token. Design: docs/design/2026-10-28-svo-adapter-workflow-token-lifecycle.md, rev 2 (security-reviewer, skeptic and QA revisions recorded in the spec).

What

• Encrypted at-rest copy of the per-run token on adapter.workflow_run (run_token_enc, Fernet via MultiFernet for rotation, ciphertext bound to the run id), plus run_token_expires_at, run_token_deadline_at, run_token_renewals. Migration 1771300000035_adapter_workflow_run_token_keeper.
• POST /runs/{run_id}/token: hands the model task the current token. Authenticated by a per-run HMAC credential in its own domain (run-token:v1:, derived from SVO_ADAPTER_RUN_SIGNAL_SECRETS; a signal token cannot fetch and vice versa). Uniform 401 for bad credential / unknown run, 404 RUN_TOKEN_UNAVAILABLE outside running/submitting or past the 72 h deadline, per-run rate limit, Cache-Control: no-store. Mints a successor on the way out when the stored token is within the renewal margin and revokes the predecessor (POST /v3/tokens/revoke, verified on portals).
• Bounds and cleanup: hard deadline from dispatch, renewal cap, revoke + clear on terminal status, cancel, failed dispatch, deadline, decrypt failure; rows stuck in cancelling are also cleared. Optional background sweep (SVO_ADAPTER_RUN_TOKEN_KEEPER_SWEEP_ENABLED) for waits longer than one TTL.
• Model task: fetches a live token before its first Tapis call when the baked token is expired or near expiry, and once on a 401 (never 403). On a 401 from /v3/jobs/submit it re-checks _existing_job() by name before resubmitting, so a late start never duplicates a job. Fails with RUN_TOKEN_EXPIRED_BEFORE_START when it cannot recover.
• Fail closed: keeper on without a usable key, run signals, or an https public URL stops the service at startup. Off by default.
• Deploy wiring: register_mint_stack.py --sync-svo-runtime manages SVO_ADAPTER_RUN_TOKEN_* only when the variable/secret is set, and refuses to enable the keeper without a key.

Verified live before coding

• The Workflows executor reaches the adapter pod (GET /health from a canary task, 0.11 s).
• A user access token mints its successor via POST /v3/tokens; POST /v3/tokens/revoke works.
• Run rows are running from dispatch, so the keeper's status filter covers the engine-wedge case.

Tests

svo-adapter-service: 233 passed. New tests/test_run_token_keeper.py covers crypto round trip and key rotation, config fail-closed, credential domains, endpoint auth / 404 / 429 / renew / cap / CAS loss, sweep, release, and runs the generated model task against a stubbed Tapis for late start, no-keeper failure code, and a mid-run 401 without a duplicate job. test_run_signal.py covers submit-time storage, arming, and failed-dispatch release. The 5 failures in test_demo_api.py and test_register_existing_etl_pieces.py pre-exist on develop.

Rollout

1. Merge: the keeper is off; only the migration and the per-run token TTL wiring take effect.
2. Set env secret SVO_ADAPTER_RUN_TOKEN_ENCRYPTION_KEYS (Fernet key) and env var SVO_ADAPTER_RUN_TOKEN_KEEPER_ENABLED=true in Tapis Dev Deploy, re-run the dev-pod deploy.
3. Acceptance: submit a composite run with the minimum TTL (900 s) and a held-back task; the task must fetch and complete, run.token.fetched recorded. Then lower SVO_ADAPTER_RUN_TOKEN_TTL_SECONDS back from the interim 48 h.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
monorepo	Ready	Preview	Nov 3, 2026 9:26pm UTC