Skip to content

serve: exchange web identity tokens at a configurable STS endpoint - #616

Open
ekohlwey wants to merge 1 commit into
Tinder:masterfrom
ekohlwey:serve-s3-sts-endpoint
Open

ekohlwey wants to merge 1 commit into
Tinder:masterfrom
ekohlwey:serve-s3-sts-endpoint

Conversation

@ekohlwey

@ekohlwey ekohlwey commented Nov 3, 2026

Copy link
Copy Markdown

Fixes #614.

Problem

serve's S3 tier resolves credentials once with Credentials::default(). With AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE set, aws-creds sends the token to https://sts.amazonaws.com/, which is hardcoded in aws-creds 0.38 and 0.40. S3-compatible stores that serve STS themselves (SeaweedFS, MinIO) can't be reached, so a pod with a Kubernetes ServiceAccount token and no static keys fails at startup with failed to resolve AWS credentials. --s3Endpoint changes only where objects are read and written.

Change

  • New --s3StsEndpoint flag, which also reads the standard AWS_ENDPOINT_URL_STS. When it is set, serve sends AssumeRoleWithWebIdentity to that endpoint as a form POST, with AWS_ROLE_ARN, the token from AWS_WEB_IDENTITY_TOKEN_FILE, and AWS_ROLE_SESSION_NAME (default bazel-diff).
  • The session is renewed five minutes before it expires, and the token file is re-read on each renewal because kubelet rotates projected tokens. Concurrent requests wait on a single exchange.
  • A failed first exchange fails startup, as an unresolvable default chain does today. A failed renewal later counts as a cache miss or an unshared write, like any other S3 error.
  • The credentials handed to rust-s3 carry no expiration. Otherwise rust-s3 would replace them, once expired, from the default chain, which goes back to sts.amazonaws.com.
  • Without an STS endpoint, credentials resolve exactly as before.
  • quick-xml 0.32 becomes a direct dependency to parse the STS answer. It was already in Cargo.lock through aws-creds.
  • RemoteCache::S3 is now boxed like the GCS and Nexus variants, because the session state tripped clippy::large_enum_variant.
  • README and tools/readme_template.md document the flag.

Testing

bazelisk test //:rust_tests //:rust_clippy_check //:rust_format_check //tools/... passes on macOS arm64 (Bazel 8.8.2). New tests:

  • web_identity::tests, against a local STS on tiny_http: the form POST contents, reuse of a session that is not near expiry, renewal of an expiring session with a rotated token, a refused exchange, an answer without credentials, a missing token file, a non-HTTP endpoint, and the renewal margin boundary.
  • server::tests::s3_cache_signs_with_the_web_identity_session: an S3 GET through S3Cache carries the session token from the STS exchange.

I checked the README help text against bazel-diff serve --help.

The S3 tier resolved credentials once through aws-creds, which sends AssumeRoleWithWebIdentity only to sts.amazonaws.com. S3-compatible stores such as MinIO and SeaweedFS serve STS themselves. With --s3StsEndpoint (or AWS_ENDPOINT_URL_STS) set, serve exchanges AWS_WEB_IDENTITY_TOKEN_FILE for AWS_ROLE_ARN's credentials there and renews the session five minutes before it expires, re-reading the token file each time.

Fixes Tinder#614
@ekohlwey

ekohlwey commented Nov 3, 2026

Copy link
Copy Markdown
Author

This change is part of the following stack:

Change managed by git-spice.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

serve: S3 tier cannot use web identity against an S3-compatible STS endpoint

2 participants