From 636b26cf78c36a62e40b8a64217ec167e5d9b29c Mon Sep 17 00:00:00 2001 From: Max Thompson Date: Thu, 22 Oct 2026 09:56:43 -0700 Subject: [PATCH 1/2] docs: say which credential each kind of ate-api caller should use The authentication guide described only the jwtProviders file, so users fell back on kubectl ate's shared ate-client token. Organize it by caller instead: people running kubectl ate, pods in the Substrate cluster, pods in another cluster, and CI jobs. Each section says what works today, including pod certificates, what kubectl ate still needs from the Kubernetes API, and what isn't supported yet. Move the per-user ServiceAccount recipe and the authentication gaps out of the authorization guide so each fact lives in one place. --- README.md | 2 +- cmd/kubectl-ate/README.md | 4 +- docs/authentication.md | 284 +++++++++++++++++++++++++++++++++----- docs/authorization.md | 34 +---- 4 files changed, 261 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index 2508d717bd..df79c88609 100644 --- a/README.md +++ b/README.md @@ -252,7 +252,7 @@ We provide several sample applications demonstrating Agent Substrate's capabilit * [Glossary](docs/glossary.md): Core terms (Actor, Atespace, ActorTemplate, WorkerPool, Worker, ate-api-server, atenet, atelet, ateworker) and how they relate. * [Integration Repositories](docs/integration-repos.md): Where integrations live, how their repositories are named, and how fixes flow back to core. * [Observability Guide](docs/observability.md): Guide to actor logging, metrics, and distributed tracing. -* [Authentication Guide](docs/authentication.md): Configure trusted JWT providers and human credentials. +* [Authentication Guide](docs/authentication.md): Which credential people, pods and CI jobs should use with the Substrate API, and how to trust more JWT issuers. * [Egress Traffic](docs/egress-traffic.md): Which protocols an Actor may reach the outside world with and which are blocked. * [Enabling man-in-the-middle (MITM) interception for Actor Egress policy](docs/egress-trust-bundle.md): Egress policies such as header injection depend on MITM interception of Actor traffic. This guide explains how an Actor should be configured to enable interception. * [Request Parking](docs/request-parking.md): How the router parks requests through transient worker-pool saturation. diff --git a/cmd/kubectl-ate/README.md b/cmd/kubectl-ate/README.md index 1f6ba2c12d..1e6748e41c 100644 --- a/cmd/kubectl-ate/README.md +++ b/cmd/kubectl-ate/README.md @@ -24,7 +24,9 @@ go run ./cmd/kubectl-ate ## Connection & Auto Port-Forwarding By default, `kubectl-ate` will automatically read your `~/.kube/config`, discover the `ate-api-server` pods in your cluster, and establish a temporary background port-forward tunnel to execute gRPC calls securely. -If you prefer to route traffic directly (e.g., through a LoadBalancer or when running natively inside a cluster pod), simply provide the `--endpoint` flag to bypass the tunnel. +If you prefer to route traffic directly (e.g., through a LoadBalancer or when running natively inside a cluster pod), simply provide the `--endpoint` flag to bypass the tunnel. Either way, `kubectl-ate` reads the CA that verifies `ate-api-server` from the cluster, so it needs a kubeconfig or in-cluster config. + +For which credential to pass with `--token-file`, see the [Authentication Guide](../../docs/authentication.md#people-running-kubectl-ate). ## Tracing The CLI supports on-demand tracing using the `--trace` flag. When enabled, the CLI will generate a trace ID and signal to the server that it wants the request to be traced. diff --git a/docs/authentication.md b/docs/authentication.md index 4bccc4fa52..cfacca8da6 100644 --- a/docs/authentication.md +++ b/docs/authentication.md @@ -1,7 +1,219 @@ # ate-api Authentication -`ate-api` accepts mTLS client certificates and bearer JWTs. JWT providers are -configured with the file passed to `--authentication-config`: +Every call to `ate-api-server` must carry a credential, which authentication +turns into a principal ID. [Authorization](authorization.md) then decides what +that principal may do. + +- **Client certificate.** If the client presents a certificate issued by the + `podidentity.podcert.ate.dev/identity` signer, the principal is its first + URI SAN, a SPIFFE ID. A bearer token sent alongside it is ignored. +- **Bearer JWT.** Otherwise the call needs an `authorization: Bearer ` + header with a token from a [trusted issuer](#configuring-jwt-providers). The + principal is the token's `sub`. +- Anything else gets `Unauthenticated`. + +## Which credential to use + +| Caller | Credential | Principal | +|---|---|---| +| [A person running `kubectl ate`](#people-running-kubectl-ate) | `--token-file` with a token for your own ServiceAccount, or from your identity provider | `system:serviceaccount::`, or the token's `sub` | +| [A pod in the Substrate cluster](#pods-in-the-substrate-cluster) | A projected ServiceAccount token, or a pod certificate | `system:serviceaccount::`, or `spiffe://cluster.local/ns//sa/` | +| [A pod in another cluster](#pods-in-another-cluster) | A projected ServiceAccount token from that cluster | `system:serviceaccount::` | +| [A CI job](#ci-jobs) | The CI system's OIDC token | The token's `sub` | + +Then grant the principal a role in an +[access policy](authorization.md#managing-access). + +## People running `kubectl ate` + +`kubectl ate` uses your kubeconfig for the Substrate cluster whatever +credential it sends to `ate-api-server`. It: + +- lists ClusterTrustBundles to verify `ate-api-server`'s serving certificate; +- without `--endpoint`, gets the `api` Service, lists pods and port-forwards in + `ate-system`; +- without `--token-file`, creates a token for the shared `ate-client` + ServiceAccount and warns that it is acting as a shared identity. + +Everyone relying on that last step is the same principal, and `ate-setup` +makes it a bootstrap owner by default (see the warning in +[authorization.md](authorization.md)). That suits a cluster only you use. On a +shared cluster, give each person their own identity in one of the two ways +below. + +`kubectl ate` rereads `--token-file` on every call but never refreshes it, so +get a new token when it expires. With `--token-file=-` it reads the token once +from stdin. + +### Your own ServiceAccount + +Create a ServiceAccount per person and let each mint tokens only for their own: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: {name: mint-alice-token, namespace: team-a} +rules: +- apiGroups: [""] + resources: ["serviceaccounts/token"] + resourceNames: ["alice"] + verbs: ["create"] +``` + +```sh +kubectl create token -n team-a alice --audience api.ate-system.svc > alice.token +kubectl ate --token-file alice.token get actors -a team-a +``` + +The principal is `system:serviceaccount:team-a:alice`. The audience must be +one the `kubernetes` provider lists; the config `ate-setup` writes lists +`api.ate-system.svc`. + +### A token from your identity provider + +If your identity provider issues OIDC ID tokens, +[add it as a JWT provider](#configuring-jwt-providers) and pass its token. +With the Google provider from that section: + +```sh +gcloud auth print-identity-token | kubectl ate --token-file=- get actors +``` + +The principal is the token's `sub`. For Google that is the numeric account ID, +not the email; [bootstrap owners](authorization.md#bootstrap-owners) shows how +to find it. + +## Pods in the Substrate cluster + +Dial `api.ate-system.svc:443` and verify the serving certificate against the +`servicedns.podcert.ate.dev/identity` trust bundle, with server name +`api.ate-system.svc`. The Service is headless, so DNS returns one address per +replica; use round-robin load balancing to spread calls across them. Go +clients can use the generated stubs in `pkg/proto/ateapipb`. + +Authenticate with a ServiceAccount token or a pod certificate. The kubelet +rotates both, so read them from the volume instead of caching them: + +```yaml +spec: + serviceAccountName: my-app + containers: + - name: app + volumeMounts: + - {name: ate-api, mountPath: /var/run/ate, readOnly: true} + volumes: + - name: ate-api + projected: + sources: + - clusterTrustBundle: + signerName: servicedns.podcert.ate.dev/identity + labelSelector: + matchLabels: + podcert.ate.dev/canarying: live + path: ca.crt + # Either a ServiceAccount token, sent as a bearer token: + - serviceAccountToken: + audience: api.ate-system.svc + path: token + # or a pod certificate, presented as the TLS client certificate: + # - podCertificate: + # signerName: podidentity.podcert.ate.dev/identity + # keyType: ECDSAP256 + # credentialBundlePath: credential-bundle.pem +``` + +- **ServiceAccount token.** The principal is + `system:serviceaccount::`. The config `ate-setup` writes + already trusts the cluster's issuer and this audience. To use `kubectl ate` + from the pod: + + ```sh + kubectl ate --endpoint=api.ate-system.svc:443 --token-file=/var/run/ate/token get actors + ``` + + It still reads the trust bundle through the Kubernetes API, so the pod's + ServiceAccount needs permission to list `clustertrustbundles`. +- **Pod certificate.** The principal is + `spiffe://cluster.local/ns//sa/`. The credential bundle is + one PEM file holding the private key followed by the certificate chain; + reread it on each TLS handshake. `kubectl ate` can't present a client + certificate. + +## Pods in another cluster + +A pod in another cluster sends a ServiceAccount token from its own cluster, +mounted as in the [previous section](#pods-in-the-substrate-cluster). + +1. **Trust the other cluster's issuer.** Add a + [JWT provider](#configuring-jwt-providers) whose `audiences` include the + audience the pod requests. Run + `kubectl get --raw /.well-known/openid-configuration | jq -r .issuer` + against the other cluster to print its issuer; on GKE it is + `https://container.googleapis.com/v1/projects//locations//clusters/`. + `ate-api-server` fetches the issuer's discovery document and keys over + HTTPS, so they must be reachable from the Substrate cluster. GKE serves + them publicly. +2. **Pick a ServiceAccount name no one else can mint.** The principal doesn't + name the cluster ([principals are not qualified by + issuer](authorization.md#known-limitations)), so + `system:serviceaccount:team-a:app` is the same principal in both clusters, + and anyone who can create a token for `team-a/app` in either one gets its + grants. If the other cluster also runs Substrate, its `ate-client` is the + same principal as yours. +3. **Reach and verify `ate-api-server`.** The install doesn't expose it + outside its cluster; exposing it is up to you. Its serving certificate + names only in-cluster Service names such as `api.ate-system.svc`, so the + client must use that as the TLS server name whatever address it dials, + and trust a copy of the serving CA. Copy it again when the CA rotates: + + ```sh + kubectl get clustertrustbundles -l podcert.ate.dev/canarying=live -o json \ + | jq -r '.items[] | select(.spec.signerName == "servicedns.podcert.ate.dev/identity") | .spec.trustBundle' \ + > ate-api-ca.crt + ``` + +`kubectl ate` works there only with a kubeconfig for the Substrate cluster, +which it needs to read the trust bundle. + +## CI jobs + +Trust the CI system's OIDC issuer and send its token. For GitHub Actions, add +this provider: + +```yaml +- name: github-actions + issuer: https://token.actions.githubusercontent.com + audiences: + - api.ate-system.svc +``` + +and request a token in the job: + +```yaml +permissions: + id-token: write +steps: +- run: | + curl -sSf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ + "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=api.ate-system.svc" \ + | jq -r .value > "$RUNNER_TEMP/ate.token" + kubectl ate --token-file "$RUNNER_TEMP/ate.token" get actors -a ci +``` + +The principal is the token's `sub`, which names the repository and the branch, +environment or pull request, such as `repo:my-org/my-repo:environment:prod`. +Its exact form depends on the repository (see GitHub's +[OIDC reference](https://docs.github.com/en/actions/reference/security/oidc)), +so decode a token before granting it. Any workflow on GitHub can request a +token for any audience, so grant only exact `sub` values. The job still needs +Kubernetes access for `kubectl ate`, as described +[above](#people-running-kubectl-ate). + +## Configuring JWT providers + +`ate-api-server` reads the issuers it trusts from the file passed to +`--authentication-config`, which the install mounts from the +`ate-api-authentication` ConfigMap: ```yaml jwtProviders: @@ -17,45 +229,51 @@ jwtProviders: - 32555940559.apps.googleusercontent.com ``` -Provider names and issuers must be unique. `issuer` must be an HTTPS URL and -`audiences` must be non-empty; a token is accepted when any configured audience -matches. `certificateAuthorityFile` and `discoveryTokenFile` are optional and -are needed for OIDC discovery against some private Kubernetes API servers. +- Provider names and issuers must be unique. `issuer` must be an HTTPS URL and + `audiences` must be non-empty. +- `ate-api-server` checks a token with the provider whose `issuer` equals its + `iss`. The signature must verify against keys from the issuer's OpenID + discovery document, and `aud` must contain one of the provider's audiences. +- `certificateAuthorityFile` and `discoveryTokenFile` are optional; some + private Kubernetes API servers need them for discovery. +- The Google values are what the Google Cloud CLI issues today: issuer + `https://accounts.google.com`, and the Cloud SDK's shared client ID as the + audience. They are examples, not built-in defaults; check the claims your + identity provider issues. -What an authenticated caller may do is decided by authorization; see -[authorization.md](authorization.md). Its principal ID does not record which -provider issued the token, so only configure providers you trust not to issue -a token whose subject is another principal's ID. +The principal ID doesn't record which provider issued the token, so only +configure providers you trust not to issue a token whose `sub` is another +principal's ID. -## Google Cloud CLI tokens +`ate-setup` creates the ConfigMap on first install with one provider, +`kubernetes`, for the cluster's ServiceAccount issuer and the audience +`api.ate-system.svc`. Set `ateapi.expectedJWTIssuer` if it detects the wrong +issuer. Later runs leave an existing ConfigMap alone, so providers you add +survive them; `ate-setup create api-authentication-config` overwrites it. -The Google Cloud CLI currently issues user identity tokens with issuer -`https://accounts.google.com` and audience -`32555940559.apps.googleusercontent.com`, the Cloud SDK's shared client ID. -These values are examples rather than built-in defaults; verify the claims -issued by your identity provider and configure them explicitly. - -With the provider configured, pipe the token to `kubectl-ate`: - -```sh -gcloud auth print-identity-token | kubectl ate --token-file=- get actors -``` - -`--token-file` accepts either a file path or `-` for stdin and only replaces the -credential sent to `ate-api`. `kubectl-ate` -still uses kubeconfig access to establish its port-forward and obtain the -server trust bundle. - -For a manifest-based installation, replace the authentication ConfigMap and -restart the deployment: +To add a provider, edit the ConfigMap and restart `ate-api-server`, which reads +it only at startup: ```sh +kubectl -n ate-system get configmap ate-api-authentication \ + -o jsonpath='{.data.authentication\.yaml}' > authentication.yaml +# Add the provider to authentication.yaml, then: kubectl -n ate-system create configmap ate-api-authentication \ --from-file=authentication.yaml \ --dry-run=client -o yaml | kubectl apply -f - kubectl -n ate-system rollout restart deployment/ate-api-server ``` -Configuration is read at process startup. Restart `ate-api` pods after changing -the ConfigMap. OIDC signing keys are cached and refreshed when an unknown key ID -is encountered. +`ate-api-server` caches signing keys and refreshes them periodically, and when +a token names an unknown key ID. + +## Not yet supported + +- **Signing in.** `kubectl ate` can't run a command to fetch or refresh a + token; people obtain tokens themselves + ([#1464](https://github.com/agent-substrate/substrate/issues/1464)). +- **`kubectl ate` without Kubernetes access** to the Substrate cluster. +- **Reaching `ate-api-server` from outside its cluster.** The install doesn't + expose it. +- **JWT providers at install time.** `ate-setup` configures only the cluster's + own ServiceAccount issuer; add others to the ConfigMap afterwards. diff --git a/docs/authorization.md b/docs/authorization.md index 8dbf7b1ffa..29f5262557 100644 --- a/docs/authorization.md +++ b/docs/authorization.md @@ -83,7 +83,7 @@ ATE_API_AUTHZ_BOOTSTRAP_OWNERS=,system:serviceaccount:ate-system:ate-client Then add the Google provider to the `ate-api-authentication` ConfigMap and restart `ate-api-server` -([authentication.md](authentication.md#google-cloud-cli-tokens)), and act as +([authentication.md](authentication.md#configuring-jwt-providers)), and act as yourself: ```sh @@ -93,11 +93,9 @@ kubectl ate --token-file me.token get atespaces ## `kubectl ate` and `ate-setup` identities -`kubectl ate` sends the token from `--token-file` (`-` for stdin). Without -one, it uses your kubeconfig to mint a token for `ate-client` (or -`ATE_CLIENT_SERVICE_ACCOUNT`) and warns that it is acting as a shared -identity. It still needs Kubernetes rights to port-forward to `ate-api-server` -unless you pass `--endpoint`. +`kubectl ate` sends the token from `--token-file`. Without one, it mints a +token for `ate-client` (or `ATE_CLIENT_SERVICE_ACCOUNT`); see +[authentication.md](authentication.md#people-running-kubectl-ate). `ate-setup` calls the API only to deploy and delete demos and benchmarks, as `ate-client`. If you leave `ate-client` out of the bootstrap owners, grant it @@ -285,24 +283,7 @@ For a shared cluster, beyond replacing the default bootstrap owner: - **Give people their own identities** in access policies, and have them pass `--token-file` with a token from a trusted JWT provider or their own - ServiceAccount. To let a user mint tokens only for their own ServiceAccount: - - ```yaml - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: {name: mint-alice-token, namespace: team-a} - rules: - - apiGroups: [""] - resources: ["serviceaccounts/token"] - resourceNames: ["alice"] - verbs: ["create"] - ``` - - ```sh - kubectl create token -n team-a alice --audience api.ate-system.svc > alice.token - kubectl ate --token-file alice.token get actors -a team-a - ``` - + ServiceAccount ([authentication.md](authentication.md#people-running-kubectl-ate)). - **Limit who can act as a ServiceAccount bootstrap owner.** Anyone who can create `serviceaccounts/token`, pods or Secrets in its namespace, or change RBAC there, can get its token. A @@ -315,11 +296,8 @@ For a shared cluster, beyond replacing the default bootstrap owner: ### Not yet supported -- **Signing in.** `kubectl ate` has no login flow or credential plugin; people - obtain tokens themselves. -- **JWT providers at install time.** `ate-setup` configures only the cluster's - ServiceAccount issuer; add others to the ConfigMap afterwards. - **Groups.** Policy members are individual principals. +- **Signing in.** See [authentication.md](authentication.md#not-yet-supported). ## Known limitations From 27e50cb64bc1585879bf3a30b4a2f2d385699fa4 Mon Sep 17 00:00:00 2001 From: Max Thompson Date: Thu, 22 Oct 2026 12:57:49 -0700 Subject: [PATCH 2/2] docs: fix authentication guide recipes that fail and tighten its warnings Several recipes in the authentication guide failed when followed. In-cluster clients need their namespace labeled ate.dev/api-access=true, since the ate-system NetworkPolicy drops everything else. `get actors` needs -a. A pod in another cluster can't project a ClusterTrustBundle from the Substrate cluster. The per-user recipe also never created the ServiceAccount, bound the Role, or granted kubectl ate its own Kubernetes access. Add the missing steps, and say where an identity is weaker than it looks: trusting another cluster's issuer lets it mint ate-client, GKE's Kubernetes Engine Developer and basic Editor roles can act as any ServiceAccount, ate-system port-forward rights reach unauthenticated Envoy admin APIs, deleting a ServiceAccount doesn't revoke its tokens, and GitHub and Google tokens authenticate anyone, so the GitHub example uses an install-specific audience and an ID-based sub. Correct what happens to a client certificate from another CA, which RBAC a ServiceAccount already has, what discoveryTokenFile sends, and how to change a wrongly detected issuer. The authorization guide no longer lists Secrets as a way to get a token ate-api accepts: Secret-based tokens carry the legacy kubernetes/serviceaccount issuer, which no provider can name. --- cmd/kubectl-ate/README.md | 4 +- docs/authentication.md | 279 +++++++++++++++++++++++++++----------- docs/authorization.md | 10 +- docs/operator-install.md | 3 +- 4 files changed, 211 insertions(+), 85 deletions(-) diff --git a/cmd/kubectl-ate/README.md b/cmd/kubectl-ate/README.md index 1e6748e41c..a228c5107d 100644 --- a/cmd/kubectl-ate/README.md +++ b/cmd/kubectl-ate/README.md @@ -24,7 +24,7 @@ go run ./cmd/kubectl-ate ## Connection & Auto Port-Forwarding By default, `kubectl-ate` will automatically read your `~/.kube/config`, discover the `ate-api-server` pods in your cluster, and establish a temporary background port-forward tunnel to execute gRPC calls securely. -If you prefer to route traffic directly (e.g., through a LoadBalancer or when running natively inside a cluster pod), simply provide the `--endpoint` flag to bypass the tunnel. Either way, `kubectl-ate` reads the CA that verifies `ate-api-server` from the cluster, so it needs a kubeconfig or in-cluster config. +If you prefer to route traffic directly (e.g., from a pod in a namespace labeled `ate.dev/api-access=true`), simply provide the `--endpoint` flag to bypass the tunnel. Either way, `kubectl-ate` reads the CA that verifies `ate-api-server` from the cluster, so it needs a kubeconfig or in-cluster config. For which credential to pass with `--token-file`, see the [Authentication Guide](../../docs/authentication.md#people-running-kubectl-ate). @@ -68,7 +68,7 @@ These flags can be appended to any command: | `--kubeconfig` | | Path to your kubeconfig file | `~/.kube/config` | | `--context` | | Name of the kubeconfig context to use | current context | | `--endpoint` | | Manual gRPC endpoint override (e.g., `localhost:8080`) | | -| `--token-file` | | Path to a bearer token for ate-api authentication, or `-` for stdin | Kubernetes ServiceAccount token | +| `--token-file` | | Path to a bearer token for ate-api authentication, or `-` for stdin | a token for the shared `ate-client` ServiceAccount | | `--output` | `-o` | Output format (`table`, `json`, `yaml`) | `table` | | `--trace` | | Enable on-demand tracing for the request | `false` | | `--verbose` | `-v` | Increase output verbosity; may be repeated or stacked (e.g., `-v`, `-vv`, `-vvvvv`) | `0` | diff --git a/docs/authentication.md b/docs/authentication.md index cfacca8da6..2d82c383fb 100644 --- a/docs/authentication.md +++ b/docs/authentication.md @@ -4,13 +4,15 @@ Every call to `ate-api-server` must carry a credential, which authentication turns into a principal ID. [Authorization](authorization.md) then decides what that principal may do. -- **Client certificate.** If the client presents a certificate issued by the - `podidentity.podcert.ate.dev/identity` signer, the principal is its first - URI SAN, a SPIFFE ID. A bearer token sent alongside it is ignored. -- **Bearer JWT.** Otherwise the call needs an `authorization: Bearer ` - header with a token from a [trusted issuer](#configuring-jwt-providers). The - principal is the token's `sub`. -- Anything else gets `Unauthenticated`. +- **Client certificate.** A client certificate must chain to the + `podidentity.podcert.ate.dev/identity` trust bundle, or the TLS handshake + fails. The principal is its first URI SAN, a SPIFFE ID. A bearer token sent + alongside it is ignored. +- **Bearer JWT.** A client that sends no certificate needs an + `authorization: Bearer ` header with a token from a + [trusted issuer](#configuring-jwt-providers). The principal is the token's + `sub`. +- A call with no certificate and no valid token gets `Unauthenticated`. ## Which credential to use @@ -26,17 +28,64 @@ Then grant the principal a role in an ## People running `kubectl ate` -`kubectl ate` uses your kubeconfig for the Substrate cluster whatever -credential it sends to `ate-api-server`. It: +`kubectl ate` needs a kubeconfig for the Substrate cluster even when you pass +`--token-file`. It uses it to: -- lists ClusterTrustBundles to verify `ate-api-server`'s serving certificate; -- without `--endpoint`, gets the `api` Service, lists pods and port-forwards in +- list ClusterTrustBundles, to verify `ate-api-server`'s serving certificate; +- without `--endpoint`, get the `api` Service, list pods and port-forward in `ate-system`; -- without `--token-file`, creates a token for the shared `ate-client` - ServiceAccount and warns that it is acting as a shared identity. +- without `--token-file`, create a token for the shared `ate-client` + ServiceAccount, warning that it is acting as a shared identity; +- for `logs actor` and `top worker`, read worker pods' logs and metrics, and + for `admin`, read and write the Secrets it manages. -Everyone relying on that last step is the same principal, and `ate-setup` -makes it a bootstrap owner by default (see the warning in +A person who isn't a cluster admin needs these roles for the first two: + +```sh +kubectl apply -f - <<'EOF' +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: ate-api-client + namespace: ate-system +rules: +- apiGroups: [""] + resources: [services] + resourceNames: [api] + verbs: [get] +- apiGroups: [""] + resources: [pods] + verbs: [list] +- apiGroups: [""] + resources: [pods/portforward] + verbs: [get, create] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: ate-api-client +rules: +- apiGroups: [certificates.k8s.io] + resources: [clustertrustbundles] + verbs: [list] +EOF +kubectl -n ate-system create rolebinding alice-ate-api-client --role=ate-api-client --user=alice@example.com +kubectl create clusterrolebinding alice-ate-api-client --clusterrole=ate-api-client --user=alice@example.com +``` + +`--user` is the person's Kubernetes user name, which on GKE is their Google +account email. + +RBAC can limit port-forwarding only by pod name, and `ate-api-server`'s pod +names change on every rollout, so in practice the grant covers every pod and +port in `ate-system`. NetworkPolicy doesn't filter it. It reaches ports that +listen only on loopback, including the router's `/statusz`, which lists recent +requests to every atespace, and the unauthenticated Envoy admin APIs of the +router and egress gateway, which can drain or stop them. Grant it only to +people you'd trust to operate Substrate. + +Without `--token-file`, everyone is the same principal, and `ate-setup` makes +it a bootstrap owner by default (see the warning in [authorization.md](authorization.md)). That suits a cluster only you use. On a shared cluster, give each person their own identity in one of the two ways below. @@ -47,28 +96,44 @@ from stdin. ### Your own ServiceAccount -Create a ServiceAccount per person and let each mint tokens only for their own: +Create each person's ServiceAccount in a namespace set aside for them, such as +`ate-users`, and let each person create tokens only for their own +ServiceAccount: -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: {name: mint-alice-token, namespace: team-a} -rules: -- apiGroups: [""] - resources: ["serviceaccounts/token"] - resourceNames: ["alice"] - verbs: ["create"] +```sh +kubectl create namespace ate-users +kubectl -n ate-users create serviceaccount alice +kubectl -n ate-users create role alice-token --verb=create --resource=serviceaccounts/token --resource-name=alice +kubectl -n ate-users create rolebinding alice-token --role=alice-token --user=alice@example.com ``` +Alice then mints a token and passes it: + ```sh -kubectl create token -n team-a alice --audience api.ate-system.svc > alice.token +kubectl create token -n ate-users alice --audience api.ate-system.svc > alice.token kubectl ate --token-file alice.token get actors -a team-a ``` -The principal is `system:serviceaccount:team-a:alice`. The audience must be +The principal is `system:serviceaccount:ate-users:alice`. The audience must be one the `kubernetes` provider lists; the config `ate-setup` writes lists `api.ate-system.svc`. +RBAC grants only add up, so these roles don't stop anyone else from acting as +alice. Anyone who can create pods or tokens in `ate-users` can, and the +built-in `edit` and `admin` roles allow both. The same rights in `ate-system` +get the `ate-client` token. On GKE a call is allowed if either IAM or RBAC +allows it, and an IAM grant covers every namespace: Kubernetes Engine Developer +can create a token for any ServiceAccount, and the basic Editor role can create +pods anywhere. +Give these people only Kubernetes Engine Cluster Viewer in IAM, and grant the +rest through RBAC. + +`ate-api-server` checks a token's signature and expiry without asking +Kubernetes whether its ServiceAccount still exists. Deleting alice's +ServiceAccount doesn't revoke tokens she already holds: they work until they +expire, plus a five-minute allowance for clock skew. To cut her off at once, +remove her from access policies. + ### A token from your identity provider If your identity provider issues OIDC ID tokens, @@ -76,23 +141,37 @@ If your identity provider issues OIDC ID tokens, With the Google provider from that section: ```sh -gcloud auth print-identity-token | kubectl ate --token-file=- get actors +gcloud auth print-identity-token | kubectl ate --token-file=- get actors -a team-a ``` The principal is the token's `sub`. For Google that is the numeric account ID, not the email; [bootstrap owners](authorization.md#bootstrap-owners) shows how -to find it. +to find it. Any Google account can get a token for the Cloud SDK's client ID, +so this provider authenticates every Google account, and access policies +decide what each may do. Any service you send one of these tokens to can +replay it here until it expires, so prefer a provider that issues tokens for +an audience of your own. ## Pods in the Substrate cluster -Dial `api.ate-system.svc:443` and verify the serving certificate against the -`servicedns.podcert.ate.dev/identity` trust bundle, with server name +Apart from Substrate's own components, the install's NetworkPolicy admits +connections to `ate-api-server` only from pods in namespaces labeled +`ate.dev/api-access=true`, and not from worker pods. Label the client's +namespace: + +```sh +kubectl label namespace ate.dev/api-access=true +``` + +Then dial `api.ate-system.svc:443` and verify the serving certificate against +the `servicedns.podcert.ate.dev/identity` trust bundle, with server name `api.ate-system.svc`. The Service is headless, so DNS returns one address per replica; use round-robin load balancing to spread calls across them. Go clients can use the generated stubs in `pkg/proto/ateapipb`. Authenticate with a ServiceAccount token or a pod certificate. The kubelet -rotates both, so read them from the volume instead of caching them: +updates all of these files, so read them from the volume instead of caching +them: ```yaml spec: @@ -128,11 +207,11 @@ spec: from the pod: ```sh - kubectl ate --endpoint=api.ate-system.svc:443 --token-file=/var/run/ate/token get actors + kubectl ate --endpoint=api.ate-system.svc:443 --token-file=/var/run/ate/token get actors -a ``` - It still reads the trust bundle through the Kubernetes API, so the pod's - ServiceAccount needs permission to list `clustertrustbundles`. + It still reads the trust bundle through the Kubernetes API, which default + RBAC lets every ServiceAccount do. - **Pod certificate.** The principal is `spiffe://cluster.local/ns//sa/`. The credential bundle is one PEM file holding the private key followed by the certificate chain; @@ -141,73 +220,109 @@ spec: ## Pods in another cluster -A pod in another cluster sends a ServiceAccount token from its own cluster, -mounted as in the [previous section](#pods-in-the-substrate-cluster). +A pod in another cluster sends a ServiceAccount token from its own cluster. +That cluster's pod certificates aren't trusted, and it has no copy of the +Substrate cluster's `servicedns.podcert.ate.dev/identity` trust bundle, so the +pod's projected volume holds the token and a copy of the serving CA from +step 3: -1. **Trust the other cluster's issuer.** Add a +```yaml + volumes: + - name: ate-api + projected: + sources: + - configMap: + name: ate-api-ca + - serviceAccountToken: + audience: api.ate-system.svc + path: token +``` + +1. **Trust only a cluster whose admins you'd trust as owners here.** + Principals don't name the cluster ([principals are not qualified by + issuer](authorization.md#known-limitations)). Anyone who can get a token + for `/` in the other cluster acts as + `system:serviceaccount::` here, whether or not that + ServiceAccount exists here. That includes `ate-system:ate-client`, the + default [bootstrap owner](authorization.md#bootstrap-owners), so replace it + first, and run the caller as a ServiceAccount that no one else in either + cluster can get a token for. +2. **Trust the other cluster's issuer.** Add a [JWT provider](#configuring-jwt-providers) whose `audiences` include the - audience the pod requests. Run - `kubectl get --raw /.well-known/openid-configuration | jq -r .issuer` - against the other cluster to print its issuer; on GKE it is + audience the pod requests. As a cluster admin of the other cluster, run + `kubectl --context get --raw /.well-known/openid-configuration | jq -r .issuer` + to print its issuer; on GKE it is `https://container.googleapis.com/v1/projects//locations//clusters/`. `ate-api-server` fetches the issuer's discovery document and keys over HTTPS, so they must be reachable from the Substrate cluster. GKE serves them publicly. -2. **Pick a ServiceAccount name no one else can mint.** The principal doesn't - name the cluster ([principals are not qualified by - issuer](authorization.md#known-limitations)), so - `system:serviceaccount:team-a:app` is the same principal in both clusters, - and anyone who can create a token for `team-a/app` in either one gets its - grants. If the other cluster also runs Substrate, its `ate-client` is the - same principal as yours. 3. **Reach and verify `ate-api-server`.** The install doesn't expose it - outside its cluster; exposing it is up to you. Its serving certificate - names only in-cluster Service names such as `api.ate-system.svc`, so the - client must use that as the TLS server name whatever address it dials, - and trust a copy of the serving CA. Copy it again when the CA rotates: + outside its cluster, and its NetworkPolicy admits only the in-cluster peers + described [above](#pods-in-the-substrate-cluster). Exposing it and admitting + that traffic is up to you; the [threat model](threat-model.md) (T-09) says + not to expose it directly to the internet. Its serving certificate names + only in-cluster Service names such as `api.ate-system.svc`, so use that as + the TLS server name whatever address you dial. Copy the serving CA into the + other cluster, and copy it again when the CA rotates: ```sh - kubectl get clustertrustbundles -l podcert.ate.dev/canarying=live -o json \ + kubectl --context get clustertrustbundles -l podcert.ate.dev/canarying=live -o json \ | jq -r '.items[] | select(.spec.signerName == "servicedns.podcert.ate.dev/identity") | .spec.trustBundle' \ - > ate-api-ca.crt + > ca.crt + kubectl --context -n create configmap ate-api-ca --from-file=ca.crt \ + --dry-run=client -o yaml | kubectl --context apply -f - ``` `kubectl ate` works there only with a kubeconfig for the Substrate cluster, -which it needs to read the trust bundle. +which it needs to read the trust bundle and, for `logs actor` and +`top worker`, worker pods' logs and metrics. ## CI jobs -Trust the CI system's OIDC issuer and send its token. For GitHub Actions, add -this provider: +Trust the CI system's OIDC issuer and send its token. Any workflow on GitHub +can request a GitHub Actions token for any audience, so this provider +authenticates every workflow on GitHub, and access policies decide which of +them may do anything. Use an audience unique to this install, so that a token +minted for it can't be replayed against another install that trusts GitHub: ```yaml - name: github-actions issuer: https://token.actions.githubusercontent.com audiences: - - api.ate-system.svc + - ate-api.prod.example.com ``` and request a token in the job: ```yaml permissions: + contents: read id-token: write steps: - run: | curl -sSf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ - "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=api.ate-system.svc" \ + "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=ate-api.prod.example.com" \ | jq -r .value > "$RUNNER_TEMP/ate.token" kubectl ate --token-file "$RUNNER_TEMP/ate.token" get actors -a ci ``` -The principal is the token's `sub`, which names the repository and the branch, -environment or pull request, such as `repo:my-org/my-repo:environment:prod`. -Its exact form depends on the repository (see GitHub's -[OIDC reference](https://docs.github.com/en/actions/reference/security/oidc)), -so decode a token before granting it. Any workflow on GitHub can request a -token for any audience, so grant only exact `sub` values. The job still needs -Kubernetes access for `kubectl ate`, as described -[above](#people-running-kubectl-ate). +The principal is the token's `sub`. Grant only a `sub` that names the +repository by ID and either a protected branch or an environment that only +protected branches can deploy to, such as +`repo:my-org@123456/my-repo@456789:ref:refs/heads/main`. Repositories created +before July 15, 2026 use names instead of IDs until they opt in or are renamed +or transferred, and a name can pass to a new owner. Don't grant a `sub` ending +in `:pull_request`, which every pull request job without an environment +shares. GitHub's +[OIDC reference](https://docs.github.com/en/actions/reference/security/oidc) +lists the forms; decode a token before granting it. + +The job also needs a kubeconfig for the Substrate cluster. On GKE, get one with +`google-github-actions/auth` and `google-github-actions/get-gke-credentials` +using a service account. As for people [above](#your-own-serviceaccount), give +it only Kubernetes Engine Cluster Viewer in IAM, and bind the `ate-api-client` +roles [above](#people-running-kubectl-ate) to it; its `--user` is the service +account's email. ## Configuring JWT providers @@ -234,8 +349,13 @@ jwtProviders: - `ate-api-server` checks a token with the provider whose `issuer` equals its `iss`. The signature must verify against keys from the issuer's OpenID discovery document, and `aud` must contain one of the provider's audiences. -- `certificateAuthorityFile` and `discoveryTokenFile` are optional; some - private Kubernetes API servers need them for discovery. +- `certificateAuthorityFile` and `discoveryTokenFile` are for an in-cluster + issuer such as `https://kubernetes.default.svc`, and `ate-setup` sets them + for one. `certificateAuthorityFile` replaces the system roots for that + provider. `discoveryTokenFile` requires it, and sends the token in that file + to the issuer and to any `/openid/v1/jwks` URL. The example points it at + `ate-api-server`'s own ServiceAccount token, so set it only for the + Substrate cluster's own issuer. - The Google values are what the Google Cloud CLI issues today: issuer `https://accounts.google.com`, and the Cloud SDK's shared client ID as the audience. They are examples, not built-in defaults; check the claims your @@ -247,9 +367,10 @@ principal's ID. `ate-setup` creates the ConfigMap on first install with one provider, `kubernetes`, for the cluster's ServiceAccount issuer and the audience -`api.ate-system.svc`. Set `ateapi.expectedJWTIssuer` if it detects the wrong -issuer. Later runs leave an existing ConfigMap alone, so providers you add -survive them; `ate-setup create api-authentication-config` overwrites it. +`api.ate-system.svc`. Later runs leave it alone, so providers you add survive +them. If it detected the wrong issuer, set `ateapi.expectedJWTIssuer`, run +`ate-setup create api-authentication-config`, which rewrites the file and +drops any providers you added, then restart `ate-api-server` as below. To add a provider, edit the ConfigMap and restart `ate-api-server`, which reads it only at startup: @@ -260,19 +381,25 @@ kubectl -n ate-system get configmap ate-api-authentication \ # Add the provider to authentication.yaml, then: kubectl -n ate-system create configmap ate-api-authentication \ --from-file=authentication.yaml \ - --dry-run=client -o yaml | kubectl apply -f - + --dry-run=client -o yaml | kubectl replace -f - kubectl -n ate-system rollout restart deployment/ate-api-server +kubectl -n ate-system rollout status deployment/ate-api-server ``` +If the rollout doesn't finish, the new pod (the one restarting in +`kubectl -n ate-system get pods -l app=ate-api-server`) logs what is wrong with +the config. Fix the ConfigMap promptly: the old pods mount it too and read the +bad file if they restart. + `ate-api-server` caches signing keys and refreshes them periodically, and when a token names an unknown key ID. ## Not yet supported - **Signing in.** `kubectl ate` can't run a command to fetch or refresh a - token; people obtain tokens themselves - ([#1464](https://github.com/agent-substrate/substrate/issues/1464)). -- **`kubectl ate` without Kubernetes access** to the Substrate cluster. + token; people obtain tokens themselves. +- **`kubectl ate` without Kubernetes access** to the Substrate cluster, or, + from outside the cluster, without port-forward rights in `ate-system`. - **Reaching `ate-api-server` from outside its cluster.** The install doesn't expose it. - **JWT providers at install time.** `ate-setup` configures only the cluster's diff --git a/docs/authorization.md b/docs/authorization.md index 29f5262557..4204ddafe1 100644 --- a/docs/authorization.md +++ b/docs/authorization.md @@ -243,10 +243,10 @@ metadata: bindings: - role: editor members: - - user:system:serviceaccount:team-a:alice + - user:system:serviceaccount:ate-users:alice - role: viewer members: - - user:system:serviceaccount:team-a:bob + - user:system:serviceaccount:ate-users:bob ``` ```sh @@ -285,10 +285,8 @@ For a shared cluster, beyond replacing the default bootstrap owner: `--token-file` with a token from a trusted JWT provider or their own ServiceAccount ([authentication.md](authentication.md#people-running-kubectl-ate)). - **Limit who can act as a ServiceAccount bootstrap owner.** Anyone who can - create `serviceaccounts/token`, pods or Secrets in its namespace, or change - RBAC there, can get its token. A - `kubernetes.io/service-account-token` Secret gets one that never expires. - Check who can, and alert on the audit events: + create `serviceaccounts/token` or pods in its namespace, or change RBAC + there, can get its token. Check who can, and alert on the audit events: ```sh kubectl auth can-i create serviceaccounts/ate-client --subresource=token -n ate-system --as alice@example.com diff --git a/docs/operator-install.md b/docs/operator-install.md index bbf0678424..f8e995164b 100644 --- a/docs/operator-install.md +++ b/docs/operator-install.md @@ -264,7 +264,8 @@ changes it. Defaults are also shown by `ate-setup --help` and by `--help` on the owning command. A setting with no default is unset unless something supplies it. The `ateapi.authz.*` settings are described in -[authorization.md](authorization.md). +[authorization.md](authorization.md), and `ateapi.expectedJWTIssuer` in +[authentication.md](authentication.md#configuring-jwt-providers). Settings for the demos, benchmarks, and e2e tests are listed with them: the [counter](../demos/counter/README.md#settings) and