Skip to content

kubectl ate: authenticate with ExecCredential plugins named by --credential-plugin - #2851

Open
Max Thompson (thompsonmax) wants to merge 2 commits into
agent-substrate:mainfrom
thompsonmax:ateconfig-exec-credential
Open

Max Thompson (thompsonmax) wants to merge 2 commits into
agent-substrate:mainfrom
thompsonmax:ateconfig-exec-credential

Conversation

@thompsonmax

@thompsonmax Max Thompson (thompsonmax) commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #1464

Support ExecCredential kubectl ate authentication.

  • --credential-plugin (or $ATE_CREDENTIAL_PLUGIN) names a kubectl credential plugin: a path to an executable, or a command line. kubectl ate passes the plugin its --kubeconfig and --context as $ATE_KUBECONFIG and $ATE_KUBE_CONTEXT.
  • internal/ateclient caches the plugin's credential until it expires and retries an RPC once on UNAUTHENTICATED. If the plugin returns the rejected token again from its own cache, the client reruns it with ATE_REFRESH_CREDENTIAL=1. The default ServiceAccount token now refreshes too, which fixes [P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated #644.
  • kubectl ate credential mints and disk-caches ServiceAccount tokens (the shared ate-client, or one named with --service-account) and wraps any command that prints a JWT.
  • kubectl ate whoami calls a new Authentication.WhoAmI RPC so users can check which principal ate-api sees.

Here's how each authentication option looks after this change:

--token-file: you supply the token yourself (Already existed before this change)

kubectl ate sends whatever token the file holds. It re-reads the file on every RPC, so a rotated file is picked up automatically, but it never runs anything to get a new token.

# A token you minted yourself
kubectl create token alice -n ate-users --audience=api.ate-system.svc > /tmp/alice.token
kubectl ate --token-file=/tmp/alice.token whoami
# A token piped through stdin, used for this one command
gcloud auth print-identity-token | kubectl ate --token-file=- get actors
# An in-cluster pod using its projected ServiceAccount token (the kubelet rotates the file)
kubectl ate --endpoint=api.ate-system.svc:443 \
  --token-file=/var/run/secrets/ate/token get actors

--credential-plugin: kubectl ate runs a command that produces the credential (What we're introducing here)

The command must print an ExecCredential (v1) as JSON. kubectl ate caches the credential until its expirationTimestamp, and runs the command again when the credential expires or ate-api rejects it.

# Built-in plugin, acting as a per-user ServiceAccount
kubectl ate --credential-plugin="kubectl-ate credential --service-account=ate-users/alice" whoami
# Same thing, set once for the whole shell session
export ATE_CREDENTIAL_PLUGIN="kubectl-ate credential --service-account=ate-users/alice"
kubectl ate get actors -a alice-space
# A standalone plugin binary; a path that exists runs as-is, with no arguments
kubectl ate --credential-plugin=/usr/local/bin/my-auth-plugin whoami
# Wrapping a tool that prints a raw JWT
kubectl ate --credential-plugin="kubectl-ate credential wrap -- gcloud auth print-identity-token" whoami
# Corporate OIDC through kubelogin
kubectl ate --credential-plugin="kubectl oidc-login get-token --oidc-issuer-url=https://idp.example.com --oidc-client-id=substrate" whoami
# Kube flags are passed to the plugin as ATE_KUBECONFIG and ATE_KUBE_CONTEXT
kubectl ate --context=prod --credential-plugin="kubectl-ate credential --service-account=ate-users/alice" get actors

Rules that apply to both

  • Precedence: --token-file comes first, then --credential-plugin or $ATE_CREDENTIAL_PLUGIN, then the shared ate-client token, which prints a warning.
  • Both flags: passing --token-file and --credential-plugin together is an error. --token-file with only $ATE_CREDENTIAL_PLUGIN set uses the token file.
  • Parsing a plugin value: a path to an existing file runs with no arguments. Anything else is split shell-style, and quotes work as they would in a shell: --credential-plugin="my-sso print-token --scope 'ate api'".

Reviewer notes: WhoAmI is a separate service because apitool requires every Control method to name a resource. The exec stanza is fixed at client.authentication.k8s.io/v1, so plugins that only answer v1beta1 fail. Passing both --token-file and --credential-plugin is an error.

Testing: unit tests; e2e checks WhoAmI with a token file and with a credential plugin (cat of an ExecCredential).

Fixes #644

  • Tests pass
  • Appropriate changes to documentation are included in the PR

@bowei Bowei Du (bowei) added area/cli area/identity area/security Security related issue/pr kind/feature An enhancement / feature request or implementation labels Oct 19, 2026
@thompsonmax Max Thompson (thompsonmax) changed the title kubectl ate: authenticate with credential plugins configured in an AteConfig file kubectl ate: authenticate with ExecCredential plugins named by --credential-plugin Oct 20, 2026
@thompsonmax
Max Thompson (thompsonmax) marked this pull request as ready for review October 20, 2026 02:41
kubectl ate minted one hour-long token for the shared ate-client
ServiceAccount when it connected and never refreshed it, so long-running
clients failed after an hour. Every user also shared that one identity,
and using another identity provider meant piping a raw JWT into
--token-file on every command.

--credential-plugin, or $ATE_CREDENTIAL_PLUGIN, names a kubectl
credential plugin. A value that names an existing file runs with no
arguments; any other value is split into a command and arguments,
honoring shell quoting. kubectl ate builds the exec stanza in memory
(client.authentication.k8s.io/v1, interactiveMode IfAvailable,
provideClusterInfo) and passes its --kubeconfig and --context to the
plugin as $ATE_KUBECONFIG and $ATE_KUBE_CONTEXT. Passing both
--token-file and --credential-plugin is an error.

ateclient runs the plugin through client-go's exec authenticator,
caches the credential until its expirationTimestamp, and retries an RPC
once with a fresh credential when ate-api returns Unauthenticated. If
the plugin returns the token that was just rejected, the client runs it
again with ATE_REFRESH_CREDENTIAL=1 so that a plugin with a cache
returns a new one. The ate-client token minted when there is no plugin
or token file is now refreshed a minute before it expires. A credential
source that fails locally is reported as a local error rather than an
Unauthenticated status.

kubectl ate credential is a built-in plugin. It mints a token for
ate-client, or for a ServiceAccount named with --service-account, and
caches it on disk until shortly before it expires; --refresh mints a new
one. -o token prints only the token, and credential wrap turns any
command that prints a JWT into a plugin. kubectl ate whoami calls a new
Authentication.WhoAmI RPC that returns the principal ate-api
authenticated the caller as. It is a separate service because it
operates on no resource.

The docs cover the flag, a per-user ServiceAccount recipe with the RBAC
a non-admin needs, what deleting a ServiceAccount does not revoke, and
in-cluster workloads.
The Unauthenticated retry also covered the minted ate-client token. A
minted token is replaced before it expires, so when ateapi rejected it,
for example because it does not trust the issuer, every failing RPC was
sent twice and minted a new token. Only exec plugin credentials are now
retried, and the token-file, stdin, and minted credentials lose their
refresh methods. All of them are still fetched before each RPC, so a
credential that cannot be fetched is still reported as a local error.

RPCs rejected together each forced a plugin rerun, because one refresh
could charge its 401 against the token another had just fetched. Refresh
is now serialized, so they share one refresh.

client-go cleans a plugin's command, which turned ./plugin into a bare
name looked up on PATH, so kubectl ate now makes a command containing a
separator absolute. It also always sets ATE_KUBECONFIG and
ATE_KUBE_CONTEXT, empty when the flag is not given, so the plugin no
longer inherits values from the user's environment that kubectl ate
itself ignores.

The e2e authz suite now checks the principal WhoAmI reports for a
ServiceAccount token sent from a token file and from a credential
plugin, minting it with ateclient.MintServiceAccountToken. The test
plugin claims its run number with an exclusive create so that concurrent
runs are counted, the concurrent refresh test runs readers alongside the
refreshers, the missing-plugin test no longer depends on client-go's
rate-limited install hint, and the credential command tests use the
package's existing runRootCLI helper.
Comment on lines +157 to +163
if c.token == "" || !c.now().Before(c.expiry.Add(-tokenExpiryMargin)) {
token, expiry, err := c.mint(ctx)
if err != nil {
return nil, err
}
c.token, c.expiry = token, expiry
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 nit 🟢 – When re-minting fails inside the one-minute margin, the RPC fails even though the cached token is still valid. Consider returning the current token while now < expiry and only surfacing the mint error once it has actually expired, so a transient TokenRequest failure does not take down a long-lived client early.

}
// Keyed by the kubeconfig too: whoever can mint here is not necessarily
// whoever can mint through another kubeconfig context on the same host.
key := []string{config.Host, kubeconfigPath, kubeContext, namespace, name, audience}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 nit 🟢 – With --kubeconfig and --context unset, this key does not distinguish different $KUBECONFIG values or a changed current-context that resolve to the same config.Host, which is the case the comment above is guarding against. Keying on the resolved context name and its AuthInfo name from RawConfig() would tie the entry to the minting identity rather than to the paths.

Comment thread docs/authentication.md
Google service account ID tokens or GitHub Actions OIDC tokens: any service
account or workflow can request one for any audience.

`kubectl ate credential` is the built-in plugin. It mints a ServiceAccount

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: in some places it says "kubectl ate credential" in others "kubectl-ate credential".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cli area/identity area/security Security related issue/pr kind/feature An enhancement / feature request or implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated

3 participants