Bump Substrate to ddb46651: templates name no sandbox, durable suspend, an identity Substrate's authorization can grant

README.md

@@ -65,28 +65,59 @@ verification, pinning and how releases are cut.
 
 Prerequisites: a cluster with [Agent Substrate](https://github.com/agent-substrate/substrate)
 installed and a snapshots bucket. ate-env is built against Substrate
-[`3a2ca623`](https://github.com/agent-substrate/substrate/commit/3a2ca623) (2026-10-16, Go module
-`go.ate.dev/substrate`). The cluster needs a Substrate recent enough to accept what ate-env writes:
-the WorkerPool declares `spec.sandboxClasses`, the ActorTemplate uses `snapshotConfig` and
-`wakeupProbe`, names its sandbox class as the string `gvisor`, and declares CPU and memory limits.
-That means Substrate at or after
-[`63a17a92`](https://github.com/agent-substrate/substrate/commit/63a17a92) (2026-10-15), which also
-renamed the worker image: `--worker-image` must be the `ateworker-gvisor` image published by that
-Substrate install. An older Substrate rejects the string sandbox class (from
-[`9ec49759`](https://github.com/agent-substrate/substrate/commit/9ec49759)) or the limits (from
-[`c58f374a`](https://github.com/agent-substrate/substrate/commit/c58f374a)); use an ate-env from before
-this bump with it.
-
-When you upgrade an existing cluster across that boundary, register your base templates again.
-Substrate cannot update an ActorTemplate in place, and a base registered earlier has no CPU and memory
-limits, thus each template `ate-env-api` derives from it for a rootfs image fails with
-`resources: Required`. Delete the base and its derived templates (named `<base>-<12 hex digits>`) with
-`kubectl-ate delete actor-template`, then register the base again with `ate-env manifest template`.
-Each environment created after that derives its template again, with the limits.
+[`ddb46651`](https://github.com/agent-substrate/substrate/commit/ddb46651) (2026-10-18, Go module
+`go.ate.dev/substrate`). The cluster needs Substrate at or after
+[`0443aec3`](https://github.com/agent-substrate/substrate/commit/0443aec3) (2026-10-17): from that
+commit the WorkerPool alone chooses the sandbox (`spec.sandboxClasses`), the ActorTemplate names
+none, and Substrate rejects a template that still sends `sandboxConfig`, while an older Substrate
+requires one. Use an ate-env from before this bump with an older Substrate. Because the template no
+longer names a class, an actor booting without a snapshot can land on any pool whose labels its
+template's `workerSelector` matches, and on any pool at all without a selector. `ate-env` labels its
+WorkerPool `ate-env/workerpool: <pool>` and each template selects that label, thus actors stay on
+their pool and its sandbox class; `--workerpool` on `ate-env manifest template` must name the same
+pool as on `ate-env manifest` (both default to `<template>-workerpool`). The templates also use
+`snapshotConfig` and `wakeupProbe` and declare CPU and memory limits, and `--worker-image` must be the
+`ateworker-gvisor` image published by the same Substrate install.
+
+`ate-env-api` suspends synchronously: a suspend returns once the snapshot is in object storage, thus a
+suspended environment survives the loss of its node. Substrate's own default became asynchronous in
+[`b40d34f1`](https://github.com/agent-substrate/substrate/commit/b40d34f1).
+
+Substrate enforces authorization by default. `ate-env-api` runs as its own ServiceAccount
+(`ate-env/ate-env-api`) and needs the editor role on each atespace it creates environments in; the
+atespace must exist first, because creating one takes a global owner. Step 1 sets both up.
+
+When you upgrade an existing cluster across these changes:
+
+- Create the atespace and the access policy (step 1) before you apply the new manifest. Without them
+  `ate-env-api` fails at startup with `PermissionDenied` while it checks the default atespace.
+- Base templates registered before keep working, because Substrate drops their old `sandboxConfig`
+  when it stores them, but they have no `workerSelector`. On a cluster with more than one
+  WorkerPool, register them again (delete the base and its `<base>-<12 hex digits>` derived
+  templates, then `ate-env manifest template`), or their actors can boot on another pool. Apply the
+  new manifest first, so the pool carries the label the templates select.
+- A base registered before the CPU and memory limits (Substrate before
+  [`c58f374a`](https://github.com/agent-substrate/substrate/commit/c58f374a)) still needs registering
+  again: Substrate cannot update an ActorTemplate in place, thus each template `ate-env-api` derives
+  from it fails with `resources: Required`. Delete the base and its derived templates (named
+  `<base>-<12 hex digits>`) with `kubectl-ate delete actor-template`, then register the base again.
 
 ### 1. Deploy the Kubernetes Resources
 
-Deploy the namespace, worker pool, and API service:
+Give `ate-env-api` access to its atespace, once, as an owner of the atespace (on a stock install the
+default `kubectl-ate` identity is a bootstrap owner):
+
+```bash
+kubectl ate create atespace ate-env
+ate-env manifest access-policy | kubectl ate create access-policy -a ate-env -f -
+```
+
+An atespace has one access policy. If `ate-env` already has one, add the binding that
+`ate-env manifest access-policy` prints to it with `kubectl ate get access-policy` and
+`kubectl ate update access-policy`. Repeat both commands for each other atespace you create
+environments in.
+
+Deploy the namespace, worker pool, ServiceAccount and API service:
 
 ```bash
 # With a release binary the API image defaults to the release's. The worker

cmd/ate-env/manifest.go

@@ -38,9 +38,21 @@ import (
 // apiName is the name of the API service Deployment and Service.
 const apiName = "ate-env-api"
 
-// sandboxConfigName is the Substrate SandboxConfig both the WorkerPool and the
-// ActorTemplate refer to. `ate-setup deploy ate-system` installs exactly this
-// one, so a pool declared with it is schedulable on a stock install.
+// workerPoolLabel labels the WorkerPool with its own name, and templates
+// select their pool by it. Substrate copies a pool's labels to its workers and
+// matches a template's workerSelector against them. Since substrate#2780 the
+// template names no sandbox class, thus without a selector an actor that
+// boots without a snapshot can land on any pool of any class in the cluster.
+const workerPoolLabel = "ate-env/workerpool"
+
+// defaultWorkerPool is the pool name both commands use unless --workerpool
+// says otherwise.
+func defaultWorkerPool(template string) string { return template + "-workerpool" }
+
+// sandboxConfigName is the Substrate SandboxConfig the WorkerPool's gvisor
+// sandbox class refers to. `ate-setup deploy ate-system` installs exactly this
+// one, so a pool declared with it is schedulable on a stock install. Since
+// substrate#2780 the ActorTemplate names no sandbox: the worker pool decides.
 const sandboxConfigName = "gvisor-default"
 
 // Default compute limits for each actor. Substrate requires both on every
@@ -84,7 +96,10 @@ binaries default it to the ate-env-api image published with the release
 
 // templateConfig holds configuration for generating a Substrate ActorTemplate manifest.
 type templateConfig struct {
-	template        string
+	template string
+	// workerPool is the WorkerPool the template's actors run on; the template
+	// selects it by workerPoolLabel.
+	workerPool      string
 	atespace        string
 	guestImage      string
 	guestCommand    []string
@@ -244,7 +259,7 @@ the "template" subcommand: ate-env manifest template`,
 				mCfg.template = apiservice.DefaultTemplate
 			}
 			if mCfg.workerPool == "" {
-				mCfg.workerPool = mCfg.template + "-workerpool"
+				mCfg.workerPool = defaultWorkerPool(mCfg.template)
 			}
 			if err := mCfg.resolveImages(); err != nil {
 				return err
@@ -264,10 +279,38 @@ the "template" subcommand: ate-env manifest template`,
 	cmd.Flags().Int32Var(&mCfg.replicas, "replicas", 5, "number of pre-warmed worker pods")
 
 	cmd.AddCommand(newManifestTemplateCommand())
+	cmd.AddCommand(newManifestAccessPolicyCommand())
 
 	return cmd
 }
 
+func newManifestAccessPolicyCommand() *cobra.Command {
+	namespace := apiservice.DefaultNamespace
+	cmd := &cobra.Command{
+		Use:   "access-policy",
+		Short: "Generate the Substrate access policy that lets ate-env-api manage environments",
+		Long: `Access-policy prints a Substrate access policy, in the form kubectl-ate reads,
+that grants ate-env-api's ServiceAccount the editor role on an atespace.
+
+Substrate enforces authorization by default: ate-env-api then needs editor on
+each atespace it creates environments in, and the atespace must exist, because
+creating one takes a global owner. As an owner of the atespace (the default
+kubectl-ate identity on a stock install), run once for each atespace:
+
+  kubectl ate create atespace ate-env
+  ate-env manifest access-policy | kubectl ate create access-policy -a ate-env -f -
+
+An atespace has one policy. If it already has one, add the binding to it with
+"kubectl ate get access-policy" and "kubectl ate update access-policy".`,
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return writeProtoYAML(cmd.OutOrStdout(), buildAccessPolicy(namespace), "access policy")
+		},
+	}
+	cmd.Flags().StringVar(&namespace, "namespace", namespace, "Kubernetes namespace ate-env-api runs in (its ServiceAccount's namespace)")
+	return cmd
+}
+
 func newManifestTemplateCommand() *cobra.Command {
 	tCfg := templateConfig{}
 
@@ -281,6 +324,9 @@ It prints YAML to stdout without touching the cluster.`,
 			if tCfg.template == "" {
 				tCfg.template = apiservice.DefaultTemplate
 			}
+			if tCfg.workerPool == "" {
+				tCfg.workerPool = defaultWorkerPool(tCfg.template)
+			}
 			if err := tCfg.resolveImages(); err != nil {
 				return err
 			}
@@ -302,6 +348,7 @@ It prints YAML to stdout without touching the cluster.`,
 	}
 
 	cmd.Flags().StringVar(&tCfg.template, "template", apiservice.DefaultTemplate, "ActorTemplate name")
+	cmd.Flags().StringVar(&tCfg.workerPool, "workerpool", "", "WorkerPool the actors run on, selected by its "+workerPoolLabel+" label (defaults to <template>-workerpool, as in ate-env manifest)")
 	cmd.Flags().StringVar(&tCfg.atespace, "atespace", apiservice.DefaultAtespace, "Substrate atespace for the ActorTemplate")
 	cmd.Flags().StringVar(&tCfg.guestImage, "guest-image", defaultGuestImage, imageDefaultHelp("digest-pinned ate-env-guest image (repo@sha256:...)", defaultGuestImage))
 	cmd.Flags().StringSliceVar(&tCfg.guestCommand, "guest-command", []string{"/ko-app/ate-env-guest"}, "guest container entrypoint")
@@ -323,29 +370,73 @@ func buildManifests(cfg manifestConfig) []any {
 	return []any{
 		buildNamespace(cfg),
 		buildWorkerPool(cfg),
+		buildAPIServiceAccount(cfg),
 		buildAPIDeployment(cfg),
 		buildAPIService(cfg),
 	}
 }
 
+// apiPrincipal is the identity ate-env-api presents to the Substrate control
+// plane: its own ServiceAccount, through a projected token. Access policies
+// name it with a "user:" prefix (see the access-policy subcommand).
+func apiPrincipal(namespace string) string {
+	return "system:serviceaccount:" + namespace + ":" + apiName
+}
+
+// buildAPIServiceAccount returns the ServiceAccount ate-env-api runs as. A
+// dedicated account gives the control plane a stable principal to authorize;
+// the namespace's default account would be shared with anything else there.
+// ate-env-api never calls the Kubernetes API, thus no API token is mounted.
+func buildAPIServiceAccount(cfg manifestConfig) *corev1.ServiceAccount {
+	automount := false
+	return &corev1.ServiceAccount{
+		TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "ServiceAccount"},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      apiName,
+			Namespace: cfg.namespace,
+			Labels:    map[string]string{"app": apiName},
+		},
+		AutomountServiceAccountToken: &automount,
+	}
+}
+
+// buildAccessPolicy returns the Substrate access policy that lets ate-env-api
+// manage environments in one atespace: the editor role creates, suspends and
+// deletes actors, and creates the per-image templates it derives.
+func buildAccessPolicy(namespace string) *ateapipb.AccessPolicy {
+	return &ateapipb.AccessPolicy{
+		Metadata: &ateapipb.ResourceMetadata{Name: "default"},
+		Bindings: []*ateapipb.Binding{{
+			Role:    "editor",
+			Members: []string{"user:" + apiPrincipal(namespace)},
+		}},
+	}
+}
+
 // writeActorTemplate writes a single Substrate ActorTemplate as protojson-shaped YAML.
 func writeActorTemplate(w io.Writer, tmpl *ateapipb.ActorTemplate) error {
+	return writeProtoYAML(w, tmpl, "actor template")
+}
+
+// writeProtoYAML writes a Substrate control-plane message as protojson-shaped
+// YAML, the form kubectl-ate reads, without its server-managed status.
+func writeProtoYAML(w io.Writer, msg proto.Message, what string) error {
 	opts := protojson.MarshalOptions{
 		UseProtoNames: false,
 	}
-	jsonBytes, err := opts.Marshal(tmpl)
+	jsonBytes, err := opts.Marshal(msg)
 	if err != nil {
-		return fmt.Errorf("encoding actor template json: %w", err)
+		return fmt.Errorf("encoding %s json: %w", what, err)
 	}
 	var m map[string]any
 	if err := json.Unmarshal(jsonBytes, &m); err != nil {
-		return fmt.Errorf("decoding actor template json: %w", err)
+		return fmt.Errorf("decoding %s json: %w", what, err)
 	}
 	delete(m, "status")
 	prune(m)
 	data, err := yaml.Marshal(m)
 	if err != nil {
-		return fmt.Errorf("encoding actor template yaml: %w", err)
+		return fmt.Errorf("encoding %s yaml: %w", what, err)
 	}
 	_, err = w.Write(data)
 	return err
@@ -440,17 +531,19 @@ func buildWorkerPool(cfg manifestConfig) *atev1alpha1.WorkerPool {
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      cfg.workerPool,
 			Namespace: cfg.namespace,
+			// Copied to the pool's workers; templates select the pool by it.
+			Labels: map[string]string{workerPoolLabel: cfg.workerPool},
 		},
 		Spec: atev1alpha1.WorkerPoolSpec{
 			Replicas:    cfg.replicas,
 			WorkerImage: cfg.workerImage,
-			// The CRD requires exactly one sandbox class per pool, and
-			// actors only schedule onto workers of their template's class,
-			// so keep this in step with buildActorTemplate. The configRef is
-			// the SandboxConfig the workers run; the template names the same
-			// one in sandboxConfig.configName.
+			// The CRD requires exactly one sandbox class per pool. Since
+			// substrate#2780 the pool alone decides the sandbox: the
+			// ActorTemplate names none, and an actor restores onto a worker of
+			// its snapshot's class. The configRef is the SandboxConfig the
+			// workers run.
 			SandboxClasses: []atev1alpha1.WorkerPoolSandboxClass{{
-				Name:      atev1alpha1.SandboxClassGvisor, // the template's sandboxConfig.sandboxClass is the same string
+				Name:      atev1alpha1.SandboxClassGvisor,
 				ConfigRef: &atev1alpha1.SandboxConfigReference{Name: sandboxConfigName},
 			}},
 		},
@@ -462,11 +555,18 @@ func buildActorTemplate(cfg templateConfig) *ateapipb.ActorTemplate {
 	if atespace == "" {
 		atespace = apiservice.DefaultAtespace
 	}
+	pool := cfg.workerPool
+	if pool == "" {
+		pool = defaultWorkerPool(cfg.template)
+	}
 	return &ateapipb.ActorTemplate{
 		Metadata: &ateapipb.ResourceMetadata{
 			Name:     cfg.template,
 			Atespace: atespace,
 		},
+		// Pin the actors to their pool, thus to its sandbox class: the
+		// template names no class since substrate#2780.
+		WorkerSelector: &ateapipb.Selector{MatchLabels: map[string]string{workerPoolLabel: pool}},
 		Containers: []*ateapipb.Container{{
 			Name:    "guest",
 			Image:   cfg.guestImage,
@@ -489,10 +589,6 @@ func buildActorTemplate(cfg templateConfig) *ateapipb.ActorTemplate {
 			// schema marks the field required, so say it explicitly.
 			PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY,
 		},
-		SandboxConfig: &ateapipb.SandboxConfig{
-			SandboxClass: string(atev1alpha1.SandboxClassGvisor),
-			ConfigName:   sandboxConfigName,
-		},
 		Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{
 			{Name: "cpu", Quantity: orDefault(cfg.cpu, defaultActorCPU)},
 			{Name: "memory", Quantity: orDefault(cfg.memory, defaultActorMemory)},
@@ -548,6 +644,7 @@ func buildTemplate(cfg templateConfig) (*ateapipb.ActorTemplate, error) {
 // buildAPIDeployment returns the ate-env-api Deployment, pointed at the
 // in-cluster Substrate endpoints.
 func buildAPIDeployment(cfg manifestConfig) *appsv1.Deployment {
+	automountAPIToken := false
 	labels := map[string]string{"app": apiName}
 	replicas := cfg.apiReplicas
 	tokenExpiration := int64(7200)
@@ -564,6 +661,11 @@ func buildAPIDeployment(cfg manifestConfig) *appsv1.Deployment {
 			Template: corev1.PodTemplateSpec{
 				ObjectMeta: metav1.ObjectMeta{Labels: labels},
 				Spec: corev1.PodSpec{
+					// The ServiceAccount is the principal the control plane
+					// authorizes (buildAccessPolicy); the projected ateapi token
+					// below is minted for it.
+					ServiceAccountName:           apiName,
+					AutomountServiceAccountToken: &automountAPIToken,
 					Containers: []corev1.Container{{
 						Name:  apiName,
 						Image: cfg.apiImage,