Krrishj18/osmo mission runner by krrishj18 · Pull Request #366 · castacks/AirStack

krrishj18 · 2026-06-11T21:00:32Z

What features did you add and/or bugs did you address?

Which GitHub issue does this address?
Additional description if not fully described in the GitHub issue
Please add videos and images to demonstrate the feature. Please upload videos to somewhere persistent (e.g. YouTube or Vimeo) for archival purposes.

How did you implement it?

Algorithm details, design decisions, engineering notes, and any other relevant information about the implementation should be included

How do you run and use it?

What commands and button presses do you use to manually launch the stack to use your new feature?
Write a detailed procedure with EXACT BASH COMMANDS so that another maintainer can replicate and understand the benefits of your feature, and reproduce the videos and images you added above.

Testing with PyTest

What pytests did you add to ensure the feature is reliable and robust? What metrics are used?
What's the exact command to run the pytests that test your feature? i.e. airstack test -m ...
What are the expected results of the tests? What should a maintainer look at to understand whether the test succeeded?

Documentation

Was mkdocs.yml updated? (y/n)
Do the docs have sufficient scope such that a newcomer can easily reproduce and use your feature?
Is there sufficient visual media?

Versioning

Did you make sure to bump the version number in the .env file according to semantic versioning?

…IA OSMO Replace the OpenStack-Nova spawn/reap backend with OSMO workflow submission. The GitHub side is unchanged (self-hosted/airstack-ephemeral labels, single-use JIT runner tokens, same-repo fork guard) and the one-job-per-worker destroy-after model is preserved; only the spawn target moved from creating a Nova VM to submitting an OSMO workflow. orchestrator.py: submit/query/cancel/list via the osmo CLI, job_id -> workflow_id state, re-login-on-auth-failure, orphan sweep via osmo workflow list; drop floating-IP/boot-volume/placement/keypair/security-group logic. runner.Dockerfile + runner-entrypoint.sh + runner-workflow.yaml.j2: prebaked privileged docker-in-docker + GPU GitHub runner image/task (replaces cloud-init.yaml.j2). config.example.yaml, setup.sh, airstack-orchestrator.service, requirements.txt: OSMO service-account token auth, install the osmo CLI, drop openstacksdk. Docs (AGENTS.md, tests/README.md, orchestrator README) updated to OSMO. Co-authored-by: Cursor <cursoragent@cursor.com>

Resolve uuid/live name after submit (OSMO returns name-only + suffix), default config to the Keycloak-backed airstack pool and Harbor runner image, and add scripts to build/push airstack-ci-runner on OSMO DinD. Co-authored-by: Cursor <cursoragent@cursor.com>

pvkumara · 2026-07-31T19:25:35Z

/pytest -m build_docker

github-actions · 2026-07-31T19:26:02Z

Running pytest tests/ -m 'build_packages or build_docker' — view run. Status will appear as a check on this PR.

Note: build_packages is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.

github-actions · 2026-07-31T19:30:46Z

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

system

Pass rates

Test	Fail	Rate
TestColconBuilds.test_colcon_build_gcs	1	0%
TestColconBuilds.test_colcon_build_ms_airsim	1	0%
TestColconBuilds.test_colcon_build_robot	1	0%
TestColconBuilds.test_colcon_test_robot	1	0%
TestDockerBuilds.test_build_gcs	1	0%
TestDockerBuilds.test_build_isaac_sim	1	0%
TestDockerBuilds.test_build_ms_airsim	1	0%
TestDockerBuilds.test_build_robot_desktop	1	0%

Metrics

Test	Metric	Value
TestDockerBuilds.test_build_robot_desktop	duration_s	7.979s
TestDockerBuilds.test_build_gcs	duration_s	45.39s
TestDockerBuilds.test_build_isaac_sim	duration_s	133.3s
TestDockerBuilds.test_build_ms_airsim	duration_s	1.287s
TestColconBuilds.test_colcon_build_robot	duration_s	1.508s
TestColconBuilds.test_colcon_test_robot	duration_s	2.837s
TestColconBuilds.test_colcon_build_gcs	duration_s	1.894s
TestColconBuilds.test_colcon_build_ms_airsim	duration_s	1.342s

pvkumara · 2026-08-03T01:41:24Z

/pytest -m build_docker

github-actions · 2026-08-03T01:42:01Z

Running pytest tests/ -m 'build_packages or build_docker' — view run. Status will appear as a check on this PR.

Note: build_packages is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.

github-actions · 2026-08-03T03:42:49Z

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

Fills in the empty ci_cd.md stub with an end-to-end guide to how CI runs the full AirStack stack on ephemeral OSMO GPU pods: architecture and job lifecycle diagrams, runner pod anatomy, the three trigger paths, what each pytest mark catches, the metrics regression gate, the security model, and layer-by-layer troubleshooting. Adds the page to the mkdocs nav (it was previously unreachable) and cross-links it from tests/README.md and the testing index. Co-authored-by: Cursor <cursoragent@cursor.com>

Every build_docker and build_packages test failed on the OSMO backend because the inner dockerd kept its data-root on the pod's overlayfs rootfs. Linux rejects a directory on overlayfs as an overlay upperdir, so image pulls still succeeded -- containerd unpacks layers with plain writes -- while every build step needing a real mount died with "mount source: overlay ... err: invalid argument", surfacing as unrelated-looking apt-get and WORKDIR failures. runner-entrypoint.sh now picks a storage backend by attempting a real overlay mount rather than trusting the filesystem type, preferring a loopback ext4 data-root (real overlay2, sparse, dies with the pod) and falling back to a pod-mounted filesystem, fuse-overlayfs, then vfs. vfs is a last resort only: it copies the whole filesystem per layer and would exhaust the storage request on the sim images. Also bumps the GitHub Actions runner to 2.336.0, since 2.334.0 stops being able to run jobs on 2026-08-10. Co-authored-by: Cursor <cursoragent@cursor.com>

Versioned cache_from entries always miss on PRs because VERSION is forced up; add a stable cache_* tag published only by docker-build.yml so system tests can reuse layers without writing the shared cache. Co-authored-by: Cursor <cursoragent@cursor.com>

Skip full compose rebuilds when a service's content fingerprint matches the previous versioned image label; registry-retag instead and only rebuild services whose Docker inputs changed. Co-authored-by: Cursor <cursoragent@cursor.com>

docker_image_plan was feeding NUM_ROBOTS with a trailing comment into compose config, which broke strconv.Atoi for deploy.replicas. Co-authored-by: Cursor <cursoragent@cursor.com>

Publish successful images even when a sibling (e.g. isaac-sim) fails, and still cosign whatever was retagged or pushed in the same run. Co-authored-by: Cursor <cursoragent@cursor.com>

Seeded gcs/ms-airsim/robot images carry content-fingerprint labels; this bump should registry-retag those digests without rebuilding. Co-authored-by: Cursor <cursoragent@cursor.com>

pvkumara · 2026-08-10T20:06:28Z

/pytest -m build_docker

github-actions · 2026-08-11T00:56:21Z

Running pytest tests/ -m 'build_packages or build_docker' — view run. Status will appear as a check on this PR.

Note: build_packages is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.

github-actions · 2026-08-11T02:53:29Z

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

system

Pass rates

Test	Pass	Fail	Rate
TestColconBuilds.test_colcon_build_gcs	1	0	100%
TestColconBuilds.test_colcon_build_ms_airsim	1	0	100%
TestColconBuilds.test_colcon_build_robot	1	0	100%
TestColconBuilds.test_colcon_test_robot	0	1	0%
TestDockerBuilds.test_build_gcs	1	0	100%
TestDockerBuilds.test_build_isaac_sim	0	1	0%
TestDockerBuilds.test_build_ms_airsim	1	0	100%
TestDockerBuilds.test_build_robot_desktop	1	0	100%

Metrics

Test	Metric	Value
TestDockerBuilds.test_build_robot_desktop	duration_s	3222s
TestDockerBuilds.test_build_gcs	duration_s	907.2s
TestDockerBuilds.test_build_isaac_sim	duration_s	1727s
TestDockerBuilds.test_build_ms_airsim	duration_s	618.5s
TestColconBuilds.test_colcon_build_robot	duration_s	248.1s
TestColconBuilds.test_colcon_test_robot	duration_s	57.72s
TestColconBuilds.test_colcon_build_gcs	duration_s	153s
TestColconBuilds.test_colcon_build_ms_airsim	duration_s	16.52s

docker

Metrics

Test	Metric	Value
docker.gcs	image_size_mb	2060MB
docker.ms-airsim	image_size_mb	4071MB

Isaac's PX4 ubuntu.sh fails dpkg configure on the NVIDIA base; pre-fix ca-certificates, drop software-properties-common, and skip NuttX/Gazebo like ms-airsim. Pin pytest<8.1 and disable launch_testing for colcon unit tests so ROS Jazzy's outdated pytest hook no longer aborts CI. Co-authored-by: Cursor <cursoragent@cursor.com>

Bring OSMO CI, floating cache, retag, and isaac/pytest fixes onto the mission-runner PR so /pytest can validate them without merging to develop. Co-authored-by: Cursor <cursoragent@cursor.com>

pvkumara · 2026-08-11T20:14:07Z

/pytest -m "build_docker or build_packages"

github-actions · 2026-08-11T20:38:40Z

Running pytest tests/ -m 'build_docker or build_packages' — view run. Status will appear as a check on this PR.

Note: build_packages is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.

github-actions · 2026-08-11T22:23:28Z

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

system

Pass rates

Test	Pass	Fail	Rate
TestColconBuilds.test_colcon_build_gcs	1	0	100%
TestColconBuilds.test_colcon_build_ms_airsim	1	0	100%
TestColconBuilds.test_colcon_build_robot	1	0	100%
TestColconBuilds.test_colcon_test_robot	0	1	0%
TestDockerBuilds.test_build_gcs	1	0	100%
TestDockerBuilds.test_build_isaac_sim	1	0	100%
TestDockerBuilds.test_build_ms_airsim	1	0	100%
TestDockerBuilds.test_build_robot_desktop	1	0	100%

Metrics

Test	Metric	Value
TestDockerBuilds.test_build_robot_desktop	duration_s	2969s
TestDockerBuilds.test_build_gcs	duration_s	516.2s
TestDockerBuilds.test_build_isaac_sim	duration_s	1751s
TestDockerBuilds.test_build_ms_airsim	duration_s	519.7s
TestColconBuilds.test_colcon_build_robot	duration_s	325.2s
TestColconBuilds.test_colcon_test_robot	duration_s	54.53s
TestColconBuilds.test_colcon_build_gcs	duration_s	67.8s
TestColconBuilds.test_colcon_build_ms_airsim	duration_s	13.54s

docker

Metrics

Test	Metric	Value
docker.gcs	image_size_mb	2060MB
docker.isaac-sim	image_size_mb	1.138e+04MB
docker.ms-airsim	image_size_mb	4076MB

A single quoted blob made pytest treat "-p no:launch_testing" as part of the -m expression, which broke lidar_point_cloud_filter colcon tests. Co-authored-by: Cursor <cursoragent@cursor.com>

pvkumara · 2026-08-11T22:37:09Z

/pytest -m build_packages

github-actions · 2026-08-11T22:37:30Z

Running pytest tests/ -m build_packages — view run. Status will appear as a check on this PR.

Note: build_packages is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.

github-actions · 2026-08-11T23:49:33Z

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

system

Pass rates

Test	Pass	Fail	Rate
TestColconBuilds.test_colcon_build_gcs	1	0	100%
TestColconBuilds.test_colcon_build_ms_airsim	1	0	100%
TestColconBuilds.test_colcon_build_robot	1	0	100%
TestColconBuilds.test_colcon_test_robot	0	1	0%

Metrics

Test	Metric	Value
TestColconBuilds.test_colcon_build_robot	duration_s	183.1s
TestColconBuilds.test_colcon_test_robot	duration_s	40.71s
TestColconBuilds.test_colcon_build_gcs	duration_s	65.41s
TestColconBuilds.test_colcon_build_ms_airsim	duration_s	13.44s

Nested single quotes around 'not linter' terminated the outer bash -ic string early, so pytest saw 'not' as a path. Use shlex.quote for the whole command and list-form pytest_args in the YAML. Co-authored-by: Cursor <cursoragent@cursor.com>

pvkumara · 2026-08-12T01:28:26Z

/pytest -m build_packages

github-actions · 2026-08-12T01:29:57Z

Running pytest tests/ -m build_packages — view run. Status will appear as a check on this PR.

Note: build_packages is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.

github-actions · 2026-08-12T02:45:06Z

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

system

Pass rates

Test	Pass	Fail	Rate
TestColconBuilds.test_colcon_build_gcs	1	0	100%
TestColconBuilds.test_colcon_build_ms_airsim	1	0	100%
TestColconBuilds.test_colcon_build_robot	1	0	100%
TestColconBuilds.test_colcon_test_robot	0	1	0%

Metrics

Test	Metric	Value
TestColconBuilds.test_colcon_build_robot	duration_s	185.8s
TestColconBuilds.test_colcon_test_robot	duration_s	39.54s
TestColconBuilds.test_colcon_build_gcs	duration_s	64.84s
TestColconBuilds.test_colcon_build_ms_airsim	duration_s	13.73s

colcon --pytest-args is a single nargs='*' option, so repeating it dropped -p and pytest treated no:launch_testing as a file path. Set PYTEST_ADDOPTS with docker exec -e instead. Co-authored-by: Cursor <cursoragent@cursor.com>

pvkumara · 2026-08-12T18:18:43Z

/pytest -m build_packages

github-actions · 2026-08-12T18:19:12Z

Running pytest tests/ -m build_packages — view run. Status will appear as a check on this PR.

Note: build_packages is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.

github-actions · 2026-08-12T19:33:39Z

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

pytest

Pass rates

Test	Pass	Fail	Skip	Rate
pytest.internal	0	1	0	0%

Metrics

Test	Metric	Value
pytest.internal	duration_s	0s

conftest functions named pytest_* are registered as hooks. pytest_addopts_env caused PluginValidationError and exit code 3. Co-authored-by: Cursor <cursoragent@cursor.com>

pvkumara · 2026-08-12T20:07:05Z

/pytest -m build_packages

github-actions · 2026-08-12T20:07:31Z

Running pytest tests/ -m build_packages — view run. Status will appear as a check on this PR.

Note: build_packages is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.

Pull and retag cache_* images instead of baking isaac/airsim on every colcon/pytest iteration. /pytest --no-image-build does the same for other marks. compose up --no-build when AIRSTACK_NO_IMAGE_BUILD=1. Co-authored-by: Cursor <cursoragent@cursor.com>

pvkumara · 2026-08-12T20:22:05Z

/pytest -m build_packages

github-actions · 2026-08-12T20:22:30Z

Running pytest tests/ -m build_packages — view run. Status will appear as a check on this PR.

Note: build_packages is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.

krrishj18 added 7 commits June 10, 2026 15:52

added mission_runner.py to automate running missions in osmo

4bd534a

updated example mission

6ad33f0

changes to recording topics

55be3f8

updates to example mission

1fdbf8c

fixed mission runner script

61b673b

fixed duplicate airstack up issue and not recording mcap files

42ecd1c

fixed iteration 1 not working

0c582b0

krrishj18 requested a review from JohnYanxinLiu June 11, 2026 21:01

krrishj18 and others added 5 commits July 1, 2026 11:45

updated git ignore

6459485

updated with airlab info

3ab223f

config updates

12a7240

pvkumara5 and others added 7 commits August 7, 2026 13:36

fix(ci): parse quoted .env values before inline comments

b093327

docker_image_plan was feeding NUM_ROBOTS with a trailing comment into compose config, which broke strconv.Atoi for deploy.replicas. Co-authored-by: Cursor <cursoragent@cursor.com>

ci(docker-build): build/push services sequentially

56a60c7

Publish successful images even when a sibling (e.g. isaac-sim) fails, and still cosign whatever was retagged or pushed in the same run. Co-authored-by: Cursor <cursoragent@cursor.com>

chore: bump VERSION to 0.19.0-alpha.8 for retag validation

624dec7

Seeded gcs/ms-airsim/robot images carry content-fingerprint labels; this bump should registry-retag those digests without rebuilding. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'ci/osmo-orchestrator' into krrishj18/osmo-mission-runner

961f9c2

Bring OSMO CI, floating cache, retag, and isaac/pytest fixes onto the mission-runner PR so /pytest can validate them without merging to develop. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(ci): pass colcon --pytest-args as separate tokens

d353eec

A single quoted blob made pytest treat "-p no:launch_testing" as part of the -m expression, which broke lidar_point_cloud_filter colcon tests. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(ci): pass colcon pytest flags via PYTEST_ADDOPTS

3d08099

colcon --pytest-args is a single nargs='*' option, so repeating it dropped -p and pytest treated no:launch_testing as a file path. Set PYTEST_ADDOPTS with docker exec -e instead. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(ci): rename helper so pytest does not treat it as a hook

51f2c8e

conftest functions named pytest_* are registered as hooks. pytest_addopts_env caused PluginValidationError and exit code 3. Co-authored-by: Cursor <cursoragent@cursor.com>

Uh oh!

Conversation

krrishj18 commented Jun 11, 2026

What features did you add and/or bugs did you address?

How did you implement it?

How do you run and use it?

Testing with PyTest

Documentation

Versioning

Uh oh!

pvkumara commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Test Metrics — e4b499d120ef5157232c6ef6b1109488a94c9641

system

Pass rates

Metrics

Uh oh!

pvkumara commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Test Metrics — e4b499d120ef5157232c6ef6b1109488a94c9641

Uh oh!

pvkumara commented Aug 10, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Test Metrics — e4b499d120ef5157232c6ef6b1109488a94c9641

system

Pass rates

Metrics

docker

Metrics

Uh oh!

pvkumara commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Test Metrics — e4b499d120ef5157232c6ef6b1109488a94c9641

system

Pass rates

Metrics

docker

Metrics

Uh oh!

pvkumara commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Test Metrics — e4b499d120ef5157232c6ef6b1109488a94c9641

system

Pass rates

Metrics

Uh oh!

pvkumara commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Test Metrics — e4b499d120ef5157232c6ef6b1109488a94c9641

system

Pass rates

Metrics

Uh oh!

pvkumara commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Test Metrics — e4b499d120ef5157232c6ef6b1109488a94c9641

pytest

Pass rates

Metrics

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`

Test Metrics — `e4b499d120ef5157232c6ef6b1109488a94c9641`