Skip to content

Removing flaky tests which are not compatible configurations and not required henceforth - #1701

Open
Kaniska (Kaniska244) wants to merge 1 commit into
devcontainers:mainfrom
Kaniska244:dind-issue-tests
Open

Removing flaky tests which are not compatible configurations and not required henceforth#1701
Kaniska (Kaniska244) wants to merge 1 commit into
devcontainers:mainfrom
Kaniska244:dind-issue-tests

Conversation

@Kaniska244

@Kaniska244 Kaniska (Kaniska244) commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Description

This PR removes flaky and incompatible test configurations from the test-all.yaml workflow that are no longer required. These combinations were producing unreliable results or testing configurations that are not supported, adding noise to CI without providing meaningful coverage.

Changes

Excluded incompatible feature / base image combinations in the test matrix:

  • oryx on ubuntu:jammy
  • oryx on mcr.microsoft.com/devcontainers/base:ubuntu
  • docker-in-docker on mcr.microsoft.com/devcontainers/base:debian
  • docker-outside-of-docker on mcr.microsoft.com/devcontainers/base:debian
  • docker-in-docker on mcr.microsoft.com/devcontainers/base:ubuntu
  • docker-outside-of-docker on mcr.microsoft.com/devcontainers/base:ubuntu

Separated iptables-isolation scenarios out of the docker-in-docker job:

  • Added a step that strips the docker_with_default_iptables and docker_with_default_iptables_ubuntu scenarios from test/docker-in-docker/scenarios.json before running, so they run in a dedicated iptables-isolation job instead of the general scenario run.

Motivation

  • Eliminates flaky test failures caused by unsupported/incompatible configurations.
  • Reduces CI noise and speeds up test runs by not exercising combinations that are not required going forward.
  • Keeps iptables-isolation coverage intact by moving it to a separate, isolated job.

Testing

  • CI (test-all workflow) exercises the updated matrix and confirms the excluded combinations are no longer run while remaining scenarios continue to pass.

@Kaniska244 Kaniska (Kaniska244) changed the title Removing flaky tests which are not compatible cases Removing flaky tests which are not compatible configurations and not required henceforth Aug 3, 2026
@Kaniska244
Kaniska (Kaniska244) marked this pull request as ready for review August 3, 2026 13:58
@Kaniska244
Kaniska (Kaniska244) requested a review from a team as a code owner August 3, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant