Skip to content

ci: trim JDK matrix, add gha layer cache, image smoke test and NDK variant #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
azlekov wants to merge 1 commit into from
Closed

ci: trim JDK matrix, add gha layer cache, image smoke test and NDK variant #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 50 additions & 11 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,42 @@ jobs:
scripts:
name: Shell scripts
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7

- name: shellcheck
run: shellcheck -x pruner.sh tests/*.sh
run: |
set -euo pipefail
# nullglob: a pattern that matches nothing (e.g. hosts/*.sh) must
# drop out instead of being passed to shellcheck as a literal.
shopt -s nullglob
files=(*.sh hosts/*.sh tests/*.sh)
if [ "${#files[@]}" -eq 0 ]; then
echo "::error::No shell scripts found to check"
exit 1
fi
shellcheck -x "${files[@]}"

- name: docker compose config
run: |
set -euo pipefail
cp .env.example .env
docker compose config -q

- name: tests
run: |
set -euo pipefail
for t in tests/*.sh; do bash "$t"; done
for t in tests/*.sh; do
# image-smoke.sh needs a built image; it runs in the build job.
[ "$(basename "$t")" = image-smoke.sh ] && continue
bash "$t"
done

matrix:
name: Generate build matrix
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
matrix: ${{ steps.generate.outputs.matrix }}
steps:
Expand All @@ -37,26 +59,29 @@ jobs:
run: |
set -euo pipefail
# PR builds deliberately cover less than deploy.yml: the full
# java x node x arch product is ~28 images, which is far more than a
# dependency bump needs. Instead we cover every JDK against the
# default Node, every Node against the default JDK, and add a single
# arm64 build of the default pair so arch-specific breakage (e.g. the
# TARGETARCH-derived JAVA_HOME) is still caught before merge.
# java x node x arch product is far more than a dependency bump
# needs. Instead we cover every JDK against the default Node, every
# Node against the default JDK, and add a single arm64 build of the
# default pair so arch-specific breakage (e.g. the TARGETARCH-derived
# JAVA_HOME) is still caught before merge. The amd64 default pair
# also gets the image smoke test (smoke: true).
MATRIX=$(jq -c '
.default_jdk as $dj | .default_node as $dn
| ( [ .java_versions[] | {java_version: ., node_version: $dn} ]
+ [ .node_versions[] | {java_version: $dj, node_version: .} ]
| unique ) as $combos
| { include:
( [ $combos[] | . + {platform: "linux/amd64", arch: "amd64", runner: "ubuntu-latest"} ]
( [ $combos[] | . + {platform: "linux/amd64", arch: "amd64", runner: "ubuntu-latest",
smoke: (.java_version == $dj and .node_version == $dn)} ]
+ [ {java_version: $dj, node_version: $dn,
platform: "linux/arm64", arch: "arm64", runner: "ubuntu-24.04-arm"} ] ) }' matrix.json)
echo "matrix=${MATRIX}" >> $GITHUB_OUTPUT
platform: "linux/arm64", arch: "arm64", runner: "ubuntu-24.04-arm", smoke: false} ] ) }' matrix.json)
echo "matrix=${MATRIX}" >> "$GITHUB_OUTPUT"

build:
name: Build ${{ matrix.arch }} JDK ${{ matrix.java_version }} Node ${{ matrix.node_version }}
needs: matrix
runs-on: ${{ matrix.runner }}
timeout-minutes: 45
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.matrix.outputs.matrix) }}
Expand All @@ -77,29 +102,43 @@ jobs:
exit 1
fi

echo "runner_tag=${RUNNER_TAG}" >> $GITHUB_OUTPUT
echo "runner_tag=${RUNNER_TAG}" >> "$GITHUB_OUTPUT"

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4

# Layer cache lives in the GitHub Actions cache, one scope per platform so
# the amd64 and arm64 legs never overwrite each other. PR builds read and
# write it too; branch scoping keeps a PR from polluting main's cache.
# `load` is only set on the smoke leg: the other legs have nothing to run.
- name: Build (no push)
uses: docker/build-push-action@v7
with:
context: .
target: runner
platforms: ${{ matrix.platform }}
push: false
load: ${{ matrix.smoke }}
tags: github-runner:smoke
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
build-args: |
VERSION=${{ steps.versions.outputs.runner_tag }}
JAVA_VERSION=${{ matrix.java_version }}
NODE_VERSION=${{ matrix.node_version }}
RUNNER_IMAGE_TAG=pr-${{ github.event.pull_request.number }}
RUNNER_IMAGE_REVISION=${{ github.event.pull_request.head.sha }}

- name: Smoke test image
if: matrix.smoke
run: bash tests/image-smoke.sh github-runner:smoke "${{ matrix.java_version }}" "${{ matrix.node_version }}"

gate:
name: All builds passed
needs: [scripts, matrix, build]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Check that all required jobs succeeded
run: |
Expand Down
100 changes: 65 additions & 35 deletions .github/workflows/check-sdk-updates.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,41 @@ on:
- cron: '0 9 * * 1' # Weekly on Monday at 9 AM UTC
workflow_dispatch:

# Writes (branch push, PR, auto-merge) go through the GitHub App token minted
# below, not GITHUB_TOKEN: PRs opened with GITHUB_TOKEN never trigger
# build.yml, so their required checks would never report.
permissions:
contents: write
pull-requests: write
contents: read

jobs:
check-updates:
name: Check for updates
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Create GitHub App token
id: app-token
uses: actions/create-github-app-token@v3
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: ${{ github.event.repository.name }}
permission-contents: write
permission-pull-requests: write

- uses: actions/checkout@v7
with:
token: ${{ steps.app-token.outputs.token }}

- name: Check for new JDK versions
id: jdk
run: |
# Get available Temurin releases >= 17
# Get available Temurin GA releases >= 17
RESPONSE=$(curl -sf https://api.adoptium.net/v3/info/available_releases || echo "")
if [ -z "$RESPONSE" ]; then
echo "Failed to fetch Adoptium releases API"
echo "has_new=false" >> $GITHUB_OUTPUT
echo "has_new=false" >> "$GITHUB_OUTPUT"
exit 0
fi

Expand All @@ -41,12 +57,12 @@ jobs:
fi
done

echo "new_versions=${NEW_VERSIONS}" >> $GITHUB_OUTPUT
echo "new_versions=${NEW_VERSIONS}" >> "$GITHUB_OUTPUT"
if [ -n "$NEW_VERSIONS" ]; then
echo "has_new=true" >> $GITHUB_OUTPUT
echo "has_new=true" >> "$GITHUB_OUTPUT"
echo "Found new JDK versions: $NEW_VERSIONS"
else
echo "has_new=false" >> $GITHUB_OUTPUT
echo "has_new=false" >> "$GITHUB_OUTPUT"
echo "No new JDK versions found"
fi

Expand All @@ -61,7 +77,7 @@ jobs:
# which falsely includes long-EOL majors (10, 12, 14, 16, 18).
if ! SCHEDULE=$(curl -sf https://raw.githubusercontent.com/nodejs/Release/main/schedule.json); then
echo "Failed to fetch Node release schedule"
echo "has_new=false" >> $GITHUB_OUTPUT
echo "has_new=false" >> "$GITHUB_OUTPUT"
exit 0
fi

Expand All @@ -88,12 +104,12 @@ jobs:
fi
done

echo "new_versions=${NEW_VERSIONS}" >> $GITHUB_OUTPUT
echo "new_versions=${NEW_VERSIONS}" >> "$GITHUB_OUTPUT"
if [ -n "$NEW_VERSIONS" ]; then
echo "has_new=true" >> $GITHUB_OUTPUT
echo "has_new=true" >> "$GITHUB_OUTPUT"
echo "Found new Node LTS versions: $NEW_VERSIONS"
else
echo "has_new=false" >> $GITHUB_OUTPUT
echo "has_new=false" >> "$GITHUB_OUTPUT"
echo "No new Node LTS versions found"
fi

Expand All @@ -107,11 +123,12 @@ jobs:
# are listed too), which then break the unattended Docker build when
# sdkmanager demands the separate Android SDK Preview license. The manifest
# carries reliable preview markers, so we parse it with a real XML parser
# and skip any package that is obsolete, carries a <codename> element, or
# uses the android-sdk-preview-license.
# and skip any package that is obsolete, carries a non-empty <codename>
# (stable platforms have an empty one), or uses the
# android-sdk-preview-license.
if ! wget -q "https://dl.google.com/android/repository/repository2-3.xml" -O /tmp/android-repo.xml; then
echo "Failed to download SDK repository manifest"
echo "updated=false" >> $GITHUB_OUTPUT
echo "updated=false" >> "$GITHUB_OUTPUT"
exit 0
fi
if ! SDK_VERSIONS=$(python3 - /tmp/android-repo.xml <<'PY'
Expand All @@ -126,7 +143,10 @@ jobs:
if pkg.get('obsolete') == 'true':
return True
for e in pkg.iter():
if local(e.tag) == 'codename':
# Stable platforms carry an *empty* <codename></codename>
# (e.g. android-37.0/37.1/37.2); previews name the next
# release there (DEV, CinnamonBun, ...).
if local(e.tag) == 'codename' and (e.text or '').strip():
return True
if local(e.tag) == 'uses-license' and e.get('ref') == 'android-sdk-preview-license':
return True
Expand Down Expand Up @@ -159,7 +179,7 @@ jobs:
PY
); then
echo "Failed to parse SDK repository manifest"
echo "updated=false" >> $GITHUB_OUTPUT
echo "updated=false" >> "$GITHUB_OUTPUT"
exit 0
fi
eval "$SDK_VERSIONS"
Expand All @@ -169,12 +189,14 @@ jobs:
CURRENT_BUILD_TOOLS=$(grep '^ARG BUILD_TOOLS=' Dockerfile | cut -d= -f2)
CURRENT_NDK=$(grep '^ARG NDK_VERSION=' Dockerfile | cut -d= -f2)

echo "latest_platform=${LATEST_PLATFORM}" >> $GITHUB_OUTPUT
echo "latest_build_tools=${LATEST_BUILD_TOOLS}" >> $GITHUB_OUTPUT
echo "latest_ndk=${LATEST_NDK}" >> $GITHUB_OUTPUT
echo "current_platform=${CURRENT_PLATFORM}" >> $GITHUB_OUTPUT
echo "current_build_tools=${CURRENT_BUILD_TOOLS}" >> $GITHUB_OUTPUT
echo "current_ndk=${CURRENT_NDK}" >> $GITHUB_OUTPUT
{
echo "latest_platform=${LATEST_PLATFORM}"
echo "latest_build_tools=${LATEST_BUILD_TOOLS}"
echo "latest_ndk=${LATEST_NDK}"
echo "current_platform=${CURRENT_PLATFORM}"
echo "current_build_tools=${CURRENT_BUILD_TOOLS}"
echo "current_ndk=${CURRENT_NDK}"
} >> "$GITHUB_OUTPUT"

# Check if updates are needed
UPDATED=false
Expand All @@ -191,21 +213,31 @@ jobs:
UPDATED=true
fi

echo "updated=${UPDATED}" >> $GITHUB_OUTPUT
echo "updated=${UPDATED}" >> "$GITHUB_OUTPUT"
echo "SDK check results:"
echo " Platform: ${CURRENT_PLATFORM} -> ${LATEST_PLATFORM}"
echo " Build tools: ${CURRENT_BUILD_TOOLS} -> ${LATEST_BUILD_TOOLS}"
echo " NDK: ${CURRENT_NDK} -> ${LATEST_NDK}"

- name: Apply JDK updates
if: steps.jdk.outputs.has_new == 'true'
env:
NEW_VERSIONS: ${{ steps.jdk.outputs.new_versions }}
run: |
for v in ${{ steps.jdk.outputs.new_versions }}; do
jq --arg v "$v" '.java_versions += [$v] | .java_versions = (.java_versions | sort_by(tonumber))' matrix.json > tmp.json && mv tmp.json matrix.json
for v in $NEW_VERSIONS; do
jq --arg v "$v" '.java_versions += [$v]' matrix.json > tmp.json && mv tmp.json matrix.json
done

# Update default to highest version
jq '.default_jdk = (.java_versions | sort_by(tonumber) | last)' matrix.json > tmp.json && mv tmp.json matrix.json
# Retention policy: keep the LTS lines (17, 21, 25, then one every
# two years: 29, 33, ...) plus the newest GA release, and drop any
# other non-LTS version once it is no longer the newest. The default
# is always the newest.
jq '
def lts: . >= 17 and ((. - 17) % 4 == 0);
([.java_versions[] | tonumber] | unique) as $all
| ($all | max) as $newest
| .java_versions = [ $all[] | select(. == $newest or lts) | tostring ]
| .default_jdk = (.java_versions | last)' matrix.json > tmp.json && mv tmp.json matrix.json

echo "Updated matrix.json:"
cat matrix.json
Expand Down Expand Up @@ -238,7 +270,7 @@ jobs:
- name: Create pull request
if: steps.jdk.outputs.has_new == 'true' || steps.sdk.outputs.updated == 'true' || steps.node.outputs.has_new == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_TOKEN: ${{ steps.app-token.outputs.token }}
run: |
BRANCH="automated/sdk-jdk-updates"

Expand All @@ -260,7 +292,7 @@ jobs:
MSG="$MSG SDK"
fi
MSG="$MSG versions"
git commit -m "$MSG"
git commit -m "chore(deps): $MSG"
git push -f origin "$BRANCH"

# Build PR body
Expand Down Expand Up @@ -301,15 +333,13 @@ jobs:
EXISTING=$(gh pr list --head "$BRANCH" --state open --json number --jq '.[0].number' 2>/dev/null || true)
if [ -z "$EXISTING" ]; then
gh pr create \
--title "Update SDK/JDK/Node versions" \
--title "chore(deps): update SDK/JDK/Node versions" \
--body-file /tmp/pr-body.txt \
--head "$BRANCH"
else
echo "PR #${EXISTING} already exists, updated with force push"
fi

# No auto-merge on purpose: this PR is opened with GITHUB_TOKEN, and GitHub
# does not trigger workflows for PRs created that way. build.yml never runs,
# so the "All builds passed" check never reports and an auto-merge request
# would hang forever. Merge manually (which runs the builds), or add a
# fine-grained PAT and restore upstream's auto-merge step.
# The PR is opened with the App token, so build.yml runs and the
# "All builds passed" check reports; auto-merge waits for it.
gh pr merge --auto --squash "$BRANCH"
Loading
Loading