Skip to content

test: run the end-to-end tests on the image - #82

Open
uniqueg wants to merge 7 commits into
devfrom
test/79-e2e-image
Open

uniqueg wants to merge 7 commits into
devfrom
test/79-e2e-image

Conversation

@uniqueg

@uniqueg uniqueg commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Closes #79

Summary

  • playwright.config.ts: with E2E_IMAGE set, Playwright starts that Docker image as the app instead of building the checkout. The image runs as it is deployed: as its non-root user, with --read-only --tmpfs /tmp, and with all settings at start (--env NAME, so values never appear on the command line). It uses the host network and gets the mock services' certificate mounted. The image reference and the certificate path are validated, because both go into a shell command.
  • CI: the new job e2e-image ("End-to-end tests (image)") runs the Chromium journey against the image of Build image on pull requests, and against the pushed digest on pushes. publish and CI passed need it, so the ruleset stays unchanged.
  • Release: the new job e2e ("End-to-end tests (release image)") runs the same tests against the exact release digest. Release image (attest, sign, tag) now needs it.
  • Docs: docs/development/index.md explains how to run the tests against a local image, and how to remove a container a killed run left behind. releasing.md and deployment.md list the new checks.

Bug found by the new tests (separate issue and PR to follow)

Against the image, 5 of 11 Chromium tests failed at the redirect after a login. The login callback built its redirects from request.url, which in the image's standalone server is its listening address. So after a login, the browser went to http://0.0.0.0:3000/dashboard, without the session.

  • The first two commits fix it: redirects go to APP_ORIGIN, and fall back to the request's origin, with a log line, if APP_ORIGIN is unset or isn't a URL. Unit tests cover all 5 redirects.
  • They are also on the branch fix/callback-origin, for their own issue and PR, which the maintainer opens. This PR will be updated from dev once that PR is merged, so that only the test: run the end-to-end tests on the image #79 changes remain.
  • The fix changes app code, so it needs Chitransh's review.
  • The deployment at krini.rahtiapp.fi serves a static nginx page, not this image, so no users are affected now.

Verification

  • Image, before the fix: E2E_IMAGE=krini:e2e-local npx playwright test --project chromium → 5 failed, 6 passed.
  • Image, after the fix: CI=1 E2E_IMAGE=krini:e2e-local npx playwright test → 22 passed, in Chromium and Firefox. Afterwards no krini-e2e container was left.
  • Checkout build: CI=1 npx playwright test --project chromium → 11 passed.
  • Leftover container: a container already named krini-e2e makes the run fail within 1 s ("Conflict", exit code 125), not after a timeout.
  • Unit tests: npm run test:coverage → 124 passed, 100% statements, branches, functions and lines.
  • Static checks:
    • bun run lint → clean.
    • bunx tsc --noEmit → clean.
    • actionlint (with shellcheck) → clean.
    • mkdocs build --strict → clean.
  • Reviews: two rounds of adversarial-reviewer and quality-reviewer. No high findings. All medium and low findings are fixed, except the one declined below.
  • CI: green on f5b39bc, including End-to-end tests (image) (57 s) and CI passed.
    • The first run failed at installing Chromium: ubuntu-latest is now Ubuntu 26.04, which Playwright 1.56 doesn't support.
    • The two image jobs therefore pin ubuntu-24.04, with a comment saying when to drop the pin.
    • The existing e2e job isn't affected, because it runs in Playwright's image.
  • Not run: the release workflow's e2e job only runs on a version tag. It uses the same steps as e2e-image, apart from pulling the image by digest, which follows the release smoke job.

Declined review findings

  • Duplicated steps (medium): the digest check with docker pull, and the Playwright setup, are repeated across the CI and release jobs. This follows the existing pattern of the image, scan and smoke jobs. Moving them into a shared script would change those existing jobs as well, which is beyond test: run the end-to-end tests on the image #79.

Open questions

None.

🤖 Generated with Claude Code

Summary by Sourcery

Validate deployed Docker images with end-to-end tests throughout CI and release workflows, while making authentication redirects reliable in standalone deployments.

New Features:

  • Run Chromium end-to-end tests against the deployed Docker image in CI and against the exact release image digest before signing and tagging.

Bug Fixes:

  • Ensure authentication callback redirects use the configured application origin, with a safe request-origin fallback for missing or invalid configuration.

Enhancements:

  • Support local end-to-end testing with Docker images using deployment-like container settings and validate the image and certificate inputs.

CI:

  • Gate image publishing, CI completion, and release image promotion on the new image-based end-to-end tests.

Documentation:

  • Document image-based end-to-end testing, leftover container cleanup, release checks, and the application-origin configuration.

Tests:

  • Add coverage for authentication callback redirects across successful and error flows, including invalid or missing application origins.

uniqueg and others added 6 commits November 14, 2026 23:31
The login callback built its redirects from request.url. The image's
standalone server sees its listening address as the origin of requests
(http://0.0.0.0:3000), so after a login it sent the browser to
http://0.0.0.0:3000/dashboard, where the session cookies (set for the
real origin) don't apply. The redirects now go to APP_ORIGIN, and to the
origin of the request only when it isn't set. Found by running the
end-to-end tests against the image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An APP_ORIGIN without a scheme made the callback throw (a 500, or a
misleading log line in the token error path). It now logs the problem
and redirects to the origin of the request, as without the setting.
The tests cover every redirect of the callback with a request at the
listening address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With E2E_IMAGE set, Playwright starts that Docker image as the app
instead of building the checkout: as the image's non-root user, with a
read-only root file system, a writable /tmp and all settings at start,
on the host network, with the mock services' certificate mounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's new e2e-image job runs the Chromium journey against the image of
the Build image job (pull requests) or the pushed digest (pushes), and
both publish and CI passed need it. The release workflow's new e2e job
does the same with the release digest before release-image attests,
signs and tags it. The development docs say how to run the tests
against a local image; the release and deployment docs list the new
checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The certificate directory is resolved from the config's directory, where
the mock services write it, and like the image it may hold only plain
characters, as both are part of a shell command. Comments say why
Playwright runs on the runner for the image, and the docs how to remove
a container that a killed run left behind and that Release image waits
for the end-to-end tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@uniqueg uniqueg self-assigned this Nov 14, 2026
@vercel

vercel Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
krini Ready Ready Preview Nov 14, 2026 10:49pm UTC

@sourcery-ai

sourcery-ai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

API

Make every authentication callback redirect use APP_ORIGIN when valid, falling back to the request origin when it is unset or invalid.

  • src/app/auth/api/callback/route.ts L6-28 (+5 small edits)

Configuration

Configure Playwright to run the app from a validated Docker image under deployment-like constraints, and add CI and release gates that test the relevant image before publishing or releasing it.

  • .github/workflows/ci.yml L6-11, L332-398 (+3 small edits)
  • .github/workflows/release.yml L8-9, L308-358
  • playwright.config.ts L16-20, L25-92, L124-127 (+2 small edits)

Documentation

Document local image-based end-to-end runs and update the deployment and release descriptions to include the new image checks.

  • docs/development/index.md L106-124
  • docs/development/releasing.md L133-137
  • docs/operator-guide/configuration.md L49-51
  • docs/operator-guide/deployment.md L49-50, L130-131

Tests

Cover callback redirects to the configured origin and the request-origin fallback across success and error cases.

  • src/app/auth/api/callback/route.test.ts L38-48, L60-63, L106-147 (+1 small edits)

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (6d208e1) to head (f5b39bc).

Additional details and impacted files
@@            Coverage Diff            @@
##               dev       #82   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           12        12           
  Lines          273       279    +6     
  Branches        77        78    +1     
=========================================
+ Hits           273       279    +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ubuntu-latest is Ubuntu 26.04 now, where Playwright 1.56 can't install
Chromium ("Playwright does not support chromium on ubuntu26.04-x64").
The image tests run Playwright on the runner, so they pin Ubuntu 24.04
until @playwright/test supports 26.04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@uniqueg
uniqueg marked this pull request as ready for review November 14, 2026 22:51
@uniqueg
uniqueg requested a review from AnonymousXC as a code owner November 14, 2026 22:51
@sourcery-ai

sourcery-ai Bot commented Nov 14, 2026

Copy link
Copy Markdown

Linked issues

Checked at commit f5b39bc

#79 test: run the end-to-end tests on the image · 3 of 4 met

# Requirement Status Notes
1 Run the end-to-end journey against the built image, started with the test settings and mock services. ✅ Met The e2e-image job runs Chromium with E2E_IMAGE set to the built image; the Playwright config starts it with the deployment constraints, test settings, and mounted mock-service certificate.
2 Make CI passed need the image end-to-end tests. ✅ Met The CI passed job includes e2e-image in its needs.
3 Run the same checks against the exact release digest before tagging and publishing it. ✅ Met The release e2e job pulls and tests the pushed digest, and release-image depends on it.
4 Document in the README's Tests section how to run the journey against a local image. 🟡 Partly met The local-image instructions are added to docs/development/index.md, but the README's Tests section is not updated.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="src/app/auth/api/callback/route.ts" line_range="26" />
<code_context>
+            );
+        }
+    }
+    return new URL(path, request.url);
+}
+
</code_context>
<issue_to_address>
**Logout fails after fallback login**

When `APP_ORIGIN` is set to a non-URL value and the callback otherwise succeeds, `appUrl` lets the callback complete with a malformed `APP_ORIGIN`, but the logout route still returns 500 for that setting without clearing session cookies. The user can log in but cannot log out through the sidebar.

Make logout handle the invalid setting consistently with the callback, or reject the invalid configuration rather than allowing login to proceed.
</issue_to_address>

### Comment 2
<location path="docs/development/index.md" line_range="106-119" />
<code_context>
     tests in the Playwright image, as CI does (see the `e2e` job in
     `.github/workflows/ci.yml`).

+    To run them against a Docker image instead of the build of the
+    checkout, set `E2E_IMAGE` to the image. Playwright then starts it as it
+    is deployed: as the image's non-root user, with a read-only root file
+    system, a writable `/tmp` and all settings at start. A difference
+    between the two, such as a file missing from the standalone output or a
+    setting read at build time, fails the tests then. This needs Docker with
+    the host network (Linux) and a free container name `krini-e2e`. CI does
+    this in Chromium for every image it builds (`e2e-image` job), and the
+    release workflow for every release image (`e2e` job):
+
+    ```bash
+    docker build -t krini:local .
+    E2E_IMAGE=krini:local bun run test:e2e --project chromium
+    ```
+
+    The container is removed when the tests end. If a run was killed and
</code_context>
<issue_to_address>
**README test instructions**

The issue asks for local-image instructions in the README's Tests section. The change adds those instructions to `docs/development/index.md` instead.
</issue_to_address>

Sourcery assessment

Approval pending. 1 finding to address first.

Blocking findings: src/app/auth/api/callback/route.ts:26


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread src/app/auth/api/callback/route.ts Resolved
Comment thread docs/development/index.md Resolved

This branch had an error being deployed

1 failed deployment
Preview – krini — f5b39bc2 Deployed Nov 14, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: run the end-to-end tests on the image

1 participant