ci: DH-20051: shard e2e tests and merge shard reports by SimonVutov · Pull Request #1361 · deephaven/deephaven-plugins

SimonVutov · 2026-06-11T19:19:11Z

This PR shards the end-to-end workflow to reduce overall time and keep full test coverage.
Before: 51-53 mins
After: 22 mins

Changes:

Workflow fan-out: e2e.yml now runs a matrix of browser/shard configs: chromium 1/2, chromium 2/2, firefox 1/2, firefox 2/2, webkit 1/2, webkit 2/2.
Shard execution: Each matrix job parses its browser and shard index and runs Playwright with project plus shard arguments.
Report fan-in: Each shard uploads a blob artifact. A merge-reports job downloads shard blobs and merges them into one final HTML report artifact.
Container output persistence: docker-compose.yml mounts blob-report so shard blob outputs are available to artifact upload.

CI reliability fixes:

playwright-docker.config.ts no longer overrides reporter in CI, allowing workflow-level blob reporter selection.
Dockerfile removes large build-essential install and keeps minimal font install to avoid no-space-left failures on runners.
docker-compose.yml removes obsolete compose version field warning.

Current e2e tests:

e2e tests with sharding:

More details, showing how long each separate shard takes:

mofojed

Have you tried this with a failed run to ensure the failure is reported correctly and you can download the report/watch the video?
Local e2e tests still work correctly?

Looking good so far

mofojed · 2026-06-12T13:08:09Z

+        with:
+          name: playwright-report
+          path: playwright-report/
+          retention-days: 90


We don't need to retain for 90 days, 30 days is fine.

mofojed · 2026-06-12T13:08:32Z

+            exit 1
+          fi
+
+          npx -y playwright@1.44.1 merge-reports --reporter html,github ./all-blob-reports


We should probably update playwright as well. v1.44.1 is pretty old. That could be a separate PR though.

mofojed · 2026-06-12T13:10:42Z


 jobs:
-  e2e-test:
+  e2e-tests:


While we should rename this step, we still should have an e2e-test step at the end that passes/fails based on the results. Reason being the branch protection requires an e2e-test step to pass before you can merge a PR:

In web-client-ui we consolidate the result into e2e-results, but I think we should just name it e2e-test here so that it matches the existing branch protection rules... otherwise I need to change those rules as well, and change it for the release branches... easier to just keep the same name.
https://github.com/deephaven/web-client-ui/blob/caeabb02f0abc824a07c505d0a7ca8693dd63cbb/.github/workflows/e2e.yml#L125

mofojed · 2026-06-12T13:12:14Z

+    needs: [e2e-tests]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4


We should use newer versions of these actions, these are old and deprecated

mofojed · 2026-06-12T13:13:40Z

+    strategy:
+      fail-fast: false
+      matrix:
+        config: [chromium-1-2, chromium-2-2, firefox-1-2, firefox-2-2, webkit-1-2, webkit-2-2]


How did you decide on this sharding? Did you take a look at how long each shard takes individually? I seem to recall webkit taking longer which is why we split up webkit on web-client-ui but not chromium/firefox.

Each shard is currently taking about 15 to 22 minutes to complete. I added a screenshot of the shard timings to the PR description for reference. We could split WebKit into 3 shards instead of 2, which might save another couple of minutes, but I think the current setup is reasonably well balanced for now.

mofojed · 2026-06-12T13:14:15Z

+        config: [chromium-1-2, chromium-2-2, firefox-1-2, firefox-2-2, webkit-1-2, webkit-2-2]
    steps:
      - name: Checkout
        uses: actions/checkout@v4


Update this action

mofojed · 2026-06-12T13:17:59Z

    apt-get update; \
-    apt-get install build-essential --yes; \
-    apt-get install fonts-dejavu-core --yes;
+    apt-get install --no-install-recommends -y fonts-dejavu-core; \


Always prefer the long form of flags in scripts - it makes it clearer to the casual reader.

Suggested change

apt-get install --no-install-recommends -y fonts-dejavu-core; \

apt-get install --no-install-recommends --yes fonts-dejavu-core; \

mofojed · 2026-06-12T13:19:51Z

    branches:
      - main
      - 'release/**'
+  workflow_dispatch:


I guess there's no reason to remove this. Just probably isn't useful unless for testing.

mofojed · 2026-06-12T13:20:32Z

      - name: Run tests
-        run: './tools/run_docker.sh ./tests/docker-compose.yml e2e-tests'
+        run: |
+          ./tools/run_docker.sh ./tests/docker-compose.yml e2e-tests \


Does each shard need to rebuild the docker image separately? Or do they share the same build image so they don't need to each re-build it?

Yes, each shard currently rebuilds the Docker image separately. We could change this to build once in a dedicated job and reuse that image across the shard jobs, which would save resources. Let me know if you want me to make that change.

I'd like to do that if possible

use newer versions of action prefer the long form of flags remove workflow_dispatch e2e-results 30 days instead of 90

Update e2e.yml

abaa0ce

SimonVutov requested a review from mofojed June 11, 2026 19:19

SimonVutov self-assigned this Jun 11, 2026

mofojed requested changes Jun 12, 2026

View reviewed changes

SimonVutov added 5 commits June 12, 2026 11:45

fix: version, flags, workflow_dispatch, e2e-results, 30 days

794dcee

use newer versions of action prefer the long form of flags remove workflow_dispatch e2e-results 30 days instead of 90

test failing run

ba7fe48

remove failing run line

e14cfbe

HTML gets produced even when tests fail

bf054b5

build once in a dedicated job and reuse that image across the shards

a7e704c

SimonVutov requested a review from mofojed June 12, 2026 17:44

fix version to v7/v8

a93e9b0

	apt-get install --no-install-recommends -y fonts-dejavu-core; \
	apt-get install --no-install-recommends --yes fonts-dejavu-core; \

Conversation

SimonVutov commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes:

Uh oh!

mofojed left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SimonVutov commented Jun 11, 2026 •

edited

Loading