[Phase1] Add E2E matrix launch smoke foundation by kieran-osgood-shopify · Pull Request #356 · Shopify/checkout-kit

kieran-osgood-shopify · 2026-06-29T15:20:55Z

Stack Context

This is Phase 1 of the BrowserStack Maestro E2E stack for Checkout Kit.

Parent issue: https://github.com/shop/issues-checkout-kit/issues/1095
Epic: https://github.com/shop/issues-checkout-kit/issues/1084

TLDR;

This step achieves taking e2e/config/matrix.yml as input and outputting a set of JSON launch tasks for browserstack to fan out tests across

What?

The E2E pipeline should be matrix-driven to allow regression testing multiple operating system versions.
In this model we define the "applications", list the os_tracks to fan out across, as well as list the suites to run against the application.

Right now the suites run for all applications, but we might want to scope some suites to individual applications (e.g. apple pay for ios devices), this might be better achieved with maestro tags but I've left those out of scope for now.

Why?

Bitrise pipelines dont have built in declarative axis based matrix support like github actions, they allow parallel builds by index, but doesn't support fanning out the way we want.

This PR intentionally avoids Bitrise and BrowserStack execution. It establishes the minimum shared smoke suite and the matrix contract used by later PRs in the stack.

Validation

shadowenv exec -- ruby e2e/test/e2e_matrix_test.rb
shadowenv exec -- ruby e2e/scripts/e2e_matrix validate
shadowenv exec -- ruby e2e/scripts/e2e_matrix expand --index 0

kieran-osgood-shopify · 2026-06-29T15:21:09Z

[Phase 7] Soft-launch E2E PR checks #364 : 2 dependent PRs (#401 , #417 )
[Phase 6] Report E2E results to GitHub #363
[Phase 5] Run BrowserStack Maestro from E2E matrix rows #362
[Phase 4] Build React Native iOS E2E IPA artifact #361
[Phase 3] Build React Native Android E2E APK artifact #360
[Phase 2] Add Bitrise E2E pipeline skeleton #357
[Phase1] Add E2E matrix launch smoke foundation #356 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Assisted-By: devx/6c1e3ad5-96c8-4972-b087-da7ff7b195c3

kiftio · 2026-07-07T15:07:12Z

+
+  def expand
+    applications.flat_map do |application|
+      os_tracks.flat_map do |os_track|


hey, likely very subjective. But I wondered a little bit about the meaning of tracks when first reading

We've got platform.. Would os or os_versions be clearer, or is this just a me problem?

Yeah fair I can make it clearer

i've replaced os_tracks with os_version_tags

kieran-osgood-shopify · 2026-07-06T12:36:43Z

+```bash
+E2E_APP_ID=com.shopify.checkoutkit.reactnativedemo \
+E2E_READY_MARKER=catalog-tab \
+maestro --platform ios test e2e/tests/shared/launch-smoke.yaml


This test will be dropped when its replaced by Kyle's full suite of tests - I just wanted to decouple from his PR's and keep it simple whilst I setup the browserstack tests

kieran-osgood-shopify · 2026-07-06T12:37:40Z

+- `config/matrix.yml`, `lib/e2e_matrix.rb`, and `scripts/` drive the CI matrix.
+- `tests/shared/launch-smoke.yaml` is the shared launch smoke suite.
+
+## Shared app contract


this'll also be removed with kyles pr's as he unifies the sample apps

kieran-osgood-shopify · 2026-07-08T08:45:36Z

+
+  def expand
+    applications.flat_map do |application|
+      os_tracks.flat_map do |os_track|


Yeah fair I can make it clearer

Assisted-By: devx/96a590ff-685a-49ce-8199-0810f0ac2379

tiagocandido · 2026-07-08T14:11:28Z

+    platform: ios
+    app_id: com.shopify.checkoutkit.reactnativedemo
+    artifact_env: E2E_REACT_NATIVE_IOS_APP_PATH
+    ready_marker: catalog-tab


Could we use the existing app-ready marker here? catalog-tab only proves the tab is visible, while checkout-kit-sample-ready is the readiness marker used by the existing E2E launch flow.

Suggested change

ready_marker: catalog-tab

ready_marker: checkout-kit-sample-ready

Good shout - i didn't know that one had merged yet!

Swapped the marker around

tiagocandido · 2026-07-08T14:13:04Z

+
+  def run_at(index)
+    runs = expand
+    runs.fetch(index)


Ruby arrays allow negative indexes, so --index -1 would return the last run instead of failing. Could we reject indexes outside the valid range before fetching?

Suggested change

runs.fetch(index)

unless index.is_a?(Integer) && index >= 0 && index < runs.length

raise IndexError, "index #{index} outside of run range 0...#{runs.length}"

end

runs.fetch(index)

Added the index check above

tiagocandido · 2026-07-08T14:14:01Z

+
+    validate_unique_ids(errors, "suite", suites)
+    suites.each do |suite|
+      id = suite.fetch("id", "<missing>")


Could suite ids be validated here too? A suite with execute but no id can pass validation, then crash later during expansion.

Suggested change

id = suite.fetch("id", "<missing>")

id = suite.fetch("id", "")

errors << "suite missing id" if id.to_s.empty?

tiagocandido · 2026-07-08T14:14:50Z

+require_relative "../lib/e2e_matrix"
+
+options = {
+  config: "e2e/config/matrix.yml"


The default config path currently depends on the caller’s working directory. Could we make the default file-relative so the script works from outside the repo root too?

Suggested change

config: "e2e/config/matrix.yml"

config: File.expand_path("../config/matrix.yml", __dir__)

kieran-osgood-shopify · 2026-07-08T15:12:46Z

Merge activity

Jul 8, 3:12 PM UTC: A user started a stack merge that includes this pull request via Graphite.
Jul 8, 3:12 PM UTC: @kieran-osgood-shopify merged this pull request with Graphite.

github-actions Bot added the #gsd:50662 Rebase Checkout Kit on UCP label Jun 29, 2026

kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-1-matrix branch 4 times, most recently from b630f4e to 1455147 Compare July 2, 2026 13:03

kieran-osgood-shopify mentioned this pull request Jul 3, 2026

[Tophat] Add Tophat integration for Checkout Kit React Native builds #401

Open

11 tasks

kieran-osgood-shopify marked this pull request as ready for review July 6, 2026 08:51

kieran-osgood-shopify requested a review from a team as a code owner July 6, 2026 08:51

kieran-osgood-shopify marked this pull request as draft July 6, 2026 09:29

kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-1-matrix branch from 1455147 to 26d5c48 Compare July 6, 2026 11:18

kieran-osgood-shopify marked this pull request as ready for review July 6, 2026 16:03

kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-1-matrix branch from 26d5c48 to 4f48aa5 Compare July 7, 2026 16:13

kieran-osgood-shopify mentioned this pull request Jul 7, 2026

[Phase 8] Add E2E path filtering for Bitrise #417

Draft

Add E2E matrix launch smoke foundation

3ebf314

Assisted-By: devx/6c1e3ad5-96c8-4972-b087-da7ff7b195c3

kiftio reviewed Jul 8, 2026

View reviewed changes

kieran-osgood-shopify commented Jul 8, 2026

View reviewed changes

kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-1-matrix branch from 4f48aa5 to 155c337 Compare July 8, 2026 10:11

kieran-osgood-shopify mentioned this pull request Jul 8, 2026

[Tophat] Add Bitrise client for Tophat artifact discovery #419

Draft

11 tasks

Document E2EMatrix expansion purpose

b733667

Assisted-By: devx/96a590ff-685a-49ce-8199-0810f0ac2379

kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-1-matrix branch from 155c337 to b733667 Compare July 8, 2026 11:41

tiagocandido reviewed Jul 8, 2026

View reviewed changes

Comment thread e2e/lib/e2e_matrix.rb

tiagocandido reviewed Jul 8, 2026

View reviewed changes

Comment thread e2e/lib/e2e_matrix.rb

tiagocandido reviewed Jul 8, 2026

View reviewed changes

Address E2E matrix launch review feedback

6ab2050

kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-1-matrix branch from 410c55c to 6ab2050 Compare July 8, 2026 15:08

tiagocandido approved these changes Jul 8, 2026

View reviewed changes

kieran-osgood-shopify merged commit 56afe53 into main Jul 8, 2026
25 checks passed

kieran-osgood-shopify deleted the kieran-osgood/e2e-browserstack/phase-1-matrix branch July 8, 2026 15:13

kieran-osgood-shopify changed the title ~~Add E2E matrix launch smoke foundation~~ [Phase1] Add E2E matrix launch smoke foundation Jul 9, 2026

This was referenced Jul 10, 2026

[Tophat] refactor: extract shared browserstack client and reuse http client #433

Draft

[Tophat] feat: add the ability to wait for a build to complete #434

Draft

	ready_marker: catalog-tab
	ready_marker: checkout-kit-sample-ready

	id = suite.fetch("id", "<missing>")
	id = suite.fetch("id", "")
	errors << "suite missing id" if id.to_s.empty?

	config: "e2e/config/matrix.yml"
	config: File.expand_path("../config/matrix.yml", __dir__)

Uh oh!

Conversation

kieran-osgood-shopify commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stack Context

TLDR;

What?

Why?

Validation

Uh oh!

kieran-osgood-shopify commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kieran-osgood-shopify commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kieran-osgood-shopify commented Jun 29, 2026 •

edited

Loading

kieran-osgood-shopify commented Jun 29, 2026 •

edited

Loading

kieran-osgood-shopify commented Jul 8, 2026 •

edited

Loading