[Phase 7] Soft-launch E2E PR checks

e2e/BITRISE.md

@@ -11,14 +11,14 @@ The pipeline runs on the allocated Bitrise app, connected to the Checkout Kit re
 
 Useful Bitrise app URLs:
 
-| Area | URL |
-|---|---|
-| App overview | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76 |
-| Workflow/config editor | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/workflow |
-| Secrets/env vars | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/secrets |
-| Code signing | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/codesigning |
-| Build triggers | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/triggers |
-| Start build | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/build/start |
+| Area                   | URL                                                                         |
+| ---------------------- | --------------------------------------------------------------------------- |
+| App overview           | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76             |
+| Workflow/config editor | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/workflow    |
+| Secrets/env vars       | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/secrets     |
+| Code signing           | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/codesigning |
+| Build triggers         | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/triggers    |
+| Start build            | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/build/start |
 
 If a direct URL does not resolve in the current Bitrise UI, open the app overview and navigate to the matching area from the sidebar.
 
@@ -40,16 +40,23 @@ Validate the configuration locally with:
 bitrise validate -c e2e/bitrise.yml
 ```
 
+## PR trigger
+
+`e2e/bitrise.yml` maps pull requests to the `e2e` pipeline with `trigger_map`. The GitHub checks are kept non-required while the suite stabilizes; merge-blocking will be controlled by branch protections when marked required.
+
+## Duplicate PR build cancellation
+
+Use Bitrise native Rolling builds instead of a repo-owned cancellation script. In Bitrise, open **Project settings > Builds > Build strategy**, enable **Abort builds triggered by pull requests**, and enable **Abort running builds** so duplicate in-progress PR pipelines are cancelled when a newer build starts.
+
 ## Required app environment variables
 
 The non-secret E2E defaults live in `e2e/bitrise.yml` under `app.envs`. Change them in this repository rather than in the Bitrise Workflow Editor.
 
-| Variable | Value | Purpose |
-|---|---|---|
-| `E2E_STRICT` | `false` | Soft/hard failure switch for BrowserStack Maestro runs. |
-| `E2E_BROWSERSTACK_API_RETRIES` | `1` | Retries for transient BrowserStack API responses. |
-| `E2E_BROWSERSTACK_TIMEOUT_SECONDS` | `1800` | BrowserStack build timeout. |
-| `E2E_BROWSERSTACK_POLL_SECONDS` | `30` | BrowserStack status polling interval. |
+| Variable                           | Value  | Purpose                                           |
+| ---------------------------------- | ------ | ------------------------------------------------- |
+| `E2E_BROWSERSTACK_API_RETRIES`     | `1`    | Retries for transient BrowserStack API responses. |
+| `E2E_BROWSERSTACK_TIMEOUT_SECONDS` | `1800` | BrowserStack build timeout.                       |
+| `E2E_BROWSERSTACK_POLL_SECONDS`    | `30`   | BrowserStack status polling interval.             |
 
 Each workflow's main `script` step sets its own wall-clock budget with the Bitrise `timeout` and `no_output_timeout` step properties instead of wrapping individual commands.
 
@@ -59,18 +66,18 @@ The `e2e-execute-browserstack-run` workflow fans out one parallel copy per Brows
 
 These secrets are configured in Bitrise.io; they cannot live in the repository. `scripts/setup_storefront_env` reads them to configure the sample app before builds.
 
-| Secret | Purpose |
-|---|---|
-| `STOREFRONT_DOMAIN` | Storefront domain for sample app builds. |
+| Secret                    | Purpose                                        |
+| ------------------------- | ---------------------------------------------- |
+| `STOREFRONT_DOMAIN`       | Storefront domain for sample app builds.       |
 | `STOREFRONT_ACCESS_TOKEN` | Storefront access token for sample app builds. |
 
 ## BrowserStack secrets
 
 The `e2e-execute-browserstack-run` workflow authenticates with BrowserStack using these secrets, configured in Bitrise.io:
 
-| Secret | Purpose |
-|---|---|
-| `BROWSERSTACK_USERNAME` | BrowserStack API username. |
+| Secret                    | Purpose                      |
+| ------------------------- | ---------------------------- |
+| `BROWSERSTACK_USERNAME`   | BrowserStack API username.   |
 | `BROWSERSTACK_ACCESS_KEY` | BrowserStack API access key. |
 
 BrowserStack artifact links in GitHub reports require access to BrowserStack App Automate. Sign in to [BrowserStack App Automate](https://app-automate.browserstack.com/dashboard/v2/builds) before opening build, video, screenshot, or log links.
@@ -81,13 +88,13 @@ React Native iOS IPA generation uses Bitrise's certificate and profile installer
 
 Upload the signing certificate and provisioning profile for the React Native sample app to the Bitrise app; the iOS artifact workflow installs them before archiving. The iOS build reads the following signing values with the defaults shown, and each can be overridden with a matching Bitrise environment variable:
 
-| Variable | Default | Purpose |
-|---|---|---|
-| `E2E_IOS_EXPORT_METHOD` | `development` | Export method for the React Native iOS IPA. |
-| `E2E_IOS_BUNDLE_ID` | `com.shopify.checkoutkit.reactnativedemo` | Bundle identifier used for iOS archive and export signing. |
-| `E2E_IOS_DEVELOPMENT_TEAM` | `A7XGC83MZE` | Apple development team used for iOS archive signing. |
-| `E2E_IOS_CODE_SIGN_IDENTITY` | `Apple Development` | Code signing identity used for iOS archive and export signing. |
-| `E2E_IOS_PROVISIONING_PROFILE_SPECIFIER` | `bitrise-checkout-kit-e2e` | Provisioning profile specifier installed by Bitrise and passed to `xcodebuild`; override it if the Bitrise-installed profile uses a different name. |
+| Variable                                 | Default                                   | Purpose                                                                                                                                             |
+| ---------------------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `E2E_IOS_EXPORT_METHOD`                  | `development`                             | Export method for the React Native iOS IPA.                                                                                                         |
+| `E2E_IOS_BUNDLE_ID`                      | `com.shopify.checkoutkit.reactnativedemo` | Bundle identifier used for iOS archive and export signing.                                                                                          |
+| `E2E_IOS_DEVELOPMENT_TEAM`               | `A7XGC83MZE`                              | Apple development team used for iOS archive signing.                                                                                                |
+| `E2E_IOS_CODE_SIGN_IDENTITY`             | `Apple Development`                       | Code signing identity used for iOS archive and export signing.                                                                                      |
+| `E2E_IOS_PROVISIONING_PROFILE_SPECIFIER` | `bitrise-checkout-kit-e2e`                | Provisioning profile specifier installed by Bitrise and passed to `xcodebuild`; override it if the Bitrise-installed profile uses a different name. |
 
 ## BrowserStack execution
 

e2e/RUNBOOK.md

@@ -0,0 +1,87 @@
+# Checkout Kit E2E Runbook
+
+## Rollout behaviour
+
+The E2E pipeline always runs and always reports; it never blocks PR merges on its
+own. Merge-blocking is controlled solely by whether the GitHub checks are marked
+**required** in branch protection. Keep them non-required until the suite is
+stable, then make them required — no code change is needed to gate or un-gate.
+
+The runner never hard-fails on test or infrastructure problems: every run writes a
+`result.json` and exits `0`, so the report workflow always has data to publish as
+GitHub commit statuses, Check Runs, and the sticky PR failure comment. Failures
+are therefore loud (red checks and a failure comment) but non-blocking.
+
+Failures land in `result.json` in one of two shapes:
+
+- **Assertion / suite failures** — a BrowserStack build that reports failed test
+  cases or a non-passing terminal status. Recorded by `normalize_result` with
+  `status` set to the build's terminal status (e.g. `"failed"`) and the failing
+  cases under `failed_tests`; there is no `error`/`error_class`.
+- **Runner exceptions** — malformed run plan, out-of-range parallel index, device
+  resolution, artifact upload, build start, polling, and timeouts. Caught and
+  recorded with `status: "error"`, `error`, and `error_class`, with the class,
+  message, and full backtrace logged to the failing step's stderr.
+
+Setup failures that happen before a run plan row is read have no run metadata
+(suite, target, status context), so they appear in the Check Run summary and the
+failure comment but do **not** post a per-suite commit status. This is separate
+from the Bitrise step's own required-env guards: a missing config variable
+(e.g. `BROWSERSTACK_ACCESS_KEY`) is validated by the step and fails fast before
+the runner starts.
+
+## Retry behavior
+
+BrowserStack API calls retry transient infrastructure responses once by default:
+
+```bash
+E2E_BROWSERSTACK_API_RETRIES=1
+```
+
+Retry applies to HTTP 429 and 5xx responses, and only to idempotent read/poll (GET) requests — build-creation and upload requests fail fast to avoid duplicate builds. Test assertion failures are not auto-retried by default so first-failure evidence is preserved.
+
+## Timeouts
+
+BrowserStack polling uses these defaults:
+
+```bash
+E2E_BROWSERSTACK_TIMEOUT_SECONDS=1800
+E2E_BROWSERSTACK_POLL_SECONDS=30
+```
+
+If polling times out, the runner attempts to stop the BrowserStack build before recording the failure.
+
+## Local rerun notes
+
+Use the BrowserStack run plan row from a failure report to identify the app target, platform, OS version tag, and suite:
+
+```bash
+ruby e2e/scripts/e2e_matrix_to_browserstack_run_plan expand --index <index>
+ruby e2e/scripts/e2e_matrix_to_browserstack_run_plan expand > /tmp/browserstack-run-plan.json
+```
+
+Use the reported resolved device to pin a rerun:
+
+```bash
+E2E_DEVICE_OVERRIDE="<resolved BrowserStack device>" \
+e2e/scripts/execute_browserstack_run --index <index> --run-plan /tmp/browserstack-run-plan.json --tests-zip <e2e-tests.zip> --output-dir <results-dir>
+```
+
+The app artifact environment variable for the run plan row must point at the `.apk` or `.ipa` artifact before rerunning.
+
+## Failure triage
+
+Use the GitHub Check Run or sticky PR failure comment first. Failure summaries should include Markdown links to:
+
+- BrowserStack build
+- failed testcase
+- video
+- screenshot
+- Maestro command log
+- Maestro log
+- device log
+- network log when enabled
+
+BrowserStack artifact links require BrowserStack App Automate access. Sign in to [BrowserStack App Automate](https://app-automate.browserstack.com/dashboard/v2/builds) before opening evidence links.
+
+Avoid posting additional PR comments for green runs.

e2e/bitrise.yml

@@ -9,7 +9,6 @@ meta:
 
 app:
   envs:
-    - E2E_STRICT: "false"
     - E2E_BROWSERSTACK_API_RETRIES: "1"
     - E2E_BROWSERSTACK_TIMEOUT_SECONDS: "1800"
     - E2E_BROWSERSTACK_POLL_SECONDS: "30"
@@ -18,6 +17,11 @@ tools:
   ruby: 3.3.6
   nodejs: 22.14.0
 
+trigger_map:
+  - pull_request_source_branch: "*"
+    draft_pull_request_enabled: false
+    pipeline: e2e
+
 pipelines:
   e2e:
     workflows:

e2e/lib/e2e_github_reporter.rb

@@ -8,13 +8,12 @@
 class E2EGitHubReporter
   COMMENT_MARKER = "<!-- checkout-kit-e2e-report -->"
 
-  def initialize(results, repository:, sha:, pr_number:, token: nil, strict: false)
+  def initialize(results, repository:, sha:, pr_number:, token: nil)
     @results = results
     @repository = repository
     @sha = sha
     @pr_number = pr_number
     @token = token
-    @strict = strict
   end
 
   def publish!
@@ -30,30 +29,31 @@ def markdown_summary
     lines << "| Status | Suite | Target | Platform | OS version tag | Device |"
     lines << "|---|---|---|---|---|---|"
     @results.each do |result|
-      lines << "| #{status_icon(result)} | `#{result.fetch("execute")}` | #{result.fetch("target")} | #{result.fetch("platform")} | #{result.fetch("os_version_tag")} | #{device_cell(result)} |"
+      lines << "| #{status_icon(result)} | `#{result["execute"]}` | #{result["target"]} | #{result["platform"]} | #{result["os_version_tag"]} | #{device_cell(result)} |"
     end
     failure_lines = failed_results.flat_map { |result| failure_details(result) }
     unless failure_lines.empty?
       lines << ""
       lines << "## Failures"
       lines << ""
-      if soft_fail?
-        lines << "> [!CAUTION]"
-        lines << "> These are currently non-blocking, but failures may indicate an issue you need to resolve before merging."
-        lines << "> If you believe there is a flaky assertion, please raise a ticket in the #checkout-kit-devs channel so it can be addressed"
-        lines << ""
-      end
+      lines << "> [!CAUTION]"
+      lines << "> These E2E checks are not yet required, so they do not block merging — but a failure may still indicate a real issue to resolve before merging."
+      lines << "> If you believe an assertion is flaky, please raise a ticket in the #checkout-kit-devs channel so it can be addressed."
+      lines << ""
       lines << "> BrowserStack artifacts require BrowserStack access. Sign in to [BrowserStack App Automate](https://app-automate.browserstack.com/dashboard/v2/builds) before opening artifact links."
       lines.concat(failure_lines)
     end
     lines.join("\n")
   end
 
   def commit_status_payloads
-    @results.map do |result|
+    @results.filter_map do |result|
+      context = result["status_context"]
+      next if blank?(context)
+
       {
-        state: result.fetch("passed") ? "success" : "failure",
-        context: result.fetch("status_context"),
+        state: result.fetch("passed", false) ? "success" : "failure",
+        context: context,
         description: status_description(result),
         target_url: browserstack_build_url(result)
       }
@@ -103,11 +103,11 @@ def existing_failure_comment
   end
 
   def failed_results
-    @results.reject { |result| result.fetch("passed") }
+    @results.reject { |result| result.fetch("passed", false) }
   end
 
-  def soft_fail?
-    !@strict
+  def blank?(value)
+    value.nil? || value.to_s.strip.empty?
   end
 
   def failure_details(result)
@@ -129,8 +129,8 @@ def failure_details(result)
   end
 
   def failure_heading(result)
-    suite = File.basename(result.fetch("execute"), ".*")
-    "#{os_label(result.fetch("platform"))} — #{suite}"
+    suite = File.basename(result["execute"].to_s, ".*")
+    "#{os_label(result["platform"])} — #{suite}"
   end
 
   def artifact_links(testcase, result)
@@ -163,26 +163,32 @@ def link_fields
   end
 
   def device_cell(result)
-    name = result.fetch("resolved_device")
+    name = result["resolved_device"]
     version = result["resolved_os_version"]
-    version && !version.empty? ? "#{name}<br>#{os_label(result.fetch("platform"))} #{version}" : name
+    version && !version.empty? ? "#{name}<br>#{os_label(result["platform"])} #{version}" : name
   end
 
   def os_label(platform)
-    platform == "ios" ? "iOS" : "Android"
+    case platform
+    when "ios" then "iOS"
+    when "android" then "Android"
+    else "Unknown"
+    end
   end
 
   def status_icon(result)
-    result.fetch("passed") ? "✅" : "❌"
+    result.fetch("passed", false) ? "✅" : "❌"
   end
 
   def status_description(result)
-    description = result.fetch("passed") ? "passed" : "#{result.fetch("status")} on #{result.fetch("resolved_device")}" 
+    description = result.fetch("passed", false) ? "passed" : "#{result["status"]} on #{result["resolved_device"]}"
     description[0, 140]
   end
 
   def browserstack_build_url(result)
-    "https://app-automate.browserstack.com/dashboard/v2/builds/#{result.fetch("build_id")}" 
+    build_id = result["build_id"]
+    base = "https://app-automate.browserstack.com/dashboard/v2/builds"
+    blank?(build_id) ? base : "#{base}/#{build_id}"
   end
 
   def issue_comments