diff --git a/e2e/README.md b/e2e/README.md index 45b8ad3e..eb942c00 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -1,19 +1,22 @@ # Checkout Kit End-to-End Tests -This directory contains Maestro end-to-end smoke flows for Checkout Kit sample -apps. +This directory contains Maestro end-to-end flows and configuration for Checkout +Kit sample apps. Two complementary setups live here: -The current runnable suite starts with React Native. It verifies a full guest -checkout from a seeded cart in the sample app, through Shopify checkout, and -back to the app after completion. +- A **local** guest-checkout smoke flow, run with `dev rn e2e`, that exercises a + full checkout from a seeded cart through Shopify checkout and back to the app. +- A **CI matrix** that expands applications, OS version tags, and suites into + BrowserStack Maestro run rows, starting with a shared launch smoke. ## Run locally -Run the matching command from the repo root. - Run `dev up` first to provision the local toolchain. Install Maestro separately and make sure `maestro --version` succeeds before running these flows. +### Full guest checkout (`dev rn e2e`) + +Run the matching command from the repo root. + React Native iOS: ```bash @@ -31,17 +34,90 @@ sample app, then run Maestro. They require the standard storefront `.env` setup, but the E2E flow seeds its own cart through the bootstrap deep link; no manual sample cart setup is required. +### Shared launch smoke + +The launch smoke launches a sample app and waits for the shared ready marker +exposed by that app, using the same environment contract used by CI. + +React Native iOS: + +```bash +E2E_APP_ID=com.shopify.checkoutkit.reactnativedemo \ +E2E_READY_MARKER=checkout-kit-sample-ready \ +maestro --platform ios test e2e/tests/shared/launch-smoke.yaml +``` + +React Native Android: + +```bash +E2E_APP_ID=com.shopify.checkoutkit.reactnativedemo \ +E2E_READY_MARKER=checkout-kit-sample-ready \ +maestro --platform android test e2e/tests/shared/launch-smoke.yaml +``` + +React Native E2E runs should use the released native SDK artifacts declared by +the React Native sample configuration, not local in-repo native SDK overrides. + +## Matrix + +CI runs are described by `config/matrix.yml`. The matrix expands applications, OS +version tags, and suites into BrowserStack Maestro run rows. Because Bitrise has no +built-in matrix support, `e2e/lib/e2e_matrix.rb` performs the expansion and the +pipeline parallelizes over the resulting rows. + +Current applications: + +- React Native iOS sample app +- React Native Android sample app + +Current OS version tags: + +- `latest` + +Current suites: + +- `tests/shared/launch-smoke.yaml` + +Validate the matrix: + +```bash +ruby e2e/scripts/e2e_matrix validate +``` + +Expand all run rows: + +```bash +ruby e2e/scripts/e2e_matrix expand +``` + +Expand a single run row by index: + +```bash +ruby e2e/scripts/e2e_matrix expand --index 0 +``` + ## Files - `config.yaml` configures Maestro for shared platform behavior. - `flows/` contains reusable Maestro subflows for app setup and checkout steps. - `tests/react-native/full-guest-checkout.yaml` composes the React Native guest checkout smoke test from those subflows. +- `config/matrix.yml`, `lib/e2e_matrix.rb`, and `scripts/` drive the CI matrix. +- `tests/shared/launch-smoke.yaml` is the shared launch smoke suite. + +## Shared app contract + +Shared flows rely on stable cross-app identifiers. The launch smoke requires each +target app to expose this ready marker: + +- `checkout-kit-sample-ready` + +Future shared flows should add identifiers here before they are used across +React Native, Swift, and Android sample apps. ## Scope -This smoke flow is intended to catch regressions in the React Native sample app -integration surface: cart bootstrap, checkout presentation, checkout -completion, and return to the sample app. It is not a replacement for -checkout-web's browser-based coverage or for future native Swift and Android -sample-app E2E coverage. +These flows catch regressions in the React Native sample app integration +surface: cart bootstrap, checkout presentation, checkout completion, and return +to the sample app. They are not a replacement for checkout-web's browser-based +coverage or for future native Swift and Android sample-app E2E coverage. diff --git a/e2e/config/matrix.yml b/e2e/config/matrix.yml new file mode 100644 index 00000000..53608764 --- /dev/null +++ b/e2e/config/matrix.yml @@ -0,0 +1,19 @@ +version: 1 +applications: + - id: react-native-ios + target: react-native + platform: ios + app_id: com.shopify.checkoutkit.reactnativedemo + artifact_env: E2E_REACT_NATIVE_IOS_APP_PATH + ready_marker: checkout-kit-sample-ready + - id: react-native-android + target: react-native + platform: android + app_id: com.shopify.checkoutkit.reactnativedemo + artifact_env: E2E_REACT_NATIVE_ANDROID_APP_PATH + ready_marker: checkout-kit-sample-ready +os_version_tags: + - latest +suites: + - id: launch-smoke + execute: tests/shared/launch-smoke.yaml diff --git a/e2e/lib/e2e_matrix.rb b/e2e/lib/e2e_matrix.rb new file mode 100644 index 00000000..3e780dab --- /dev/null +++ b/e2e/lib/e2e_matrix.rb @@ -0,0 +1,168 @@ +# frozen_string_literal: true + +require "yaml" + +# Expands the compact E2E test matrix in config/matrix.yml into one fully-resolved +# run per (application x os_version_tag x suite) combination. +# +# Unlike GitHub Actions, Bitrise has no built-in matrix/strategy support, so we +# do the fan-out ourselves: `expand` produces the JSON list of runs that the +# pipeline consumes, and Bitrise parallelizes over it via `run_at`/`count`. +# +# Today the numbers make the config and the expansion look equivalent - +# 2 applications + 1 os_version_tag + 1 suite, and 2 * 1 * 1 = 2 runs - so it can look +# like a plain YAML-to-JSON copy. The point is the multiplication, not the copy: +# adding a single os_version_tag (e.g. a minimum-supported OS) transparently duplicates +# every application x suite across that OS, turning an additive config change into +# a multiplicative set of runs without hand-writing each one. +class E2EMatrix + attr_reader :config_path + + def self.load(config_path) + new(config_path, YAML.safe_load_file(config_path)) + end + + def initialize(config_path, config) + @config_path = config_path + @config = config || {} + end + + def expand + applications.flat_map do |application| + os_version_tags.flat_map do |os_version_tag| + suites.map do |suite| + build_run(application, os_version_tag, suite) + end + end + end + end + + def run_at(index) + runs = expand + unless index.is_a?(Integer) && index >= 0 && index < runs.length + raise IndexError, "index #{index} outside of run range 0...#{runs.length}" + end + + runs.fetch(index) + end + + def validation_errors + errors = [] + errors << "version must be 1" unless @config.fetch("version", nil) == 1 + validate_collection(errors, "applications", applications) + validate_collection(errors, "os_version_tags", os_version_tags) + validate_collection(errors, "suites", suites) + validate_applications(errors) + validate_os_version_tags(errors) + validate_suites(errors) + errors + end + + private + + def build_run(application, os_version_tag, suite) + platform = application.fetch("platform") + os_version_tag_id = os_version_tag_id(os_version_tag) + suite_id = suite.fetch("id") + application_id = application.fetch("id") + + { + "id" => "#{application_id}-#{os_version_tag_id}-#{suite_id}", + "application_id" => application_id, + "target" => application.fetch("target"), + "platform" => platform, + "os_version_tag" => os_version_tag_id, + "device_selector" => device_selector(platform, os_version_tag), + "app_id" => application.fetch("app_id"), + "artifact_env" => application.fetch("artifact_env"), + "execute" => suite.fetch("execute"), + "ready_marker" => application.fetch("ready_marker"), + "status_context" => "checkout-kit/e2e/#{application_id}/#{os_version_tag_id}/#{suite_id}" + } + end + + def device_selector(platform, os_version_tag) + return os_version_tag.fetch("device_selector") if os_version_tag.is_a?(Hash) && os_version_tag.key?("device_selector") + + "#{platform}:phone:#{os_version_tag_id(os_version_tag)}" + end + + def os_version_tag_id(os_version_tag) + os_version_tag.is_a?(Hash) ? os_version_tag.fetch("id") : os_version_tag + end + + def applications + @config.fetch("applications", []) || [] + end + + def os_version_tags + @config.fetch("os_version_tags", []) || [] + end + + def suites + @config.fetch("suites", []) || [] + end + + def validate_collection(errors, name, collection) + errors << "#{name} must be a non-empty array" unless collection.is_a?(Array) && !collection.empty? + end + + def validate_applications(errors) + return unless applications.is_a?(Array) + + validate_unique_ids(errors, "application", applications) + applications.each do |application| + id = application.fetch("id", "") + required_application_keys.each do |key| + errors << "application #{id} missing #{key}" if application.fetch(key, "").to_s.empty? + end + platform = application.fetch("platform", nil) + errors << "application #{id} platform must be ios or android" unless ["ios", "android"].include?(platform) + end + end + + def validate_os_version_tags(errors) + return unless os_version_tags.is_a?(Array) + + ids = os_version_tags.map { |os_version_tag| safe_os_version_tag_id(os_version_tag) } + errors << "os_version_tag ids must be unique" unless ids.compact.uniq.length == ids.compact.length + os_version_tags.each do |os_version_tag| + id = safe_os_version_tag_id(os_version_tag) + errors << "os_version_tag missing id" if id.to_s.empty? + end + end + + def safe_os_version_tag_id(os_version_tag) + os_version_tag_id(os_version_tag) + rescue KeyError + nil + end + + def validate_suites(errors) + return unless suites.is_a?(Array) + + validate_unique_ids(errors, "suite", suites) + suites.each do |suite| + id = suite.fetch("id", "") + errors << "suite missing id" if id.to_s.empty? + execute = suite.fetch("execute", "") + errors << "suite #{id} missing execute" if execute.empty? + next if execute.empty? + + errors << "suite #{id} execute path does not exist: #{execute}" unless File.exist?(File.join(e2e_root, execute)) + end + end + + def validate_unique_ids(errors, label, collection) + ids = collection.map { |item| item.fetch("id", nil) } + errors << "#{label} ids must be unique" unless ids.compact.uniq.length == ids.compact.length + end + + def required_application_keys + ["id", "target", "platform", "app_id", "artifact_env", "ready_marker"] + end + + def e2e_root + File.expand_path("..", File.dirname(config_path)) + end +end diff --git a/e2e/scripts/e2e_matrix b/e2e/scripts/e2e_matrix new file mode 100755 index 00000000..de97ae08 --- /dev/null +++ b/e2e/scripts/e2e_matrix @@ -0,0 +1,51 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +require "json" +require "optparse" +require_relative "../lib/e2e_matrix" + +options = { + config: File.expand_path("../config/matrix.yml", __dir__) +} + +parser = OptionParser.new do |opts| + opts.banner = "Usage: e2e/scripts/e2e_matrix validate|expand [options]" + opts.on("--config PATH") { |path| options[:config] = path } + opts.on("--index INDEX", Integer) { |index| options[:index] = index } +end + +command = ARGV.shift +parser.parse!(ARGV) + +unless ["validate", "expand"].include?(command) + warn parser + exit 1 +end + +matrix = E2EMatrix.load(options.fetch(:config)) + +case command +when "validate" + errors = matrix.validation_errors + if errors.empty? + puts "E2E matrix is valid" + else + warn errors.join("\n") + exit 1 + end +when "expand" + errors = matrix.validation_errors + unless errors.empty? + warn errors.join("\n") + exit 1 + end + + output = if options.key?(:index) + matrix.run_at(options.fetch(:index)) + else + matrix.expand + end + + puts JSON.pretty_generate(output) +end diff --git a/e2e/tests/shared/launch-smoke.yaml b/e2e/tests/shared/launch-smoke.yaml new file mode 100644 index 00000000..5f534a86 --- /dev/null +++ b/e2e/tests/shared/launch-smoke.yaml @@ -0,0 +1,7 @@ +appId: ${E2E_APP_ID} +--- +- launchApp +- extendedWaitUntil: + visible: + id: ${E2E_READY_MARKER} + timeout: 60000