-
Notifications
You must be signed in to change notification settings - Fork 1
[Phase1] Add E2E matrix launch smoke foundation #356
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,19 +1,22 @@ | ||
| # Checkout Kit End-to-End Tests | ||
|
|
||
| This directory contains Maestro end-to-end smoke flows for Checkout Kit sample | ||
| apps. | ||
| This directory contains Maestro end-to-end flows and configuration for Checkout | ||
| Kit sample apps. Two complementary setups live here: | ||
|
|
||
| The current runnable suite starts with React Native. It verifies a full guest | ||
| checkout from a seeded cart in the sample app, through Shopify checkout, and | ||
| back to the app after completion. | ||
| - A **local** guest-checkout smoke flow, run with `dev rn e2e`, that exercises a | ||
| full checkout from a seeded cart through Shopify checkout and back to the app. | ||
| - A **CI matrix** that expands applications, OS version tags, and suites into | ||
| BrowserStack Maestro run rows, starting with a shared launch smoke. | ||
|
|
||
| ## Run locally | ||
|
|
||
| Run the matching command from the repo root. | ||
|
|
||
| Run `dev up` first to provision the local toolchain. Install Maestro separately | ||
| and make sure `maestro --version` succeeds before running these flows. | ||
|
|
||
| ### Full guest checkout (`dev rn e2e`) | ||
|
|
||
| Run the matching command from the repo root. | ||
|
|
||
| React Native iOS: | ||
|
|
||
| ```bash | ||
|
|
@@ -31,17 +34,90 @@ sample app, then run Maestro. They require the standard storefront `.env` setup, | |
| but the E2E flow seeds its own cart through the bootstrap deep link; no manual | ||
| sample cart setup is required. | ||
|
|
||
| ### Shared launch smoke | ||
|
|
||
| The launch smoke launches a sample app and waits for the shared ready marker | ||
| exposed by that app, using the same environment contract used by CI. | ||
|
|
||
| React Native iOS: | ||
|
|
||
| ```bash | ||
| E2E_APP_ID=com.shopify.checkoutkit.reactnativedemo \ | ||
| E2E_READY_MARKER=checkout-kit-sample-ready \ | ||
| maestro --platform ios test e2e/tests/shared/launch-smoke.yaml | ||
| ``` | ||
|
|
||
| React Native Android: | ||
|
|
||
| ```bash | ||
| E2E_APP_ID=com.shopify.checkoutkit.reactnativedemo \ | ||
| E2E_READY_MARKER=checkout-kit-sample-ready \ | ||
| maestro --platform android test e2e/tests/shared/launch-smoke.yaml | ||
| ``` | ||
|
|
||
| React Native E2E runs should use the released native SDK artifacts declared by | ||
| the React Native sample configuration, not local in-repo native SDK overrides. | ||
|
|
||
| ## Matrix | ||
|
|
||
| CI runs are described by `config/matrix.yml`. The matrix expands applications, OS | ||
| version tags, and suites into BrowserStack Maestro run rows. Because Bitrise has no | ||
| built-in matrix support, `e2e/lib/e2e_matrix.rb` performs the expansion and the | ||
| pipeline parallelizes over the resulting rows. | ||
|
|
||
| Current applications: | ||
|
|
||
| - React Native iOS sample app | ||
| - React Native Android sample app | ||
|
|
||
| Current OS version tags: | ||
|
|
||
| - `latest` | ||
|
|
||
| Current suites: | ||
|
|
||
| - `tests/shared/launch-smoke.yaml` | ||
|
|
||
| Validate the matrix: | ||
|
|
||
| ```bash | ||
| ruby e2e/scripts/e2e_matrix validate | ||
| ``` | ||
|
|
||
| Expand all run rows: | ||
|
|
||
| ```bash | ||
| ruby e2e/scripts/e2e_matrix expand | ||
| ``` | ||
|
|
||
| Expand a single run row by index: | ||
|
|
||
| ```bash | ||
| ruby e2e/scripts/e2e_matrix expand --index 0 | ||
| ``` | ||
|
|
||
| ## Files | ||
|
|
||
| - `config.yaml` configures Maestro for shared platform behavior. | ||
| - `flows/` contains reusable Maestro subflows for app setup and checkout steps. | ||
| - `tests/react-native/full-guest-checkout.yaml` composes the React Native guest | ||
| checkout smoke test from those subflows. | ||
| - `config/matrix.yml`, `lib/e2e_matrix.rb`, and `scripts/` drive the CI matrix. | ||
| - `tests/shared/launch-smoke.yaml` is the shared launch smoke suite. | ||
|
|
||
| ## Shared app contract | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this'll also be removed with kyles pr's as he unifies the sample apps |
||
|
|
||
| Shared flows rely on stable cross-app identifiers. The launch smoke requires each | ||
| target app to expose this ready marker: | ||
|
|
||
| - `checkout-kit-sample-ready` | ||
|
|
||
| Future shared flows should add identifiers here before they are used across | ||
| React Native, Swift, and Android sample apps. | ||
|
|
||
| ## Scope | ||
|
|
||
| This smoke flow is intended to catch regressions in the React Native sample app | ||
| integration surface: cart bootstrap, checkout presentation, checkout | ||
| completion, and return to the sample app. It is not a replacement for | ||
| checkout-web's browser-based coverage or for future native Swift and Android | ||
| sample-app E2E coverage. | ||
| These flows catch regressions in the React Native sample app integration | ||
| surface: cart bootstrap, checkout presentation, checkout completion, and return | ||
| to the sample app. They are not a replacement for checkout-web's browser-based | ||
| coverage or for future native Swift and Android sample-app E2E coverage. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| version: 1 | ||
| applications: | ||
| - id: react-native-ios | ||
| target: react-native | ||
| platform: ios | ||
| app_id: com.shopify.checkoutkit.reactnativedemo | ||
| artifact_env: E2E_REACT_NATIVE_IOS_APP_PATH | ||
| ready_marker: checkout-kit-sample-ready | ||
| - id: react-native-android | ||
| target: react-native | ||
| platform: android | ||
| app_id: com.shopify.checkoutkit.reactnativedemo | ||
| artifact_env: E2E_REACT_NATIVE_ANDROID_APP_PATH | ||
| ready_marker: checkout-kit-sample-ready | ||
| os_version_tags: | ||
| - latest | ||
| suites: | ||
| - id: launch-smoke | ||
| execute: tests/shared/launch-smoke.yaml |
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,168 @@ | ||||||||||||||
| # frozen_string_literal: true | ||||||||||||||
|
|
||||||||||||||
| require "yaml" | ||||||||||||||
|
|
||||||||||||||
| # Expands the compact E2E test matrix in config/matrix.yml into one fully-resolved | ||||||||||||||
| # run per (application x os_version_tag x suite) combination. | ||||||||||||||
| # | ||||||||||||||
| # Unlike GitHub Actions, Bitrise has no built-in matrix/strategy support, so we | ||||||||||||||
| # do the fan-out ourselves: `expand` produces the JSON list of runs that the | ||||||||||||||
| # pipeline consumes, and Bitrise parallelizes over it via `run_at`/`count`. | ||||||||||||||
| # | ||||||||||||||
| # Today the numbers make the config and the expansion look equivalent - | ||||||||||||||
| # 2 applications + 1 os_version_tag + 1 suite, and 2 * 1 * 1 = 2 runs - so it can look | ||||||||||||||
| # like a plain YAML-to-JSON copy. The point is the multiplication, not the copy: | ||||||||||||||
| # adding a single os_version_tag (e.g. a minimum-supported OS) transparently duplicates | ||||||||||||||
| # every application x suite across that OS, turning an additive config change into | ||||||||||||||
| # a multiplicative set of runs without hand-writing each one. | ||||||||||||||
| class E2EMatrix | ||||||||||||||
|
tiagocandido marked this conversation as resolved.
|
||||||||||||||
| attr_reader :config_path | ||||||||||||||
|
|
||||||||||||||
| def self.load(config_path) | ||||||||||||||
| new(config_path, YAML.safe_load_file(config_path)) | ||||||||||||||
|
tiagocandido marked this conversation as resolved.
|
||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def initialize(config_path, config) | ||||||||||||||
| @config_path = config_path | ||||||||||||||
| @config = config || {} | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def expand | ||||||||||||||
| applications.flat_map do |application| | ||||||||||||||
| os_version_tags.flat_map do |os_version_tag| | ||||||||||||||
| suites.map do |suite| | ||||||||||||||
| build_run(application, os_version_tag, suite) | ||||||||||||||
| end | ||||||||||||||
| end | ||||||||||||||
| end | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def run_at(index) | ||||||||||||||
| runs = expand | ||||||||||||||
| unless index.is_a?(Integer) && index >= 0 && index < runs.length | ||||||||||||||
| raise IndexError, "index #{index} outside of run range 0...#{runs.length}" | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| runs.fetch(index) | ||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ruby arrays allow negative indexes, so
Suggested change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added the index check above |
||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def validation_errors | ||||||||||||||
| errors = [] | ||||||||||||||
| errors << "version must be 1" unless @config.fetch("version", nil) == 1 | ||||||||||||||
| validate_collection(errors, "applications", applications) | ||||||||||||||
| validate_collection(errors, "os_version_tags", os_version_tags) | ||||||||||||||
| validate_collection(errors, "suites", suites) | ||||||||||||||
| validate_applications(errors) | ||||||||||||||
| validate_os_version_tags(errors) | ||||||||||||||
| validate_suites(errors) | ||||||||||||||
| errors | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| private | ||||||||||||||
|
|
||||||||||||||
| def build_run(application, os_version_tag, suite) | ||||||||||||||
| platform = application.fetch("platform") | ||||||||||||||
| os_version_tag_id = os_version_tag_id(os_version_tag) | ||||||||||||||
| suite_id = suite.fetch("id") | ||||||||||||||
| application_id = application.fetch("id") | ||||||||||||||
|
|
||||||||||||||
| { | ||||||||||||||
| "id" => "#{application_id}-#{os_version_tag_id}-#{suite_id}", | ||||||||||||||
| "application_id" => application_id, | ||||||||||||||
| "target" => application.fetch("target"), | ||||||||||||||
| "platform" => platform, | ||||||||||||||
| "os_version_tag" => os_version_tag_id, | ||||||||||||||
| "device_selector" => device_selector(platform, os_version_tag), | ||||||||||||||
| "app_id" => application.fetch("app_id"), | ||||||||||||||
| "artifact_env" => application.fetch("artifact_env"), | ||||||||||||||
| "execute" => suite.fetch("execute"), | ||||||||||||||
| "ready_marker" => application.fetch("ready_marker"), | ||||||||||||||
| "status_context" => "checkout-kit/e2e/#{application_id}/#{os_version_tag_id}/#{suite_id}" | ||||||||||||||
| } | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def device_selector(platform, os_version_tag) | ||||||||||||||
| return os_version_tag.fetch("device_selector") if os_version_tag.is_a?(Hash) && os_version_tag.key?("device_selector") | ||||||||||||||
|
|
||||||||||||||
| "#{platform}:phone:#{os_version_tag_id(os_version_tag)}" | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def os_version_tag_id(os_version_tag) | ||||||||||||||
| os_version_tag.is_a?(Hash) ? os_version_tag.fetch("id") : os_version_tag | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def applications | ||||||||||||||
| @config.fetch("applications", []) || [] | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def os_version_tags | ||||||||||||||
| @config.fetch("os_version_tags", []) || [] | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def suites | ||||||||||||||
| @config.fetch("suites", []) || [] | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def validate_collection(errors, name, collection) | ||||||||||||||
| errors << "#{name} must be a non-empty array" unless collection.is_a?(Array) && !collection.empty? | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def validate_applications(errors) | ||||||||||||||
| return unless applications.is_a?(Array) | ||||||||||||||
|
|
||||||||||||||
| validate_unique_ids(errors, "application", applications) | ||||||||||||||
| applications.each do |application| | ||||||||||||||
| id = application.fetch("id", "<missing>") | ||||||||||||||
| required_application_keys.each do |key| | ||||||||||||||
| errors << "application #{id} missing #{key}" if application.fetch(key, "").to_s.empty? | ||||||||||||||
| end | ||||||||||||||
| platform = application.fetch("platform", nil) | ||||||||||||||
| errors << "application #{id} platform must be ios or android" unless ["ios", "android"].include?(platform) | ||||||||||||||
| end | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def validate_os_version_tags(errors) | ||||||||||||||
| return unless os_version_tags.is_a?(Array) | ||||||||||||||
|
|
||||||||||||||
| ids = os_version_tags.map { |os_version_tag| safe_os_version_tag_id(os_version_tag) } | ||||||||||||||
| errors << "os_version_tag ids must be unique" unless ids.compact.uniq.length == ids.compact.length | ||||||||||||||
| os_version_tags.each do |os_version_tag| | ||||||||||||||
| id = safe_os_version_tag_id(os_version_tag) | ||||||||||||||
| errors << "os_version_tag missing id" if id.to_s.empty? | ||||||||||||||
| end | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def safe_os_version_tag_id(os_version_tag) | ||||||||||||||
| os_version_tag_id(os_version_tag) | ||||||||||||||
| rescue KeyError | ||||||||||||||
| nil | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def validate_suites(errors) | ||||||||||||||
| return unless suites.is_a?(Array) | ||||||||||||||
|
|
||||||||||||||
| validate_unique_ids(errors, "suite", suites) | ||||||||||||||
| suites.each do |suite| | ||||||||||||||
| id = suite.fetch("id", "") | ||||||||||||||
| errors << "suite missing id" if id.to_s.empty? | ||||||||||||||
| execute = suite.fetch("execute", "") | ||||||||||||||
| errors << "suite #{id} missing execute" if execute.empty? | ||||||||||||||
| next if execute.empty? | ||||||||||||||
|
|
||||||||||||||
| errors << "suite #{id} execute path does not exist: #{execute}" unless File.exist?(File.join(e2e_root, execute)) | ||||||||||||||
| end | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def validate_unique_ids(errors, label, collection) | ||||||||||||||
| ids = collection.map { |item| item.fetch("id", nil) } | ||||||||||||||
| errors << "#{label} ids must be unique" unless ids.compact.uniq.length == ids.compact.length | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def required_application_keys | ||||||||||||||
| ["id", "target", "platform", "app_id", "artifact_env", "ready_marker"] | ||||||||||||||
| end | ||||||||||||||
|
|
||||||||||||||
| def e2e_root | ||||||||||||||
| File.expand_path("..", File.dirname(config_path)) | ||||||||||||||
| end | ||||||||||||||
| end | ||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| #!/usr/bin/env ruby | ||
| # frozen_string_literal: true | ||
|
|
||
| require "json" | ||
| require "optparse" | ||
| require_relative "../lib/e2e_matrix" | ||
|
|
||
| options = { | ||
| config: File.expand_path("../config/matrix.yml", __dir__) | ||
| } | ||
|
|
||
| parser = OptionParser.new do |opts| | ||
| opts.banner = "Usage: e2e/scripts/e2e_matrix validate|expand [options]" | ||
| opts.on("--config PATH") { |path| options[:config] = path } | ||
| opts.on("--index INDEX", Integer) { |index| options[:index] = index } | ||
| end | ||
|
|
||
| command = ARGV.shift | ||
| parser.parse!(ARGV) | ||
|
|
||
| unless ["validate", "expand"].include?(command) | ||
| warn parser | ||
| exit 1 | ||
| end | ||
|
|
||
| matrix = E2EMatrix.load(options.fetch(:config)) | ||
|
|
||
| case command | ||
| when "validate" | ||
| errors = matrix.validation_errors | ||
| if errors.empty? | ||
| puts "E2E matrix is valid" | ||
| else | ||
| warn errors.join("\n") | ||
| exit 1 | ||
| end | ||
| when "expand" | ||
| errors = matrix.validation_errors | ||
| unless errors.empty? | ||
| warn errors.join("\n") | ||
| exit 1 | ||
| end | ||
|
|
||
| output = if options.key?(:index) | ||
| matrix.run_at(options.fetch(:index)) | ||
| else | ||
| matrix.expand | ||
| end | ||
|
|
||
| puts JSON.pretty_generate(output) | ||
| end |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| appId: ${E2E_APP_ID} | ||
| --- | ||
| - launchApp | ||
| - extendedWaitUntil: | ||
| visible: | ||
| id: ${E2E_READY_MARKER} | ||
| timeout: 60000 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test will be dropped when its replaced by Kyle's full suite of tests - I just wanted to decouple from his PR's and keep it simple whilst I setup the browserstack tests