diff --git a/e2e/BITRISE.md b/e2e/BITRISE.md index a6f717ac..38184b55 100644 --- a/e2e/BITRISE.md +++ b/e2e/BITRISE.md @@ -11,14 +11,14 @@ The pipeline runs on the allocated Bitrise app, connected to the Checkout Kit re Useful Bitrise app URLs: -| Area | URL | -|---|---| -| App overview | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76 | -| Workflow/config editor | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/workflow | -| Secrets/env vars | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/secrets | -| Code signing | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/codesigning | -| Build triggers | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/triggers | -| Start build | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/build/start | +| Area | URL | +| ---------------------- | --------------------------------------------------------------------------- | +| App overview | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76 | +| Workflow/config editor | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/workflow | +| Secrets/env vars | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/secrets | +| Code signing | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/codesigning | +| Build triggers | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/triggers | +| Start build | https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/build/start | If a direct URL does not resolve in the current Bitrise UI, open the app overview and navigate to the matching area from the sidebar. @@ -40,16 +40,23 @@ Validate the configuration locally with: bitrise validate -c e2e/bitrise.yml ``` +## PR trigger + +`e2e/bitrise.yml` maps pull requests to the `e2e` pipeline with `trigger_map`. The GitHub checks are kept non-required while the suite stabilizes; merge-blocking will be controlled by branch protections when marked required. + +## Duplicate PR build cancellation + +Use Bitrise native Rolling builds instead of a repo-owned cancellation script. In Bitrise, open **Project settings > Builds > Build strategy**, enable **Abort builds triggered by pull requests**, and enable **Abort running builds** so duplicate in-progress PR pipelines are cancelled when a newer build starts. + ## Required app environment variables The non-secret E2E defaults live in `e2e/bitrise.yml` under `app.envs`. Change them in this repository rather than in the Bitrise Workflow Editor. -| Variable | Value | Purpose | -|---|---|---| -| `E2E_STRICT` | `false` | Soft/hard failure switch for BrowserStack Maestro runs. | -| `E2E_BROWSERSTACK_API_RETRIES` | `1` | Retries for transient BrowserStack API responses. | -| `E2E_BROWSERSTACK_TIMEOUT_SECONDS` | `1800` | BrowserStack build timeout. | -| `E2E_BROWSERSTACK_POLL_SECONDS` | `30` | BrowserStack status polling interval. | +| Variable | Value | Purpose | +| ---------------------------------- | ------ | ------------------------------------------------- | +| `E2E_BROWSERSTACK_API_RETRIES` | `1` | Retries for transient BrowserStack API responses. | +| `E2E_BROWSERSTACK_TIMEOUT_SECONDS` | `1800` | BrowserStack build timeout. | +| `E2E_BROWSERSTACK_POLL_SECONDS` | `30` | BrowserStack status polling interval. | Each workflow's main `script` step sets its own wall-clock budget with the Bitrise `timeout` and `no_output_timeout` step properties instead of wrapping individual commands. @@ -59,18 +66,18 @@ The `e2e-execute-browserstack-run` workflow fans out one parallel copy per Brows These secrets are configured in Bitrise.io; they cannot live in the repository. `scripts/setup_storefront_env` reads them to configure the sample app before builds. -| Secret | Purpose | -|---|---| -| `STOREFRONT_DOMAIN` | Storefront domain for sample app builds. | +| Secret | Purpose | +| ------------------------- | ---------------------------------------------- | +| `STOREFRONT_DOMAIN` | Storefront domain for sample app builds. | | `STOREFRONT_ACCESS_TOKEN` | Storefront access token for sample app builds. | ## BrowserStack secrets The `e2e-execute-browserstack-run` workflow authenticates with BrowserStack using these secrets, configured in Bitrise.io: -| Secret | Purpose | -|---|---| -| `BROWSERSTACK_USERNAME` | BrowserStack API username. | +| Secret | Purpose | +| ------------------------- | ---------------------------- | +| `BROWSERSTACK_USERNAME` | BrowserStack API username. | | `BROWSERSTACK_ACCESS_KEY` | BrowserStack API access key. | BrowserStack artifact links in GitHub reports require access to BrowserStack App Automate. Sign in to [BrowserStack App Automate](https://app-automate.browserstack.com/dashboard/v2/builds) before opening build, video, screenshot, or log links. @@ -81,13 +88,13 @@ React Native iOS IPA generation uses Bitrise's certificate and profile installer Upload the signing certificate and provisioning profile for the React Native sample app to the Bitrise app; the iOS artifact workflow installs them before archiving. The iOS build reads the following signing values with the defaults shown, and each can be overridden with a matching Bitrise environment variable: -| Variable | Default | Purpose | -|---|---|---| -| `E2E_IOS_EXPORT_METHOD` | `development` | Export method for the React Native iOS IPA. | -| `E2E_IOS_BUNDLE_ID` | `com.shopify.checkoutkit.reactnativedemo` | Bundle identifier used for iOS archive and export signing. | -| `E2E_IOS_DEVELOPMENT_TEAM` | `A7XGC83MZE` | Apple development team used for iOS archive signing. | -| `E2E_IOS_CODE_SIGN_IDENTITY` | `Apple Development` | Code signing identity used for iOS archive and export signing. | -| `E2E_IOS_PROVISIONING_PROFILE_SPECIFIER` | `bitrise-checkout-kit-e2e` | Provisioning profile specifier installed by Bitrise and passed to `xcodebuild`; override it if the Bitrise-installed profile uses a different name. | +| Variable | Default | Purpose | +| ---------------------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `E2E_IOS_EXPORT_METHOD` | `development` | Export method for the React Native iOS IPA. | +| `E2E_IOS_BUNDLE_ID` | `com.shopify.checkoutkit.reactnativedemo` | Bundle identifier used for iOS archive and export signing. | +| `E2E_IOS_DEVELOPMENT_TEAM` | `A7XGC83MZE` | Apple development team used for iOS archive signing. | +| `E2E_IOS_CODE_SIGN_IDENTITY` | `Apple Development` | Code signing identity used for iOS archive and export signing. | +| `E2E_IOS_PROVISIONING_PROFILE_SPECIFIER` | `bitrise-checkout-kit-e2e` | Provisioning profile specifier installed by Bitrise and passed to `xcodebuild`; override it if the Bitrise-installed profile uses a different name. | ## BrowserStack execution diff --git a/e2e/RUNBOOK.md b/e2e/RUNBOOK.md new file mode 100644 index 00000000..03c247f8 --- /dev/null +++ b/e2e/RUNBOOK.md @@ -0,0 +1,98 @@ +# Checkout Kit E2E Runbook + +## Rollout behaviour + +The E2E pipeline always runs and always reports; it never blocks PR merges on its +own. Merge-blocking is controlled solely by whether the single **"Checkout Kit E2E"** +GitHub Check Run is marked **required** in branch protection. Keep it non-required +until the suite is stable, then make it required — no code change is needed to gate +or un-gate. This single umbrella check stays stable across matrix changes, so +requiring it never churns as applications, OS versions, or suites are added. + +The runner never hard-fails on test or infrastructure problems: every run writes a +`result.json` and exits `0`, so the report workflow always has data to publish. The +report posts one **"Checkout Kit E2E"** Check Run and a sticky PR failure comment; it +does not post per-suite commit statuses. Failures are therefore loud (one red check +and a failure comment) but non-blocking. + +Failures land in `result.json` in one of two shapes: + +- **Assertion / suite failures** — a BrowserStack build that reports failed test + cases or a non-passing terminal status. Recorded by `normalize_result` with + `status` set to the build's terminal status (e.g. `"failed"`) and the failing + cases under `failed_tests`; there is no `error`/`error_class`. +- **Runner exceptions** — malformed run plan, out-of-range parallel index, device + resolution, artifact upload, build start, polling, and timeouts. Caught and + recorded with `status: "error"`, `error`, and `error_class`, with the class, + message, and full backtrace logged to the failing step's stderr. + +Setup failures that happen before a run plan row is read have no run metadata +(suite, target), so they appear in the Check Run summary and the failure comment +with an `error` status but reduced detail. This is separate from the Bitrise +step's own required-env guards: a missing config variable +(e.g. `BROWSERSTACK_ACCESS_KEY`) is validated by the step and fails fast before +the runner starts. + +The report also enforces a **completeness check**: it compares the number of +`result.json` files against `E2E_BROWSERSTACK_RUN_PLAN_COUNT` (the run plan row +count, shared across the pipeline). If a run never reports — for example a whole +execute workflow that failed to upload — the "Checkout Kit E2E" check is forced red +and the failure comment notes the shortfall, so a missing run can never silently +pass. When the expected count is unavailable the completeness check is skipped +rather than reporting a false failure. + +## Retry behavior + +BrowserStack API calls retry transient infrastructure responses once by default: + +```bash +E2E_BROWSERSTACK_API_RETRIES=1 +``` + +Retry applies to HTTP 429 and 5xx responses **and to transient network exceptions** (connection timeouts, resets, TLS/socket errors), and only to idempotent read/poll (GET) requests — build-creation and upload requests fail fast to avoid duplicate builds. Connections use a 10s open timeout and a 120s read timeout. Test assertion failures are not auto-retried by default so first-failure evidence is preserved. + +## Timeouts + +BrowserStack polling uses these defaults: + +```bash +E2E_BROWSERSTACK_TIMEOUT_SECONDS=1800 +E2E_BROWSERSTACK_POLL_SECONDS=30 +``` + +If polling times out, the runner attempts to stop the BrowserStack build before recording the failure. + +## Local rerun notes + +Use the BrowserStack run plan row from a failure report to identify the app target, platform, OS version tag, and suite: + +```bash +ruby e2e/scripts/e2e_matrix_to_browserstack_run_plan expand --index +ruby e2e/scripts/e2e_matrix_to_browserstack_run_plan expand > /tmp/browserstack-run-plan.json +``` + +Use the reported resolved device to pin a rerun: + +```bash +E2E_DEVICE_OVERRIDE="" \ +e2e/scripts/execute_browserstack_run --index --run-plan /tmp/browserstack-run-plan.json --tests-zip --output-dir +``` + +The app artifact environment variable for the run plan row must point at the `.apk` or `.ipa` artifact before rerunning. + +## Failure triage + +Use the GitHub Check Run or sticky PR failure comment first. Failure summaries should include Markdown links to: + +- BrowserStack build +- failed testcase +- video +- screenshot +- Maestro command log +- Maestro log +- device log +- network log when enabled + +BrowserStack artifact links require BrowserStack App Automate access. Sign in to [BrowserStack App Automate](https://app-automate.browserstack.com/dashboard/v2/builds) before opening evidence links. + +Avoid posting additional PR comments for green runs. diff --git a/e2e/bitrise.yml b/e2e/bitrise.yml index 059de0d7..c2130d70 100644 --- a/e2e/bitrise.yml +++ b/e2e/bitrise.yml @@ -9,7 +9,6 @@ meta: app: envs: - - E2E_STRICT: "false" - E2E_BROWSERSTACK_API_RETRIES: "1" - E2E_BROWSERSTACK_TIMEOUT_SECONDS: "1800" - E2E_BROWSERSTACK_POLL_SECONDS: "30" @@ -18,6 +17,11 @@ tools: ruby: 3.3.6 nodejs: 22.14.0 +trigger_map: + - pull_request_source_branch: "*" + draft_pull_request_enabled: false + pipeline: e2e + pipelines: e2e: workflows: @@ -219,6 +223,7 @@ workflows: --tests-zip "$E2E_TESTS_ZIP" \ --output-dir "$results_dir" - deploy-to-bitrise-io@2: + is_always_run: true inputs: - pipeline_intermediate_files: |- $BITRISE_DEPLOY_DIR/e2e/results:E2E_BROWSERSTACK_RESULTS_DIR diff --git a/e2e/lib/e2e_github_reporter.rb b/e2e/lib/e2e_github_reporter.rb index 1e3af973..c3ccd609 100644 --- a/e2e/lib/e2e_github_reporter.rb +++ b/e2e/lib/e2e_github_reporter.rb @@ -8,17 +8,16 @@ class E2EGitHubReporter COMMENT_MARKER = "" - def initialize(results, repository:, sha:, pr_number:, token: nil, strict: false) + def initialize(results, repository:, sha:, pr_number:, token: nil, expected: nil) @results = results @repository = repository @sha = sha @pr_number = pr_number @token = token - @strict = strict + @expected = expected end def publish! - commit_status_payloads.each { |payload| client.post_json("/repos/#{@repository}/statuses/#{@sha}", payload) } client.post_json("/repos/#{@repository}/check-runs", check_run_payload) sync_failure_comment end @@ -30,38 +29,30 @@ def markdown_summary lines << "| Status | Suite | Target | Platform | OS version tag | Device |" lines << "|---|---|---|---|---|---|" @results.each do |result| - lines << "| #{status_icon(result)} | `#{result.fetch("execute")}` | #{result.fetch("target")} | #{result.fetch("platform")} | #{result.fetch("os_version_tag")} | #{device_cell(result)} |" + lines << "| #{status_icon(result)} | `#{result["execute"]}` | #{result["target"]} | #{result["platform"]} | #{result["os_version_tag"]} | #{device_cell(result)} |" + end + unless complete? + lines << "" + lines << "> [!WARNING]" + lines << "> Expected #{@expected} run#{@expected == 1 ? "" : "s"}, received #{@results.length} — #{missing_count} did not report. Missing runs count as failures until every run reports." end failure_lines = failed_results.flat_map { |result| failure_details(result) } unless failure_lines.empty? lines << "" lines << "## Failures" lines << "" - if soft_fail? - lines << "> [!CAUTION]" - lines << "> These are currently non-blocking, but failures may indicate an issue you need to resolve before merging." - lines << "> If you believe there is a flaky assertion, please raise a ticket in the #checkout-kit-devs channel so it can be addressed" - lines << "" - end + lines << "> [!CAUTION]" + lines << "> These E2E checks are not yet required, so they do not block merging — but a failure may still indicate a real issue to resolve before merging." + lines << "> If you believe an assertion is flaky, please raise a ticket in the #checkout-kit-devs channel so it can be addressed." + lines << "" lines << "> BrowserStack artifacts require BrowserStack access. Sign in to [BrowserStack App Automate](https://app-automate.browserstack.com/dashboard/v2/builds) before opening artifact links." lines.concat(failure_lines) end lines.join("\n") end - def commit_status_payloads - @results.map do |result| - { - state: result.fetch("passed") ? "success" : "failure", - context: result.fetch("status_context"), - description: status_description(result), - target_url: browserstack_build_url(result) - } - end - end - def failure_comment_body - return nil if failed_results.empty? + return nil if failed_results.empty? && complete? [COMMENT_MARKER, markdown_summary].join("\n") end @@ -69,7 +60,7 @@ def failure_comment_body private def check_run_payload - conclusion = failed_results.empty? ? "success" : "failure" + conclusion = failed_results.empty? && complete? ? "success" : "failure" { name: "Checkout Kit E2E", head_sha: @sha, @@ -82,6 +73,16 @@ def check_run_payload } end + def complete? + @expected.nil? || @results.length >= @expected + end + + def missing_count + return 0 if @expected.nil? + + [@expected - @results.length, 0].max + end + def sync_failure_comment body = failure_comment_body existing = existing_failure_comment @@ -103,11 +104,11 @@ def existing_failure_comment end def failed_results - @results.reject { |result| result.fetch("passed") } + @results.reject { |result| result.fetch("passed", false) } end - def soft_fail? - !@strict + def blank?(value) + value.nil? || value.to_s.strip.empty? end def failure_details(result) @@ -129,8 +130,8 @@ def failure_details(result) end def failure_heading(result) - suite = File.basename(result.fetch("execute"), ".*") - "#{os_label(result.fetch("platform"))} — #{suite}" + suite = File.basename(result["execute"].to_s, ".*") + "#{os_label(result["platform"])} — #{suite}" end def artifact_links(testcase, result) @@ -163,26 +164,27 @@ def link_fields end def device_cell(result) - name = result.fetch("resolved_device") + name = result["resolved_device"] version = result["resolved_os_version"] - version && !version.empty? ? "#{name}
#{os_label(result.fetch("platform"))} #{version}" : name + version && !version.empty? ? "#{name}
#{os_label(result["platform"])} #{version}" : name end def os_label(platform) - platform == "ios" ? "iOS" : "Android" + case platform + when "ios" then "iOS" + when "android" then "Android" + else "Unknown" + end end def status_icon(result) - result.fetch("passed") ? "✅" : "❌" - end - - def status_description(result) - description = result.fetch("passed") ? "passed" : "#{result.fetch("status")} on #{result.fetch("resolved_device")}" - description[0, 140] + result.fetch("passed", false) ? "✅" : "❌" end def browserstack_build_url(result) - "https://app-automate.browserstack.com/dashboard/v2/builds/#{result.fetch("build_id")}" + build_id = result["build_id"] + base = "https://app-automate.browserstack.com/dashboard/v2/builds" + blank?(build_id) ? base : "#{base}/#{build_id}" end def issue_comments diff --git a/e2e/lib/json_http_client.rb b/e2e/lib/json_http_client.rb index 092e223b..6b409ce3 100644 --- a/e2e/lib/json_http_client.rb +++ b/e2e/lib/json_http_client.rb @@ -2,15 +2,36 @@ require "json" require "net/http" +require "openssl" # Small JSON-over-HTTPS client shared by the BrowserStack executor and the # GitHub reporter. Each caller supplies its host, an error label, default # headers, and an authenticator block that stamps credentials onto the request. +# Callers that want transient-failure retries pass a retries count and a +# retryable predicate; the default is a single attempt with no retries. class JsonHttpClient - def initialize(host:, error_label:, default_headers: {}, &authenticator) + MAX_BACKOFF_SECONDS = 30 + OPEN_TIMEOUT_SECONDS = 10 + READ_TIMEOUT_SECONDS = 120 + RETRYABLE_EXCEPTIONS = [ + Net::OpenTimeout, + Net::ReadTimeout, + Errno::ECONNRESET, + Errno::ECONNREFUSED, + Errno::EHOSTUNREACH, + Errno::ETIMEDOUT, + SocketError, + OpenSSL::SSL::SSLError, + EOFError, + IOError + ].freeze + + def initialize(host:, error_label:, default_headers: {}, retries: 0, retryable: nil, &authenticator) @host = host @error_label = error_label @default_headers = default_headers + @retries = retries + @retryable = retryable @authenticator = authenticator end @@ -28,16 +49,55 @@ def patch_json(path, body) def execute(request) @default_headers.each { |name, value| request[name] = value } - @authenticator&.call(request) - response = Net::HTTP.start(@host, 443, use_ssl: true) { |http| http.request(request) } - body = response.body.to_s.empty? ? {} : JSON.parse(response.body) - return body if response.is_a?(Net::HTTPSuccess) + attempts = 0 + loop do + attempts += 1 + @authenticator&.call(request) + begin + response = perform(request) + rescue *RETRYABLE_EXCEPTIONS + raise unless retry_allowed?(request, attempts) + + sleep(backoff_seconds(attempts)) + next + end + return parse_body(response) if response.is_a?(Net::HTTPSuccess) + raise "#{@error_label} request failed #{response.code}: #{response.body.to_s[0, 500]}" unless retryable?(response) && retry_allowed?(request, attempts) - raise "#{@error_label} request failed #{response.code}: #{body}" + sleep(backoff_seconds(attempts)) + end end private + def perform(request) + Net::HTTP.start(@host, 443, use_ssl: true, open_timeout: OPEN_TIMEOUT_SECONDS, read_timeout: READ_TIMEOUT_SECONDS) { |http| http.request(request) } + end + + def retry_allowed?(request, attempts) + idempotent_request?(request) && attempts <= @retries + end + + def retryable?(response) + @retryable&.call(response) || false + end + + def idempotent_request?(request) + request.is_a?(Net::HTTP::Get) + end + + def backoff_seconds(attempt) + [attempt, MAX_BACKOFF_SECONDS].min + rand + end + + def parse_body(response) + return {} if response.body.to_s.empty? + + JSON.parse(response.body) + rescue JSON::ParserError + raise "#{@error_label} request returned non-JSON response" + end + def with_json_body(request, body) request["Content-Type"] = "application/json" request.body = JSON.generate(body) diff --git a/e2e/scripts/execute_browserstack_run b/e2e/scripts/execute_browserstack_run index 63e91a27..b2f34f55 100755 --- a/e2e/scripts/execute_browserstack_run +++ b/e2e/scripts/execute_browserstack_run @@ -16,32 +16,96 @@ class BrowserStackRunExecutor API_HOST = "api-cloud.browserstack.com" TERMINAL_STATUSES = %w[passed failed error timedout stopped done].freeze + def self.failure_result(context, error) + context.merge( + "status" => "error", + "passed" => false, + "failed_tests" => [], + "error" => error.message, + "error_class" => error.class.name + ) + end + + def self.record_failure!(error, results_dir:, identifier:, context: {}) + warn "Recording failure for #{identifier}: #{error.class}: #{error.message}" + warn error.backtrace.join("\n") if error.backtrace + FileUtils.mkdir_p(results_dir) + File.write(File.join(results_dir, "result.json"), JSON.pretty_generate(failure_result(context, error))) + end + + def self.build_options(argv, env: ENV) + cli = parse_cli_options(argv) + { + run_plan_path: cli[:run_plan_path] || env["E2E_BROWSERSTACK_RUN_PLAN_JSON"] || "browserstack-run-plan.json", + run_index: resolve_run_index(cli, env), + tests_zip: resolve_tests_zip(cli, env), + output_dir: cli[:output_dir] || env["E2E_BROWSERSTACK_RESULTS_DIR"] || File.join(Dir.pwd, "e2e-results") + } + end + + def self.parse_cli_options(argv) + cli = {} + OptionParser.new do |opts| + opts.on("--run-plan PATH") { |path| cli[:run_plan_path] = path } + opts.on("--index INDEX", Integer) { |index| cli[:run_index] = index } + opts.on("--tests-zip PATH") { |path| cli[:tests_zip] = path } + opts.on("--output-dir PATH") { |path| cli[:output_dir] = path } + end.parse!(argv) + cli + end + + def self.resolve_run_index(cli, env) + return cli[:run_index] if cli.key?(:run_index) + + raw = env["BITRISE_IO_PARALLEL_INDEX"] + raise ArgumentError, "run_index is required: pass --index or set BITRISE_IO_PARALLEL_INDEX" if raw.nil? || raw.strip.empty? + + Integer(raw) + end + + def self.resolve_tests_zip(cli, env) + value = cli[:tests_zip] || env["E2E_TESTS_ZIP"] + raise ArgumentError, "tests_zip is required: pass --tests-zip or set E2E_TESTS_ZIP" if value.nil? || value.strip.empty? + + value + end + def initialize(options) @options = options @username = ENV.fetch("BROWSERSTACK_USERNAME") @access_key = ENV.fetch("BROWSERSTACK_ACCESS_KEY") - @client = JsonHttpClient.new(host: API_HOST, error_label: "BrowserStack") do |request| + @client = JsonHttpClient.new( + host: API_HOST, + error_label: "BrowserStack", + retries: ENV.fetch("E2E_BROWSERSTACK_API_RETRIES", "1").to_i, + retryable: ->(response) { response.code.to_i == 429 || response.code.to_i >= 500 } + ) do |request| request.basic_auth(@username, @access_key) end end def run FileUtils.mkdir_p(output_dir) - run = run_plan.fetch(run_index) - app_path = ENV.fetch(run.fetch("artifact_env")) - device = resolve_device(run) - app = upload_file("/app-automate/maestro/v2/app", app_path, custom_id(run, "app")) - suite = upload_file("/app-automate/maestro/v2/test-suite", tests_zip, custom_id(run, "suite")) - build = start_build(run, app.fetch("app_url"), suite.fetch("test_suite_url"), device.fetch("browserstack_device")) - build_status = poll_build(build.fetch("build_id")) + @run = run_plan.fetch(run_index) + app_path = ENV.fetch(@run.fetch("artifact_env")) + @device = resolve_device(@run) + @app = upload_file("/app-automate/maestro/v2/app", app_path, custom_id(@run, "app")) + @suite = upload_file("/app-automate/maestro/v2/test-suite", tests_zip, custom_id(@run, "suite")) + @build = start_build(@run, @app.fetch("app_url"), @suite.fetch("test_suite_url"), @device.fetch("browserstack_device")) + build_status = poll_build(@build.fetch("build_id")) sessions = fetch_sessions(build_status) - result = normalize_result(run, device, app, suite, build, build_status, sessions) + result = normalize_result(@run, @device, @app, @suite, @build, build_status, sessions) write_json("result.json", result) - exit(result.fetch("passed") ? 0 : 1) + rescue StandardError => error + self.class.record_failure!(error, results_dir: output_dir, identifier: failure_identifier, context: failure_context) end private + def failure_identifier + (@run && @run["id"]) || "run #{run_index}" + end + def resolve_device(run) override = ENV["E2E_DEVICE_OVERRIDE"] || ENV["E2E_#{run.fetch("platform").upcase}_DEVICE_OVERRIDE"] if override && !override.empty? @@ -90,12 +154,24 @@ class BrowserStackRunExecutor response = @client.get("/app-automate/maestro/v2/builds/#{build_id}") write_json("build-status.json", response) return response if TERMINAL_STATUSES.include?(response.fetch("status").to_s.downcase) - raise "BrowserStack build timed out: #{build_id}" if Time.now >= deadline + if Time.now >= deadline + stop_build(build_id) + raise "BrowserStack build timed out: #{build_id}" + end sleep ENV.fetch("E2E_BROWSERSTACK_POLL_SECONDS", "30").to_i end end + # BrowserStack's Maestro stop endpoint omits the "/v2/" segment that the + # build-status endpoint uses; this asymmetry is intentional per BrowserStack. + # https://www.browserstack.com/docs/app-automate/api-reference/maestro/builds + def stop_build(build_id) + @client.post_json("/app-automate/maestro/builds/#{build_id}/stop", {}) + rescue StandardError => error + warn "Unable to stop BrowserStack build #{build_id}: #{error.message}" + end + def fetch_sessions(build_status) build_id = build_status.fetch("id") build_status.fetch("devices", []).flat_map do |device| @@ -133,6 +209,26 @@ class BrowserStackRunExecutor } end + def failure_context + run = @run || {} + device = @device || {} + { + "id" => run["id"], + "status_context" => run["status_context"], + "platform" => run["platform"], + "target" => run["target"], + "os_version_tag" => run["os_version_tag"], + "execute" => run["execute"], + "device_selector" => run["device_selector"], + "resolved_device" => device["resolved_device"] || run["device_selector"], + "resolved_os_version" => device["resolved_os_version"], + "browserstack_device" => device["browserstack_device"], + "app_url" => @app && @app["app_url"], + "test_suite_url" => @suite && @suite["test_suite_url"], + "build_id" => @build && @build["build_id"] + } + end + def multipart_post(path, file_path, custom_id) boundary = "----checkout-kit-#{SecureRandom.hex(12)}" file = File.binread(file_path) @@ -175,18 +271,13 @@ class BrowserStackRunExecutor end end -options = { - run_plan_path: ENV.fetch("E2E_BROWSERSTACK_RUN_PLAN_JSON", "browserstack-run-plan.json"), - run_index: Integer(ENV.fetch("BITRISE_IO_PARALLEL_INDEX")), - tests_zip: ENV.fetch("E2E_TESTS_ZIP"), - output_dir: ENV.fetch("E2E_BROWSERSTACK_RESULTS_DIR", File.join(Dir.pwd, "e2e-results")) -} - -OptionParser.new do |opts| - opts.on("--run-plan PATH") { |path| options[:run_plan_path] = path } - opts.on("--index INDEX", Integer) { |index| options[:run_index] = index } - opts.on("--tests-zip PATH") { |path| options[:tests_zip] = path } - opts.on("--output-dir PATH") { |path| options[:output_dir] = path } -end.parse! - -BrowserStackRunExecutor.new(options).run +if $PROGRAM_NAME == __FILE__ + begin + options = BrowserStackRunExecutor.build_options(ARGV) + BrowserStackRunExecutor.new(options).run + rescue StandardError => error + results_dir = (options && options[:output_dir]) || + ENV.fetch("E2E_BROWSERSTACK_RESULTS_DIR", File.join(Dir.pwd, "e2e-results")) + BrowserStackRunExecutor.record_failure!(error, results_dir: results_dir, identifier: "e2e setup") + end +end diff --git a/e2e/scripts/report_e2e_results b/e2e/scripts/report_e2e_results index 9b359ebb..d7260129 100755 --- a/e2e/scripts/report_e2e_results +++ b/e2e/scripts/report_e2e_results @@ -11,7 +11,7 @@ options = { sha: ENV["BITRISE_GIT_COMMIT"], pr_number: ENV["BITRISE_PULL_REQUEST"], token: ENV["GITHUB_TOKEN"], - strict: ENV.fetch("E2E_STRICT", "false") == "true" + expected: ENV["E2E_BROWSERSTACK_RUN_PLAN_COUNT"] } OptionParser.new do |opts| @@ -19,6 +19,7 @@ OptionParser.new do |opts| opts.on("--repository REPOSITORY") { |repository| options[:repository] = repository } opts.on("--sha SHA") { |sha| options[:sha] = sha } opts.on("--pr NUMBER") { |number| options[:pr_number] = number } + opts.on("--expected COUNT", Integer) { |count| options[:expected] = count } end.parse! pr_number = options[:pr_number].to_s.strip @@ -34,10 +35,15 @@ end end options[:pr_number] = Integer(options[:pr_number]) +expected_raw = options[:expected] +expected = expected_raw.nil? || expected_raw.to_s.strip.empty? ? nil : Integer(expected_raw) + result_paths = Dir.glob(File.join(options.fetch(:results_root), "**", "result.json")) if result_paths.empty? warn "No E2E result files found under #{options.fetch(:results_root)}" - exit 1 + exit 1 if expected.nil? + + warn "Publishing an incomplete report: expected #{expected} run(s), found 0" end results = result_paths.sort.map { |path| JSON.parse(File.read(path)) } @@ -47,7 +53,7 @@ reporter = E2EGitHubReporter.new( sha: options.fetch(:sha), pr_number: options.fetch(:pr_number), token: options.fetch(:token), - strict: options.fetch(:strict) + expected: expected ) reporter.publish! puts reporter.markdown_summary