[Phase 5] Run BrowserStack Maestro from E2E matrix rows #362
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| # frozen_string_literal: true | ||
|
|
||
| # Resolves abstract E2E device selectors, such as `ios:phone:latest`, to a | ||
| # concrete BrowserStack device and OS version that is currently available. | ||
| class BrowserStackDeviceResolver | ||
| def initialize(devices, limits = []) | ||
| @devices = devices | ||
| @limits = limits | ||
| end | ||
|
|
||
| def resolve(selector) | ||
| platform, form_factor, version_tag = selector.split(":") | ||
| raise ArgumentError, "unsupported device selector: #{selector}" unless platform && form_factor == "phone" && version_tag | ||
|
|
||
| candidates = available_phone_devices(platform) | ||
| os_version = os_version_for_tag(candidates, version_tag) | ||
| device = candidates.select { |candidate| candidate.fetch("os_version") == os_version } | ||
| .sort_by { |candidate| candidate.fetch("device") } | ||
| .first | ||
|
|
||
| raise ArgumentError, "no BrowserStack device available for #{selector}" unless device | ||
|
|
||
| { | ||
| "device_selector" => selector, | ||
| "platform" => platform, | ||
| "os_version_tag" => version_tag, | ||
| "resolved_device" => device.fetch("device"), | ||
| "resolved_os_version" => device.fetch("os_version"), | ||
| "browserstack_device" => "#{device.fetch("device")}-#{device.fetch("os_version")}" | ||
| } | ||
| end | ||
|
|
||
| private | ||
|
|
||
| def available_phone_devices(platform) | ||
| @devices.select do |device| | ||
| device.fetch("os") == platform && | ||
| device.fetch("realMobile", true) && | ||
| phone?(device.fetch("device")) && | ||
| available?(device) | ||
| end | ||
| end | ||
|
|
||
| def phone?(device_name) | ||
| !device_name.match?(/ipad|tablet| tab\b/i) | ||
| end | ||
|
|
||
| def available?(device) | ||
| limit = @limits.find do |candidate| | ||
| candidate.fetch("os") == device.fetch("os") && | ||
| candidate.fetch("os_version") == device.fetch("os_version") && | ||
| candidate.fetch("device") == device.fetch("device") | ||
| end | ||
|
|
||
| return true unless limit | ||
|
|
||
| limit.fetch("group_usage", 0).to_i < limit.fetch("device_limit", 1).to_i | ||
| end | ||
|
|
||
| def os_version_for_tag(candidates, version_tag) | ||
| versions = candidates.map { |candidate| candidate.fetch("os_version") }.uniq.sort_by { |version| version_segments(version) } | ||
| raise ArgumentError, "no BrowserStack devices available" if versions.empty? | ||
|
|
||
| case version_tag | ||
| when "latest" | ||
| versions.last | ||
| when "previous" | ||
| major_versions = versions.group_by { |version| version_segments(version).first }.keys.sort | ||
| previous_major = major_versions[-2] | ||
| raise ArgumentError, "no previous BrowserStack OS version available" unless previous_major | ||
|
|
||
| versions.select { |version| version_segments(version).first == previous_major }.last | ||
| else | ||
| raise ArgumentError, "unsupported OS version tag: #{version_tag}" | ||
| end | ||
| end | ||
|
|
||
| def version_segments(version) | ||
| version.to_s.split(".").map(&:to_i) | ||
| end | ||
| end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,206 @@ | ||
| #!/usr/bin/env ruby | ||
| # frozen_string_literal: true | ||
|
|
||
| require "base64" | ||
| require "fileutils" | ||
| require "json" | ||
| require "net/http" | ||
| require "optparse" | ||
| require "securerandom" | ||
| require "uri" | ||
| require_relative "../lib/browserstack_device_resolver" | ||
|
|
||
| # Executes one row from the BrowserStack run plan, from artifact upload through | ||
| # build polling and normalized result persistence. | ||
| class BrowserStackRunExecutor | ||
| API_HOST = "api-cloud.browserstack.com" | ||
| TERMINAL_STATUSES = %w[passed failed error timedout stopped done].freeze | ||
|
|
||
| def initialize(options) | ||
| @options = options | ||
| @username = ENV.fetch("BROWSERSTACK_USERNAME") | ||
| @access_key = ENV.fetch("BROWSERSTACK_ACCESS_KEY") | ||
| end | ||
|
|
||
| def run | ||
| FileUtils.mkdir_p(output_dir) | ||
| run = run_plan.fetch(run_index) | ||
| app_path = ENV.fetch(run.fetch("artifact_env")) | ||
| device = resolve_device(run) | ||
| app = upload_file("/app-automate/maestro/v2/app", app_path, custom_id(run, "app")) | ||
| suite = upload_file("/app-automate/maestro/v2/test-suite", tests_zip, custom_id(run, "suite")) | ||
| build = start_build(run, app.fetch("app_url"), suite.fetch("test_suite_url"), device.fetch("browserstack_device")) | ||
| build_status = poll_build(build.fetch("build_id")) | ||
| sessions = fetch_sessions(build_status) | ||
| result = normalize_result(run, device, app, suite, build, build_status, sessions) | ||
| write_json("result.json", result) | ||
| exit(result.fetch("passed") ? 0 : 1) | ||
| end | ||
|
|
||
| private | ||
|
|
||
| def resolve_device(run) | ||
| override = ENV["E2E_DEVICE_OVERRIDE"] || ENV["E2E_#{run.fetch("platform").upcase}_DEVICE_OVERRIDE"] | ||
|
Contributor
Author
There was a problem hiding this comment. These will be coming in to play with another PR I am preparing to give us a way to trigger runs for specific devices and OS's (e.g. iOS 27 Beta) |
||
| if override && !override.empty? | ||
| return { | ||
| "device_selector" => run.fetch("device_selector"), | ||
| "browserstack_device" => override, | ||
| "resolved_device" => override.split("-").first, | ||
| "resolved_os_version" => override.split("-").last | ||
| } | ||
| end | ||
|
|
||
| devices = get_json("/app-automate/devices.json") | ||
| limits = get_json("/app-automate/device_tier_limits.json") | ||
| BrowserStackDeviceResolver.new(devices, limits).resolve(run.fetch("device_selector")) | ||
| end | ||
|
|
||
| def upload_file(path, file_path, custom_id) | ||
| response = multipart_post(path, file_path, custom_id) | ||
| write_json("#{custom_id}.json", response) | ||
| response | ||
| end | ||
|
|
||
| def start_build(run, app_url, test_suite_url, device) | ||
| body = { | ||
| app: app_url, | ||
| testSuite: test_suite_url, | ||
| project: ENV.fetch("E2E_BROWSERSTACK_PROJECT", "checkout-kit-e2e"), | ||
| buildTag: ENV.fetch("BITRISE_GIT_COMMIT", "local"), | ||
| customBuildName: run.fetch("id"), | ||
| devices: [device], | ||
| execute: [run.fetch("execute")], | ||
| setEnvVariables: { | ||
| E2E_APP_ID: run.fetch("app_id"), | ||
| E2E_READY_MARKER: run.fetch("ready_marker") | ||
| } | ||
| } | ||
| response = post_json("/app-automate/maestro/v2/#{run.fetch("platform")}/build", body) | ||
| write_json("build-start.json", response) | ||
| response | ||
| end | ||
|
|
||
| def poll_build(build_id) | ||
| deadline = Time.now + ENV.fetch("E2E_BROWSERSTACK_TIMEOUT_SECONDS", "1800").to_i | ||
| loop do | ||
| response = get_json("/app-automate/maestro/v2/builds/#{build_id}") | ||
| write_json("build-status.json", response) | ||
| return response if TERMINAL_STATUSES.include?(response.fetch("status").to_s.downcase) | ||
| raise "BrowserStack build timed out: #{build_id}" if Time.now >= deadline | ||
|
|
||
| sleep ENV.fetch("E2E_BROWSERSTACK_POLL_SECONDS", "30").to_i | ||
| end | ||
| end | ||
|
|
||
| def fetch_sessions(build_status) | ||
| build_id = build_status.fetch("id") | ||
| build_status.fetch("devices", []).flat_map do |device| | ||
| device.fetch("sessions", []).map do |session| | ||
| get_json("/app-automate/maestro/v2/builds/#{build_id}/sessions/#{session.fetch("id")}") | ||
| end | ||
| end | ||
| end | ||
|
|
||
| def normalize_result(run, device, app, suite, build, build_status, sessions) | ||
| status = build_status.fetch("status").to_s.downcase | ||
| failed_tests = sessions.flat_map do |session| | ||
| session.dig("testcases", "data").to_a.flat_map do |group| | ||
| group.fetch("testcases", []).select { |testcase| testcase.fetch("status", "") != "passed" } | ||
| end | ||
| end | ||
|
|
||
| { | ||
| "id" => run.fetch("id"), | ||
| "status_context" => run.fetch("status_context"), | ||
| "platform" => run.fetch("platform"), | ||
| "target" => run.fetch("target"), | ||
| "os_version_tag" => run.fetch("os_version_tag"), | ||
| "execute" => run.fetch("execute"), | ||
| "device_selector" => device.fetch("device_selector"), | ||
| "resolved_device" => device.fetch("browserstack_device"), | ||
| "app_url" => app.fetch("app_url"), | ||
| "test_suite_url" => suite.fetch("test_suite_url"), | ||
| "build_id" => build.fetch("build_id"), | ||
| "status" => status, | ||
| "passed" => status == "passed" && failed_tests.empty?, | ||
| "failed_tests" => failed_tests | ||
| } | ||
| end | ||
|
|
||
| def multipart_post(path, file_path, custom_id) | ||
| boundary = "----checkout-kit-#{SecureRandom.hex(12)}" | ||
| file = File.binread(file_path) | ||
| body = +"" | ||
| body << "--#{boundary}\r\n" | ||
| body << "Content-Disposition: form-data; name=\"file\"; filename=\"#{File.basename(file_path)}\"\r\n\r\n" | ||
| body << file | ||
| body << "\r\n--#{boundary}\r\n" | ||
| body << "Content-Disposition: form-data; name=\"custom_id\"\r\n\r\n" | ||
| body << custom_id | ||
| body << "\r\n--#{boundary}--\r\n" | ||
| request = Net::HTTP::Post.new(path) | ||
| request["Content-Type"] = "multipart/form-data; boundary=#{boundary}" | ||
| request.body = body | ||
| execute_request(request) | ||
| end | ||
|
|
||
| def get_json(path) | ||
| execute_request(Net::HTTP::Get.new(path)) | ||
| end | ||
|
|
||
| def post_json(path, body) | ||
| request = Net::HTTP::Post.new(path) | ||
| request["Content-Type"] = "application/json" | ||
| request.body = JSON.generate(body) | ||
| execute_request(request) | ||
| end | ||
|
|
||
| def execute_request(request) | ||
| request.basic_auth(@username, @access_key) | ||
| response = Net::HTTP.start(API_HOST, 443, use_ssl: true) { |http| http.request(request) } | ||
| parsed = JSON.parse(response.body) | ||
| return parsed if response.is_a?(Net::HTTPSuccess) | ||
|
|
||
| raise "BrowserStack request failed #{response.code}: #{parsed}" | ||
| end | ||
|
|
||
| def custom_id(run, suffix) | ||
| ["checkout-kit", run.fetch("id"), ENV.fetch("BITRISE_GIT_COMMIT", "local"), suffix].join("-").gsub(/[^A-Za-z0-9._-]/, "-")[0, 100] | ||
| end | ||
|
|
||
| def write_json(name, object) | ||
| File.write(File.join(output_dir, name), JSON.pretty_generate(object)) | ||
| end | ||
|
|
||
| def output_dir | ||
| @options.fetch(:output_dir) | ||
| end | ||
|
|
||
| def run_plan | ||
| @run_plan ||= JSON.parse(File.read(@options.fetch(:run_plan_path))) | ||
| end | ||
|
|
||
| def run_index | ||
| @options.fetch(:run_index) | ||
| end | ||
|
|
||
| def tests_zip | ||
| @options.fetch(:tests_zip) | ||
| end | ||
| end | ||
|
|
||
| options = { | ||
| run_plan_path: ENV.fetch("E2E_BROWSERSTACK_RUN_PLAN_JSON", "browserstack-run-plan.json"), | ||
| run_index: Integer(ENV.fetch("BITRISE_IO_PARALLEL_INDEX")), | ||
| tests_zip: ENV.fetch("E2E_TESTS_ZIP"), | ||
| output_dir: ENV.fetch("E2E_BROWSERSTACK_RESULTS_DIR", File.join(Dir.pwd, "e2e-results")) | ||
| } | ||
|
|
||
| OptionParser.new do |opts| | ||
| opts.on("--run-plan PATH") { |path| options[:run_plan_path] = path } | ||
| opts.on("--index INDEX", Integer) { |index| options[:run_index] = index } | ||
| opts.on("--tests-zip PATH") { |path| options[:tests_zip] = path } | ||
| opts.on("--output-dir PATH") { |path| options[:output_dir] = path } | ||
| end.parse! | ||
|
|
||
| BrowserStackRunExecutor.new(options).run | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.