Action is flaky, any resolution?

[chris-rutkowski]

Usually, the action completes nicely in about 1min or so, but about 25% of the times the iOS simulator simply doesn't boot:

There is nothing extraordinary in the action. Clone, setup dev stack (Flutter in this case), start emulator and execute integration tests.

name: Integration test

on:
  pull_request:
    branches: [ main ]

jobs:
  ios:
    runs-on: macos-26
    timeout-minutes: 20

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4

    - name: Setup Flutter
      uses: subosito/flutter-action@v2
      with:
        flutter-version: "3.35.4"

    - name: Get Flutter dependencies
      run: flutter pub get

    - name: Start iOS Simulator
      uses: futureware-tech/simulator-action@v4
      with:
        model: 'iPhone 17 Pro'
        os_version: '26.0'
        wait_for_boot: true
      id: simulator

    - name: Run integration tests
      run: flutter test integration_test -d ${{ steps.simulator.outputs.udid }} -v

Please let me know if this is a common issue and if there is a known walkaround?

[dotdoom]

Thanks for the report!

I haven't heard of any common condition which would cause this to flake.

In my experience some combinations of MacOS + Simulator versions can take a lot of resources from the actions runner. If you're using GitHub Actions runner / Free tier, the action is limited in RAM and CPU. Would you like to try a different model/os_version (e.g. iPhone 16 Pro) for the runner to see if it helps stabilize the environment, as a workaround?

[chris-rutkowski]

I will give it a go. I tried macOS-26 and iPhone 17 Pro after experiencing other build freezing issues (apparently Xcode bug).

Now I've just switched to iPhone 16 Pro with iOS 18.6. I will keep you posted if I experience similar issues.

Do you happen to have a matrix of safe macOS, Xcode, iOS and simular model versions that the action is tested?

Thank you for your contributions.

[dotdoom]

I don't have one today, but it's a good idea for FR to build the matrix automatically.

For the current 5¹ macOS versions and e.g. 20 different iPhone OS+model configurations², if we run 10 random workflows per day (easily within GitHub limits), we will have a sample size of almost 10 runs for each within a quarter.

Footnotes

1. https://github.com/actions/runner-images?tab=readme-ov-file#available-images ↩

2. https://github.com/futureware-tech/simulator-action/wiki/Devices-macos-latest ↩