Add command line option allowing to skip instead of fail

[cedelavergne-ledger]

Change default behavior to only fail the test where the elf is missing, and run the other ones
A dedicated warning is displayed if the library is missing

Example usage with app-boilerplate

For this test, I have not compiled the NanoS+ app

$ pytest --device=all_nano test_pubkey_cmd.py
======================= test session starts ========================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /work/apps/app-boilerplate/tests/standalone/venv/bin/python3
cachedir: .pytest_cache
rootdir: /work/apps/app-boilerplate/tests/standalone
configfile: setup.cfg
collected 9 items                                                                                                                                                                                                                                                          

test_pubkey_cmd.py::test_get_public_key_no_confirm[nanos] SKIPPED (Device nanos is not supported according to the manifest)
test_pubkey_cmd.py::test_get_public_key_confirm_accepted[nanos] SKIPPED (Device nanos is not supported according to the manifest)
test_pubkey_cmd.py::test_get_public_key_confirm_refused[nanos] SKIPPED (Device nanos is not supported according to the manifest)
test_pubkey_cmd.py::test_get_public_key_no_confirm[nanosp] ERROR
test_pubkey_cmd.py::test_get_public_key_confirm_accepted[nanosp] ERROR
test_pubkey_cmd.py::test_get_public_key_confirm_refused[nanosp] ERROR 
test_pubkey_cmd.py::test_get_public_key_no_confirm[nanox] PASSED
test_pubkey_cmd.py::test_get_public_key_confirm_accepted[nanox] PASSED
test_pubkey_cmd.py::test_get_public_key_confirm_refused[nanox] PASSED

============== short test summary info ===============
ERROR test_pubkey_cmd.py::test_get_public_key_no_confirm[nanosp] - Failed: Missing ELF: File '/work/apps/app-boilerplate/build/nanos2/bin/app.elf' missing. Did you compile for this target?
ERROR test_pubkey_cmd.py::test_get_public_key_confirm_accepted[nanosp] - Failed: Missing ELF: File '/work/apps/app-boilerplate/build/nanos2/bin/app.elf' missing. Did you compile for this target?
ERROR test_pubkey_cmd.py::test_get_public_key_confirm_refused[nanosp] - Failed: Missing ELF: File '/work/apps/app-boilerplate/build/nanos2/bin/app.elf' missing. Did you compile for this target?
======= 3 passed, 3 skipped, 3 errors in 3.14s =======

Adding --ignore-missing-binaries doesn't change the behavior

Test with app-exchange

For this test, I have removed the Cardano App from the directory test/python/lib_binaries

$ pytest --device=stax -k "test_c and fund_valid_1 and not cosmos"
======================= test session starts ========================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /work/apps/app-exchange/test/python/venv/bin/python3
cachedir: .pytest_cache
rootdir: /work/apps/app-exchange/test/python
configfile: setup.cfg
plugins: anyio-4.12.0, cov-7.0.0
collected 574 items / 572 deselected / 2 selected                                                                                                                                                                                                        

test_cardano.py::TestsCardanoClient::test_cardano_shelley[stax-fund_valid_1] ERROR 
test_celo.py::TestsCelo::test_celo[stax-fund_valid_1] ERROR

================ short test summary info ================
ERROR test_cardano.py::TestsCardanoClient::test_cardano_shelley[stax-fund_valid_1] - Failed: Missing ELF: File '/work/apps/app-exchange/test/python/lib_binaries/cardano_stax.elf' missing. Did you compile for this target?
ERROR test_celo.py::TestsCelo::test_celo[stax-fund_valid_1] - Failed: Missing ELF: File '/work/apps/app-exchange/test/python/lib_binaries/cardano_stax.elf' missing. Did you compile for this target?
=========== 572 deselected, 2 errors in 4.66s ===========

$ pytest --device=stax -k "test_c and fund_valid_1 and not cosmos" --ignore-missing-binaries
================== test session starts ==================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /work/apps/app-exchange/test/python/venv/bin/python3
cachedir: .pytest_cache
rootdir: /work/apps/app-exchange/test/python
configfile: setup.cfg
plugins: anyio-4.12.0, cov-7.0.0
collected 574 items / 572 deselected / 2 selected                                                                                                                                                                                                           

test_cardano.py::TestsCardanoClient::test_cardano_shelley[stax-fund_valid_1] FAILED
test_celo.py::TestsCelo::test_celo[stax-fund_valid_1] PASSED

=================== warnings summary ====================
test_cardano.py::TestsCardanoClient::test_cardano_shelley[stax-fund_valid_1]
test_celo.py::TestsCelo::test_celo[stax-fund_valid_1]
  /work/apps/tools/ragger/src/ragger/conftest/base_conftest.py:310: UserWarning: Could not find sideloaded app library for 'Cardano ADA': File '/work/apps/app-exchange/test/python/lib_binaries/cardano_stax.elf' missing. Did you compile for this target?
    warnings.warn(f"Could not find sideloaded app library for '{lib_name}': {e}", UserWarning)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================ short test summary info ================
FAILED test_cardano.py::TestsCardanoClient::test_cardano_shelley[stax-fund_valid_1] - TimeoutError
===== 1 failed, 1 passed, 572 deselected, 2 warnings in 11.13s =====

Test with app-plugin-boilerplate

For this test, I have not compiled the Stax plugin neither the NanoS+ Ethereum App

$ pytest --device=all
================== test session starts ==================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /work/apps/app-plugin-boilerplate/tests/venv/bin/python3
cachedir: .pytest_cache
rootdir: /work/apps/app-plugin-boilerplate/tests
configfile: setup.cfg
collected 7 items                                                                                                                                                                                                                                           

test_swap.py::test_swap_exact_eth_for_token[nanos] SKIPPED (Device nanos is not supported according to the manifest) 
test_swap.py::test_swap_exact_eth_for_token[nanosp] ERROR
test_swap.py::test_swap_exact_eth_for_token[nanox] PASSED
test_swap.py::test_swap_exact_eth_for_token[flex] PASSED
test_swap.py::test_swap_exact_eth_for_token[stax] ERROR
test_swap.py::test_swap_exact_eth_for_token[apex_p] PASSED
test_swap.py::test_swap_exact_eth_for_token[apex_m] SKIPPED (Device apex_m is not supported according to the manifest)

================ short test summary info ================
ERROR test_swap.py::test_swap_exact_eth_for_token[nanosp] - Failed: Missing ELF: File '/work/apps/app-plugin-boilerplate/tests/.test_dependencies/ethereum/build/nanos2/bin/app.elf' missing. Did you compile for this target?
ERROR test_swap.py::test_swap_exact_eth_for_token[stax] - Failed: Missing ELF: File '/work/apps/app-plugin-boilerplate/build/stax/bin/app.elf' missing. Did you compile for this target?
======== 3 passed, 2 skipped, 2 errors in 6.67s =========

Adding --ignore-missing-binaries doesn't change the behavior

[codecov-commenter]

Codecov Report

❌ Patch coverage is 60.00000% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.69%. Comparing base (1f5eb06) to head (647705f).

Files with missing lines	Patch %	Lines
src/ragger/conftest/base_conftest.py	53.57%	13 Missing ⚠️
src/ragger/utils/misc.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #264      +/-   ##
==========================================
- Coverage   77.94%   77.69%   -0.26%     
==========================================
  Files          36       36              
  Lines        2145     2170      +25     
==========================================
+ Hits         1672     1686      +14     
- Misses        473      484      +11     

Flag	Coverage Δ
apex_p-py3.10	77.60% <60.00%> (-0.26%)	⬇️
apex_p-py3.11	77.60% <60.00%> (-0.26%)	⬇️
apex_p-py3.12	77.60% <60.00%> (-0.26%)	⬇️
apex_p-py3.13	77.60% <60.00%> (-0.26%)	⬇️
apex_p-py3.9	77.57% <60.00%> (-0.26%)	⬇️
flex-py3.10	77.60% <60.00%> (-0.26%)	⬇️
flex-py3.11	77.60% <60.00%> (-0.26%)	⬇️
flex-py3.12	77.60% <60.00%> (-0.26%)	⬇️
flex-py3.13	77.60% <60.00%> (-0.26%)	⬇️
flex-py3.9	77.57% <60.00%> (-0.26%)	⬇️
nanos-py3.10	75.94% <60.00%> (-0.24%)	⬇️
nanos-py3.11	75.94% <60.00%> (-0.24%)	⬇️
nanos-py3.12	75.94% <60.00%> (-0.24%)	⬇️
nanos-py3.13	75.94% <60.00%> (-0.24%)	⬇️
nanos-py3.9	76.05% <60.00%> (-0.24%)	⬇️
nanosp-py3.10	75.99% <60.00%> (-0.24%)	⬇️
nanosp-py3.11	75.99% <60.00%> (-0.24%)	⬇️
nanosp-py3.12	75.99% <60.00%> (-0.24%)	⬇️
nanosp-py3.13	75.99% <60.00%> (-0.24%)	⬇️
nanosp-py3.9	76.09% <60.00%> (-0.24%)	⬇️
nanox-py3.10	75.94% <60.00%> (-0.24%)	⬇️
nanox-py3.11	75.94% <60.00%> (-0.24%)	⬇️
nanox-py3.12	75.94% <60.00%> (-0.24%)	⬇️
nanox-py3.13	75.94% <60.00%> (-0.24%)	⬇️
nanox-py3.9	76.05% <60.00%> (-0.24%)	⬇️
stax-py3.10	77.60% <60.00%> (-0.26%)	⬇️
stax-py3.11	77.60% <60.00%> (-0.26%)	⬇️
stax-py3.12	77.60% <60.00%> (-0.26%)	⬇️
stax-py3.13	77.60% <60.00%> (-0.26%)	⬇️
stax-py3.9	77.57% <60.00%> (-0.26%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud