Skip to content

Address flaky instrumentation tests#3754

Open
avazirna wants to merge 10 commits into
masterfrom
address-flaky-tests
Open

Address flaky instrumentation tests#3754
avazirna wants to merge 10 commits into
masterfrom
address-flaky-tests

Conversation

@avazirna

@avazirna avazirna commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Technical Summary

Addresses recurring flakiness in the instrumentation (BrowserStack) test suite. Each fix targets state that leaked between tests or a flaky UI assertions:

  • InstrumentationUtility.changeWifi() now also toggles the wifi_wakeup global setting alongside svc wifi enable/disable. Without disabling Wi-Fi wakeup, the OS could silently re-enable Wi-Fi (auto-scan/reconnect) mid-test, breaking offline scenarios.
  • DemoUserOfflineTest and ManualQuarantineTest — add an @After teardown that re-enables Wi-Fi (changeWifi(true)), so a test that runs offline restores network state and doesn't poison subsequent tests.
  • SessionExpirationTests — dismiss the notification drawer with pressBack() after verifying the login-expired notification, so the open drawer doesn't interfere with the next test.
  • InstallFromListTest — make the "Install An App" matcher in tearDown case-insensitive (equalToIgnoringCase), since the label's casing varies and an exact match intermittently failed to find the button.

Safety Assurance

Safety story

Test-only change — no production/application code is touched (all changes are under app/instrumentation-tests/). There is no impact on app behavior or existing data; the blast radius is limited to the CI/instrumentation suite. Confidence comes from re-running the previously flaky tests and getting a successful build during the first attempt: https://app-automate.browserstack.com/projects/CommCare/builds/Build+app-commcare-release+apk/2559

Automated test coverage

This PR stabilizes existing instrumentation coverage rather than adding new logic — the affected tests (DemoUserOfflineTest, ManualQuarantineTest, SessionExpirationTests, InstallFromListTest) continue to assert the same behavior, now with deterministic setup/teardown of Wi-Fi and notification-drawer state.

Labels and Review

  • Do we need to enhance the manual QA test coverage ? If yes, RELEASES.md is updated accordingly
  • Does the PR introduce any major changes worth communicating ? If yes, RELEASES.md is updated accordingly
  • Risk label is set correctly
  • The set of people pinged as reviewers is appropriate for the level of risk of the change

@coderabbitai

coderabbitai Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This change modifies CaseClaimTest to add explicit synchronization in the testCaseClaimByDifferentUser test flow. It introduces imports for Espresso view matchers and a waitForView utility, then inserts a wait action that pauses the test for up to 10 seconds after case creation, blocking until the case name text appears in the view hierarchy. This gating ensures subsequent test interactions proceed only after the UI has stabilized.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The PR description follows the template and includes technical summary, safety, testing, and review sections, with only the optional product description missing.
Title check ✅ Passed The title matches the main change: a test stabilization fix for flaky instrumentation tests.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch address-flaky-tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@app/instrumentation-tests/src/org/commcare/androidTests/CaseClaimTest.java`:
- Around line 131-132: The current wait uses waitForView(withText(name)) via
onView(isRoot()) which can match the search EditText; change the matcher to
exclude EditText so it targets the result row (e.g., combine with
not(isAssignableFrom(EditText.class)) or equivalent) and use that same matcher
both in the waitForView(...) call and when performing the subsequent click so
you synchronize on and interact with the actual case result rather than the
search field.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 24419774-58eb-4cb6-8ff8-68fe080fdd5c

📥 Commits

Reviewing files that changed from the base of the PR and between 3b36b92 and df088d8.

📒 Files selected for processing (1)
  • app/instrumentation-tests/src/org/commcare/androidTests/CaseClaimTest.java

Comment on lines +131 to +132
onView(isRoot())
.perform(waitForView(withText(name), 10000, true));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Wait condition is too broad and can pass before results load.

waitForView(withText(name), ...) can match the already-filled search EditText, so this may not actually wait for the result row. Use a matcher that excludes EditText (and reuse it for the click) so synchronization targets the case result itself.

Proposed fix
+import static org.hamcrest.Matchers.not;
@@
-        onView(isRoot())
-                .perform(waitForView(withText(name), 10000, true));
-        onView(withText(name))
+        onView(isRoot())
+                .perform(waitForView(allOf(withText(name), not(withClassName(endsWith("EditText")))), 10000, true));
+        onView(allOf(withText(name), not(withClassName(endsWith("EditText")))))
                 .perform(click());
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/instrumentation-tests/src/org/commcare/androidTests/CaseClaimTest.java`
around lines 131 - 132, The current wait uses waitForView(withText(name)) via
onView(isRoot()) which can match the search EditText; change the matcher to
exclude EditText so it targets the result row (e.g., combine with
not(isAssignableFrom(EditText.class)) or equivalent) and use that same matcher
both in the waitForView(...) call and when performing the subsequent click so
you synchronize on and interact with the actual case result rather than the
search field.

@codecov

codecov Bot commented Jun 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 26.51%. Comparing base (05be72f) to head (543a42f).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3754      +/-   ##
============================================
- Coverage     26.52%   26.51%   -0.02%     
+ Complexity     4526     4523       -3     
============================================
  Files           964      965       +1     
  Lines         57703    57724      +21     
  Branches       6871     6873       +2     
============================================
- Hits          15307    15303       -4     
- Misses        40522    40546      +24     
- Partials       1874     1875       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@avazirna avazirna force-pushed the address-flaky-tests branch from df088d8 to d340d76 Compare June 7, 2026 18:20
@avazirna avazirna force-pushed the address-flaky-tests branch 3 times, most recently from f957af2 to 21690c3 Compare June 21, 2026 15:59
@avazirna avazirna force-pushed the address-flaky-tests branch from 0ef9508 to d56c6fc Compare June 22, 2026 22:24
@avazirna avazirna force-pushed the address-flaky-tests branch from fd291b1 to 38ec167 Compare June 29, 2026 12:53
@avazirna avazirna changed the title [Draft] Address flaky tests Address flaky instrumentation tests Jun 30, 2026
@avazirna avazirna force-pushed the address-flaky-tests branch from 2601096 to b412597 Compare June 30, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant