ci: run some sharness tests under asan - #7656
Conversation
424c559 to
7f6ec81
Compare
|
Just some quick feedback -- I haven't looked at the whole PR yet. Needing to add If we do need to go with this approach, maybe it would work to do something like:
This drops the requirement to touch every existing test for the wrapped commands, test authors don't have to worry about which commands to wrap - they just write tests, and it allows us easily to add new wrapped commands in the future. |
We can certainly go with this approach with a few limited number of tests. I think the issue was that with python presumably having false positives (#7624) along with a number of these issues (ps, dd, etc.), the number of "safe" ones shrank by a pretty decent amount. The number of false positives recently shrank, due to unknown reasons, but I think it should be assumed they could come back.
Your idea is a good one. I'll change #7538 accordingly. |
79d754b to
71c5011
Compare
Problem: A test in t2714-python-cli-batch.t assumes a prior test has been executed. But that prior test will not run if ASAN is configured. Set NO_ASAN on the test to ensure it isn't executed like prior dependent ones.
Problem: Tests in t0006-module-exec.t and t2614-job-shell-doom.t test how segfaults are reported. Under ASAN, segfault signals may be handled / reported differently. Skip tests that expect a specific message when a segfault occurs. Under ASAN, that specific message cannot be expected.
Problem: A test in t0006-module-exec.t and t3306-system-routercrash.t simulate a segfault by sending a SIGSEGV signal to crash a module / broker. With ASAN, the signal could be captured, the expected result may not happen, and an asan log will be generated indicating a segfault happened. Follow up tests depend on the SIGSEGV crashing a module / broker, so we can't just skip the test. These tests are specifically covering SIGSEGV, so we don't want to just change the signal. Under ASAN, instead send a SIGKILL to crash the broker / module. This will ensure follow on tests continue to work as expected under ASAN. We will still get ample coverage of the SIGSEGV case under non-ASAN workflows.
Problem: Several tests are skipped when ASAN is enabled, but they have no comments / explanation why. Add comments explaining that the tests in t0005-exec.t are skipped because ASAN causes a segfault to be reported differently than we normally would expect. Tests in t0016-cron-faketime.t and t3001-mpi-personalities.t are skipped because they change LD_PRELOAD. Tests in t2714-python-cli-batch.t are skipped because of slowness.
Problem: Some tests fail under ASAN due to problems with underlying commands. This can be worked around if ASAN was not set on those specific commands. Prefix a special directory asan-scripts to PATH when running with ASAN. In this directory add special wrapper scripts for ps, pkill, and dd that will run without ASAN. These wrapper scripts simply unset LD_PRELOAD and ASAN_OPTIONS before running the command.
Problem: A few tests fail or hang under ASAN for test specific reasons. Set NO_ASAN on those tests.
Problem: The asan CI tests only run against unit tests in the src/ directory. This is because tests did not pass or some tests hung. Many of those issues have been fixed, although some issues still linger. Update CI and scripts to run asan over unit tests and a subset of tests in sharness. Tests with "ci=asan" tag will be run under asan. The tags will be set in a following commit.
Add the '# ci=asan' comment to a number of test files in the t/ directory. This enables these tests to run in the ASAN CI workflow. Skip a number of test files that have shown to have problems under CI with ASAN. Do not add to system related tests, regression tests, and valgrind tests. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
71c5011 to
549dd31
Compare
chu11
changed the title
|
the next iteration of our ASAN in CI attempt. Using a ci=asan tag, tag a large number (but not all) CI tests to run with ASAN. Built on top of #7538. At this point in time, it currently passes CI! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7656 +/- ##
==========================================
+ Coverage 83.91% 83.93% +0.02%
==========================================
Files 587 587
Lines 99979 99979
==========================================
+ Hits 83901 83921 +20
+ Misses 16078 16058 -20 🚀 New features to boost your workflow:
|
2 participants
Next iteration of work, built on top of #7538.
Per suggestion, run a subset of sharness tests using a "ci=asan" tag.
WIP, I'm sure there will be iteration as we figure out how long this takes, which tests we want to keep or not keep. For the time being, removed valgrind (obviously), system tests, python/lua tests, and regression tests. But kept everything else for now. I imagine I'll remove some as I iterate. (example: do i really need to run "fake resources" tests under asan? probably not)