tools: add more options to track flaky tests

[aduh95]

• For tests that are listed as flaky, try running them 100 times until we get a passing one. If it keeps failing after 100 attempts, it's not flaky, it's broken.
• For test failures from tests that are not marked as flaky, try to run them 10 times to get an idea if the test is flaky or broken.

Those changes only apply to GHA, Jenkins config is not on this repo AFAIK, and local runs are not affected as it doesn't change the defaults.

My hope is with such changes, we would be lenient on marking tests as flaky because the CI would check that are indeed flaky (currently, it would simply ignore its result).

Refs: #43929 (comment)
/cc @tniessen

[nodejs-github-bot]

Review requested:

• @nodejs/actions

[tniessen]

Oh wow! You were quick to act on my suggestion!

[tniessen]

cc @nodejs/build

[mcollina]

lgtm

[tniessen]

The approach here looks like what I suggested, but I'd prefer if someone who's more familiar with make run-ci and tools/test.py takes a look, too.

[lpinca]

The problem I see here is that it does not run tests on platforms where tests are flaky, for example arm or smartOS.

[aduh95]

The problem I see here is that it does not run tests on platforms where tests are flaky, for example arm or smartOS.

If this PR lands, the config of the Jenkins runners can be updated to take advantage of these changes, and can be applied to arm and/or SmartOS. Or do you mean something else?

[lpinca]

If a test is flaky only a particular platform and we don't measure its flakiness on that particular platform how can we determine if it is flaky or broken?

[tniessen]

I might be missing something; my suggestion was to measure flakiness on any platform where tests are marked as flaky.

[lpinca]

That makes sense but from my understanding this uses only GHA platforms which is a small subset of the supported platforms.

[tniessen]

My understanding is that this is limited to GHA only because Jenkins config lives outside of the repo, and once this lands, Jenkins could be adapted accordingly. Is that correct @aduh95?

[lpinca]

Ah ok, I guess that's what @aduh95 wrote in #43954 (comment).

[tniessen]

This needs to be adapted:

node/tools/test.py

Lines 1436 to 1438 in d29e78a

	if options.flaky_tests not in [RUN, SKIP, DONTCARE]:
	print("Unknown flaky-tests mode %s" % options.flaky_tests)
	return False

[nodejs-github-bot]

Landed in 2e4bcfc