fix: fix flakiness in test com.dianping.cat.message.context.MessageIdFactoryTest#testDefaultDomainInParallel

[hofi1]

Problem:

This test is flaky due to the fact that the time which is provided (to shut the pool down) can be sufficient or not. It can fail to shut the pool down in the provided time (what results in a failed test)
If the code remains the way it is, it results in a flaky test which sometimes passes and might fail other times (non-deterministic behavior).

Solution:

I added the condition if the shutdown is finished or if it failed. Furthermore, the duration for the shutdown got increased to make sure, that it is possible to finish the process without running in a timeout. If the shutdown reaches the timeout, the test fails as intended. If the shutdown is finished successfully, the same conditions (asserts) apply as before.

Result:

The test is deterministic and not flaky. This improves the quality of the test and reduces the time to search for the bug during future development.

[prathyushreddylpr]

Can you briefly explain why the tests are flaky? Like what happens with the Assert conditions when they run before the process is completed.

[hofi1]

@prathyushreddylpr I just updated my comment, thank you for the feedback!

[zzjas]

Without the fix, does this test fail deterministically with NonDex? i.e. with a particular seed, the test always fails.

[hofi1]

@zzjas it does not fail using the NonDex engine, it can fail depending on the machine you are running this test on (depending on the speed/computational power of the computer – it failed normally on my machine even tho I have a pretty powerful machine).
This is the result of one of the patches distributed to the students by your PhD student.

[njain2208]

The Solution looks good to me and the description perfectly defines the flaky test and its fix.

[zzjas]

The "fix" essentially increased the timeout but did not solve the problem. The patch was generated by machines (AI actually) so you don't have to trust it 100%. Feel free to just open a real PR, but I would recommend changing the timeout to something shorter, not 1 hour. Or you can explore ways to rerun the test instead of failing immediately if it times out. Anyway, the same reminder: Once you open a real PR, please mark this tentative PR as Opened in your tentative_pr.csv file and also raise a PR to IDoFT marking this as Opened. Thanks!