fix RMATest failure and reduce testing cost (#2002)

[tianfengfrank]

Summary:

The NvlEnabledTestParam test was hanging because it had two NCCL communicators alive simultaneously — and it only needed one.

How it happened: D95122239 migrated RMATest from ::testing::Test to NcclxBaseTest, which made SetUp() create a full NCCL communicator (this->comm) for every test. This was correct for RMATestParam and MultiWindowTestParam which use this->comm. But NvlEnabledTestParam never uses it — it creates its own communicator with parameterized backends (NVL+IB or IB-only). Nobody noticed the test body was now running with two active comms.

Why it hangs: After 97 prior test cycles (each creating/destroying a comm with ~54MB pinned memory across 8 ranks), the CUDA pinned memory pool is fragmented. When NvlEnabledTestParam tries to bootstrap its second comm while the fixture's comm is still alive, one or more ranks can't complete cudaHostAlloc → ncclSocketAccept blocks waiting for that rank → all ranks hang → parent process sends SIGTERM.

The fix: Override SetUp()/TearDown() in NvlEnabledTestParam to skip the unnecessary fixture comm creation. Now only one comm exists at a time, and the bootstrap completes cleanly.

NOTE: we also decrease the NumIter to save test resources. And add re_timeout to avoid false alarm.

Reviewed By: dolpm

Differential Revision: D100051366

[meta-codesync]

@tianfengfrank has exported this pull request. If you are a Meta employee, you can view the originating Diff in D100051366.