Memory issue with running verifiers - RL

[ADharaUTEXAS123007]

When I run uv run vf-rl @ configs/vf-rl/wiki-search.toml,

memory accumulates with each step,
And I get the error
orch.OutOfMemoryError: CUDA out of memory. Tried to allocate 36.98 GiB. GPU 0 has a total capacity of 95.00 GiB of which 35.86 GiB is free. Including non-PyTorch memory, this process has 59.13 GiB memory in use. Of the allocated memory 56.35 GiB is allocated by PyTorch, and 99.55 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[W1222 01:23:56.231734873 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())

[shyampathak22]

@ADharaUTEXAS123007 Disable gradient tracking on the entropy! Go to verifiers/rl/trainer/trainer.py and on line 306, change

entropies = entropy_from_logits(logits)

to

entropies = entropy_from_logits(logits.detach())

[shyampathak22]

#662 Opened this PR last night to address it!

[ADharaUTEXAS123007]

@shyampathak22 : Still getting this error 👍🏻
Doesn't completely solve the issue, still getting this error when you run gsm8k toml at 16th step.

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.52 GiB. GPU 0 has a total capacity of 95.00 GiB of which 16.98 GiB is free. Including non-PyTorch memory, this process has 78.02 GiB memory in use. Of the allocated memory 71.27 GiB is allocated by PyTorch, and 4.07 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)