Fabric error: connection not established

[Lime-Cakes]

I got the following out of the blue while testing some code on free pod4 machines. After restarting machine, the same loading code seemed to run fine, being stable and produce inferenceModel that can be run and produce correct results.

  File "<ipython-input-10-d80f14a13956>", line 54, in restoreSavedModel
    inference_model.loadExecutable(
  File "/usr/local/lib/python3.8/dist-packages/poptorch/_impl.py", line 358, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/poptorch/_poplar_executor.py", line 1000, in loadExecutable
    self._on_device_attach()
  File "/usr/local/lib/python3.8/dist-packages/poptorch/_poplar_executor.py", line 383, in _on_device_attach
    self.copyWeightsToDevice()
  File "/usr/local/lib/python3.8/dist-packages/poptorch/_poplar_executor.py", line 562, in copyWeightsToDevice
    poptorch_core.copyWeightsToDevice_impl(self._executable,
poptorch.poptorch_core.UnrecoverableError: In unknown:0: 'poplar_unrecoverable_runtime_error': fabric_atomic_increment_hsp_register: Could not complete operation. Fabric error: connection not established: CONNECTION_KILL, device 3, IP 10.5.17.11:50052. IPUoF Server unreachable at 2023-02-03T15:32:48.181819Z
Error raised in:
  [0] copyWeightsToDevice

[payoto]

Hello @Lime-Cakes,
This issue is due to a problem with the machine. I will reach out to Paperspace Gradient to investigate which machine is affected but it would be very useful if you could contact their support with this error and provide them with the "notebook ID" if you still have it. The "notebook ID" can be found in your list of notebooks:

Thanks for reporting this error,
Alexandre