Cannot infer, UnpicklingError

[thelabcat]

python infer.py config_filename=configs/LRS3_V_WER19.1.ini data_filename=video.mp4

/home/wilbur/bin/Visual_Speech_Recognition_for_Multiple_Languages/pipelines/model.py:47: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  self.model.load_state_dict(torch.load(model_path, map_location=lambda storage, loc: storage))
/home/wilbur/bin/Visual_Speech_Recognition_for_Multiple_Languages/espnet/asr/asr_utils.py:773: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  model_state_dict = torch.load(path, map_location=lambda storage, loc: storage)
Error executing job with overrides: ['config_filename=configs/LRS3_V_WER19.1.ini', 'data_filename=video.mp4']
Traceback (most recent call last):
  File "infer.py", line 20, in <module>
    main()
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "infer.py", line 15, in main
    output = InferencePipeline(cfg.config_filename, device=device, detector=cfg.detector, face_track=True)(cfg.data_filename, cfg.landmarks_filename)
  File "/home/wilbur/bin/Visual_Speech_Recognition_for_Multiple_Languages/pipelines/pipeline.py", line 52, in __init__
    self.landmarks_detector = LandmarksDetector(device="cuda:0")
  File "/home/wilbur/bin/Visual_Speech_Recognition_for_Multiple_Languages/pipelines/detectors/retinaface/detector.py", line 16, in __init__
    self.face_detector = RetinaFacePredictor(
  File "/home/wilbur/bin/Visual_Speech_Recognition_for_Multiple_Languages/face_detection/ibug/face_detection/retina_face/retina_face_predictor.py", line 28, in __init__
    pretrained_dict = torch.load(model.weights, map_location=self.device)
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/torch/serialization.py", line 1114, in load
    return _legacy_load(
  File "/home/wilbur/.conda/envs/autoavsr/lib/python3.8/site-packages/torch/serialization.py", line 1338, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

I was trying to use the AutoAVSR models, but IDK what to use honestly.

[xduzhangjiayu]

@thelabcat Have you solve that? I met the same issue

[thelabcat]

No. I did manage to get something "working" using their Google Colab example, but it turns out it is a ridiculously large amount of VRAM for even the tiniest video clips.

[xduzhangjiayu]

@thelabcat I found the solution, you need to download the checkpoint here "https://github.com/hhj1897/face_detection/blob/master/ibug/face_detection/retina_face/weights/Resnet50_Final.pth", and place it on the right place in your project