Why same model can adapt Hubert and Deepspeech？

[ranck626]

Even aud_ds and aud_hu has different shape.
for may:
hubert:(6072, 2, 1024)
deepspeech:(6073, 16, 29)
@Fictionarry Thanks for your attention!

[ranck626]

Actually， i want to use another audio feature extraction network...

[Fictionarry]

The first dimension should be the same as the frame length. It seems a common problem that different features have a different length of 1 frame, but it is not a big problem if the results are as expected. For the other dimensions, you can refer to the code to see how to handle different shapes of HuBERT, DeepSpeech and so on.

[ranck626]

@Fictionarry
Thank you for your reply. I found that when training with hubert features, the L1 value is relatively large, while the PNSR value is relatively small compared to Deepspeech features. Is this normal? Even around 20,000 steps, the value does not change much.

How does the Loss change when using May's video and hubert to extract audio features?

Thank you for your attention and wish you all the best.

[ranck626]

by the way, i set preload=False in the scene/dataset_readers.py - readCamerasFromTransforms(·)

[Fictionarry]

@Fictionarry Thank you for your reply. I found that when training with hubert features, the L1 value is relatively large, while the PNSR value is relatively small compared to Deepspeech features. Is this normal? Even around 20,000 steps, the value does not change much.

How does the Loss change when using May's video and hubert to extract audio features?

Thank you for your attention and wish you all the best.

I do not really remember the particular values but generally, HuBERT is better at the generalizability and DeepSpeech is specific for English with much fewer dimensions, so I think it's explainable that the model using DeepSpeech is easier to converge. It's probably to find some English cases that DeepSpeech keeps a similar performance or even better than HuBERT, while for most other languages may not. Besides, L1 or PSNR do not always directly reflect the dynamic performance, as it may more relate to the hair, skin, or some more irrelevant parts.