First, thank you for this amazing work!
I am suspecting that an indentation is missing at the following position of the code:
|
# Compute attention scores based on dot products between |
The reason why it raises my suspicion is that, if the code is executed as it is, it seems like the actual key values (to_tensor) are never involved in the computation of the attention scores when k means is enabled. If I am mistaken, would you mind explain why line 787 replaces the original attention scores with the values computed here (where the embedding "to_centroids" seems to be initialized to be a mapping of the queries)?
First, thank you for this amazing work!
I am suspecting that an indentation is missing at the following position of the code:
gansformer/pytorch_version/training/networks.py
Line 784 in 3a9efa4
The reason why it raises my suspicion is that, if the code is executed as it is, it seems like the actual key values (to_tensor) are never involved in the computation of the attention scores when k means is enabled. If I am mistaken, would you mind explain why line 787 replaces the original attention scores with the values computed here (where the embedding "to_centroids" seems to be initialized to be a mapping of the queries)?