Fix MoE expert outputs not weighted by gate probabilities

[MRPRESIDENT66]

The MoEBlock computes softmax gate probabilities for each expert but only uses them for top-k selection. The expert outputs are summed without being scaled by their gate weights, which diverges from the standard MoE formulation (result = Σ gate_weight_i × expert_i(input)).

This matches the reference HuggingFace Transformers implementation in MixtralSparseMoeBlock where each expert's output is multiplied by its routing weight before accumulation.

[edwardcapriolo]

There are no unit tests here. I will ty with mixtral and see how it goes

[edwardcapriolo]

Changes output to useless:

Was:
Edward Capriolo is a writer, director, and producer of theatrical films, television, and documentaries. He is also a professor of film and media studies at the University of North Carolina, Wilmington. He is the founder and director of the Wilmington Film Festival, a juried competition for independent filmmakers. He is also the founder and director of the Wilmington Film Commission, a non-profit organization that supports the local film industry. He>

[main] INFO io.teknek.deliverance.model.AbstractModel - Tensor provider = Native SIMD Operations, parallelSplitSize = 32 
[main] INFO io.teknek.deliverance.model.AbstractModel - Model type = Q4, Working memory type = F32, Quantized memory type = I8
 
1
0
0
0
0
0
0
0
0

 mvn test -Dtest=MixralIT

[edwardcapriolo]

Moving the scale BEFORE the results doesnt blow up the output.

 model.configurableTensorProvider.get().scale(gateWeight, moeResult, 0, model.config.embeddingLength);        
                                                                                                                                 
                    // matmul the projection and sum into result                                                                 
                    try (AbstractTensor bufq = model.maybeQuantize(buf)) { 
                    ```
                    
                   But in my limited testing the result is unchanged

[MRPRESIDENT66]

Moving the scale BEFORE the results doesnt blow up the output.

 model.configurableTensorProvider.get().scale(gateWeight, moeResult, 0, model.config.embeddingLength);        
                                                                                                                                 
                    // matmul the projection and sum into result                                                                 
                    try (AbstractTensor bufq = model.maybeQuantize(buf)) { 
                    ```
                    
                   But in my limited testing the result is unchanged

[MRPRESIDENT66]

Hi, sorry for not being able to test locally with a full Mixtral model.

Regarding your suggestion of moving scale before the matmul — I believe it has no effect because dotProductChunk uses result.set(...) which overwrites moeResult entirely, so any scaling applied before it gets lost.

I also realized my first version of this fix was incomplete. Multiplying by the raw softmax probabilities without re-normalizing causes the output to shrink, because the top-k weights (e.g. top-2 out of 8 experts) sum to less than 1. This is why the output broke in your test.

The updated fix adds a re-normalization step before applying the weights, ensuring the selected top-k weights sum to 1

[edwardcapriolo]

Send the PR here. edwardcapriolo/deliverance#87

I dont have merge ability for this project and it is not being maintained ATM