The repo already covers autograd, linear/logistic regression, an MLP with hand-written backprop, activations/init, and first-order optimizers. The next building block toward a tiny transformer is the attention formula itself.
Goal
Implement scaled dot-product self-attention for a single head in numpy only: softmax(Q Kᵀ / √d_k) V, with optional causal masking, a SelfAttentionHead that projects the same sequence into Q/K/V, and pytest coverage for shapes, scale, masks, and edge cases (empty / single token).
Why
Attention is the core of modern transformers. Showing the math without a framework is the point of this lab.
The repo already covers autograd, linear/logistic regression, an MLP with hand-written backprop, activations/init, and first-order optimizers. The next building block toward a tiny transformer is the attention formula itself.
Goal
Implement scaled dot-product self-attention for a single head in numpy only:
softmax(Q Kᵀ / √d_k) V, with optional causal masking, aSelfAttentionHeadthat projects the same sequence into Q/K/V, and pytest coverage for shapes, scale, masks, and edge cases (empty / single token).Why
Attention is the core of modern transformers. Showing the math without a framework is the point of this lab.