Skip to content

Implement fast_erf() and fast_gelu()#45

Open
certik wants to merge 1 commit into
mainfrom
fast_erf
Open

Implement fast_erf() and fast_gelu()#45
certik wants to merge 1 commit into
mainfrom
fast_erf

Conversation

@certik

@certik certik commented Mar 22, 2023

Copy link
Copy Markdown
Owner

This approximates the erf() function directly.

To use it, apply the following patch:

--- a/gpt2.f90
+++ b/gpt2.f90
@@ -109,7 +109,7 @@ real(sp), intent(in) :: x(:,:), fc_w(:,:), fc_b(:), proj_w(:,:), proj_b(:)
 real(sp) :: y(size(x,1),size(x,2))
 !real(sp) :: a(4*size(x,1),size(x,2))
 !a = gelu(linear(x, fc_w, fc_b))
-y = linear(gelu(linear(x, fc_w, fc_b)), proj_w, proj_b)
+y = linear(fast_gelu(linear(x, fc_w, fc_b)), proj_w, proj_b)
 end function
 
 function attention(n_embd_head,n_seq,n_seq_x, q, k, v, mask) result(y)

This approximates the erf() function directly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant