squall/flash-attention
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #105 from jamaliki/patch-1

Browse Source 
Change default dropout value in documentation
This commit is contained in:
Tri Dao 2023-01-15 23:01:20 -08:00 committed by GitHub
commit 2ec7d3f72c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
flash_attn/flash_attention.py
View File

@ -16,7 +16,7 @@ class FlashAttention(nn.Module):
(default: 1/sqrt(d_keys) where d_keys is computed at
runtime)
attention_dropout: The dropout rate to apply to the attention
(default: 0.1)
(default: 0.0)
"""
def __init__(self, softmax_scale=None, attention_dropout=0.0, device=None, dtype=None):
super().__init__()