squall/flash-attention
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Change default dropout value in documentation

Browse Source 
Documentation says default is 0.1, but the code has attention_dropout default at 0.0
This commit is contained in:
Kiarash Jamali 2023-01-13 10:50:07 +00:00 committed by GitHub
parent d509832426
commit 41cb909741
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
flash_attn/flash_attention.py
View File

@ -16,7 +16,7 @@ class FlashAttention(nn.Module):
(default: 1/sqrt(d_keys) where d_keys is computed at
runtime)
attention_dropout: The dropout rate to apply to the attention
(default: 0.1)
(default: 0.0)
"""
def __init__(self, softmax_scale=None, attention_dropout=0.0, device=None, dtype=None):
super().__init__()