squall/flash-attention
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

README typo

Browse Source
This commit is contained in:
Dan Fu 2022-05-27 22:38:20 +01:00
parent dc6d130088
commit 4decc3c166
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
README.md
View File

@ -35,6 +35,7 @@ We display FlashAttention speedup using these parameters (similar to BERT-base):
* Batch size 8 * Batch size 8
* Head dimension 64 * Head dimension 64
* 12 attention heads * 12 attention heads
Our graphs show sequence lengths between 128 and 4096 (when standard attention runs out of memory on an A100), but FlashAttention can scale up to sequence length 64K. Our graphs show sequence lengths between 128 and 4096 (when standard attention runs out of memory on an A100), but FlashAttention can scale up to sequence length 64K.
#### Speedup #### Speedup