squall/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Bugfix] Fix logit soft cap in flash-attn backend (#7425)

Browse Source
This commit is contained in:
Woosuk Kwon 2024-08-12 09:58:28 -07:00 committed by GitHub
parent d2bc4510a4
commit cfba4def5d
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
vllm/attention/backends/flash_attn.py
View File

@ -563,6 +563,7 @@ class FlashAttentionImpl(AttentionImpl):
softmax_scale=self.scale, softmax_scale=self.scale,
causal=True, causal=True,
alibi_slopes=self.alibi_slopes, alibi_slopes=self.alibi_slopes,
softcap=self.logits_soft_cap,
).squeeze(1) ).squeeze(1)
# Reshape the output tensor. # Reshape the output tensor.