squall/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Bugfix] Fix mrope_position_delta in non-last prefill chunk (#10403)

Browse Source 
Signed-off-by: imkero <kerorek@outlook.com>
This commit is contained in:
电脑星人 2024-11-17 16:50:24 +08:00 committed by GitHub
parent 76aab90ab6
commit 80d85c5d7b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
vllm/model_executor/layers/rotary_embedding.py
View File

@ -922,9 +922,9 @@ class MRotaryEmbedding(RotaryEmbedding):
torch.arange(text_len).view(1, -1).expand(3, -1) + st_idx)
llm_positions = torch.cat(llm_pos_ids_list, dim=1).reshape(3, -1)
llm_positions = llm_positions[:, context_len:seq_len]
mrope_position_delta = (llm_positions.max() + 1 -
len(input_tokens)).item()
llm_positions = llm_positions[:, context_len:seq_len]
return llm_positions.tolist(), mrope_position_delta