squall/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Bugfix] Allow prefill of assistant response when using mistral_common (#9446)

Browse Source
This commit is contained in:
sasha0552 2024-10-17 15:06:37 +00:00 committed by GitHub
parent 9d30a056e7
commit 5e443b594f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
vllm/transformers_utils/tokenizers/mistral.py
View File

@ -166,6 +166,10 @@ class MistralTokenizer:
tools: Optional[Dict[str, Any]] = None, tools: Optional[Dict[str, Any]] = None,
**kwargs) -> List[int]: **kwargs) -> List[int]:
last_message = messages[-1]
if last_message["role"] == "assistant":
last_message["prefix"] = True
request = ChatCompletionRequest(messages=messages, request = ChatCompletionRequest(messages=messages,
tools=tools) # type: ignore[type-var] tools=tools) # type: ignore[type-var]
encoded = self.mistral.encode_chat_completion(request) encoded = self.mistral.encode_chat_completion(request)