History

Atlas a62bc0109c [Misc] Add Gamma-Distribution Request Generation Support for Serving Benchmark. (#10105 ) Signed-off-by: Mozhou <spli161006@gmail.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>		2024-11-07 11:20:30 +00:00
..
cutlass_benchmarks	[Kernel] Add per-tensor and per-token AZP epilogues (#5941 )	2024-08-06 18:17:08 +00:00
kernels	[CI/Build] drop support for Python 3.8 EOL (#8464 )	2024-11-06 07:11:55 +00:00
overheads	[Core] Deprecating block manager v1 and make block manager v2 default (#8704 )	2024-10-17 11:38:15 -05:00
backend_request_func.py	[CI/Build] drop support for Python 3.8 EOL (#8464 )	2024-11-06 07:11:55 +00:00
benchmark_latency.py	[Misc] Make benchmarks use EngineArgs (#9529 )	2024-10-22 15:40:38 -07:00
benchmark_prefix_caching.py	[Misc] Make benchmarks use EngineArgs (#9529 )	2024-10-22 15:40:38 -07:00
benchmark_prioritization.py	[Misc] Make benchmarks use EngineArgs (#9529 )	2024-10-22 15:40:38 -07:00
benchmark_serving.py	[Misc] Add Gamma-Distribution Request Generation Support for Serving Benchmark. (#10105 )	2024-11-07 11:20:30 +00:00
benchmark_throughput.py	[Feature] Update benchmark_throughput.py to support image input (#9851 )	2024-11-05 19:30:02 +00:00
launch_tgi_server.sh	[benchmark] Update TGI version (#7917 )	2024-08-27 15:07:53 -07:00
README.md	[Feature] Update benchmark_throughput.py to support image input (#9851 )	2024-11-05 19:30:02 +00:00
sonnet.txt	feat(benchmarks): Add Prefix Caching Benchmark to Serving Benchmark (#3277 )	2024-03-27 13:39:26 -07:00

README.md

Benchmarking vLLM

Downloading the ShareGPT dataset

You can download the dataset by running:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

Downloading the ShareGPT4V dataset

The json file refers to several image datasets (coco, llava, etc.). The benchmark scripts will ignore a datapoint if the referred image is missing.

wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json
mkdir coco -p
wget http://images.cocodataset.org/zips/train2017.zip -O coco/train2017.zip
unzip coco/train2017.zip -d coco/