Woosuk Kwon
|
7a7929abe8
|
Implement preemption via recomputation & Refactor scheduling logic (#12)
|
2023-03-30 14:51:46 -07:00 |
|
Woosuk Kwon
|
80a2f812f1
|
Implement LLaMA (#9)
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
|
2023-03-30 12:25:32 +08:00 |
|
Woosuk Kwon
|
64e0e38314
|
Add cache watermark to avoid frequent cache eviction (#11)
|
2023-03-29 16:38:48 -07:00 |
|
Zhuohan Li
|
721fa3df15
|
FastAPI-based working frontend (#10)
|
2023-03-29 14:48:56 +08:00 |
|
Woosuk Kwon
|
d359cda5fa
|
Minor
|
2023-03-26 08:00:39 +00:00 |
|
Woosuk Kwon
|
cfae35b861
|
Add miscellaneous updates (#8)
|
2023-03-13 13:48:38 -07:00 |
|
Woosuk Kwon
|
e9d3f2ff77
|
Add memory analyzer & utomatically configure KV cache size (#6)
|
2023-03-11 23:23:14 -08:00 |
|
Woosuk Kwon
|
1a7eb7da61
|
Support beam search & parallel generation (#7)
|
2023-03-10 09:58:21 -08:00 |
|
Woosuk Kwon
|
0deacbce6e
|
Implement single_query_cached_kv_attention kernel (#3)
|
2023-03-01 15:02:19 -08:00 |
|
Woosuk Kwon
|
6aef2278f4
|
[Minor] Fix printing format
|
2023-02-24 11:56:06 +00:00 |
|
Woosuk Kwon
|
1132fae0ca
|
Add Frontend
|
2023-02-24 11:46:43 +00:00 |
|
Woosuk Kwon
|
ef6098ec51
|
Merge pre_step and step
|
2023-02-24 10:36:08 +00:00 |
|
Woosuk Kwon
|
53f70e7334
|
Reduce the number of states in scheduler
|
2023-02-24 10:22:39 +00:00 |
|
Woosuk Kwon
|
afdbe5d373
|
[WIP] Add server script
|
2023-02-24 01:33:37 +00:00 |
|
Woosuk Kwon
|
fdd0f2f472
|
Minor
|
2023-02-23 20:23:47 +00:00 |
|
Woosuk Kwon
|
4f6f4967f6
|
Add get_block_table
|
2023-02-23 07:55:14 +00:00 |
|
Woosuk Kwon
|
331fa0b042
|
Implement scheduler.step & Add a threshold for batch size
|
2023-02-23 07:54:20 +00:00 |
|
Woosuk Kwon
|
2729087efe
|
Fix a ref count bug in swapping
|
2023-02-14 09:20:12 +00:00 |
|
Woosuk Kwon
|
c128c2ed03
|
Fix double free
|
2023-02-14 09:14:05 +00:00 |
|
Woosuk Kwon
|
c80ed212a1
|
Minor
|
2023-02-14 09:06:27 +00:00 |
|
Woosuk Kwon
|
263e91b386
|
Minor
|
2023-02-14 09:05:39 +00:00 |
|
Woosuk Kwon
|
be1e2163c9
|
Fix memory leak in swapping
|
2023-02-14 08:46:33 +00:00 |
|
Woosuk Kwon
|
7e5f604e68
|
Fix bugs in scheduler
|
2023-02-14 02:25:32 +00:00 |
|
Woosuk Kwon
|
1f739f9b03
|
Fix a bug in swap_in and swap_out
|
2023-02-14 02:12:58 +00:00 |
|
Woosuk Kwon
|
b1644f764f
|
Fix can_swap_in
|
2023-02-14 01:57:40 +00:00 |
|
Woosuk Kwon
|
531e1c74e8
|
Fix typo
|
2023-02-13 18:51:33 +00:00 |
|
Woosuk Kwon
|
d3d317665b
|
Fix scheduler
|
2023-02-13 09:37:00 +00:00 |
|
Woosuk Kwon
|
5a309bb588
|
Add scheduler
|
2023-02-13 02:39:53 +00:00 |
|
Woosuk Kwon
|
eb52db1bea
|
Fix can_swap_in
|
2023-02-13 02:38:50 +00:00 |
|
Woosuk Kwon
|
a2a9869cb7
|
SERVING -> RUNNING
|
2023-02-12 08:25:05 +00:00 |
|
Woosuk Kwon
|
46958cf941
|
BlockAllocator -> BlockManager
|
2023-02-12 08:15:18 +00:00 |
|
Woosuk Kwon
|
38ed06a846
|
Minor
|
2023-02-09 11:30:55 +00:00 |
|
Woosuk Kwon
|
aa78aeaa0f
|
Add block manager
|
2023-02-09 11:27:33 +00:00 |
|