Xuechen Li bb4cded17b support when num_heads is not divisible by world_size; resolves #459 (#461) ... * uneql rank. * trim. * enable passing in number of heads for each rank. * simplify. * simplify. * cleanup. * fix col parallel. * fix bug with row parallel. * fit out proj. * refac. * fix sharding logic. * refac sharding. * refac. * support multiple of. * make fn reuseable. * fix bug in dimensions. * scaffold. * test uneven heads. * fix test by adding barrier. * refac. * reuse code. * clean up.		2023-08-18 14:10:35 -07:00
..
test_bert.py	[FusedDense] Support relu, rename FusedDenseGeluDense -> FusedMLP	2023-01-17 18:12:27 -08:00
test_falcon.py	[GPT] Implement parallel LLaMa	2023-07-28 15:52:48 -10:00
test_gpt_generation_cg.py	[Gen] Fix FT kernel smem size, CG when batch size changed	2023-04-20 17:03:13 -07:00
test_gpt_generation_parallel.py	Implement GPT-J	2023-03-22 16:16:58 -07:00
test_gpt_generation.py	[MHA] Implement MQA/GQA	2023-07-23 00:06:58 -07:00
test_gpt_neox.py	Implement LLaMa	2023-04-18 21:51:35 -07:00
test_gpt_parallel.py	[FusedDense] Support relu, rename FusedDenseGeluDense -> FusedMLP	2023-01-17 18:12:27 -08:00
test_gpt.py	Implement GPT-J	2023-03-22 16:16:58 -07:00
test_gptj.py	[Rotary] Fix tests when loading state dict with rotary inv_freqs	2023-07-26 07:16:33 -10:00
test_llama.py	support when num_heads is not divisible by world_size; resolves #459 (#461)	2023-08-18 14:10:35 -07:00
test_opt.py	Implement GPT-J	2023-03-22 16:16:58 -07:00
test_vit.py	[ViT] Minor fix so it runs	2023-08-17 17:25:34 -07:00