Xuechen Li bb4cded17b support when num_heads is not divisible by world_size; resolves #459 (#461) ... * uneql rank. * trim. * enable passing in number of heads for each rank. * simplify. * simplify. * cleanup. * fix col parallel. * fix bug with row parallel. * fit out proj. * refac. * fix sharding logic. * refac sharding. * refac. * support multiple of. * make fn reuseable. * fix bug in dimensions. * scaffold. * test uneven heads. * fix test by adding barrier. * refac. * reuse code. * clean up.		2023-08-18 14:10:35 -07:00
..
__init__.py	Add __init__.py files to subdirectories for installation	2022-11-17 16:55:44 -08:00
bert.py	Allow rotary embeddings for Bert (#363)	2023-07-23 00:21:45 -07:00
falcon.py	[GPT] Implement Falcon	2023-07-23 10:32:29 -07:00
gpt_neox.py	Implement GPT-NeoX	2023-03-29 01:21:25 -07:00
gpt.py	support when num_heads is not divisible by world_size; resolves #459 (#461)	2023-08-18 14:10:35 -07:00
gptj.py	Implement LLaMa	2023-04-18 21:51:35 -07:00
llama.py	enable loading hf llama checkpoints for training (#446)	2023-08-15 08:33:15 -07:00
opt.py	Implement LLaMa	2023-04-18 21:51:35 -07:00
vit.py	[ViT] Run black on vit.py	2023-08-17 17:45:09 -07:00