Xuechen Li bb4cded17b support when num_heads is not divisible by world_size; resolves #459 (#461) ... * uneql rank. * trim. * enable passing in number of heads for each rank. * simplify. * simplify. * cleanup. * fix col parallel. * fix bug with row parallel. * fit out proj. * refac. * fix sharding logic. * refac sharding. * refac. * support multiple of. * make fn reuseable. * fix bug in dimensions. * scaffold. * test uneven heads. * fix test by adding barrier. * refac. * reuse code. * clean up.		2023-08-18 14:10:35 -07:00
..
__init__.py	Add __init__.py files to subdirectories for installation	2022-11-17 16:55:44 -08:00
block.py	[MLP] Add ParallelMLP	2023-07-22 23:45:51 -07:00
embedding.py	Reorder LN in Block, support OPT	2023-01-15 22:14:31 -08:00
mha.py	support when num_heads is not divisible by world_size; resolves #459 (#461)	2023-08-18 14:10:35 -07:00
mlp.py	[MLP] Change the check for out_features being None	2023-08-10 00:04:38 -07:00