Tri Dao
|
f5b308e258
|
[LayerNorm] Rename layernorm.py -> layer_norm.py
|
2024-01-05 00:21:03 -08:00 |
|
Tri Dao
|
665b55e2e2
|
[LayerNorm] Implement parallel layer norm in Triton
|
2024-01-04 23:15:35 -08:00 |
|
Tri Dao
|
aa5c6438c5
|
[LayerNorm] Implement rowscale in Triton layernorm
|
2024-01-04 01:07:03 -08:00 |
|
Tri Dao
|
cd089597fd
|
[LayerNorm] Implement dropout in fused residual + LN/RMSNorm
|
2023-12-19 16:26:07 -08:00 |
|
Tri Dao
|
9356a1c038
|
[LayerNorm] Implement layer_norm_linear
|
2023-11-30 21:46:07 -08:00 |
|
Tri Dao
|
017716451d
|
[LayerNorm] Add postnorm residual + LayerNorm/RMSNorm in Triton
|
2023-11-13 22:37:55 -08:00 |
|
Tri Dao
|
79bd1a2d5d
|
[LayerNorm] Implement residual + LayerNorm/RMSNorm in Triton
|
2023-11-13 02:04:49 -08:00 |
|
Tri Dao
|
0e8c46ae08
|
Run isort and black on test files
|
2023-08-18 20:59:35 -07:00 |
|
Tri Dao
|
2a2a3c4bfd
|
[LayerNorm] Add test for randomness
|
2023-07-23 12:31:55 -10:00 |
|
Tri Dao
|
d2f4324f4c
|
[LayerNorm] Make sure memory addresses are aligned to 16 bytes
|
2023-07-04 14:53:12 -07:00 |
|
Tri Dao
|
393882bc08
|
[LayerNorm] Implement LN with parallel residual, support dim 8k
|
2023-03-31 14:23:45 -07:00 |
|
Tri Dao
|
88173a1aaf
|
[FusedDense] Support relu, rename FusedDenseGeluDense -> FusedMLP
|
2023-01-17 18:12:27 -08:00 |
|
Tri Dao
|
93383bd55b
|
[TP] Implement TensorParallel without sequence parallel
|
2023-01-07 13:45:22 -08:00 |
|
Tri Dao
|
6738d9477d
|
[LayerNorm] Implement RMS Norm
|
2023-01-06 17:34:22 -08:00 |
|
Tri Dao
|
226a1b721d
|
Implement TensorParallel for FusedDense and FusedDenseGeluDense
|
2022-12-24 11:48:56 -08:00 |
|
Tri Dao
|
e68ebbe89a
|
Simplify FusedDense
|
2022-12-22 21:25:31 -08:00 |
|
Tri Dao
|
5db330519a
|
[LayerNorm] Support taking subset of input or subset of output
|
2022-12-12 22:16:14 -08:00 |
|
Tri Dao
|
ae137ed17a
|
[LayerNorm] Fuse LayerScale
|
2022-12-10 23:28:23 -08:00 |
|
Tri Dao
|
8c6609ae1a
|
[LayerNorm] Support all dimensions up to 6k (if divisible by 8)
|
2022-12-09 02:06:22 -08:00 |
|
Tri Dao
|
fa6d1ce44f
|
Add fused_dense and dropout_add_layernorm CUDA extensions
|
2022-11-13 21:59:20 -08:00 |
|