Commit Graph

17 Commits

Author SHA1 Message Date
Kai Londenberg
b443207c1f
Paged Attention support for FA3 (#1268) 2024-11-09 17:05:01 -08:00
milesvant
c1d146cbd5
Fix copy-paste error in hopper tests (#1279) 2024-10-15 13:54:40 -07:00
jayhshah
a5a75274bc
FA3 kvcache + split kv + gqa parallelization (#1236) 2024-10-15 00:21:22 -07:00
Ying Zhang
be6c1b98c4 small fixes 2024-09-16 16:13:00 -07:00
Ying Zhang
dff976a84a fixes 2024-09-16 15:44:44 -07:00
Ying Zhang
8cbc8a042f small fixes 2024-09-16 14:54:39 -07:00
Ying Zhang
db80387343 Add seqused_q in fwd / bwd and seqused_k in bwd. 2024-09-16 14:24:11 -07:00
jayhshah
c92ca63268
FA3 FP8 qkv descales + restore max offset for h128 causal + added sync for producer WG (#1173) 2024-08-25 12:18:04 -07:00
Ying Zhang
53537da422 add a unittest 2024-08-17 13:23:50 -07:00
Tri Dao
c33de664a1 Fix import in test 2024-08-01 02:14:25 -07:00
Tri Dao
bafe253042 [FA3] Bwd 2024-08-01 01:57:06 -07:00
jayhshah
5018ac6ac5
Fp8 kernel with "in-kernel" transpose of V in producer (#1100)
* base version

* restructure pipelines, add special fp8 epilogue

* add variants

* add fp8 causal and modify dynamic tile scheduler

* better causal schedule

* maintain two schedules for non causal and causal

* removing macros

* fix regression

* clean up unneeded methods and variants

* fix mistake with NumProducerThreads

* base version

* restructure pipelines, add special fp8 epilogue

* add variants

* add fp8 causal and modify dynamic tile scheduler

* better causal schedule

* maintain two schedules for non causal and causal

* removing macros

* fix regression

* clean up unneeded methods and variants

* fix mistake with NumProducerThreads

* use seqlen traits

* add fp8 .cu files and benchmark script

* fix merge issue

* fix merge issue

* fix merge issue

* remove duplicate code

* fix regression with varseqlen

* move varseqlen init in constexpr

* fix test script

* more constexpr on varseqlen and add max offset

* add back test cases
2024-07-30 14:14:14 -07:00
Tri Dao
3aae9c18c1 Revert "Changes For FP8 (#1075)"
This reverts commit 1899c970c8.
2024-07-25 01:28:44 -07:00
ganeshcolfax
1899c970c8
Changes For FP8 (#1075)
* adding files for fp8 changes.

* removed contiguous check.

* enable all tests except odd-seq-lengths, where it crashes now.

* undid clang formatting.

* change to correct tile size for headdim=128.

* fixed odd-seq-len-k.

* minor formatting.

* minor reformatting.

---------

Co-authored-by: Tri Dao <tridao@users.noreply.github.com>
2024-07-23 13:51:14 -07:00
Ying Zhang
dfe1a59e4b
Add var-seq-len to FA3 fp16 / bf16 fwd (#1072)
* fwd var-seq-len

* fixes

* benchmark

* fixes

---------

Co-authored-by: Tri Dao <tridao@users.noreply.github.com>
2024-07-22 21:32:41 -07:00
Tri Dao
74b0761ff7 [FA3] BF16 forward 2024-07-14 23:39:46 -07:00
Tri Dao
7f67966cc7 FA3 initial code release 2024-07-11 09:53:36 -07:00