5f1ae4a34b
* check in the two ways of approaching backwards for softcapping, both functional * prepare the softcap switch for backwards * temporary * cleanup to the way Tri prefers * calculate dtanh when copying from scores -> dtanh Tensor * no ternary operators allowed for constexpr, so just use some hack found online * fix maybe_dtanh, restore some files * restore another file * move calculate_dtanh to utils and colocate with apply_softcap * cleanup * maybe last cleanup * save for another pr * remove a stray line * fix spacing * fix an issue, and make test_flash_attn.py ready to test softcapping backwards |
||
|---|---|---|
| .. | ||
| cutlass@756c351b49 | ||
| flash_attn | ||
| ft_attention | ||
| fused_dense_lib | ||
| fused_softmax | ||
| layer_norm | ||
| rotary | ||
| xentropy | ||