SIGN IN SIGN UP
hpcaitech / ColossalAI UNCLAIMED

Making large AI models cheaper, faster and more accessible

0 0 96 Python

[Inference]Support FP16/BF16 Flash Attention 2 And Add high_precision Flag To Rotary Embedding (#5461)

* Support FP16/BF16 Flash Attention 2

* fix bugs in test_kv_cache_memcpy.py

* add context_kv_cache_memcpy_kernel.cu

* rm typename MT

* add tail process

* add high_precision

* add high_precision to config.py

* rm unused code

* change the comment for the high_precision parameter

* update test_rotary_embdding_unpad.py

* fix vector_copy_utils.h

* add comment for self.high_precision when using float32
Y
yuehuayingxueluo committed
87079cffe8e006d4949aa7ca7cb60e6b813ff701
Parent: 7ff42cc
Committed by GitHub <noreply@github.com> on 3/25/2024, 5:40:34 AM