SIGN IN SIGN UP

llama : fix Gemma-2 Query scaling factors (#8473)

* 9B - query_pre_attn_scalar = 256 not 224

See https://github.com/google/gemma_pytorch/commit/03e657582d17cb5a8617ebf333c1c16f3694670e

Gemma 9b should use 256 and not 224 (self.config.hidden_size // self.config.num_attention_heads)

* llama : fix Gemma-2 Query scaling factor

ggml-ci

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
G
Georgi Gerganov committed
73cf442e7bc9baca7b3e213b261551812f1676c9
Parent: e236528
Committed by GitHub <noreply@github.com> on 7/14/2024, 11:05:09 AM