use --use_sdpa_with_kv_cache for 1B/3B bf16 (#5861)
Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/5861 We should use this option during exporting 1B/3B models as bf16 because KVCache is always fp32. Otherwise, we see regressed performance for 1B/3B in bf16 format. ghstack-source-id: 246391007 Reviewed By: mergennachin Differential Revision: D63871048 fbshipit-source-id: 6b3ff80dbc689a04c70e2fcc5c98698bb74f899b
L
Lunwen He committed
2726bdb87efc956298e683aca4f6ddd0039f6030
Parent: 478a9b6
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 10/7/2024, 6:12:59 PM