modify model to use sdpa_with_kv_cache op (#2154)
Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/2154 This diff adds new option of `use_sdpa_with_kv_cache` to do model surgery and make Attention layer use sdpa_with_kv_cache op. NB: This will likely conflict with some upcoming refacotring. NB: Model exported with this op is not runnable in aten mode ghstack-source-id: 216820234 exported-using-ghexport validated oss ci is clean. have to by pass because ci think its needs internal linter to pass. bypass-github-export-checks Reviewed By: mergennachin Differential Revision: D54075828 fbshipit-source-id: 3f89c77c3315fbf0f0ed49f9a4cc5ddb231cbe6c
K
Kimish Patel committed
c67f0ff57993239c5881d979e267a2ec560f208f
Parent: 92ac890
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 2/29/2024, 5:50:29 PM