Dont memory plan for inputs (#2155)
Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/2155 For KV cache with IO tHis results in 1. allocating kv cache in the memory plan but also allocated by llama runner 2. Doing actual copy of kv cache Also we should really make plan_input = false by default. I dont imagine a case where this does not result in making copies. Planning for output is fine but still dangerous as people may assume having reference to output tensor is all good without realizing the underlying memory being shared. ghstack-source-id: 216889056 exported-using-ghexport validated oss ci is clean. have to by pass because ci think its needs internal linter to pass. bypass-github-export-checks Reviewed By: mergennachin Differential Revision: D54161288 fbshipit-source-id: b5e7aa42d4a72e455550af5d7467f46f2a1017f8
K
Kimish Patel committed
76cbfb7cce22b6794a5c7787d9a5caca049204ea
Parent: c67f0ff
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 2/29/2024, 5:50:29 PM