Added optimizer implementation (#3699)
Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/3699 This adds the optimizer logic, reusing much of the logic from [LiteInterpreter](https://fburl.com/code/t5dqeyje). The main differences being: 1. SGDParamGroup takes in a Span<char*> and a Span<Tensor> which represents named parameters. unlike LI or core PT, portable tensors don't use the autograd framework and we won't be supporting it either. instead, we're likely to use the backwards graph to calculate the gradients of the parameters. in that case, we need a way to map the gradients to its appropriate parameter. We expect that the sizes of the two spans are equal, and the index of a specific parameter is the same in both spans. 2. SGD step takes in a Span<char*> and a Span<Tensor> which represents the named gradients. We use this to match the gradient to the appropriate parameter. Similar to above, we expect that the spans are equal sizes and the index of a gradient data is the same as its parameter name. 3. Uses the out variant operations rather than the inplace or functional variants since those are already implemented. I *believe* since we're only using clone, add (same sized tensor), and mul_scalar, there isn't any harm in overwriting the data. 4. For the momentum buffer, I allocate memory for the underlying data and TensorImpl. This gets cleaned up when the SGD destructor is called. Reviewed By: iseeyuan Differential Revision: D57216865 fbshipit-source-id: 5ab49b6f584debc15976982a2e9eb964515e5c54
D
David Lin committed
d44877beb583d46c39ce2b6e0065d5e1e5c7a530
Parent: 1343224
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 5/24/2024, 6:21:15 PM