remove torchao dependency (#2599)

Summary:
X-link: https://github.com/pytorch/pytorch/pull/122524

Pull Request resolved: https://github.com/pytorch/executorch/pull/2599

bypass-github-export-checks
bypass-github-pytorch-ci-checks
bypass-github-executorch-ci-checks

Reviewed By: jerryzh168

Differential Revision: D55263008

fbshipit-source-id: e62d8df96fb64b811cd3ecf4e547bbb9e9fefcc7