Cut dependencies and clean up Arm backend unit tester (#2231)

Summary:
bypass-github-pytorch-ci-checks
bypass-github-export-checks

Pull Request resolved: https://github.com/pytorch/executorch/pull/2231

Reviewed By: mergennachin

Differential Revision: D54640970

Pulled By: digantdesai

fbshipit-source-id: 5bab38b60cff1ceb74d1a0b06694e240af1ba9d1