Forward fix macOS job after test-infra #5086 (#2980)

Summary:
After https://github.com/pytorch/test-infra/pull/5086, the working directory is now set correctly, so `pushd` isn't needed anymore.  More importantly, trying to change the directory ends up failing all macOS CI jobs because that subdirectory doesn't exist.

Pull Request resolved: https://github.com/pytorch/executorch/pull/2980

Reviewed By: larryliu0820

Differential Revision: D55996299

Pulled By: huydhn

fbshipit-source-id: 05758603d7628cc0a01fd577a49202d45c84e6c5