SIGN IN SIGN UP

Use cached PyTorch wheels on MacOS jobs (#9484)

One of the current drawback of using pinned PyTorch commit on CI is that
we need to build PyTorch wheel on all MacOS jobs because it doesn't have
Docker image. Building PyTorch wheel is usually not too bad because we
have sccache in place to make the compilation faster. However, it's
still slower than using a prebuilt wheel, and sccache is also not
available on GitHub MacOS runner `macos-latest-xlarge` (no access to
S3).

As all MacOS jobs are building exactly the same PyTorch wheel, the
proposal here is to cache the wheel on S3 `gha-artifacts` bucket which
is publicly readable, i.e.
https://gha-artifacts.s3.us-east-1.amazonaws.com/cached_artifacts/pytorch/executorch/pytorch_wheels/Darwin/311/torch-2.7.0a0%2Bgit295f2ed-cp311-cp311-macosx_14_0_arm64.whl.
The job can check for matching wheel from S3 and use it instead. If
there is no such wheel, it will continue building PyTorch normally. Once
a new wheel is built and if the runner has write access to S3, it will
upload the wheel so that other jobs can pick it up going forward.

### Testing

All CI jobs pass (failures are pre-existing from trunk). Here are some
quick number on how this helps reduce the durations of different MacOS
jobs.

* Apple workflow:
* build-benchmark-app:
[BEFORE](https://github.com/pytorch/executorch/actions/runs/14002229786/job/39210715922)
~80m →
[AFTER](https://github.com/pytorch/executorch/actions/runs/14001343158/job/39214390843)
~44m
* build-frameworks-ios:
[BEFORE](https://github.com/pytorch/executorch/actions/runs/14002229786/job/39210732212)
~80m →
[AFTER](https://github.com/pytorch/executorch/actions/runs/14001343158/job/39214394644)
~ 44m
* build-demo-ios:
[BEFORE](https://github.com/pytorch/executorch/actions/runs/14003433493/job/39213882743)
~ 55m →
[AFTER](https://github.com/pytorch/executorch/actions/runs/14001343158/job/39214390955)
~23m
* Apple perf workflow:
* build-benchmark-app:
[BEFORE](https://github.com/pytorch/executorch/actions/runs/13982706236/job/39208203350)
~80m →
[AFTER](https://github.com/pytorch/executorch/actions/runs/14001347585/job/39214401072)
~48m
* export model (llama):
[BEFORE](https://github.com/pytorch/executorch/actions/runs/13982706236/job/39150917351)
~30m →
[AFTER](https://github.com/pytorch/executorch/actions/runs/14001347585/job/39214401617)
~13m
* All MacOS jobs in pull and trunk:
* BEFORE ~417 on commit b195ed9a2564e5c2cd3554518213c4769d3ad022 → AFTER
~268m

Overall, I'm seeing the duration for all MacOS jobs reducing by close to
2x. This is very useful to reduce the cost running MacOS jobs (remember
the budget request to OSS team because of the $$$ GitHub MacOS runners)
H
Huy Do committed
5c5b84e94fd04bd9cd74a4d327d72f6f98eb04ff
Parent: 60280d9
Committed by GitHub <noreply@github.com> on 3/24/2025, 8:26:52 PM