Use cached PyTorch wheels on MacOS jobs (#9484)
One of the current drawback of using pinned PyTorch commit on CI is that we need to build PyTorch wheel on all MacOS jobs because it doesn't have Docker image. Building PyTorch wheel is usually not too bad because we have sccache in place to make the compilation faster. However, it's still slower than using a prebuilt wheel, and sccache is also not available on GitHub MacOS runner `macos-latest-xlarge` (no access to S3). As all MacOS jobs are building exactly the same PyTorch wheel, the proposal here is to cache the wheel on S3 `gha-artifacts` bucket which is publicly readable, i.e. https://gha-artifacts.s3.us-east-1.amazonaws.com/cached_artifacts/pytorch/executorch/pytorch_wheels/Darwin/311/torch-2.7.0a0%2Bgit295f2ed-cp311-cp311-macosx_14_0_arm64.whl. The job can check for matching wheel from S3 and use it instead. If there is no such wheel, it will continue building PyTorch normally. Once a new wheel is built and if the runner has write access to S3, it will upload the wheel so that other jobs can pick it up going forward. ### Testing All CI jobs pass (failures are pre-existing from trunk). Here are some quick number on how this helps reduce the durations of different MacOS jobs. * Apple workflow: * build-benchmark-app: [BEFORE](https://github.com/pytorch/executorch/actions/runs/14002229786/job/39210715922) ~80m → [AFTER](https://github.com/pytorch/executorch/actions/runs/14001343158/job/39214390843) ~44m * build-frameworks-ios: [BEFORE](https://github.com/pytorch/executorch/actions/runs/14002229786/job/39210732212) ~80m → [AFTER](https://github.com/pytorch/executorch/actions/runs/14001343158/job/39214394644) ~ 44m * build-demo-ios: [BEFORE](https://github.com/pytorch/executorch/actions/runs/14003433493/job/39213882743) ~ 55m → [AFTER](https://github.com/pytorch/executorch/actions/runs/14001343158/job/39214390955) ~23m * Apple perf workflow: * build-benchmark-app: [BEFORE](https://github.com/pytorch/executorch/actions/runs/13982706236/job/39208203350) ~80m → [AFTER](https://github.com/pytorch/executorch/actions/runs/14001347585/job/39214401072) ~48m * export model (llama): [BEFORE](https://github.com/pytorch/executorch/actions/runs/13982706236/job/39150917351) ~30m → [AFTER](https://github.com/pytorch/executorch/actions/runs/14001347585/job/39214401617) ~13m * All MacOS jobs in pull and trunk: * BEFORE ~417 on commit b195ed9a2564e5c2cd3554518213c4769d3ad022 → AFTER ~268m Overall, I'm seeing the duration for all MacOS jobs reducing by close to 2x. This is very useful to reduce the cost running MacOS jobs (remember the budget request to OSS team because of the $$$ GitHub MacOS runners)
H
Huy Do committed
5c5b84e94fd04bd9cd74a4d327d72f6f98eb04ff
Parent: 60280d9
Committed by GitHub <noreply@github.com>
on 3/24/2025, 8:26:52 PM