ci: cache e2e prod preparation to run once instead of per-shard, various improvements and flake fixes (#15368)

Moves `prepare-run-test-against-prod:ci` into a dedicated `e2e-prep` job
that runs once and caches the result. Previously this 2-minute step ran
for every E2E shard.

Saves ~2 minutes per shard (~140+ minutes total across all E2E jobs).

Additionally, this entails a few other test suite improvements and
flakes fixes.

All together, **this reduces the amount of time our entire CI takes to
run from > 1 hour if you factor in manual retries, to about 15
minutes.**

- about 2x faster `waitForCompilation` checking, by checking more
frequently and without waiting for the load event. This previously took
about 2 seconds in CI, often running before every single test. After
`waitForCompilation` we usually navigate to a different page, so waiting
for a full page load was a waste of time.
- Reduce test timeouts. Quicker failures => quicker retries => quicker
test suite passes. The previous config was excessive
- Introduces a shorter navigation timeout. By default, it used to be
equal to `TEST_TIMEOUT`. `TEST_TIMEOUT` is high to accommodate long e2e
tests with multiple steps, not to allow page.goto to try for 3 minutes
before it fails.
- Reduces console spam due to `waitForCompilation`
- Reduce retries of `waitForCompilation`. If the dev server fails for
some reason, we shouldn't have to wait 50 (!!) minutes until the e2e
test finally fails and is ready to be retried
- Lots of flake fixes for individual tests and test suites
- Fixes flaky behavior of checkbox component. This **fixes flakiness
EVERYWHERE any checkbox is clicked**.
- Fixes upload directory clearing and seed restoration not working
correctly if there were multiple upload dirs