CI: Fix cache key and include base branch

The base branch is hardcoded as an env variable as I couldn't find a simple
way to just get either `3.2` or `master`. But it's easy to change when we
branch off from `master` to a new stable branch, which doesn't happen often.

(There's `{{github.base_ref}}` but it's probably more verbose like
`ref/heads/master`, and only valid for PRs.)