Remove build files in source

They aren't used in subsequent files and just slow down CI since it has
to extract from the cache and then package the changes.