Rollup merge of #118909 - Urgau:cleanup-improvement-invalid_ref_casting, r=est31

Some cleanup and improvement for invalid ref casting impl

This PR makes some cleanups and improvements to the `invalid_reference_casting` implementation in preparation for linting on new patterns, while reusing most of the logic.

r? `@est31` (feel free to re-assign)