Few fixes to export to enable exporting sam (#1204)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/1204

Spec violation happens when exporting SAM model where mixed dtype on aten op
triggers spec violation.

In the process I realized that turning off ir_validity does not fully work.
So in this diff, I first made changes that respect check_ir_validity flag and
added tests for the same.

Next PR will address mixed dtype
ghstack-source-id: 207644974
exported-using-ghexport

Reviewed By: JacobSzwejbka

Differential Revision: D51293091

fbshipit-source-id: 070dca358a0c01c9c0317bb8348646106b233c8f