Remove constraint on having all the input arg dtypes to be (#1205)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/1205

This seems like not the right constraint
ghstack-source-id: 207644973
exported-using-ghexport

Reviewed By: larryliu0820

Differential Revision: D51293090

fbshipit-source-id: e3eb526756e65fa6699fcef88fe8ec08afd1ca81