Bug fix partitioner (#5239)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/5239

Forgot to copy over this part of the constraint when I refactored the partitioner

Reviewed By: balakv504, kirklandsign

Differential Revision: D62471496

fbshipit-source-id: d2fb76ffd2dea5671b38f89e050beac3a35aff4e