Enable optimized build for portable configuration of test-models (#5317)

Summary:
PR 5244 probably broke it, because it makes optimized ops a requirement to run llama without xnnpack.

Pull Request resolved: https://github.com/pytorch/executorch/pull/5317

Test Plan: `bash .ci/scripts/test_model.sh llama2 cmake portable` was broken and now succeeds

Reviewed By: larryliu0820

Differential Revision: D62591634

Pulled By: swolchok

fbshipit-source-id: 9170bc00107bea28f1ff1bcfd3b8735b959ae573