Fixing minor issues in llama2 7b repro (#2926)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/2926

Fixing issues we've seen in https://github.com/pytorch/executorch/issues/2907 and https://github.com/pytorch/executorch/issues/2805

bypass-github-export-checks
bypass-github-pytorch-ci-checks
bypass-github-executorch-ci-checks

Reviewed By: iseeyuan, cccclai

Differential Revision: D55893925

fbshipit-source-id: c6e0264d868cb487faf02f95ff1bd223cbcc97ac