[llama-mm] Fix vision encoder model test (#6842)

* [llama-mm] Fix vision encoder model test

Summary: As titled. We need a smaller model config to make it work on CI
jobs.

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix pull.yml and trunk.yml

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Remove torchao

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Small fixes

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Add torchao back

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags: