SIGN IN SIGN UP

Skip quantizing output linear layer (#1644)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/1644

Skip quantizing output linear layer

Required memory for activation in bytes:  [0, x]

| Configuration.                             | memory size | exec log  | checkpoint name |
|--|--|--|--|
|FP32 activation & weights              |x = 5.5 GB  | P1042687346 | new_llama_checkpoint.pte [ ] |
|FP32 activation, qint8 weights:    |x = 3 GB      | P1042717078 | new_llama_checkpoint_q.pte [-Q] |
|FP16 activation, qint8 weights:      |x = 1.9 GB  | P1042645801 | new_llama_checkpoint_qh.pte [-Q -H] |
|FP16 activation & weights            |x = 1.5 GB  | P1042661553 | new_llama_checkpoint_h.pte [-H] |
|(old) FP32 activation, FP16 scales, qint8 weights |x = 3 GB | P1042678025  | new_llama_checkpoint_qhf.pte [-Q ] |

Reviewed By: hl475

Differential Revision: D52827084

fbshipit-source-id: 8c0b6aebdcbd1f48ca1c7ce704c9f77248e944a3
M
Michael Gschwind committed
1468f57005baa45f450173befd13b49ca88935c8
Parent: a4b54f7
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 1/19/2024, 9:15:47 PM