Weight-Only Quantization

Source: machinelearningmastery.com

Read original source

At a glance

During autoregressive decoding, a transformer reads its weights for every new token. At small batch sizes, moving these weights can take more time than computation. Quantization reduces this cost by storing weights with fewer bits. For example, an INT8 weight…