AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Quantized LLMs: A Detailed Examination of Their Performance

Despite the impressive performance of large language models (LLMs), deploying these models in resource-constrained environments remains a challenge due to their immense size. Low-bit quantization is a promising technique for compressing LLMs, reducing memory and computational demands during inference. However, prior research has primarily focused on limited metrics and outdated datasets, leaving many questions unanswered about the efficacy of quantization on the latest, instruction-tuned LLMs. This paper delves into a comprehensive evaluation of quantized instruction-tuned LLMs, ranging from 7B to 405B parameters, using 13 benchmarks across six task types.

The researchers discovered that quantizing a larger LLM to a similar size as a smaller FP16 LLM generally outperforms across most benchmarks, highlighting the value of model scaling. However, performance varied significantly based on the chosen quantization method, model size, and bit-width. For instance, weight-only methods like GPTQ and AWQ often yielded better results in larger models, while methods quantizing both weights and activations sometimes resulted in significant accuracy drops. The study also found that task difficulty did not significantly impact accuracy degradation due to quantization. This indicates that quantization impacts performance fairly consistently across various task complexities.

The researchers compared the performance of quantized LLMs across 13 benchmarks, encompassing a wide range of tasks, including commonsense question answering, knowledge and language understanding, instruction following, hallucination detection, mathematics, and dialogue. Interestingly, they found that the MT-Bench evaluation method, while commonly used, has limited discriminatory power among recent high-performing LLMs.

The study’s findings emphasize the importance of selecting the right quantization method and bit-width for specific tasks and models. Weight-only methods like GPTQ and AWQ were found to be more effective for larger models, while FP8, supported by appropriate hardware, showed promise for both weight and activation quantization. This comprehensive evaluation provides crucial insights for researchers and developers working with quantized LLMs, paving the way for more efficient and effective deployment of these models in resource-constrained environments.