Importance Of Quantization In Large Language Models
The paper states that quantization is used to accelerate matrix multiplication — the main component of large language models (LLMs) — because matrix multiplication is often bottlenecked by the speed of loading matrice...
The paper states that quantization is used to accelerate matrix multiplication — the main component of large language models (LLMs) — because matrix multiplication is often bottlenecked by the speed of loading matrices from memory.[:cite[1]{ln=2}] It explains that for LLM inference many A⊤B products are computed and that quantization (bits/entry) must be reduced so compute units are fully utilized given the ops:bytes (FLOPs to memory bandwidth) ratio; this motivates targeting quantization rates of a few bits/entry to fully leverage modern matrix‑multiplication hardware.[:cite[3]{ln=1}][:cite[2]{ln=5}] The paper also notes that quantization can dramatically reduce power consumption since reading bytes from DRAM costs orders of magnitude more energy than small integer arithmetic, so memory limited matrix multiplication benefits both speed and energy efficiency from quantization.[:cite[4]{ln=1}][:cite[4]{ln=2}]