LLM Weight Quantization (AWQ, GPTQ, GGUF)

LLM weight quantization is what makes it possible to run a model trained in 16-bit precision on hardware with a fraction of the memory that training used, by storing each weight in fewer bits after training is done. A weight originally stored as a 16-bit floating point number (2 bytes) can be quantized down to 8, 4, or even 3 bits, and since a model’s weights dominate its memory footprint, that directly translates into smaller downloads, lower VRAM requirements, and, because reading weights off memory is what actually bottlenecks decoding on many setups, faster inference. AWQ, GPTQ, and GGUF are the three names practitioners run into constantly, but they answer different questions: GPTQ and AWQ are quantization algorithms that decide which numbers to keep, while GGUF is primarily a file format (paired with llama.cpp’s own quantization schemes) built for CPU and consumer-GPU inference rather than a competing algorithm in the same sense.

Why 16 Bits Is More Than Most Weights Need

A model’s weights are the product of a training process that never specifically asked for 16-bit precision, that precision is simply what gradient descent happened to run in. Empirically, most trained weights can be represented with far less precision without meaningfully changing the model’s output distribution, because the important signal in a weight matrix is concentrated in a relatively small number of large-magnitude values, and small-magnitude noise is exactly what the network learned to be robust to during training. Quantization exploits this: round most weights down to a coarse grid of representable values, spend a little extra care on the values that matter most, and the resulting error, while nonzero, stays small enough that downstream task performance barely moves.

The three widely used approaches differ mainly in how they decide what “care about the values that matter most” means in practice.

GPTQ: Second-Order Error Correction, One Layer at a Time

GPTQ (2022) was the first method to push post-training quantization down to 4-bit while keeping accuracy close to the unquantized baseline, and it did so by treating quantization as an optimization problem rather than simple rounding. For each layer, GPTQ quantizes weights one column at a time, and after rounding a column, it updates the remaining not-yet-quantized columns to compensate for the error just introduced, using the layer’s Hessian (a second-order measure of how sensitive the output is to each weight) computed from a small calibration dataset. This error-compensation step is what separates GPTQ from naive round-to-nearest quantization: instead of letting rounding error accumulate independently across every weight, each column’s rounding error gets partially absorbed by the columns quantized after it.

# scenario: quantizing a downloaded fp16 checkpoint to 4-bit GPTQ
# before serving it on a GPU with limited VRAM
from transformers import AutoModelForCausalLM, GPTQConfig

quant_config = GPTQConfig(
    bits=4,
    dataset="c4",              # calibration data used to compute the Hessian
    group_size=128,            # weights quantized in groups of 128, sharing one scale
    desc_act=True,             # quantize columns in order of decreasing activation impact
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=quant_config,
    device_map="auto",
)
model.save_pretrained("./llama-2-7b-gptq-4bit")

GPTQ’s group_size is itself a tunable trade-off: a smaller group size (e.g. 32) means each small block of weights gets its own scale factor, improving accuracy at the cost of extra metadata to store; a larger group size (e.g. 128) shares one scale across more weights, saving space but averaging over more variation. A 2026 theoretical analysis showed GPTQ’s column-by-column error correction, run back-to-front, is mathematically equivalent to Babai’s nearest-plane algorithm for finding the closest point on a lattice, formalizing why the method works rather than just observing that it does empirically.

AWQ: Protect the Weights the Activations Actually Use

AWQ (Activation-aware Weight Quantization, 2023) starts from a different observation: not all weights matter equally, and the ones that matter most can be identified cheaply, before quantizing anything, by looking at which weights consistently multiply against large-magnitude activations. Roughly the top 1% of weight channels by this criterion are identified as salient, and rather than storing those specific weights at higher precision (which would require mixed-precision storage, an implementation headache), AWQ instead rescales the salient channels up and the corresponding activations down by a per-channel factor before quantizing everything uniformly. The rescaling shrinks the relative quantization error on the channels that matter, without needing any weights to actually be stored at a different bit-width than the rest.

# scenario: quantizing the same checkpoint with AWQ instead, favoring
# faster quantization (no backprop-style error correction) and strong
# throughput on GPU serving engines like vLLM
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

model.quantize(
    tokenizer,
    quant_config={"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"},
    # calib_data defaults to a small pile-derived sample used only to
    # measure activation magnitudes, not to run error correction
)
model.save_quantized("./llama-2-7b-awq-4bit")

Because AWQ doesn’t need to run a per-layer optimization loop the way GPTQ does, it’s typically faster to quantize a model with, and because its output is a uniformly-quantized tensor rather than one with mixed group-wise correction terms, it tends to be the fastest option to actually serve on GPU inference engines with optimized kernels (notably vLLM’s Marlin kernel), at accuracy that’s competitive with, and on some instruction-tuned model benchmarks slightly ahead of, GPTQ.

GGUF: A File Format Built for CPUs and Consumer GPUs

GGUF (GPT-Generated Unified Format) is the file format used by llama.cpp and its ecosystem (Ollama, LM Studio, koboldcpp), and it matters for a different reason than GPTQ or AWQ: it’s optimized for running quantized models on CPUs and mixed CPU/GPU setups, not just data-center GPUs. The quantization schemes bundled with GGUF, known as “K-quants,” use block-based rounding: weights are grouped into small blocks (typically 32 weights per block, 8 blocks per superblock), each block gets its own scale and minimum value, and formats like Q4_K_M selectively keep certain sensitive tensors (attention outputs, feed-forward down-projections) at higher precision than the rest of the model, at a small size cost, which is what the _M (“medium”) suffix signals versus a uniform _S (“small”) variant.

# scenario: converting a downloaded fp16 checkpoint to a 4-bit GGUF
# file that runs well split across a laptop's CPU and integrated GPU
python convert_hf_to_gguf.py ./llama-2-7b-hf --outfile llama-2-7b-f16.gguf
./llama-quantize llama-2-7b-f16.gguf llama-2-7b-Q4_K_M.gguf Q4_K_M
# Q4_K_M: ~4.5 bits/weight average, keeps select tensors at 6-8 bits

Unlike GPTQ and AWQ, which typically ship model-specific config metadata alongside standard safetensors weights, a GGUF file is fully self-contained: architecture, tokenizer, and quantized weights all live in one file, which is a large part of why it became the format of choice for the local-inference tooling ecosystem rather than the datacenter-serving one.

Comparing the Three

None of these three is a strict winner across every benchmark. GPTQ and GGUF’s Q4_K_M land within a fraction of a perplexity point of each other on many models; AWQ tends to hold up better than both on instruction-tuned, chat-style models specifically, since its activation-driven salience criterion better tracks what those models actually use at inference time. The practical decision usually comes down to where the model needs to run, not which method is “more accurate” in the abstract.

Try It: Bit-Width vs. Model Size and Estimated Quality Loss

The calculator below applies to any of the three methods, since bits-per-weight is the dominant factor in both memory footprint and accuracy impact, with the specific algorithm mostly affecting how much of that impact is recovered at a given bit-width, not whether the trade-off exists. Model size is exact (params × bits / 8); the quality-loss curve is illustrative of the well-documented pattern that accuracy degrades slowly down to about 4 bits and then drops off sharply below it, not a benchmark result for a specific model.

Drag bits-per-weight down from 16 and watch the checkpoint shrink linearly while the illustrative quality-loss curve stays flat until around 4 bits, then bends sharply, the empirical reason 4-bit is the default sweet spot across AWQ, GPTQ, and GGUF alike.

What’s New (2025-2026)

• Importance-matrix GGUF quantization. Newer GGUF quantization runs (imatrix) compute an activation-based importance weighting before assigning bits, closing much of the gap to GPTQ/AWQ’s calibration-aware accuracy while keeping GGUF’s CPU-friendly block format.
• Sub-4-bit methods maturing. Formats and methods targeting 2- and 3-bit weights (GGUF’s IQ2_XS/IQ3_XXS variants, and research methods building on GPTQ’s error-correction idea) have narrowed the accuracy gap at extreme compression, though a 2026 systematic evaluation on Llama-3.1-8B found the “best” quant level still depends heavily on which specific benchmark and use case is being optimized for, not a single universal ranking.
• Formal understanding of why GPTQ works. The 2026 result recasting GPTQ as Babai’s nearest-plane lattice algorithm gives the method a rigorous optimality characterization it previously lacked, and is already informing new solver variants that exploit the same lattice structure more directly.
• Quantization-aware fine-tuning combined with LoRA. Rather than quantizing a fully fine-tuned model after the fact, pipelines that quantize first and then apply LoRA adapters on top of the frozen quantized weights (the QLoRA pattern) have become a standard way to fine-tune large models on a single consumer GPU, treating quantization and adaptation as one combined workflow rather than sequential, independent steps.
• Quantization spreading beyond weights to KV caches and activations, a related but distinct target: KV cache optimization techniques and dedicated methods like TurboQuant apply the same bits-per-value logic to the attention cache rather than the static weight matrices covered here.

from transformers import AutoModelForCausalLM, AutoTokenizer

# scenario: choosing which pre-quantized checkpoint of a 70B model
# fits a single 24GB consumer GPU for local inference
model_id = "TheBloke/Llama-2-70B-GPTQ"      # 4-bit GPTQ, ~35GB -> won't fit
model_id = "TheBloke/Llama-2-70B-AWQ"       # 4-bit AWQ, ~37GB -> won't fit
model_id = "TheBloke/Llama-2-70B-GGUF"      # pick a K-quant that fits instead

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    # For GGUF specifically, the quant level (Q4_K_M, Q5_K_S, etc.)
    # is baked into the downloaded file, not chosen at load time.
)