AI Model Quantization Techniques (2026) | GPTQ, AWQ, SmoothQuant, QLoRA, GGUF & Tools

Foundations & Surveys

5 entries

Papers and guides that explain how quantization works.

01Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only InferenceThe paper that established integer-only inference and quantization-aware training for neural networks. 02A Survey of Quantization Methods for Efficient Neural Network InferenceSurvey of quantization methods, covering uniform versus non-uniform, per-channel scaling and PTQ versus QAT. 03A White Paper on Neural Network QuantizationQualcomm AI Research's practical guide to PTQ and QAT. 04Hugging Face: Quantization OverviewGuide to the quantization methods supported in Transformers and how to choose between them. 05LLM Model Formats Explained: GGUF, GPTQ, AWQ, EXL2Explains the difference between quantized model containers and quantization methods.

Post-training Quantization (PTQ) Methods

6 entries

Quantize a trained model with little or no retraining.

06LLM.int8()8-bit matrix multiplication for transformers that handles outlier features, behind bitsandbytes 8-bit. 07GPTQOne-shot weight quantization using approximate second-order information, typically to 3 or 4 bits. 08AWQ: Activation-aware Weight QuantizationProtects the small share of salient weights found from activation statistics to keep 4-bit accuracy. 09SmoothQuantMigrates quantization difficulty from activations to weights to enable W8A8 inference. 10AQLMAdditive quantization of language models for extreme compression at around 2 bits per weight. 11HQQ: Half-Quadratic QuantizationFast calibration-free weight quantization from Dropbox.

Quantization-aware Training & Low-bit Methods

5 entries

Train or fine-tune with quantization in the loop.

12QLoRAFine-tunes a 4-bit quantized model with low-rank adapters, introducing the NF4 data type. 13The Era of 1-bit LLMs: BitNet b1.58Ternary-weight language models trained from scratch with weights in {-1, 0, 1}. 14Microsoft BitNetInference framework for 1-bit LLMs on CPUs. 158-bit Optimizers via Block-wise QuantizationBlock-wise quantization of optimizer states to reduce training memory. 16TensorFlow Quantization-aware Training GuideHow to apply quantization-aware training with the TensorFlow Model Optimization Toolkit.

Libraries & Toolkits

13 entries

Open-source implementations you can use today.

17bitsandbytes8-bit and 4-bit quantization, NF4 and 8-bit optimizers for PyTorch. 18GPTQModelActively maintained toolkit for GPTQ and related quantization methods. 19AutoGPTQ (archived)Earlier GPTQ library; the repository is archived. 20AutoAWQ (archived)AWQ implementation for Hugging Face models; the repository is archived. 21MIT HAN Lab llm-awqReference implementation of AWQ. 22ExLlamaV3Inference library with EXL3 variable bits-per-weight quantization; ExLlamaV2 is its predecessor. 23llm-compressorCompression library for producing quantized models served by vLLM. 24NVIDIA Model OptimizerNVIDIA's toolkit for quantization, pruning and distillation, formerly TensorRT Model Optimizer. 25PyTorch ao (torchao)PyTorch-native quantization, sparsity and low-precision training. 26Hugging Face OptimumHardware-optimization toolkit for Transformers, including quantization backends. 27optimum-quantoPyTorch quantization backend for Optimum. 28Intel Neural CompressorQuantization, pruning and distillation across PyTorch, TensorFlow and ONNX. 29UnslothMemory-efficient fine-tuning and quantization of open models.

Formats & Runtimes

8 entries

Where quantized models are stored and run.

30GGUF format specificationThe single-file format used by llama.cpp for quantized models. 31llama.cppC/C++ inference engine with many quantization types for CPUs and GPUs. 32vLLM QuantizationQuantization methods supported when serving with vLLM. 33TensorRT-LLMNVIDIA's library for optimised LLM inference with FP8, INT4 and INT8. 34ONNX Runtime QuantizationStatic and dynamic INT8 quantization for ONNX models. 35LiteRTGoogle's runtime for on-device inference, formerly TensorFlow Lite. 36OpenVINOIntel toolkit for optimising and deploying models on Intel hardware. 37MLXApple's array framework with quantized inference on Apple silicon.

7 entries

Related directories on this site.

38LLM Training and Inferencing Optimization TechniquesTraining and inference optimisation methods. 39Small Language ModelsCompact models for on-device use. 40Open-Source LLMsOpen-weight models and local tools. 41LLM Inference PlatformsPlatforms for serving LLMs. 42Production AI Inference StackvLLM, llm-d and GPU load balancing. 43Edge AIAI at the edge. 44LLMOpsOperating LLM applications.

Frequently Asked Questions

What is model quantization?

Quantization represents a model's weights, and sometimes activations, with fewer bits, such as 8-bit or 4-bit integers instead of 16- or 32-bit floats, which reduces memory use and often speeds up inference.

What is the difference between PTQ and QAT?

Post-training quantization converts an already trained model, using little or no data. Quantization-aware training simulates low precision during training or fine-tuning, which usually preserves more accuracy at very low bit widths.

Which method should I use for an LLM?

For GPU serving, AWQ or GPTQ at 4 bits and FP8 or INT8 on supporting hardware are common; for CPU and edge, GGUF with llama.cpp; for fine-tuning on limited memory, QLoRA with bitsandbytes.

Does quantization hurt accuracy?

Usually slightly. 8-bit is often near lossless, 4-bit costs a small accuracy drop that good methods minimise, and below 3 bits quality drops more, so evaluate on your own tasks.

Is GGUF a quantization method?

No. GGUF is a file format that stores quantized weights; GPTQ and AWQ are quantization methods. The LLM Model Formats Explained page covers the difference.

AI Model Quantization Techniques: PTQ, QAT, GPTQ, AWQ, GGUF & Low-bit Methods