Techniques for AI Model Optimization logopeterindia.net
Knowledge Hub

Techniques for AI Model Optimization

How to make models smaller and faster — pruning, quantisation, knowledge distillation, low-rank and parameter-efficient methods, efficient architectures and inference-time techniques, with surveys, papers and libraries.

83 topics•10 categories•Updated October 2026

Choosing an Optimisation Technique

6 entries

Model optimisation makes a trained network cheaper to store and faster or less costly to run, ideally without losing much accuracy. The main families are pruning (removing weights, neurons or whole structures), quantisation (storing weights and activations in fewer bits), knowledge distillation (training a small student to imitate a larger teacher), low-rank factorisation and parameter-efficient fine-tuning (such as LoRA), efficient architectures and neural architecture search, sparse and mixture-of-experts designs, and inference-time techniques such as efficient attention, KV-cache management and speculative decoding. They are often combined, and the right mix depends on whether the bottleneck is memory, compute or latency and on the target hardware. Always re-measure accuracy and real speed on the target device, because a smaller model is not always a faster one. This page collects surveys, papers, courses and libraries for each family.

01AI Model Quantization TechniquesTechniques for reducing model precision, with LLM-focused notes. 02LLM Training and Inference Optimization TechniquesParallelism, KV cache, batching and cost cutting. 03LLM Model Formats ExplainedGGUF, GPTQ, AWQ and EXL2 compared. 04Deployment Strategies for Deep Learning ModelsServing, rollout, edge and LLM deployment. 05Linear Algebra for Deep LearningWeblog covering SVD and LoRA. 06Edge AIRunning models on devices at the edge.

Surveys, Courses & Overviews

8 entries

Where to start before choosing techniques.

07A Survey of Model Compression and Acceleration for Deep Neural NetworksOverview of pruning, quantisation, low-rank methods and distillation. 08Efficient Processing of Deep Neural NetworksSze et al. tutorial and survey covering hardware and algorithm co-design. 09Efficient Transformers: A SurveyTaxonomy of efficient Transformer variants. 10A Survey on Efficient Inference for Large Language ModelsCovers data, model and system-level inference optimisation. 11Efficient Large Language Models: A SurveyModel-, data- and framework-centric efficiency techniques. 12MIT 6.5940 TinyML and Efficient Deep Learning ComputingSong Han's course on pruning, quantisation, distillation and efficient inference. 13Large Transformer Model Inference Optimization (Lilian Weng)Readable overview of quantisation, pruning, sparsity, distillation and architectural tricks. 14Green AIArgues for reporting efficiency alongside accuracy.

Pruning & Sparsity

10 entries

Removing weights, neurons or structures.

15Learning both Weights and Connections for Efficient Neural NetworksHan et al. on magnitude pruning and retraining. 16Pruning Filters for Efficient ConvNetsStructured pruning of convolutional filters. 17The Lottery Ticket HypothesisFinds small trainable subnetworks inside large networks. 18What is the State of Neural Network Pruning?Meta-analysis of pruning papers and their comparability. 19Accelerating Sparse Deep Neural NetworksNVIDIA's 2:4 structured sparsity for Ampere GPUs. 20SparseGPTOne-shot pruning of large language models. 21Wanda: A Simple and Effective Pruning Approach for LLMsPruning by weight magnitude times input activation. 22Sheared LLaMAStructured pruning followed by continued pretraining. 23PyTorch pruning tutorialUsing torch.nn.utils.prune to prune models. 24Pruning (artificial neural network)Background on pruning methods.

Quantisation

10 entries

Representing weights and activations with fewer bits.

25Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only InferenceJacob et al. on 8-bit quantisation-aware training. 26A Survey of Quantization Methods for Efficient Neural Network InferenceComprehensive survey of quantisation approaches. 27LLM.int8()8-bit matrix multiplication for Transformers with outlier handling. 28GPTQAccurate post-training quantisation for generative Transformers. 29SmoothQuantMigrates quantisation difficulty from activations to weights. 30AWQ: Activation-aware Weight QuantizationProtects salient weights when quantising LLMs. 31QLoRAFine-tuning on 4-bit quantised models with LoRA adapters. 32The Era of 1-bit LLMs (BitNet b1.58)Ternary-weight language models. 33FP8 Formats for Deep Learning8-bit floating-point formats for training and inference. 34bitsandbytesLibrary for 8-bit and 4-bit quantisation in PyTorch.

Knowledge Distillation

8 entries

Training small students from larger teachers.

35Distilling the Knowledge in a Neural NetworkHinton et al. introduce soft-target distillation. 36FitNetsDistilling intermediate representations. 37Knowledge Distillation: A SurveySurvey of distillation methods and applications. 38DistilBERTA smaller, faster BERT trained by distillation. 39TinyBERTTransformer distillation across layers. 40Distilling Step-by-StepUsing rationales from large models to train smaller ones. 41MiniLLMDistillation of large language models using reverse KL. 42LLM Pruning and Distillation in Practice: The Minitron ApproachCombining pruning and distillation to compress LLMs.

Low-Rank Methods & Parameter-Efficient Fine-Tuning

5 entries

Adapting models by training few parameters.

43LoRA: Low-Rank Adaptation of Large Language ModelsTrains low-rank update matrices while freezing the base model. 44DoRAWeight-decomposed low-rank adaptation. 45Parameter-Efficient Transfer Learning for NLP (Adapters)Small adapter modules inserted into Transformers. 46Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-TuningWhy few parameters can suffice for fine-tuning. 47Hugging Face PEFTLibrary implementing LoRA and other parameter-efficient methods. GitHub lists v0.21.2 as the latest release, from September 2026.

Efficient Architectures, NAS & Sparse Models

8 entries

Designing smaller or sparser networks from the start.

48MobileNetV2Inverted residuals and linear bottlenecks for mobile vision. 49EfficientNetCompound scaling of depth, width and resolution. 50Neural Architecture Search: A SurveyOverview of search spaces and strategies. 51DARTSDifferentiable architecture search. 52Once-for-AllTrain one supernet and specialise it for many devices. 53Switch TransformersSparse mixture-of-experts scaling. 54Mixtral of ExpertsOpen sparse mixture-of-experts language model. 55MambaState-space sequence model with linear-time scaling.

Efficient Attention & Inference-Time Techniques

8 entries

Speeding up generation without changing the model's weights.

56FlashAttentionIO-aware exact attention that reduces memory traffic. 57FlashAttention-2Better parallelism and work partitioning. 58Fast Transformer Decoding: One Write-Head is All You Need (Multi-Query Attention)Sharing keys and values across heads to shrink the KV cache. 59GQA: Grouped-Query AttentionInterpolates between multi-head and multi-query attention. 60Efficient Memory Management for LLM Serving with PagedAttentionThe vLLM paper on KV-cache memory management. 61Fast Inference from Transformers via Speculative DecodingUsing a draft model to accelerate sampling. 62MedusaMultiple decoding heads for faster generation. 63H2O: Heavy-Hitter OracleKV-cache eviction keeping the most important tokens.

Libraries, Compilers & Toolkits

11 entries

Software that applies these techniques.

64PyTorch torchaoPyTorch library for quantisation and sparsity. 65Hugging Face OptimumHardware-specific optimisation for Hugging Face models. 66LLM CompressorCompression recipes for deployment with vLLM. GitHub lists v0.14.0.1 as the latest release, from October 2026. 67NVIDIA TensorRT-LLMOptimised LLM inference on NVIDIA GPUs. 68Intel Neural CompressorQuantisation, pruning and distillation tools across frameworks. 69OpenVINO NNCFNeural Network Compression Framework for OpenVINO. GitHub lists v3.4.0 as the latest release, from August 2026. 70Microsoft OliveModel optimisation toolkit targeting ONNX Runtime. 71ONNX RuntimeCross-platform inference engine with quantisation tools. 72Apache TVMCompiler stack for deploying optimised models to diverse hardware. 73MLXArray framework for efficient ML on Apple silicon. GitHub lists v0.32.4 as the latest release, from September 2026. 74MLPerf Inference BenchmarkStandard benchmark for measuring inference speed.

Related Pages

9 entries

Neighbouring pages on this site.

75Deep LearningFrameworks, algorithms and datasets. 76Evaluation Metrics for Deep LearningMetrics, benchmarks and libraries. 77Inference EnginesEngines for running LLM inference. 78LLM Inference PlatformsAPIs, GPU clouds, local apps and engines. 79ML Model Serving ToolsServing trained models behind an API. 80How vLLM WorksPagedAttention and continuous batching. 81Top Deep Learning AlgorithmsCore deep learning algorithms. 82Mathematics for Deep LearningHub for the mathematics used in deep learning. 83State Space Models (SSMs) like MambaMamba, S4, hybrids such as Jamba, RWKV and other linear-time sequence models.

Frequently Asked Questions

Why does deep learning need statistics?

Training objectives such as cross-entropy and squared error are maximum likelihood estimates under assumed noise models, and questions about overfitting, evaluation, uncertainty and data shift are statistical questions. Without them it is hard to tell a real improvement from noise.

What statistics topics should I learn first for deep learning?

Start with random variables and distributions, expectation and variance, maximum likelihood and Bayes' theorem, then the bias-variance tradeoff, cross-validation and the bootstrap. After that, calibration, hypothesis testing for model comparison and distribution shift matter most in practice.

How do I know whether one model is really better than another?

Evaluate on held-out data, report uncertainty (for example confidence intervals from the bootstrap or variation across random seeds), and use an appropriate test when comparing models. The evaluation section lists papers on comparing classifiers and on error bars for language-model evaluations.

What is calibration and why does it matter?

A model is calibrated when its stated probabilities match observed frequencies. Guo et al. found that modern neural networks are often overconfident, and the uncertainty section lists methods such as temperature scaling, deep ensembles and conformal prediction.

Does the bias-variance tradeoff still apply to large neural networks?

The classical U-shaped curve describes many models, but work on double descent shows that test error can fall again as models grow past the point of fitting the training data. The generalisation section lists the main papers.

How is this page different from Probability Theory for AI?

That page covers probability, Bayesian inference and generative models. This page focuses on statistical practice for deep learning: estimation and regularisation, generalisation, evaluation and significance, uncertainty, distribution shift and experiment design.