Small Language Models

Peterindia · Technology Resources

Link-Checked Directory · 2026

11 Small Language Models Worth Knowing

Small Language Models (SLMs) trade raw scale for speed, privacy, and cost — running on a phone, laptop, or single GPU instead of a cluster. This guide catalogues 11 SLMs spanning frontier-grade compact instruct models, research and reproducibility suites, and distilled ultra-lightweight specialists, plus the compression techniques and 2026 economics that put them at the center of agentic AI.

11Models Catalogued
0.5B–72BParameter Range
100%Links Verified

Why 2026 Is the Year of the SLM

Small language models — generally under ~10B parameters — became the default deployment class for on-device, edge, and cost-sensitive enterprise workloads in 2026. NVIDIA's 2025 research position paper, "Small Language Models are the Future of Agentic AI," argues that most agentic subtasks — extraction, formatting, routing, tool calls — don't need a frontier model at all: fine-tuned SLMs handle an estimated 80–90% of these subtasks with lower latency and more deterministic behavior, at roughly 10–30× lower serving cost and a fraction of the energy draw of a general-purpose LLM.

Analysts now expect the majority of AI inference calls to run on-device rather than in the cloud, driven by better NPUs, model compression, and runtimes like llama.cpp, ONNX Runtime, and Ollama that make a quantized SLM as easy to ship as a mobile app. The emerging pattern for production AI systems is heterogeneous: reserve a large frontier model for the "plan/decide" step, and route the repetitive language work to a small, fast, cheap model everywhere else.

Frontier-Grade Compact Instruct Models

4 Models

Full model families from major labs that ship purpose-built small tiers alongside their flagship sizes — strong general capability in a footprint that fits on a single GPU or a high-end device.

  1. 0.5B – 72B family

    Qwen2 Collection

    Alibaba's Qwen2 series spans five sizes — 0.5B, 1.5B, 7B, a 57B-A14B mixture-of-experts variant, and 72B — sharing one tokenizer and architecture. The 0.5B and 1.5B tiers are purpose-built for lightweight, resource-constrained deployment while inheriting the same training recipe as the flagship models.

    Visit →
  2. 7B · 32K context

    Mistral-7B-Instruct-v0.2

    An instruction-tuned fine-tune of Mistral 7B v0.2 with a 32K context window and Apache-2.0 licensing. A popular compact-but-capable chat and reasoning backbone that runs comfortably on a single consumer GPU, with over a million downloads a month.

    Visit →
  3. 8B / 70B · Ollama

    Llama 3 (via Ollama)

    Meta's Llama 3 in 8B and 70B sizes, packaged for one-command local deployment through Ollama. The 8B instruction-tuned variant is a common starting point for on-device and self-hosted chat applications — run it locally with a single ollama run llama3.

    Visit →
  4. On-device family

    Gemma

    Google DeepMind's open model family, built from the same research as Gemini and spanning ultra-small on-device variants up through larger sizes. Selective-activation variants such as Gemma 3n run with a memory footprint far smaller than their nominal parameter count.

    Visit →

Research & Reproducibility Suites

2 Suites

Model suites released primarily to make small-model science reproducible — identical training data and order across sizes, full checkpoint histories, and permissive licensing for interpretability and scaling-law research.

  1. 14M – 12B · 8 sizes

    Pythia (EleutherAI)

    A suite of 8 model sizes from 14M to 12B parameters, all trained on identical data in identical order with 154 saved checkpoints per model. Built specifically to study how small language models learn during training, rather than to top a benchmark leaderboard.

    Visit →
  2. Model Zoo · Wafer-scale

    Cerebras

    Cerebras' GitHub organization hosts its open Cerebras-GPT model family and ModelZoo training code, trained on its wafer-scale hardware and released across a range of small-to-mid parameter counts under permissive Apache-2.0 licensing for reproducible research.

    Visit →

Distilled & Ultra-Lightweight Specialists

5 Models

Models engineered to punch above their parameter count — through curated "textbook-quality" data, DPO-style distillation, or architectural downscaling — for phones, browsers, and other tightly resource-constrained targets.

  1. 2.7B · Microsoft

    Phi-2

    A 2.7B-parameter model trained on curated synthetic "textbook-quality" data and filtered web text rather than raw internet scale. At release it approached the performance of models several times its size on common-sense and reasoning benchmarks, without any RLHF fine-tuning.

    Visit →
  2. 3B · DPO-tuned

    StableLM Zephyr 3B

    Stability AI's 3B instruction-tuned model, trained with the Zephyr Direct Preference Optimization recipe. At release it outperformed several 7B-class chat models on MT-Bench and AlpacaEval win rate despite having under half their parameter count.

    Visit →
  3. 1.4B · Mobile-first

    MobileLLaMA 1.4B Base

    A LLaMA-architecture model downscaled to 1.4B parameters specifically for off-the-shelf mobile deployment, trained on 1.3T tokens from RedPajama. It also serves as the language backbone for the MobileVLM vision-language assistant project.

    Visit →
  4. 774M · MBZUAI

    LaMini-GPT 774M

    Part of MBZUAI's LaMini-LM series, which distills large-scale instructions into small, fast base architectures. This GPT-2-large-based 774M model is fine-tuned on 2.58 million instruction-response pairs for lightweight instruction following.

    Visit →
  5. Sub-3B · OpenBMB

    MiniCPM

    OpenBMB's "flagship performance in your pocket" series — sub-3B models engineered through careful data curation and training-dynamics research to match much larger models on benchmarks while running efficiently on phones and other edge devices.

    Visit →

Key Techniques Behind SLMs

How Small Gets Smart

The engineering that lets an SLM approach the quality of a model many times its size.

Knowledge Distillation
A small "student" model is trained to mimic the outputs (or logits) of a larger "teacher" model, transferring capability without the teacher's parameter count.
Quantization (INT8/INT4)
Weights are stored in lower-precision integers instead of 16/32-bit floats, shrinking memory footprint and speeding up inference with minimal accuracy loss.
Structured Pruning
Redundant weights, attention heads, or entire layers are removed from a larger model, then the pruned network is retrained to recover lost capability.
Curated "Textbook" Data
Training on smaller volumes of high-quality, filtered, or synthetic data (as with Phi-2) can outperform brute-force training on raw internet-scale corpora.
Selective / Sparse Activation
Mixture-of-experts and selective-parameter designs (Qwen2's 57B-A14B, Gemma 3n) activate only a fraction of total parameters per token, cutting real inference cost.
Edge Runtimes
GGUF-format models via llama.cpp, ONNX Runtime, and Ollama package quantized SLMs for one-command deployment on laptops, phones, and browsers.

SLM vs. LLM at a Glance

Choosing the Right Scale

Small and large models aren't competitors so much as complementary tools in the same system.

DimensionSmall Language Model (SLM)Large Language Model (LLM)
Typical Parameters0.5B – 14B70B – 1T+
Deployment TargetPhone, laptop, edge device, single GPUMulti-GPU cluster, hosted cloud API
LatencyMilliseconds; often works fully offlineSeconds; typically network-dependent
Cost per TokenFractions of a cent; free to self-hostMetered API pricing, higher per-token cost
Best Task FitNarrow, repetitive subtasks: extraction, routing, tool calls, classificationOpen-ended reasoning, novel problems, broad world knowledge
Fine-Tuning TimeHours on modest hardwareDays to weeks on large clusters
Data PrivacyFully on-device; no data leaves the machineData typically sent to a hosted API

© Peterindia — Small Language Models (SLMs) · Compiled and verified August 2026