Reasoning models for edge devices logopeterindia.net
Knowledge Hub

Reasoning Models for Edge Devices

Small and distilled reasoning models that run on vehicles, robots, phones and embedded boards, with the runtimes, techniques and papers behind them. Grouped by role, not ranked.

50 topics•6 categories•Updated October 2026

What Reasoning Models for Edge Devices Are

A reasoning model works through intermediate steps before it answers or acts, and an edge device is the phone, robot, vehicle computer or embedded board where the model has to run without a round trip to a data center. Putting the two together is hard because reasoning adds computation at inference time, while edge hardware is limited in memory, power and latency. The usual answer is to train a large teacher model, distil its behaviour into a small student, compress the student with quantization, and run it on an optimised on-device runtime. Some small models also let you switch reasoning on or off, which trades answer quality against delay.

Spotlight: NVIDIA Alpamayo & In-Vehicle Reasoning

1 entry

NVIDIA's reasoning model for in-vehicle use. Specifications are NVIDIA's own; no independent benchmarks were found.

01Alpamayo 2 EdgePer NVIDIA, a 2-billion-parameter reasoning model built on NVIDIA Cosmos, optimized for the DRIVE AGX Thor platform and designed to run at 10 Hz for in-vehicle scene understanding and motion planning.

Small Reasoning Language Models

6 entries

Open-weight models small enough for laptops, phones and edge boards. Parameter counts are from each model's Hugging Face page.

02Phi-4-mini-reasoningMicrosoft's reasoning-tuned small model of about 3.8B parameters, released under the MIT licence. 03DeepSeek-R1-Distill-Qwen-1.5BSmall Qwen-based model distilled from DeepSeek-R1 reasoning outputs; MIT licence. 04Qwen3-0.6BSmallest Qwen3 model; the Qwen3 family offers thinking and non-thinking modes. Apache 2.0. 05SmolLM3-3BHugging Face's 3B model with a switchable reasoning mode. Apache 2.0. 06Llama-3.1-Nemotron-Nano-4BNVIDIA model of roughly 4.5B parameters aimed at reasoning tasks; published under NVIDIA's own licence. 07gpt-oss-20bOpenAI open-weight reasoning model of about 21B parameters under Apache 2.0; larger than the others here, so suited to higher-end edge hardware.

Embodied & Robot Reasoning Models

5 entries

Vision-language-action and robot foundation models that reason and act on the device.

08Gemini Robotics On-DeviceGoogle DeepMind's robotics model designed to run locally on robotic devices. 09NVIDIA Isaac GR00TNVIDIA's open foundation model for generalist humanoid robots. 10SmolVLACompact open vision-language-action model from Hugging Face's LeRobot team, built to run on modest hardware. 11openpiPhysical Intelligence's open-source models and code for robot policies. 12LeRobotHugging Face library for robotics models, datasets and tools. GitHub lists v0.6.1 as the latest release, from August 2026.

On-Device Runtimes & Toolchains

11 entries

Engines that run compressed models on CPUs, GPUs and NPUs.

13llama.cppC/C++ inference engine for GGUF models on CPUs and many GPUs. GitHub lists v0.8.0 as the latest release, from October 2026. 14ONNX RuntimeCross-platform inference engine for ONNX models, including on-device. 15ExecuTorchPyTorch runtime for mobile and edge devices. 16LiteRTGoogle's on-device runtime, the successor to TensorFlow Lite. 17MLC LLMCompiler-based engine for deploying LLMs on phones, browsers and GPUs. 18MLXApple's array framework for machine learning on Apple silicon. GitHub lists v0.32.4 as the latest release, from October 2026. 19OllamaTool for running open models locally. 20TensorRT-LLMNVIDIA library for optimized LLM inference on NVIDIA GPUs. GitHub lists v1.2.1 as the latest release, from April 2026. 21TensorRT Edge-LLMNVIDIA's C++ LLM and VLM inference software for physical AI. GitHub lists v0.11.1 as the latest release, from October 2026. 22Qualcomm AI HubQualcomm's hub for optimizing and deploying models on Snapdragon devices. 23BitNetMicrosoft's official inference framework for 1-bit LLMs.

Techniques & Key Papers

7 entries

How reasoning is taught to models and then compressed to fit the edge.

24Chain-of-Thought Prompting (arXiv)Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. 25Scaling Test-Time Compute (arXiv)Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. 26DeepSeek-R1 (arXiv)DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, which also reports distilling reasoning into smaller models. 27Distilling the Knowledge in a Neural Network (arXiv)The foundational paper on knowledge distillation. 28GPTQ (arXiv)Accurate post-training quantization for generative pre-trained transformers. 29AWQ (arXiv)Activation-aware weight quantization for LLM compression and acceleration. 30Speculative Decoding (arXiv)Fast Inference from Transformers via Speculative Decoding.

Related Pages on This Site

20 entries

Reasoning models, small models, edge AI, quantization, inference, robotics and chip pages that relate to on-device reasoning.

31Large Reasoning Models17 large reasoning models compared. 32Small Language Models (SLMs)Phi, Gemma, Qwen, SmolLM and other small models. 33Edge AIJetson, LiteRT, ONNX Runtime, OpenVINO and more. 34Physical AIRobot foundation models, world models and simulation. 35World Action ModelsModels that predict future states and actions for robots. 36Large Action ModelsModels that take actions on behalf of users. 37Vision Language ModelsOpen and proprietary vision-language models. 38AI Model Quantization TechniquesGPTQ, AWQ, SmoothQuant and related methods. 39AI Model Optimization TechniquesPruning, quantization and distillation. 40LLM Model Formats ExplainedGGUF, GPTQ, AWQ and EXL2 compared. 41LLM Training & Inference OptimizationKV cache, batching and other techniques. 42LLM Inference Platforms and Tools55 inference platforms compared. 43Platforms for Running LLMs LocallyOllama, LM Studio and other local tools. 44Open-Source LLMsDeepSeek, Qwen, Llama, Kimi and GLM. 45AI Chip ProvidersMakers of AI accelerators and chips. 46Edge / Fog ComputingEdge platforms, Kubernetes and edge AI. 47Real-Time Systems & RTOSDeterministic software for time-critical systems. 48Robotics TechnologiesRobotics resources, research and products. 49AI-Driven Robots and DronesEmbodied AI architecture for robots and drones. 50Reinforcement Learning GuideFrom MDPs to deep reinforcement learning.

Frequently Asked Questions

What is a reasoning model for edge devices?

A model that works through intermediate steps before answering or acting, small and optimised enough to run on a vehicle computer, robot, phone or embedded board instead of a data center. They are usually produced by distilling a larger teacher model, compressing the student with quantization, and serving it with an on-device runtime.

What is Alpamayo 2 Edge?

According to an NVIDIA public statement, a 2-billion-parameter reasoning model built on NVIDIA Cosmos, optimized for the NVIDIA DRIVE AGX Thor platform and designed to run at 10 Hz for in-vehicle scene understanding and motion planning. NVIDIA's Alpamayo page lists it as a 2B vision-language-action model for in-car reasoning. These are NVIDIA's own specifications; no independent benchmarks were found.

How is Alpamayo 2 Edge related to Alpamayo 2 Super?

NVIDIA lists Alpamayo 2 Super as a larger reasoning model (34B on its Alpamayo page) used for planning, auto-labelling and evaluation, and describes it as a teacher that can be distilled into compact models that run in the vehicle. Edge is the 2B in-car model.

How are the models on this page chosen and ordered?

They are grouped by role, such as vehicle reasoning models, small reasoning language models, robot models, runtimes and papers. They are not ranked, the list is not complete, and model sizes and licences come from each model's own page, so check them before use.