What Are Edge-Native Inference Engines?
12 entriesAn edge-native inference engine runs trained models on the hardware where users and data already are, such as laptops, desktops, phones and embedded devices, and is designed around that hardware's limits rather than assuming datacenter GPUs. For large mixture-of-experts (MoE) language models, that means managing a small GPU memory, a larger host RAM pool and the PCIe link between them. A 31 October 2026 LinkedIn post by Sumanth describes UC Berkeley's open-source FreeToken as one such engine; this page checks its claims against the project's README and paper, then lists related engines. Descriptions are paraphrased, are not rankings, and release facts come from GitHub on 1 November 2026.
FreeToken: An Edge-Native MoE Engine
4 entriesSources for the project described in the post.
Other Edge and Local Inference Engines
16 entriesEngines for running models on personal and embedded hardware.
Frequently Asked Questions
What is an edge-native inference engine?
An engine designed to run models on personal or embedded hardware, working within its limited and uneven memory, compute and interconnect resources, instead of assuming a datacenter GPU. FreeToken uses the term for an engine that treats a machine's GPU, CPU, host RAM and PCIe link as one platform.
How can a large MoE model run on a consumer GPU?
Mixture-of-experts models activate only some experts per token, so an engine can keep always-needed weights on the GPU, hold the larger expert pool in host RAM, and fetch or compute experts as needed. FreeToken's README describes bandwidth-adaptive CPU-GPU co-execution and expert caching for this; check the paper for its measured results.
Do the reported speeds and model sizes apply to my machine?
Not necessarily. The paper reports results on specific hardware, from an 8GB laptop GPU to a single workstation GPU, and its numbers are the authors' own. Performance depends on GPU, PCIe bandwidth, RAM and the model, so test on your hardware. The README lists native support for NVIDIA RTX 30, 40 and 50 series GPUs.
How does FreeToken differ from llama.cpp or Ollama?
Per its README, FreeToken targets frontier-scale MoE models with expert residency and CPU-GPU co-execution, loads Hugging Face weights directly and exposes Anthropic- and OpenAI-compatible APIs. llama.cpp and Ollama are broader local runtimes. This page does not rank them; see each project's documentation.
Is FreeToken open source?
Its GitHub repository carries the Apache-2.0 licence. Check the licences of the models you run with it, since open-weight models have their own terms.
Which engines suit phones and embedded devices?
ExecuTorch, LiteRT, MNN, ONNX Runtime and OpenVINO target mobile, embedded and NPU hardware, while llama.cpp-based tools and MLX target laptops and desktops. See the Edge AI and LLM Inference Engines pages.
How current is this page?
Links and release facts were checked on 1 November 2026 from GitHub and project sites. The page cites the LinkedIn post by Sumanth and verifies its FreeToken details against the README and paper; LinkedIn may ask visitors to sign in. MLC LLM has no recent GitHub release. Descriptions follow each source's own pages and are not rankings.