Open-Source LLM Stack logopeterindia.net
Knowledge Hub

Open-Source Tools to Build a Full LLM Stack

Eight layers of an LLM stack, and the open-source tools for each: from open-weight models and local runners to serving engines, fine-tuning, gateways, vector databases, evals and chat interfaces.

38 sources•9 sections•Updated November 22, 2026

You can build a full LLM stack without paying for a single API call. Every layer has capable open-source options: you run an open-weight model with a local runner, serve it at scale with an inference engine, adapt it with a fine-tuning library, route traffic through a gateway, retrieve context from a vector database, measure quality with an evaluation framework and give users a chat interface. The tools are grouped below by layer. Cards show the latest GitHub release and licence where they were checked on 22 November 2026; where GitHub detects the licence as "Other", read the repository licence before use. Hardware, hosting and engineering time still cost money, and the projects differ in maturity, so check recent activity before you commit.

1. Open-Weight Models

4 entries

The base of the stack. These families publish weights that you can download and run on your own hardware; check each model card for its licence and size before use.

2. Local Runners

4 entries

Programs that load a model on a laptop, workstation or edge device and expose it for use, without a cloud account.

3. Serving Engines

4 entries

Servers built for throughput and latency when many users share a model, on GPUs or a Kubernetes cluster.

4. Fine-Tuning

4 entries

Tools that adapt an open model to your data with full fine-tuning, LoRA, QLoRA or reinforcement learning methods.

5. Gateways

4 entries

A single entry point between applications and many model servers or providers, handling routing, fallbacks and keys.

6. Vector Databases

4 entries

Stores that index embeddings so an application can retrieve relevant context for a model.

7. Evaluation

5 entries

Frameworks that test prompts, RAG pipelines and models, so changes are measured instead of guessed.

8. Chat Interfaces

4 entries

The front end your users see: chat applications that connect to the models and stores below them.