You can build a full LLM stack without paying for a single API call. Every layer has capable open-source options: you run an open-weight model with a local runner, serve it at scale with an inference engine, adapt it with a fine-tuning library, route traffic through a gateway, retrieve context from a vector database, measure quality with an evaluation framework and give users a chat interface. The tools are grouped below by layer. Cards show the latest GitHub release and licence where they were checked on 22 November 2026; where GitHub detects the licence as "Other", read the repository licence before use. Hardware, hosting and engineering time still cost money, and the projects differ in maturity, so check recent activity before you commit.
1. Open-Weight Models
4 entriesThe base of the stack. These families publish weights that you can download and run on your own hardware; check each model card for its licence and size before use.
2. Local Runners
4 entriesPrograms that load a model on a laptop, workstation or edge device and expose it for use, without a cloud account.
3. Serving Engines
4 entriesServers built for throughput and latency when many users share a model, on GPUs or a Kubernetes cluster.
4. Fine-Tuning
4 entriesTools that adapt an open model to your data with full fine-tuning, LoRA, QLoRA or reinforcement learning methods.
5. Gateways
4 entriesA single entry point between applications and many model servers or providers, handling routing, fallbacks and keys.
6. Vector Databases
4 entriesStores that index embeddings so an application can retrieve relevant context for a model.
7. Evaluation
5 entriesFrameworks that test prompts, RAG pipelines and models, so changes are measured instead of guessed.
8. Chat Interfaces
4 entriesThe front end your users see: chat applications that connect to the models and stores below them.