The open-source ecosystem (vLLM, llama.cpp, Ollama, LiteLLM, LangChain, LlamaIndex)

Models and offerings #

Who they are

This hub is for anyone who has weights and wants a service, or has an application and wants to stop depending on one API vendor. None of these projects trains models. They are the plumbing between the model makers on the other platform hubs and your code, and most of them started as one person’s project in 2023 before growing companies around them.

vLLM came out of UC Berkeley’s Sky Computing Lab in 2023 with the paged attention paper and is now the most common way to serve an open model on GPUs. llama.cpp began in March 2023 as Georgi Gerganov’s port of Llama to plain C++ and grew the GGUF format and the ggml tensor library that every local runner uses. Ollama wrapped llama.cpp in a Docker-like command line later that year. LangChain (October 2022) and LlamaIndex (November 2022) appeared with ChatGPT to orchestrate calls and retrieval, and LiteLLM arrived in 2023 to make every provider look like OpenAI. The open-source ML topic covers the licensing landscape these projects sit in.

Tooling and APIs

Serving on GPUs: pip install vllm, then vllm serve <hub-id> gives an OpenAI-compatible endpoint. vLLM v0.33.0 (2 November 2026) ships CUDA 13.0-only wheels, moved its default ROCm build to 10.0 and dropped Python 3.10. It is Apache-2.0 and a PyTorch Foundation project. SGLang is the main alternative with similar performance and a different scheduler. Both read Hugging Face checkpoints directly and both carry NVIDIA-contributed kernels, so the gap to TensorRT-LLM is small for most models.

Running locally: llama.cpp reached v1.0.0 on 13 November 2026, removing its long-deprecated public API, and continues to publish numbered builds several times a day. It runs on CPU, Apple Metal, CUDA, ROCm, Vulkan and SYCL, reads GGUF files, and its llama-server speaks the OpenAI chat, embeddings and now audio formats. Ollama (MIT, v0.40.16 on 9 November 2026) installs on macOS, Windows and Linux, pulls quantised models from its library with ollama pull, serves them at http://127.0.0.1:11434, uses an MLX backend on Apple Silicon, and also brokers hosted “cloud” models through the same client. Our tiny RAG project runs against it.

# needs-api
import httpx

body = {
    "model": "smollm2:135m",
    "messages": [{"role": "user", "content": "In one sentence, what does a learning rate control?"}],
    "stream": False,
}
reply = httpx.post("http://127.0.0.1:11434/api/chat", json=body, timeout=120).json()
print(reply["message"]["content"])

Gateways and orchestration: LiteLLM’s Python SDK maps completion(model="anthropic/...") onto each provider’s API, and its proxy adds virtual keys, budgets and logging in front of all of them. The core is MIT, and features in its enterprise directory are under a commercial licence. LangChain 1.x (langchain 1.5.0, 11 November 2026, Python 3.11 or newer) is the chain and agent library, with LangGraph for stateful, cyclic agent graphs and LangSmith as the paid tracing service. LlamaIndex (0.14.25, 21 September 2026) focuses on loading, chunking, indexing and querying documents, with LlamaCloud as its paid parsing and hosting service. The RAG pipeline checklist shows where each fits.

Everything above is free. Pain points: vLLM’s CUDA and PyTorch version coupling, llama.cpp’s build matrix and its relentless API churn before 1.0, Ollama’s default quantisations hiding quality loss, LiteLLM’s size, and the abstraction layers in LangChain and LlamaIndex that make a simple call hard to debug.

Migration notes

Between serving engines: vLLM, SGLang, llama.cpp, Ollama, Foundry Local, NIM and TensorRT-LLM all expose an OpenAI-compatible chat endpoint, so the application sees a base_url change. What differs is sampling parameters, tool-calling and structured-output support, and which quantisation formats each loads. GGUF is llama.cpp’s and Ollama’s format, vLLM reads safetensors plus GPTQ, AWQ, FP8 and NVFP4 checkpoints, and TensorRT-LLM builds its own engines. The quantisation formats reference maps them.

From a hosted provider to self-hosting: keep the openai SDK, point it at vLLM, and expect to lose hosted web search, file stores and the provider’s safety filters, which you now own. LiteLLM makes the switch a config change and keeps both paths live for fallback.

Framework to no framework: a LangChain or LlamaIndex pipeline is a few hundred lines of plain Python once you know which calls it makes, and many teams migrate that way as a project matures. Going in, LangChain 1.x and LlamaIndex 0.14 broke import paths from their 0.x days, so old tutorials do not run unchanged.

Watch for

Licensing is clean at the tool layer and messy at the model layer: an MIT runner happily serves a model whose licence forbids your use. The model’s licence governs. Ollama’s model library shows licences on each model page and the models reference records them for the ones we track.

Open core is the business model of the companies behind these projects. LiteLLM’s enterprise features, LangSmith, LlamaCloud and Ollama’s cloud models are paid and may change terms, while the MIT and Apache-2.0 cores cannot be taken away but can stop being developed. Hugging Face’s Text Generation Inference, once the obvious vLLM alternative, is in maintenance mode according to its repository, which is what that looks like.

Version churn is the operational risk. vLLM minor releases drop CUDA versions and Python versions (0.32 dropped Python 3.10, 0.33 dropped CUDA 12.9 wheels), llama.cpp removed 36 deprecated functions in 1.0.0, and LangChain’s major version changed its package layout. Pin everything and read release notes. Telemetry: Ollama does not phone home by default, LangChain and LlamaIndex send nothing unless you configure LangSmith or LlamaCloud, and LiteLLM’s proxy has opt-out usage reporting, so check each project’s documentation before deploying in a sensitive environment.