LLM Inference Engines Explained: llama.cpp, vLLM, SGLang, mistral.rs, PyTorch and LiteRT-LM

A model file on its own does nothing. It is a big pile of numbers, and some other program has to load those numbers, feed text in and get answers out. That program is called an inference engine, and picking the right one matters about as much as picking the model.

Today is November 12, 2026. Open models now come close to what paid AI services can do, and many teams want to run them on their own hardware for privacy, cost or offline use. Then they meet a list of strange names: llama.cpp, vLLM, SGLang, mistral.rs, PyTorch, LiteRT-LM. This post explains what that family of tools is called, why each one exists, and when Rohit, our example engineer, would reach for each.

What we call these tools

The general name is LLM inference engines. You will also see them called model runtimes, serving engines or serving frameworks. Some quick definitions:

• Inference means using a trained model to produce answers. Training teaches the model; inference is asking it things.
• Serving means running a model as a service, usually an HTTP API, so many users or apps can call it at once.
• Runtime is the low-level code that does the maths on the CPU, GPU or phone chip.

All six tools on the list take the same model weights and turn them into answers. They differ in where they run best (laptop, big GPU server or phone), how many users they handle at once, and how much memory they need.

Why the engine matters so much

The same model can feel fast or painfully slow depending on the engine. Three things decide that:

1. Memory. A model has to fit in RAM or GPU memory. Some engines support quantization, which stores the numbers with fewer bits (for example 4 bits instead of 16). The model gets roughly three to four times smaller and runs on cheaper hardware, at a small cost in quality.
2. Throughput. This is how many answers per second the engine can produce for many users together. Server engines use tricks such as continuous batching (new requests join a running batch instead of waiting their turn) to keep the GPU busy.
3. Latency. This is how quickly one user sees the first word. For chat it matters more than raw throughput.

A wrong choice is expensive. Rohit could rent a costly GPU and still serve only a few users, or try to load a model that never fits on the laptop.

The six engines, one by one

llama.cpp: runs almost anywhere

llama.cpp is a C and C++ engine built to run models on ordinary hardware. It uses its own file format, GGUF, which packs a quantized model into a single file. It runs on plain CPUs, Apple Silicon, NVIDIA and AMD GPUs and more. Popular desktop tools such as Ollama and LM Studio are built on top of it.

Practical example. Rohit is a consultant who reviews client contracts and cannot paste them into a public chatbot. With llama.cpp he runs a small quantized model on his laptop, fully offline, and summarises documents without any text leaving the machine.

# Start a local OpenAI-compatible server on port 8080
llama-server -m ./models/example-model-Q4_K_M.gguf --port 8080

vLLM: the workhorse for GPU servers

vLLM is a Python engine for serving models on GPUs to many users at once. Its key idea is PagedAttention. While answering, a model keeps a scratchpad of earlier tokens called the KV cache. vLLM stores that scratchpad in small pages, much like an operating system manages memory, so very little GPU memory is wasted and far more requests fit at the same time. Together with continuous batching, this lets one GPU serve many users.

Practical example. Rohit's company, Acme, wants an internal assistant for 300 staff. Calling a paid API for every question is getting expensive, and some data must stay in-house. Rohit puts vLLM on one GPU server, and because it exposes an OpenAI-compatible API, the existing chat app only needs a new base URL.

vllm serve example-org/example-model --port 8000

SGLang: fast for agents and repeated prompts

SGLang is another high-performance serving engine. It shines when many requests share the same beginning, such as a long system prompt, a shared document, or the earlier turns of a chat. Its RadixAttention keeps those shared parts cached and reuses them instead of computing them again. It also has strong support for structured output, meaning answers forced into valid JSON or a fixed format.

Practical example. Rohit builds a support-ticket triage agent. Every ticket is sent with the same 3,000-word policy document, and the model must reply with JSON fields such as category and priority. SGLang reuses the cached policy part across tickets and keeps the output in a shape his code can parse.

python -m sglang.launch_server --model-path example-org/example-model --port 30000

mistral.rs: a Rust engine for Rust shops

mistral.rs is a community-built inference engine written in Rust. It is not an official product of the company Mistral AI, despite the name, and it runs many model families. It supports quantization, an OpenAI-compatible server, and can be used as a library inside a Rust program. Rust gives memory safety and a single compiled binary with no Python environment to manage.

Practical example. Rohit's team has a log-analysis tool written in Rust that runs on locked-down servers where installing Python is not allowed. With mistral.rs they add a "summarise this incident" feature directly into the same binary, so there is no extra service to deploy or patch.

Plain PyTorch: the reference, not the race car

PyTorch is the framework most models are built and trained in. Running a model with plain PyTorch, often through the Hugging Face Transformers library and its generate() function, is the most direct way to use it. It is easy to change and debug, and new models usually work there first. It is not tuned for serving many users, so it is slower and uses more memory than the engines above.

Practical example. Rohit is testing whether a fine-tuned model is any good. He loads it in a notebook with plain PyTorch, tries different prompts, changes code and compares results. Once he is happy, he moves the model to vLLM or SGLang for production.

Google's LiteRT-LM: models on the phone

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite. LiteRT-LM is its layer for running language models such as Gemma on phones, browsers and small edge devices. It is built for limited memory, battery life and mobile chips, and it lets an app run AI with no network connection and no server bill.

Practical example. Rohit builds a field-inspection Android app for technicians who often have no signal. With LiteRT-LM and a small Gemma model on the phone, the app turns spoken notes into a clean report offline, and the data never leaves the device.

How Rohit picks one: a simple decision guide

Step 1: Decide where the model runs

On a phone or small device, start with LiteRT-LM. On a laptop or a CPU-only server, start with llama.cpp. On GPU servers, look at vLLM or SGLang. Inside a Rust program, consider mistral.rs.

Step 2: Count the users

One person or a few: llama.cpp is simple and enough. Dozens or hundreds at once: vLLM or SGLang, because batching is what keeps the cost per answer low.

Step 3: Look at the workload shape

Lots of shared prompts, agents or strict JSON output point to SGLang. General chat and API traffic point to vLLM. Experiments and research point to plain PyTorch.

Step 4: Test on your own hardware

Published numbers depend on the GPU, the model and the settings. Run your own prompts on your own machine with two candidate engines, and measure time to first word, words per second and memory use before deciding.

Conclusion

llama.cpp, vLLM, SGLang, mistral.rs, PyTorch and LiteRT-LM all do the same basic job: they run a trained model and turn input into answers. That family is called LLM inference engines. What separates them is where they run and who they serve: llama.cpp for laptops and small servers, vLLM and SGLang for busy GPU servers, mistral.rs for Rust-native apps, PyTorch for experiments, and LiteRT-LM for phones and edge devices. Choose the engine as carefully as the model.

Merits

• Running open models yourself keeps sensitive data in-house and can work fully offline.
• Quantization in llama.cpp and mistral.rs lets capable models run on ordinary laptops.
• vLLM and SGLang serve many users from one GPU, cutting the cost per answer.
• OpenAI-compatible APIs make switching from a paid service a small code change.
• LiteRT-LM brings AI features to phones with no server bill and no network needed.

Demerits

• Each engine has its own setup, flags and model formats to learn.
• Server engines need capable GPUs, which are costly to buy or rent.
• Quantized models lose some quality, and the loss differs by model and task.
• Not every engine supports every new model on day one.
• Self-hosting means you own updates, security patches and monitoring.

Caution

This article is educational. Model names, ports and paths in the examples are placeholders and must be replaced with your own. These projects change quickly, so features, flags and supported hardware may differ in the version you install. Check the official documentation and test on your own hardware before relying on any claim here for a production decision.

Frequently asked questions

• What is an LLM inference engine? — It is the software that loads a trained language model and runs it to produce answers, on a CPU, a GPU or a phone chip.
• Is llama.cpp or vLLM better? — Neither is better everywhere: llama.cpp is best for laptops and small servers, and vLLM is best for serving many users on GPU servers.
• What is the difference between vLLM and SGLang? — Both are fast GPU serving engines; SGLang is especially strong when requests share long prompts or need structured output such as JSON.
• Can I run an LLM without a GPU? — Yes, llama.cpp runs quantized models on ordinary CPUs, although more slowly than on a GPU.
• Is mistral.rs made by Mistral AI? — No, it is a separate community project written in Rust that can run many model families.
• Why not just use PyTorch in production? — Plain PyTorch is flexible but not tuned for many users, so dedicated engines usually give much more speed from the same hardware.
• What is LiteRT-LM used for? — It is Google's runtime for running language models such as Gemma directly on phones and edge devices, offline.
• What does quantization do? — It stores model numbers with fewer bits, so the model uses far less memory and runs on cheaper hardware with a small loss in quality.

Tags

#llm #inferenceengine #llamacpp #vllm #sglang #mistralrs #pytorch #litertlm #selfhostedai #aiengineering