Six Serving Engines on One Tesla T4: Gemma 4 E2B on a Turing GPU

In today's tech landscape, understanding how different inference serving engines perform on hardware like the Tesla T4 GPU is crucial. This topic is particularly relevant as more organizations look to optimize their AI workloads on cloud platforms. This article explores a comparison of six engines running the Gemma 4 E2B model on a Tesla T4 GPU.

Overview of the Tesla T4 GPU

The Tesla T4 is an NVIDIA data-center graphics card that many cloud providers still offer. It is based on the Turing architecture and is known for being one of the most affordable options available. However, it has some limitations, such as a lack of support for certain advanced features like bf16 (a data type) and only 64 KiB of shared memory per block.

The Engines Tested

The article compares the following six inference serving engines:

• llama.cpp
• vLLM
• SGLang
• mistral.rs
• Plain PyTorch engine
• Google's LiteRT-LM

Each engine serves the Gemma 4 E2B model using Google's Quantization-Aware Training (QAT) 4-bit weights. The tests were conducted on a Google Cloud Compute Engine virtual machine (VM) with a Tesla T4 GPU.

Performance Metrics

Each engine was evaluated based on several performance metrics, including throughput (the number of tokens processed per second) and answer quality. Here are some key findings:

Single Request Throughput

Without using a drafter, the engines achieved the following token processing rates:

• vLLM: 193.0 tokens per second
• llama.cpp: 113.5 tokens per second
• SGLang: 108.8 tokens per second
• mistral.rs: 121.1 tokens per second
• LiteRT-LM: 44.6 tokens per second

Batch Processing Performance

When processing 64 requests with specific input and output token configurations, the throughput varied:

• vLLM: 1,165.5 tokens per second
• SGLang: 908.6 tokens per second
• mistral.rs: 238.0 tokens per second
• llama.cpp: 633.0 tokens per second
• LiteRT-LM: 308.0 tokens per second

Answer Quality

The engines were also assessed for their answer quality using the GSM8K benchmark. The scores ranged from 0.8954 to 0.9030 for the five CUDA engines, while LiteRT-LM scored slightly lower at 0.8734.

Technical Specifications

The setup for the tests included:

• GPU: NVIDIA Tesla T4 (Turing architecture, sm_75)
• Shared Memory per Block: 64 KiB
• Driver Version: 620.70.06
• Model Weights: Google's QAT 4-bit weights

Conclusion

This comparison highlights the performance differences among various inference serving engines on a Tesla T4 GPU. While vLLM showed the best throughput, all engines had their strengths and weaknesses in terms of speed and accuracy.

Merits

• Provides a clear comparison of multiple engines.
• Highlights performance metrics that are crucial for AI workloads.
• Offers insights into the capabilities of the Tesla T4 GPU.

Demerits

• Limited to only six engines, which may not represent all available options.
• Performance may vary based on specific use cases and workloads.

Caution

This article is meant for educational purposes. Any placeholder values must be replaced with actual data in practical applications. Readers should verify claims against the original source before relying on them.

Frequently asked questions

• What is the Tesla T4 GPU? — The Tesla T4 is an NVIDIA data-center GPU designed for AI workloads, known for its affordability and efficiency.
• What are inference serving engines? — These are software frameworks that allow machine learning models to serve predictions based on input data.
• What is the Gemma 4 E2B model? — It is a machine learning model used for various AI tasks, evaluated in this comparison.
• How does QAT improve model performance? — Quantization-Aware Training (QAT) helps optimize models for lower precision, improving performance without significantly sacrificing accuracy.
• What were the results of the performance tests? — The tests showed varying throughput rates and answer quality across the six engines.
• Why is answer quality important? — High answer quality ensures that the predictions made by the model are accurate and reliable, which is crucial for real-world applications.