Benchmarking Ollama and vLLM for Concurrent LLM Serving: A Multi-Scenario Evaluation of Performance and Scalability

Abstract

Serving LLMs to many concurrent users gives rise to significant challenges for latency, throughput, and scalability. This paper presents a systematic and reproducible benchmark of two widely used large language model (LLM) serving frameworks, Ollama and vLLM, under concurrent workloads. We conducted a controlled comparison by running both frameworks on a single NVIDIA H100 80 GB GPU with the same model weights (Qwen3-4B) and inference configurations. We then evaluated them using five open benchmark datasets across four scenarios consisting of baseline question answering, complex reasoning, streaming interaction, and stress testing. Each scenario was executed under increasing concurrency levels, ranging from light loads to high-concurrency stress, to measure end-to-end latency, throughput, time-to-first-token (TTFT), success rate, and resource usage. Our experiments show that vLLM clearly outperforms Ollama across all four scenarios, achieving a 100% request success rate. It delivers 20-29 times higher throughput and 8-19 times lower P95 latency, and it completes every request successfully. Additionally, vLLM produced the first token within 0.5-3.5 s and remained stable at up to 100 concurrent users. Ollama, by contrast, required 54-122 s for the first token, hit a concurrency bottleneck near 10 users, and exhibited a timeout-based error rate of 13-30.06% under heavier loads. Notably, both frameworks demonstrated significant memory growth during extended endurance tests, necessitating careful monitoring in long-running deployments. Overall, for a single model on a single GPU, vLLM is highly suitable for high-concurrency serving, while Ollama remains a practical choice for lightweight, local, or developmental workflows.