Сравнение LLM Serving Systems: vLLM, Ollama, LM Studio, llama.cpp

Введение: LLM Serving

Проблема

Running LLMs in production requires:
  - High throughput (many requests/sec)
  - Low latency (fast response)
  - Easy deployment (Docker, CLI)
  - API compatibility (OpenAI-style)
  - Resource efficiency (GPU memory)

Many options exist. Which one to choose?

Факт: vLLM achieves 24x throughput compared to naive serving (research, 2025).

Обзор систем

Сравнительная таблица

Feature            vLLM      Ollama    LM Studio   llama.cpp
────────────────────────────────────────────────────────────────
Throughput         ★★★★★     ★★★       ★★          ★★★
Latency            ★★★★      ★★★       ★★★         ★★★★
GPU Support        CUDA      CUDA      CUDA        CUDA/Metal
CPU Support        ✅        ✅          ✅          ✅
Mac Support        ❌        ✅          ✅          ✅
Quantization       INT4-8    INT4-8    INT4-8      Q2-Q8
PagedAttention     ✅        ❌          ❌          ❌
Speculative Dec.   ✅        ✅          ✅          ✅
Continuous Batch   ✅        ✅          ❌          ❌
API Format         OpenAI    OpenAI    OpenAI      OpenAI
Deployment         Docker    Docker    GUI         CLI
Monitoring         ✅        ❌          ❌          ❌
Multi-GPU          ✅        ✅          ✅          ✅

vLLM

Архитектура

vLLM Architecture:

┌─────────────────────────────────────┐
│          API Server (FastAPI)        │
├─────────────────────────────────────┤
│       Scheduler (Continuous Batch)   │
├─────────────────────────────────────┤
│     PagedAttention Engine            │
│  ┌───────────────────────────────┐   │
│  │ KV Cache Manager               │   │
│  │ - Block Manager                │   │
│  │ - Prefix Caching               │   │
│  └───────────────────────────────┘   │
├─────────────────────────────────────┤
│       Model Executor                 │
│  - Tensor Parallel                   │
│  - Pipeline Parallel                 │
└─────────────────────────────────────┘

Установка

# Install vLLM
pip install vllm

# Docker
docker run -d \
  --gpus all \
  -p 8000:8000 \
  vllm/vllm-openai \
  --model meta-llama/Llama-3-70B

# Verify
curl http://localhost:8000/v1/models

Ключевые фичи

1. PagedAttention:
   - Memory fragmentation elimination
   - 24x throughput vs naive serving
   - Dynamic KV cache management

2. Continuous Batching:
   - Request-level batching (not token-level)
   - Low latency for short prompts
   - High throughput for long generations

3. Speculative Decoding:
   - Small model drafts large model tokens
   - 2x speedup typical

4. Tensor Parallelism:
   - Split model across multiple GPUs
   - NVLink for fast inter-GPU communication

Производительность

Benchmark: Llama-3-70B, A100-80GB

Metric              vLLM      Naive
──────────────────────────────────────
Throughput          2400 tok/s  100 tok/s
Memory efficiency   95%         60%
P99 latency         500ms       2000ms
GPU utilization     85%         40%

Конфигурация

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3-70B",
    tensor_parallel_size=4,      # 4 GPUs
    gpu_memory_utilization=0.9,   # 90% GPU memory
    max_model_len=4096,           # Max context
    swap_space=16,                # CPU swap (GB)
    cache_size=16,                # KV cache (GB)
)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=1024,
    stop=["\n\n"]
)

output = llm.generate("Hello, world!", sampling_params)

Ollama

Архитектура

Ollama Architecture:

┌─────────────────────────────────────┐
│          REST API Server             │
├─────────────────────────────────────┤
│          Model Manager               │
│  - Pull/Push models                  │
│  - Model registry                    │
├─────────────────────────────────────┤
│       Inference Engine               │
│  - GGUF loader                       │
│  - Quantization support              │
│  - Metal/CUDA backend                │
├─────────────────────────────────────┤
│       Runtime                        │
│  - System tray                       │
│  - Auto-start                        │
└─────────────────────────────────────┘

Установка

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull model
ollama pull llama3:70b

# Run server
ollama serve

# API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3:70b",
  "prompt": "Hello, world!",
  "stream": false
}'

Модели

Ollama model library:

llama3:70b      - Meta Llama 3 (70B)
mistral:7b      - Mistral 7B
codellama:70b   - Code Llama (70B)
phi3:mini       - Microsoft Phi-3
gemma:7b        - Google Gemma (7B)
neural-chat:7b  - Intel Neural Chat
dolphin-mixtral - Dolphin Mixtral

Quantization:
  :q4_0 - Q4_K_M (default)
  :q3_0 - Q3_K_M
  :q2_0 - Q2_K
  :f16  - FP16

Модификация моделей

Modelfile example:

FROM llama3:70b

SYSTEM You are a helpful assistant.

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>
{{ .Response }}<|eot_id|>"""

Производительность

Benchmark: Llama-3-70B, A100-80GB

Metric              Ollama      vLLM
──────────────────────────────────────
Throughput          180 tok/s   2400 tok/s
Latency (TTFT)      200ms       50ms
Memory usage        75%         60%
Ease of use         ★★★★★     ★★★

LM Studio

Архитектура

LM Studio Architecture:

┌─────────────────────────────────────┐
│          GUI Application             │
│  - Model browser                     │
│  - Chat interface                    │
│  - Settings panel                    │
├─────────────────────────────────────┤
│          Local API Server            │
│  - OpenAI-compatible API             │
│  - Localhost:1234                    │
├─────────────────────────────────────┤
│       Inference Engine               │
│  - llama.cpp backend                 │
│  - Metal/CUDA                        │
└─────────────────────────────────────┘

Установка

# Download from website
# https://lmstudio.ai/

# Or via Homebrew (macOS)
brew install --cask lmstudio

# Start server
# GUI: Server → Start Server
# API: http://localhost:1234/v1/models

Особенности

1. GUI-first approach:
   - Browse models visually
   - Test models before deploying
   - Chat interface for testing

2. Model search:
   - HuggingFace integration
   - Filter by quantization
   - Preview model cards

3. Local API:
   - OpenAI-compatible
   - Works with any OpenAI client
   - No server setup needed

API совместимость

# LM Studio API (OpenAI-compatible)
curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'

llama.cpp

Архитектура

llama.cpp Architecture:

┌─────────────────────────────────────┐
│          C++ Core                    │
├─────────────────────────────────────┤
│       GGUF Format                    │
│  - Model serialization               │
│  - Quantization metadata             │
├─────────────────────────────────────┤
│       Backends                       │
│  - CPU (AVX2, AVX-512)              │
│  - GPU (CUDA, Metal, Vulkan)        │
│  - OpenCL (AMD)                     │
├─────────────────────────────────────┤
│          Python Bindings             │
│  - llama-cpp-python                  │
│  - LangChain integration             │
└─────────────────────────────────────┘

Установка

# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make

# Python bindings
pip install llama-cpp-python

# Docker
docker pull ghcr.io/ggerganov/llama.cpp:server

GGUF формат

GGUF (GGML Universal Format):

Header:
  - Magic number
  - Tensor names
  - Tensor shapes
  - Quantization type

Tensors:
  - Weight matrices
  - Bias vectors
  - Layer norm params

Quantization:
  Q0:  raw FP32
  Q1:  raw FP16
  Q2:  INT2
  Q3:  INT3
  Q4:  Q4_0 (default)
  Q5:  Q5_0
  Q6:  INT6
  Q7:  INT8
  Q8:  FP16

Серверный режим

# Start llama.cpp server
./server \
  -m models/llama-3-70b.Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 4096 \
  --n-gpu-layers 99

# API
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Hello, world!",
    "max_tokens": 1024,
    "temperature": 0.7
  }'

Производительность

Benchmark: Llama-3-70B, A100-80GB

Metric              llama.cpp   vLLM
──────────────────────────────────────
Throughput          200 tok/s   2400 tok/s
CPU-only            15 tok/s    N/A
Mac (M2)            30 tok/s    N/A
Memory efficiency   90%         95%

Сравнение производительности

Throughput

Llama-3-70B, A100-80GB, batch=32

System              Throughput (tok/s)
─────────────────────────────────────
vLLM                2400
llama.cpp           200
Ollama              180
LM Studio           170
Naive (HuggingFace) 100

Latency

Llama-3-8B, A100-80GB, prompt=128 tokens

System              TTFT      First token
────────────────────────────────────────
vLLM                50ms      20ms
llama.cpp           80ms      25ms
Ollama              150ms     40ms
LM Studio           200ms     50ms

Memory Usage

Llama-3-70B, FP16

System              VRAM Used   Efficiency
────────────────────────────────────────
vLLM                145 GB      95%
llama.cpp           150 GB      90%
Ollama              160 GB      85%
LM Studio           165 GB      82%
Naive               240 GB      58%

Выбор системы

Для production

Choose vLLM when:
  ✓ High throughput required
  ✓ Multiple GPUs available
  ✓ Kubernetes deployment
  ✓ Monitoring needed
  ✓ OpenAI API compatibility

Choose llama.cpp when:
  ✓ CPU-only deployment
  ✓ Edge deployment
  ✓ Custom hardware
  ✓ Minimal dependencies

Для development

Choose Ollama when:
  ✓ Quick prototyping
  ✓ macOS development
  ✓ Simple deployment
  ✓ Model experimentation

Choose LM Studio when:
  ✓ GUI preferred
  ✓ Model browsing
  ✓ Testing models
  ✓ Non-technical users

Для edge

Choose llama.cpp when:
  ✓ Raspberry Pi
  ✓ Embedded devices
  ✓ CPU-only servers
  ✓ Custom form factors

Choose Ollama when:
  ✓ Mac Mini
  ✓ Consumer laptops
  ✓ Simple setup needed

Интеграция

LangChain

# vLLM
from langchain.llms import VLLM
llm = VLLM(model="meta-llama/Llama-3-70B")

# Ollama
from langchain.llms import OllamaLLM
llm = OllamaLLM(model="llama3:70b")

# llama.cpp
from langchain.llms import LlamaCpp
llm = LlamaCpp("models/llama-3-70b.Q4_K_M.gguf")

# LM Studio
from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI(
    base_url="http://localhost:1234/v1",
    model="local-model"
)

OpenAI Client

from openai import OpenAI

# vLLM
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="n/a"
)

# Ollama
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="n/a"
)

# llama.cpp
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="n/a"
)

# LM Studio
client = OpenAI(
    base_url="http://localhost:1234/v1",
    api_key="n/a"
)

# All use the same API:
response = client.chat.completions.create(
    model="...",
    messages=[{"role": "user", "content": "Hello!"}]
)

Заключение

Выбор системы зависит от use case:

Production serving:

vLLM: highest throughput, best for production
llama.cpp: good for CPU-only or edge

Development:

Ollama: simplest setup
LM Studio: best GUI experience

Edge:

llama.cpp: only option for constrained environments
Ollama: good for consumer devices

Введение: LLM Serving

Проблема

Обзор систем

Сравнительная таблица

vLLM

Архитектура

Установка

Ключевые фичи

Производительность

Конфигурация

Ollama

Архитектура

Установка

Модели

Модификация моделей

Производительность

LM Studio

Архитектура

Установка

Особенности

API совместимость

llama.cpp

Архитектура

Установка

GGUF формат

Серверный режим

Производительность

Сравнение производительности

Throughput

Latency

Memory Usage

Выбор системы

Для production

Для development

Для edge

Интеграция

LangChain

OpenAI Client

Заключение

Ресурсы