Мониторинг LLM: Observability для production систем
opensourceaillmmonitoringobservabilityproductionit
Введение: LLM Monitoring
Проблема
Production LLM systems need monitoring:
- Track latency (TTFT, throughput)
- Monitor costs (tokens/sec)
- Detect drift (quality degradation)
- Alert on errors (timeouts, OOM)
- Debug requests (traces, logs)
Traditional monitoring isn't enough:
- Need token-level metrics
- Need prompt/response tracking
- Need quality evaluation
Факт: Companies using LLM monitoring see 40% cost reduction (industry benchmark, 2025).
Что мониторить?
Ключевые метрики
LLM Metrics Categories:
1. Performance:
- Time to First Token (TTFT)
- Tokens per second (throughput)
- Total generation time
- P50/P90/P99 latency
2. Quality:
- Response length
- Perplexity
- Toxicity score
- Relevance score
3. Cost:
- Input tokens/sec
- Output tokens/sec
- Total cost/day
- Cost per request
4. Reliability:
- Error rate
- Timeout rate
- Retry rate
- Availability
Dashboard example
┌─────────────────────────────────────────────────────┐
│ LLM Service Dashboard │
├──────────────┬──────────────┬───────────────────────┤
│ TTFT (P99) │ Throughput │ Error Rate │
│ 450ms │ 240 tok/s │ 0.5% │
│ ┌───┐ │ ┌───┐ │ ┌───┐ │
│ │ │ │ │ │ │ │ │ │
│ └───┘ │ └───┘ │ └───┘ │
├──────────────┼──────────────┼───────────────────────┤
│ Token Usage │ Cost/Day │ Quality Score │
│ In: 1.2M │ $142.50 │ 94.2% │
│ Out: 3.8M │ │ ┌───┐ │
│ │ │ │ │ │
└──────────────┴──────────────┴───────────────────────┘
Performance Metrics
Time to First Token (TTFT)
TTFT = time from request to first token
Why it matters:
- User-perceived latency
- Affects UX (longer = worse)
- Indicates queue wait time
Target:
- Interactive: < 200ms
- Acceptable: < 500ms
- Warning: > 1000ms
Monitoring:
histogram(
name: "llm_ttft_seconds",
buckets: [0.1, 0.2, 0.5, 1.0, 2.0, 5.0]
)
Tokens per Second
Throughput metrics:
Input throughput:
tokens_in = prompt_tokens / processing_time
Output throughput:
tokens_out = generated_tokens / generation_time
Total throughput:
total = tokens_in + tokens_out
Per-model breakdown:
model_a: 240 tok/s
model_b: 180 tok/s
model_c: 90 tok/s
GPU utilization correlation:
high_throughput + low_gpu = bottleneck elsewhere
high_throughput + high_gpu = GPU bound
Latency Percentiles
Latency distribution:
Metric P50 P90 P99
──────────────────────────────────
TTFT 50ms 200ms 800ms
Gen/s 45ms 120ms 500ms
Total 200ms 800ms 3000ms
Alert thresholds:
P99 TTFT > 2s → warning
P99 Total > 10s → critical
Cost Monitoring
Token Tracking
# Track token usage
class TokenTracker:
def __init__(self):
self.usage = defaultdict(lambda: {
"prompt_tokens": 0,
"completion_tokens": 0,
"total_tokens": 0
})
def record(self, model, prompt, completion):
prompt_tokens = len(tokenizer.encode(prompt))
completion_tokens = len(tokenizer.encode(completion))
self.usage[model]["prompt_tokens"] += prompt_tokens
self.usage[model]["completion_tokens"] += completion_tokens
self.usage[model]["total_tokens"] += prompt_tokens + completion_tokens
# Pricing per 1M tokens
PRICING = {
"gpt-4": {"input": 30, "output": 60},
"gpt-3.5": {"input": 3, "output": 6},
"llama-3-70b": {"input": 0, "output": 0}, # self-hosted
}
Cost Calculation
def calculate_cost(usage, pricing):
"""Calculate cost from token usage"""
total = 0
breakdown = {}
for model, tokens in usage.items():
price = pricing.get(model, {"input": 0, "output": 0})
model_cost = (
tokens["prompt_tokens"] * price["input"] / 1e6 +
tokens["completion_tokens"] * price["output"] / 1e6
)
breakdown[model] = model_cost
total += model_cost
return total, breakdown
# Example
usage = {
"gpt-4": {"prompt_tokens": 500000, "completion_tokens": 100000},
"gpt-3.5": {"prompt_tokens": 2000000, "completion_tokens": 500000},
}
total, breakdown = calculate_cost(usage, PRICING)
# gpt-4: $21.00
# gpt-3.5: $9.00
# total: $30.00
Cost Optimization
Cost reduction strategies:
1. Model routing:
- Simple queries → cheap model
- Complex queries → expensive model
2. Caching:
- Cache frequent prompts
- 30% reduction typical
3. Quantization:
- INT4 models → 50% memory savings
- Enables cheaper hardware
4. Batch processing:
- Group similar requests
- Better GPU utilization
Quality Monitoring
Response Quality Metrics
Quality dimensions:
1. Relevance:
- How well does response match prompt?
- Score: 1-5 (LLM-evaluated)
2. Coherence:
- Is response logically consistent?
- Score: 1-5
3. Toxicity:
- Harmful/offensive content?
- Score: 0-1 (lower is better)
4. Factual accuracy:
- Are claims correct?
- Hard to measure automatically
Automated Quality Checks
class QualityChecker:
def __init__(self):
self.toxicity_model = load_toxicity_model()
self.evaluator = load_evaluator()
def check(self, prompt, response):
return {
"toxicity": self.toxicity_model.predict(response),
"relevance": self.evaluator.relevance(prompt, response),
"coherence": self.evaluator.coherence(response),
"length": len(response.split()),
"unusual_chars": response.count('\u200b'), # zero-width
}
def flag_issues(self, metrics):
issues = []
if metrics["toxicity"] > 0.5:
issues.append("HIGH_TOXICITY")
if metrics["length"] < 10:
issues.append("SHORT_RESPONSE")
if metrics["unusual_chars"] > 5:
issues.append("SUSPICIOUS_CHARS")
return issues
Drift Detection
Drift types:
1. Input drift:
- User prompts changing over time
- Monitor prompt distribution
2. Output drift:
- Response quality changing
- Monitor quality scores
3. Model drift:
- Model behavior changing (updates)
- A/B test before/after
Detection:
- Statistical tests (KS test)
- Baseline comparison
- Alert on significant changes
Tracing and Debugging
Request Tracing
# OpenTelemetry integration
from opentelemetry import trace
tracer = trace.get_tracer("llm-service")
def generate_with_trace(prompt, model):
with tracer.start_as_current_span("llm.generate") as span:
span.set_attribute("model", model)
span.set_attribute("prompt_length", len(prompt))
with tracer.start_as_current_span("api.call") as api_span:
response = llm.generate(prompt, model)
api_span.set_attribute("tokens", len(response))
span.set_attribute("ttft_ms", ttft * 1000)
span.set_attribute("total_ms", total * 1000)
return response
Prompt/Response Logging
# Structured logging
import json
import logging
logger = logging.getLogger("llm")
def log_request(prompt, response, metrics):
log_entry = {
"timestamp": datetime.utcnow().isoformat(),
"model": "llama-3-70b",
"prompt": prompt[:1000], # truncate for storage
"response": response[:1000],
"metrics": {
"ttft": metrics["ttft"],
"tokens": metrics["tokens"],
"latency": metrics["latency"],
},
"user_id": "anonymous",
}
logger.info(json.dumps(log_entry))
Debugging Common Issues
Issue: High TTFT
- Check: GPU utilization
- Check: Queue length
- Check: Model loading time
- Fix: Increase batch size, add GPUs
Issue: Low throughput
- Check: Token generation rate
- Check: GPU memory usage
- Check: Network latency
- Fix: Optimize batch size, use tensor parallel
Issue: High error rate
- Check: OOM errors
- Check: Timeout errors
- Check: API rate limits
- Fix: Increase limits, add retries
Issue: Quality degradation
- Check: Model version
- Check: Temperature setting
- Check: Prompt format
- Fix: Rollback, adjust parameters
Tools and Stack
Monitoring Stack
Recommended stack:
1. Metrics:
- Prometheus (collection)
- Grafana (visualization)
- vLLM metrics exporter
2. Tracing:
- OpenTelemetry (instrumentation)
- Jaeger (storage)
- Tempo (alternative)
3. Logging:
- ELK Stack (Elasticsearch, Logstash, Kibana)
- Loki (lightweight)
- CloudWatch (AWS)
4. Alerting:
- Alertmanager (Prometheus)
- PagerDuty (on-call)
- Slack notifications
vLLM Metrics
# vLLM exposes Prometheus metrics
# Enable with: --metrics-host 0.0.0.0 --metrics-port 8001
# Key metrics:
# vllm:cpu_cache_usage_perc
# vllm:gpu_cache_usage_perc
# vllm:prompt_tokens_total
# vllm:generation_tokens_total
# vllm:time_to_first_token_seconds
# vllm:time_per_output_token_seconds
# vllm:e2e_request_latency_seconds
# vllm:num_requests_running
# vllm:num_requests_swapped
Grafana Dashboard
{
"dashboard": {
"panels": [
{
"title": "TTFT Distribution",
"type": "histogram",
"query": "histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))"
},
{
"title": "Throughput",
"type": "graph",
"query": "rate(vllm:generation_tokens_total[5m])"
},
{
"title": "GPU Cache Usage",
"type": "gauge",
"query": "vllm:gpu_cache_usage_perc"
},
{
"title": "Request Latency P99",
"type": "graph",
"query": "histogram_quantile(0.99, rate(vllm:e2e_request_latency_seconds_bucket[5m]))"
}
]
}
}
Alerting
Alert Rules
# Prometheus alerting rules
groups:
- name: llm-alerts
rules:
- alert: HighTTFT
expr: histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m])) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "High TTFT detected"
- alert: LowThroughput
expr: rate(vllm:generation_tokens_total[5m]) < 100
for: 10m
labels:
severity: critical
annotations:
summary: "Low throughput detected"
- alert: HighErrorRate
expr: rate(llm_requests_total{status="error"}[5m]) / rate(llm_requests_total[5m]) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate detected"
- alert: GPUCacheFull
expr: vllm:gpu_cache_usage_perc > 0.95
for: 1m
labels:
severity: warning
annotations:
summary: "GPU cache nearly full"
Slack Integration
# Slack notifications
import requests
def send_slack_alert(message, channel="#llm-alerts"):
webhook = "https://hooks.slack.com/services/..."
payload = {
"channel": channel,
"text": message,
"username": "LLM Monitor",
"icon_emoji": ":robot_face:"
}
requests.post(webhook, json=payload)
# Usage
send_slack_alert("⚠️ High TTFT: 2.5s (threshold: 2s)")
send_slack_alert("🔴 Low throughput: 50 tok/s (threshold: 100 tok/s)")
Заключение
LLM monitoring is essential for production:
Key metrics:
- TTFT (user-perceived latency)
- Throughput (tokens/sec)
- Cost (tokens × pricing)
- Quality (automated checks)
Best practices:
- Set up dashboards early
- Alert on SLO violations
- Log prompts/responses for debugging
- Monitor costs daily
- Track quality metrics
Tools:
- Prometheus + Grafana for metrics
- OpenTelemetry for tracing
- ELK/Loki for logging
- Custom quality checkers