Vector Databases: хранение и поиск embedding
opensourceaillmvectordatabasesearchit
Введение: почему обычные базы не работают
Проблема
Обычная база данных:
SELECT * FROM documents WHERE text LIKE '%машинное обучение%'
✗ Только exact match
✗ Не понимает семантику
✗ Не работает с embeddings
Vector database:
SELECT * FROM documents WHERE embedding NEAREST [0.23, -0.45, ...]
✓ Semantic search
✓ Fast approximate nearest neighbor
✓ Built for embeddings
Факт: Для поиска среди 1M+ embeddings нужен special index. Linear scan = 1M distance calculations per query — слишком медленно.
Что такое Vector Database?
Core Concepts
Vector Database = база данных для векторов
Основные операции:
1. Upsert: добавить/обновить вектор
2. Search: найти N ближайших векторов
3. Filter: поиск с metadata filter
Пример:
Upsert: id=1, embedding=[0.23, -0.45, ...], metadata={title: "..."}
Search: query=[0.21, -0.43, ...], top_k=10, filter={date > 2025-01-01}
Distance Metrics
Как измеряем "близость" векторов?
1. Cosine Distance:
1 - cosine_similarity(A, B)
Range: [0, 2]
✓ Нормализованная длина
✓ Для LLM embeddings
2. L2 (Euclidean):
sqrt(sum((a - b)^2))
Range: [0, ∞)
✓ Для позиционных данных
3. Inner Product:
dot(A, B)
Range: [-∞, +∞]
✓ Для нормализованных векторов
4. Hamming Distance:
count of different bits
✓ Для binary embeddings
Уровень 1: Индексы для ANN
Почему не linear scan?
1M vectors, 768 dimensions:
Linear scan: 1M * 768 = 768M operations per query
At 1000 queries/sec: 768B ops/sec
Need: ~100 GPUs just for search!
ANN (Approximate NN):
Index: pre-compute structure
Query: 10K * 768 = 7.68M operations
Speedup: 100x
HNSW (Hierarchical Navigable Small World)
HNSW — самый популярный ANN алгоритм.
Структура: multi-layer graph
Layer 3: ●────●────●
╲ ╱
Layer 2: ●──●──●──●──●
╲ │ ╱
Layer 1: ●──●──●──●──●──●
╲ │ ╱
Layer 0: ●─●─●─●─●─●─●
Search: start from top layer, narrow down, finish at layer 0.
HNSW Parameters
M: max connections per node (2-16)
M=2: sparse graph, memory efficient
M=16: dense graph, faster search
efConstruction: build quality (64-512)
Higher = better index, slower build
efSearch: query time (10-500)
Higher = more accurate, slower query
Trade-off:
M=16, efSearch=128: ~99% recall, 1ms
M=8, efSearch=64: ~97% recall, 0.5ms
M=4, efSearch=32: ~95% recall, 0.3ms
IVF (Inverted File Index)
IVF: k-means clustering + per-cluster search
Step 1: k-means clustering
Cluster 1: ● ● ●
Cluster 2: ● ● ●
Cluster 3: ● ●
Step 2: Search
Assign query to N nearest clusters
Search only those vectors
nlist: number of clusters (128-65536)
nprobe: number of clusters to search (1-1024)
Trade-off:
nprobe=1: fast, inaccurate
nprobe=100: slow, accurate
Уровень 2: Популярные Vector Databases
Chroma
import chromadb
# In-memory (для dev)
client = chromadb.Client()
# Persistent (для production)
client = chromadb.PersistentClient(path="./chroma-db")
# Collection
collection = client.create_collection(
name="documents",
metadata={"hnsw:space": "cosine"}
)
# Add documents
collection.add(
documents=[
"Что такое LLM?",
"LLM — большая языковая модель",
"Embedding — вектор представления"
],
metadatas=[
{"source": "faq", "category": "intro"},
{"source": "faq", "category": "intro"},
{"source": "glossary", "category": "tech"},
],
ids=["doc1", "doc2", "doc3"]
)
# Query
results = collection.query(
query_texts=["языковые модели"],
n_results=2,
where={"category": "intro"}
)
Qdrant
from qdrant_client import QdrantClient
from qdrant_client.models import (
Distance, VectorParams, PointStruct
)
client = QdrantClient(path="./qdrant-db")
client.create_collection(
"documents",
vectors_config=VectorParams(
size=768,
distance=Distance.COSINE
)
)
# Upsert
client.upsert(
collection_name="documents",
points=[
PointStruct(
id=1,
vector=embedding,
payload={"title": "Doc 1", "date": "2026-01-01"}
)
]
)
# Search
hits = client.search(
collection_name="documents",
query_vector=embedding,
limit=10,
query_filter={"date": {"gt": "2025-01-01"}}
)
Weaviate
import weaviate
client = weaviate.Client("http://localhost:8080")
class Object:
pass
# Schema
client.schema.create_class({
"class": "Document",
"vectorizer": "none",
"properties": [
{"name": "title", "type": "string"},
{"name": "content", "type": "text"},
{"name": "date", "type": "date"}
]
})
# Add
client.data_object.create({
"title": "Doc 1",
"content": "Content here",
"date": "2026-01-01"
}, vector=embedding)
# Query
results = (
client.query
.get("Document", ["title", "date"])
.with_near_vector({"vector": embedding})
.with_where({"path": ["date"], ...})
.with_limit(10)
.do()
)
Milvus
from pymilvus import connections, Collection, CollectionSchema, Field, DataType
connections.connect("default", host="localhost", port="19530")
# Schema
fields = [
Field(name="id", dtype=INT64, is_primary=True),
Field(name="embedding", dtype=FLOAT_VECTOR, dim=768),
Field(name="title", dtype=VARCHAR, max_length=500),
]
schema = CollectionSchema(fields)
collection = Collection("documents", schema)
# Create index
index_params = {
"index_type": "HNSW",
"metric_type": "COSINE",
"params": {"M": 16, "efConstruction": 256}
}
collection.create_index("embedding", index_params)
# Insert
collection.insert([ids, embeddings, titles])
collection.load()
# Search
results = collection.search(
data=[query_embedding],
anns_field="embedding",
param={"ef": 64},
limit=10,
output_fields=["title"]
)
Уровень 3: Self-hosted vs Managed
Self-hosted
Self-hosted:
- Qdrant (Docker)
- Chroma (local)
- Milvus (K8s)
- Weaviate (Docker)
Плюсы:
✓ Полный контроль
✓ Нет cost per vector
✓ Data privacy
✓ Custom configuration
Минусы:
✗运维 overhead
✗ Scaling manually
✗ Backup manually
Managed Services
Managed:
- Pinecone
- Weaviate Cloud
- Qdrant Cloud
- Chroma Cloud
Плюсы:
✓ Zero运维
✓ Auto scaling
✓ Backup included
✓ HA out of the box
Минусы:
✗ Cost per vector
✗ Data leaves your infra
✗ Vendor lock-in
Уровень 4: Hybrid Search
Text + Vector
Pure vector search:
"bank" → finds "river bank", "bank account"
✗ Не точный для exact keywords
Pure text search:
"bank" → finds all docs with "bank"
✗ Не понимает синонимы
Hybrid search:
0.3 * text_score + 0.7 * vector_score
✓ Best of both worlds
BM25 + Vector
class HybridSearch:
def __init__(self, vector_db, text_db):
self.vector_db = vector_db
self.text_db = text_db
def search(self, query: str, top_k: int = 10):
# Text search (BM25)
text_results = self.text_db.search(query, top_k=50)
# Vector search
vector_results = self.vector_db.search(query, top_k=50)
# Reciprocal rank fusion
scores = {}
for i, doc in enumerate(text_results):
scores[doc.id] += 1 / (i + 60)
for i, doc in enumerate(vector_results):
scores[doc.id] += 1 / (i + 60)
# Sort and return top_k
return sorted(scores.items(), key=lambda x: -x[1])[:top_k]
RRF (Reciprocal Rank Fusion)
RRF: объединяем ранги из разных источников
score(doc) = Σ 1 / (k + rank_i(doc))
k = 60 (default)
rank_i(doc) = rank of doc in result set i
Пример:
Doc A: rank 3 in text, rank 7 in vector
score = 1/(60+3) + 1/(60+7) = 0.0159 + 0.0149 = 0.0308
Doc B: rank 10 in text, rank 5 in vector
score = 1/(60+10) + 1/(60+5) = 0.0143 + 0.0154 = 0.0297
Doc A wins! Better combined ranking.
Уровень 5: Metadata Filtering
Filtered Search
Filter → Vector Search → Results
where = {
"category": {"$eq": "tech"},
"date": {"$gte": "2025-01-01"},
"author": {"$in": ["Alice", "Bob"]}
}
Реализация:
1. Pre-filter: filter IDs first, then search
✓ Точно
✗ Может не найти ничего
2. Post-filter: search, then filter results
✓ Всегда находит top-K
✗ Может не хватить релевантных
3. HNSW with predicates (Qdrant)
✓ Combines index + filter
✓ Best performance
Qdrant Payload Index
client.create_payload_index(
collection_name="documents",
field_name="category",
field_schema="keyword" # exact match
)
client.create_payload_index(
collection_name="documents",
field_name="date",
field_schema="integer" # range queries
)
# Now filtered search uses index
results = client.search(
query_vector=embedding,
query_filter=Filter(
must=[
FieldCondition(
key="category",
match=MatchValue(value="tech")
)
]
)
)
Уровень 6: Scaling
Sharding
100M vectors, 768 dim, float32:
Size: 100M * 768 * 4 bytes = 307 GB
One machine:
Memory: 307 GB (need ~512GB RAM)
Search: slow
Sharding (4 nodes):
Each: ~77 GB
Search: 4x faster (parallel)
Sharding strategies:
- By ID: hash(id) % num_shards
- By metadata: category-based
- Round-robin: even distribution
Replication
Replication = copies for HA
Primary: read + write
Replica 1: read only
Replica 2: read only
Benefits:
✓ Read scaling
✓ Fault tolerance
✓ Zero-downtime updates
Trade-off:
✗ More storage
✗ Replication lag
Уровень 7: Real-world Patterns
RAG Pipeline
class RAGPipeline:
def __init__(self, vector_db, embedding_model):
self.db = vector_db
self.model = embedding_model
def ingest(self, documents: list[dict]):
"""Добавляем документы в базу"""
embeddings = self.model.encode([d['text'] for d in documents])
self.db.upsert(
ids=[d['id'] for d in documents],
vectors=embeddings.tolist(),
payloads=[{k: v for k, v in d.items() if k != 'text'}]
)
def retrieve(self, query: str, top_k: int = 5) -> list[dict]:
"""Извлекаем релевантные документы"""
query_embedding = self.model.encode(query).tolist()
hits = self.db.search(
query_vector=query_embedding,
limit=top_k
)
return [
{"id": h.id, "text": h.payload['text'], "score": h.score}
for h in hits
]
def answer(self, query: str) -> str:
"""RAG: retrieve + generate"""
docs = self.retrieve(query)
context = "\n\n".join([d['text'] for d in docs])
prompt = f"""
Ответь на вопрос используя контекст:
<context>{context}</context>
<question>{query}</question>
"""
return llm.generate(prompt)
Incremental Updates
Batch ingestion:
1. Collect documents
2. Generate embeddings
3. Upsert to vector DB
4. Repeat every N minutes
Real-time:
1. Document updated
2. Generate embedding (async)
3. Upsert to vector DB
4. Notify consumers
Challenges:
- Stale embeddings during generation
- Duplicate documents
- Deleted documents
Уровень 8: Monitoring
Key Metrics
Metrics:
1. Index size: number of vectors
2. Index memory: RAM usage
3. Query latency: p50, p95, p99
4. Recall: % of true NN in results
5. Throughput: queries/sec
Alerts:
- Latency > 100ms (p99)
- Recall < 95%
- Memory > 80%
A/B Testing
def compare_databases(db_a, db_b, test_queries, ground_truth):
results_a = []
results_b = []
for query in test_queries:
res_a = db_a.search(query, top_k=10)
res_b = db_b.search(query, top_k=10)
recall_a = recall_at_k(res_a, ground_truth[query], k=10)
recall_b = recall_at_k(res_b, ground_truth[query], k=10)
results_a.append(recall_a)
results_b.append(recall_b)
print(f"DB A recall: {sum(results_a)/len(results_a):.2%}")
print(f"DB B recall: {sum(results_b)/len(results_b):.2%}")
Уровень 9: Embedding Compression
Dimensionality Reduction
768-dim → 256-dim:
- PCA: principal component analysis
- Truncated SVD: faster
- Autoencoder: best quality
Quality trade-off:
768-dim: 100% recall
512-dim: 98% recall, 0.67x size
256-dim: 95% recall, 0.33x size
128-dim: 90% recall, 0.17x size
Quantization
Float32 → Int8:
4x compression
~1-2% accuracy drop
Float32 → Binary:
32x compression
~10-15% accuracy drop
Float32 → Ternary:
10.7x compression
~3-5% accuracy drop
Example:
768-dim float32: 3072 bytes
768-dim int8: 768 bytes
768-dim binary: 96 bytes (bits)
Уровень 10: Best Practices
Model Selection
Small project (< 1M vectors):
- Chroma (simplest)
- Qdrant single-node
Medium (1M - 100M):
- Qdrant (cluster)
- Milvus
- Weaviate
Large (> 100M):
- Milvus (K8s)
- Pinecone (managed)
- Custom (FAISS + Redis)
Production Checklist
Production Vector DB:
[ ] HNSW index with proper M/ef values
[ ] Metadata indexes for filters
[ ] Replication for HA
[ ] Monitoring (latency, recall, memory)
[ ] Regular backup
[ ] A/B test different configs
[ ] Plan for scaling
[ ] Handle deletions gracefully
Заключение
Vector databases — критичная инфраструктура для RAG и semantic search.
Ключевые выводы:
- HNSW — лучший универсальный индекс
- Hybrid search (text + vector) > pure vector
- Metadata filtering — must-have для production
- Quantization даёт 4x compression с минимальной потерей
- Monitoring recall — критично для качества
Ресурсы: