Многомодальные LLM: Vision + Language модели
opensourceaillmmultimodalvisionimageit
Введение: Multimodal LLM
Проблема
Text-only LLMs are limited:
- Can't understand images
- Can't analyze charts
- Can't read documents visually
- Can't describe scenes
Solution: Multimodal LLM
- Understand text + images
- Analyze charts and diagrams
- Read and extract text from images
- Generate descriptions and answers
Факт: Multi-modal models can achieve 80%+ of GPT-4V quality with open-source alternatives.
Что такое Multimodal?
Основная идея
Multimodal Model Architecture:
Input: Model: Output:
┌──────────┐ ┌──────────────┐ ┌──────────┐
│ Text │ │ │ │ Text │
└──────────┘ │ LLM + │─────────→└──────────┘
┌──────────┐ │ Vision │ ┌──────────┐
│ Image │─────────────→│ Encoder │─────────→│ Image │
└──────────┘ │ + Projector │ │ Answer │
└──────────────┘ └──────────┘
Types of Multimodal Input
1. Text + Image:
- Image captioning
- Visual question answering
- Document understanding
2. Text + Image + Video:
- Video understanding
- Action recognition
- Temporal reasoning
3. Text + Audio:
- Speech recognition
- Audio understanding
- Music analysis
4. Text + Image + Audio:
- Full video understanding
- Multi-sensory reasoning
Vision Encoder
Image Processing
import torch
from transformers import CLIPVisionModel, AutoImageProcessor
# Load vision encoder
vision_encoder = CLIPVisionModel.from_pretrained("openai/clip-vit-large-patch14")
image_processor = AutoImageProcessor.from_pretrained("openai/clip-vit-large-patch14")
def process_image(image):
"""Process image for vision encoder"""
inputs = image_processor(
images=image,
return_tensors="pt"
)
# Get image embeddings
with torch.no_grad():
image_features = vision_encoder(
inputs["pixel_values"]
).last_hidden_state
return image_features # [batch, num_patches, hidden_size]
# Image processing pipeline:
# 1. Resize to 224x224 or 336x336
# 2. Normalize (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])
# 3. Convert to tensor
# 4. Split into patches (14x14 or 16x16)
Vision Encoder Options
Encoder Parameters Quality Speed
─────────────────────────────────────────────────────
CLIP-ViT-B/32 150M Good Fast
CLIP-ViT-L/14 300M Better Medium
CLIP-ViT-H/14 632M Best Slow
SigLIP-SO400M 400M Best Medium
EVA-02-7B 7B Excellent Very Slow
For local deployment:
- CLIP-ViT-L/14 is good balance
- SigLIP for better text-image alignment
Projection Layer
Connecting Vision to Text
class VisionProjector(nn.Module):
"""Connect vision encoder to LLM"""
def __init__(
self,
vision_hidden_size: int = 1024,
llm_hidden_size: int = 4096,
num_patches: int = 576, # 24x24 for 224x224
num_learnable: int = 64, # Learnable queries
):
super().__init__()
# Learnable queries (like attention)
self.learnable_queries = nn.Parameter(
torch.randn(num_learnable, vision_hidden_size)
)
# MLP projection
self.projection = nn.Sequential(
nn.Linear(vision_hidden_size, llm_hidden_size),
nn.GELU(),
nn.Linear(llm_hidden_size, llm_hidden_size),
)
# Cross-attention (optional)
self.cross_attention = nn.MultiheadAttention(
embed_dim=llm_hidden_size,
num_heads=8,
batch_first=True,
)
def forward(self, image_features):
"""
image_features: [batch, num_patches, vision_hidden_size]
Returns: [batch, num_output_tokens, llm_hidden_size]
"""
batch = image_features.shape[0]
# Repeat learnable queries for each batch
queries = self.learnable_queries.unsqueeze(0).repeat(batch, 1, 1)
# Cross-attention: queries attend to image features
attended, _ = self.cross_attention(
query=queries,
key=image_features,
value=image_features,
)
# Project to LLM dimension
output = self.projection(attended)
return output # [batch, num_learnable, llm_hidden_size]
Architecture Variants
1. Simple Linear Projection:
projector = nn.Linear(vision_dim, llm_dim)
- Fast to train
- Less flexible
2. MLP Projection:
projector = Linear → GELU → Linear
- Better capacity
- Standard choice
3. Q-Former (BLIP-2):
Learnable queries + cross-attention
- Most flexible
- More parameters
4. Perceiver Resampler:
Learned compression
- Good for many images
Popular Multi-modal Models
Open Source Options
Model Size Vision Text MME
────────────────────────────────────────────────────────
LLaVA-1.5 7B CLIP-ViT-L Llama-2 1485
LLaVA-1.5 13B CLIP-ViT-L Mistral 1519
LLaVA-1.6 7B SigLIP Mistral 1552
LLaVA-1.6 13B SigLIP Mistral 1574
LLaVA-NeXT 7B CLIP-ViT-H Llama-3 1634
LLaVA-NeXT 8B SigLIP Qwen2 1658
MiniCPM-V 2B EVA-02 MiniCPM 1523
Qwen2-VL 7B SigLIP Qwen2 1689
Yi-VL 34B CLIP-ViT-H Yi 1598
MME score: higher is better (1600+ is GPT-4 level)
LLaVA Architecture
LLaVA (Large Language-and-Vision Assistant):
1. Vision Encoder:
CLIP ViT-L/14 → image embeddings
2. Projector:
Linear projection to LLM space
3. LLM:
Llama-2 / Mistral / Qwen2
Training stages:
Stage 1: Freeze LLM, train projector
- Align image features to text space
- Use image-text pairs
Stage 2: Fine-tune entire model
- Train projector + LLM (optional)
- Use instruction tuning data
Training Multi-modal Models
Stage 1: Pre-training
# Pre-training: align vision to text
from transformers import AutoModelForCausalLM, Trainer
# Load components
vision_encoder = CLIPVisionModel.from_pretrained("openai/clip-vit-large-patch14")
projector = VisionProjector(vision_hidden_size=1024, llm_hidden_size=4096)
llm = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
# Freeze LLM
for param in llm.parameters():
param.requires_grad = False
# Train only projector
optimizer = AdamW(list(projector.parameters()), lr=1e-3)
# Data: image-text pairs
for image, text in dataloader:
# Vision encoding
image_features = vision_encoder(
image_processor(image, return_tensors="pt")["pixel_values"]
).last_hidden_state
# Projection
projected = projector(image_features)
# LLM forward (with projected features as context)
loss = llm(
inputs_embeds=projected,
labels=text_ids,
).loss
loss.backward()
optimizer.step()
Stage 2: Instruction Tuning
# Instruction tuning data format:
{
"image": "path/to/image.jpg",
"conversations": [
{"from": "human", "value": "What is in this image?"},
{"from": "gpt", "value": "This image shows a cat sitting on a sofa."}
]
}
# Fine-tune with instructions
for batch in instruction_dataloader:
image_features = get_image_features(batch["images"])
# Combine text and image embeddings
input_embeds = combine_text_image(
batch["text_ids"],
image_features,
llm,
)
loss = llm(
inputs_embeds=input_embeds,
labels=batch["labels"],
).loss
loss.backward()
optimizer.step()
Inference
Visual Question Answering
from transformers import AutoProcessor, AutoModelForVision2Seq
# Load LLaVA model
model = AutoModelForVision2Seq.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
processor = AutoProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
# Process input
image = Image.open("cat.jpg")
prompt = "USER: <image>\nWhat is in this image?\nASSISTANT:"
inputs = processor(text=prompt, images=image, return_tensors="pt")
# Generate
output = model.generate(**inputs, max_new_tokens=512)
# Decode
response = processor.decode(output[0], skip_special_tokens=True)
print(response)
# "This image shows a fluffy orange cat sitting on a gray sofa."
Image Captioning
def caption_image(model, processor, image_path):
"""Generate image caption"""
image = Image.open(image_path).convert("RGB")
prompt = "USER: <image>\nDescribe this image in detail.\nASSISTANT:"
inputs = processor(text=prompt, images=image, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=256)
caption = processor.decode(output[0], skip_special_tokens=True)
return caption.split("ASSISTANT:")[1].strip()
# Example output:
# "A golden retriever dog running through a field of yellow flowers"
Document Understanding
def analyze_document(model, processor, image_path, question):
"""Extract information from documents"""
image = Image.open(image_path).convert("RGB")
prompt = f"USER: <image>\n{question}\nASSISTANT:"
inputs = processor(text=prompt, images=image, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=512)
answer = processor.decode(output[0], skip_special_tokens=True)
return answer.split("ASSISTANT:")[1].strip()
# Use cases:
# - Extract text from invoices
# - Read forms and extract fields
# - Understand charts and graphs
# - OCR-like capabilities
Advanced Techniques
Multiple Images
# Handle multiple images in one prompt
def process_multiple_images(model, processor, images, question):
"""Process multiple images"""
# Create image placeholders
image_placeholders = "\n".join(["<image>" for _ in images])
prompt = f"USER: {image_placeholders}\n{question}\nASSISTANT:"
inputs = processor(text=prompt, images=images, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=512)
return processor.decode(output[0], skip_special_tokens=True)
# Use cases:
# - Image comparison
# - Before/after analysis
# - Object detection across images
Video Understanding
# Process video frames
def understand_video(model, processor, video_path, question):
"""Analyze video content"""
import cv2
# Extract frames (e.g., 1 frame per second)
cap = cv2.VideoCapture(video_path)
fps = cap.get(cv2.CAP_PROP_FPS)
frame_interval = int(fps) # 1 frame per second
frames = []
while True:
ret, frame = cap.read()
if not ret:
break
frames.append(Image.fromarray(frame[:, :, ::-1]))
# Use first N frames
frames = frames[:8]
# Process with model
prompt = f"USER: {'<image>' * len(frames)}\n{question}\nASSISTANT:"
inputs = processor(text=prompt, images=frames, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=512)
return processor.decode(output[0], skip_special_tokens=True)
Evaluation
Benchmarks
Benchmark What it measures Max Score
────────────────────────────────────────────────
MME Perception + Cognition 4125
MMBench General understanding 1000
SEED-Bench Visual reasoning 1000
MM-Vet Complex reasoning 861
LLaVA-Bench Conversation quality 100
Target scores for good model:
MME: > 1500
MMBench: > 650
SEED-Bench: > 700
Comparison
Model MME MMBench SEED
────────────────────────────────────────────────
GPT-4V 2328 75.8 78.2
LLaVA-NeXT 1634 68.4 72.1
Qwen2-VL 1689 71.2 74.5
MiniCPM-V 1523 65.3 69.8
Заключение
Multi-modal LLMs extend text models to vision:
Key components:
- Vision encoder (CLIP, SigLIP)
- Projection layer (linear, MLP, Q-Former)
- LLM backbone (Llama, Mistral, Qwen)
Popular models:
- LLaVA: best open-source quality
- Qwen2-VL: best benchmark scores
- MiniCPM-V: smallest, fastest
Use cases:
- Visual question answering
- Image captioning
- Document understanding
- Chart analysis
- OCR-like extraction