Speech Recognition: Распознавание речи с LLM
speechasrllmopensourcewhisper
Введение
Speech Recognition (ASR — Automatic Speech Recognition) — задача преобразования речи в текст. LLM и большие модели, такие как Whisper, произвели революцию в распознавании речи.
Эволюция распознавания речи
1. HMM-based (Hidden Markov Models)
Hidden Markov Model подход:
P(text|speech) ∝ P(speech|text) × P(text)
Компоненты:
- Acoustic Model (AM): P(speech|text)
- Language Model (LM): P(text)
- Pronunciation Dictionary
Пример:
Audio → [phonemes] → [words] → [text]
2. DNN-HMM
Deep Neural Network HMM:
DNN заменяет Gaussian Mixture Model
Архитектура:
Audio Features → DNN → Phone Probabilities → HMM → Words
Фичи:
- MFCC (Mel-frequency cepstral coefficients)
- Filter banks
- Delta and Delta-Delta
3. End-to-End (E2E)
End-to-End модели:
Audio → [Neural Network] → Text
Подходы:
- CTC (Connectionist Temporal Classification)
- Attention-based Seq2Seq
- Transducer (RNN-T)
4. Modern LLM-based
Modern approaches:
- Whisper (OpenAI) — multi-lingual ASR
- Whisper.cpp — C++ inference
- LLM post-processing — исправление текста
Преимущества:
- Zero-shot для 100+ языков
- Speech-to-text, translation, transcription
- Robust к шуму и акцентам
Whisper — OpenAI ASR
Установка и базовое использование
import whisper
# Загрузка модели
model = whisper.load_model("base")
# Распознавание
result = model.transcribe("audio.wav")
print(result["text"])
# → "Hello, world!"
# С деталями
print(result)
# {
# "text": "Hello, world!",
# "language": "english",
# "segments": [...]
# }
Размеры моделей Whisper
# Доступные модели:
models = ["tiny", "base", "small", "medium", "large"]
# tiny: ~39M параметров, ~1GB RAM, ~50MB модель
# base: ~74M параметров, ~1GB RAM, ~50MB модель
# small: 244M параметров, ~2GB RAM, ~150MB модель
# medium: 769M параметров, ~5GB RAM, ~500MB модель
# large: 1.5B параметров, ~10GB RAM, ~1GB модель
# Выбор модели
model = whisper.load_model("small") # баланс скорость/точность
Transcription с деталями
def detailed_transcription(audio_file):
model = whisper.load_model("medium")
result = model.transcribe(
audio_file,
verbose=True,
word_timestamps=True,
suppress_tokens="-1"
)
for segment in result["segments"]:
print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s]")
print(f" Text: {segment['text']}")
print(f" Words:")
for word in segment.get("words", []):
print(f" [{word['start']:.2f}s - {word['end']:.2f}s]: "
f"{word['word']} (p={word['probability']:.2f})")
return result
Языковое распознавание
def detect_and_transcribe(audio_file):
model = whisper.load_model("base")
# Автоопределение языка
result = model.transcribe(audio_file)
print(f"Language: {result['language']}")
print(f"Confidence: {result.get('language_probability', 'N/A')}")
print(f"Text: {result['text']}")
return result
Перевод речи на английский
def speech_to_text(audio_file, source_lang):
"""Распознавание речи с переводом на английский"""
model = whisper.load_model("medium")
result = model.transcribe(
audio_file,
language=source_lang,
task="translate" # translate to English
)
return result["text"]
# Использование
print(speech_to_text("russian_audio.wav", "ru"))
# → "Hello, world!" (даже если оригинал на русском)
Whisper.cpp — локальное распознавание
Установка
# Клонирование
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
# Сборка
mkdir build
cd build
cmake ..
cmake --build . --config Release
# Конвертация модели
../models/ggml-base.bin → ggml-base.bin
CLI использование
# Базовое распознавание
./whisper audio.wav -m models/ggml-base.bin -l ru
# С субтитрами
./whisper audio.wav -m models/ggml-medium.bin --srt
# В реальном времени
./whisper -m models/ggml-base.bin --steps 3 -t 8
# Параметры
./whisper audio.wav \
-m models/ggml-medium.bin \
-l ru \
-t 8 \ # threads
-osrt \ # output SRT
-otxt \ # output TXT
-f 10 \ # offset (seconds)
-d 30 # duration (seconds)
Python API для whisper.cpp
import whisper_cpp
# Инициализация
params = whisper_cpp_default_params()
params.n_threads = 8
params.language = "auto"
params.translate = False
ctx = whisper_cpp_from_file("models/ggml-base.bin", params)
# Распознавание
audio = load_audio("audio.wav", whisper_cpp_full_sample_rate)
result = ctx.process_audio(audio)
print(result["text"])
Real-time speech recognition
Распознавание в реальном времени
import sounddevice as sd
import numpy as np
import whisper
class RealTimeASR:
def __init__(self, model_size="base"):
self.model = whisper.load_model(model_size)
self.buffer = []
self.chunk_size = 16000 * 10 # 10 секунд
def audio_callback(self, indata, frames, time, status):
"""Callback для аудио потока"""
if status:
print(status)
self.buffer.append(indata.copy())
if len(self.buffer) >= self.chunk_size:
self.process_buffer()
def process_buffer(self):
"""Обработка буфера"""
audio = np.concatenate(self.buffer[:self.chunk_size])
result = self.model.transcribe(
audio,
fp16=False
)
print(f"Recognized: {result['text']}")
self.buffer = []
def start(self):
"""Захват аудио"""
with sd.InputStream(
channels=1,
samplerate=16000,
callback=self.audio_callback
):
sd.sleep(2**31)
# Использование
asr = RealTimeASR("base")
asr.start()
WebRTC VAD для активации
import webrtcvad
import sounddevice as sd
import numpy as np
class VoiceActivatedASR:
def __init__(self, vad_mode=3):
self.vad = webrtcvad.Vad()
self.vad.set_mode(vad_mode) # 0-3 (aggressiveness)
self.model = whisper.load_model("base")
self.is_speaking = False
self.audio_buffer = []
def detect_speech(self, audio_chunk):
"""Обнаружение речи"""
# 16kHz, 16-bit, mono
frame = audio_chunk.astype(np.uint8)
# VAD detection
is_speech = self.vad.is_speech(frame, 16000)
if is_speech and not self.is_speaking:
self.is_speaking = True
self.audio_buffer = []
print("Speech started...")
if not is_speech and self.is_speaking:
self.is_speaking = False
self.transcribe_buffer()
print("Speech ended.")
if self.is_speaking:
self.audio_buffer.append(audio_chunk)
return is_speech
def transcribe_buffer(self):
"""Распознавание буфера"""
if not self.audio_buffer:
return
audio = np.concatenate(self.audio_buffer)
result = self.model.transcribe(audio)
print(f"Result: {result['text']}")
Speaker diarization
Разделение спикеров
import torch
from pyannote.audio import Pipeline
# Загрузка модели
diarization = Pipeline.from_pretrained(
"pyannote/speaker-diarization",
use_auth_token="your_hf_token"
)
# Применение
diarization("audio.wav")
# Результат:
# [00:00.000 -> 00:03.200] SPEAKER_00: "Hello, how are you?"
# [00:03.200 -> 00:06.500] SPEAKER_01: "I'm fine, thanks!"
# [00:06.500 -> 00:09.100] SPEAKER_00: "Great to hear!"
Diarization + ASR
def transcribe_with_speakers(audio_file):
"""Распознавание с разделением спикеров"""
# 1. Diarization
diarization_pipeline = Pipeline.from_pretrained(
"pyannote/speaker-diarization"
)
diarization = diarization_pipeline(audio_file)
# 2. ASR для каждого сегмента
asr_model = whisper.load_model("medium")
segments = []
for turn, _, speaker in diarization.itertracks(yield_label=True):
audio_segment = extract_segment(audio_file, turn.start, turn.end)
result = asr_model.transcribe(audio_segment)
segments.append({
"speaker": speaker,
"start": turn.start,
"end": turn.end,
"text": result["text"]
})
return segments
Speech-to-text API
OpenAI Whisper API
from openai import OpenAI
client = OpenAI()
def whisper_api_transcribe(audio_file):
"""Использование Whisper API"""
with open(audio_file, "rb") as f:
transcript = client.audio.transcriptions.create(
model="whisper-1",
file=f,
language="ru",
response_format="text"
)
return transcript
# С JSON response
def whisper_api_detailed(audio_file):
transcript = client.audio.transcriptions.create(
model="whisper-1",
file=audio_file,
response_format="verbose_json",
timestamp_granularities=["segment"]
)
return {
"text": transcript.text,
"language": transcript.language,
"duration": transcript.duration,
"segments": transcript.segments
}
Google Cloud Speech-to-Text
from google.cloud import speech
def google_speech_transcribe(audio_file):
client = speech.SpeechClient()
with open(audio_file, "rb") as f:
audio = speech.RecognitionAudio(content=f.read())
config = speech.RecognitionConfig(
encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=16000,
language_code="ru-RU",
enable_automatic_punctuation=True,
model="default",
use_enhanced=True
)
response = client.recognize(config=config, audio=audio)
for result in response.results:
print(f"Transcript: {result.alternatives[0].transcript}")
print(f"Confidence: {result.alternatives[0].confidence}")
Fine-tuning ASR
Fine-tuning Whisper
import datasets
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from transformers import WhisperTrainingArguments, WhisperTrainer
# Загрузка данных
dataset = datasets.load_dataset("librispeech_asr", "all")
# Загрузка модели
processor = WhisperProcessor.from_pretrained("openai/whisper-small")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
# Подготовка данных
def prepare_audio(batch):
audio = batch["audio"]
input_features = processor(
audio["array"],
sampling_rate=audio["sampling_rate"],
return_tensors="pt"
).input_features
with processor.as_target_processor():
labels = processor(batch["text"]).input_ids
batch["input_features"] = input_features
batch["labels"] = labels
return batch
dataset = dataset.map(prepare_audio)
# Обучение
training_args = WhisperTrainingArguments(
output_dir="./whisper-finetuned",
per_device_train_batch_size=8,
learning_rate=1e-4,
num_train_epochs=5,
warmup_steps=500,
fp16=True,
)
trainer = WhisperTrainer(
model=model,
args=training_args,
train_dataset=dataset["train"],
processor=processor,
)
trainer.train()
trainer.save_model("./whisper-finetuned-final")
Fine-tuning для домена
# Медицинская транскрипция
medical_dataset = load_medical_audio_dataset()
# Финансовая транскрипция
financial_dataset = load_financial_audio_dataset()
# Юридическая транскрипция
legal_dataset = load_legal_audio_dataset()
# Fine-tuning улучшает терминологию домена
model = whisper.load_model("medium")
model.finetune(domain_dataset)
Практические применения
1. Транскрипция подкастов
def transcribe_podcast(audio_file):
"""Транскрипция подкаста"""
model = whisper.load_model("medium")
result = model.transcribe(
audio_file,
word_timestamps=True,
verbose=True
)
# Генерация SRT
generate_srt(result, "podcast.srt")
# Генерация HTML с таймкодами
generate_html(result, "podcast.html")
return result
2. Виртуальный помощник
class VoiceAssistant:
def __init__(self):
self.asr = whisper.load_model("base")
self.tts = pyttsx3.init()
self.llm = OpenAI()
def listen(self):
"""Слушать голос"""
with sd.InputStream(samplerate=16000, channels=1):
audio = sd.rec(int(16000 * 5))
sd.wait()
result = self.asr.transcribe(audio)
return result["text"]
def respond(self, text):
"""Голосовой ответ"""
self.tts.say(text)
self.tts.runAndWait()
def run(self):
"""Основной цикл"""
while True:
query = self.listen()
response = self.llm.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": query}]
)
self.respond(response.choices[0].message.content)
3. Субтитры для видео
def add_subtitles(video_file, language="ru"):
"""Добавление субтитров к видео"""
# Распознавание
model = whisper.load_model("medium")
result = model.transcribe(video_file)
# Генерация SRT
srt_content = generate_srt_content(result)
save_file("subtitles.srt", srt_content)
# Встраивание в видео
subprocess.run([
"ffmpeg", "-i", video_file,
"-vf", "subtitles=subtitles.srt",
"-c:v", "libx264",
"-c:a", "aac",
"video_subtitled.mp4"
])
4. Meeting notes
def transcribe_meeting(audio_file):
"""Транскрипция встречи"""
# Diarization
diarization = get_speaker_diarization(audio_file)
# ASR
model = whisper.load_model("medium")
segments = []
for turn, _, speaker in diarization.itertracks(yield_label=True):
audio_segment = extract(audio_file, turn.start, turn.end)
result = model.transcribe(audio_segment)
segments.append({
"speaker": speaker,
"text": result["text"],
"start": turn.start,
"end": turn.end
})
# Генерация заметок
notes = generate_meeting_notes(segments)
return {
"transcript": segments,
"notes": notes,
"action_items": extract_action_items(notes)
}
Open Source инструменты
Модели:
✅ Whisper (OpenAI) — 100+ языков
✅ Whisper.cpp — C++ inference
✅ wav2vec 2.0 (Meta)
✅ HuBERT (Meta)
✅ Conformer (Google)
Библиотеки:
✅ whisper.py (OpenAI)
✅ speechrecognition (Python)
✅ Kaldi
✅ ESPnet
API:
✅ OpenAI Whisper API
✅ Google Cloud Speech
✅ Azure Speech
✅ AWS Transcribe
Итоги
Speech Recognition с Whisper и LLM стала значительно доступнее. Локальные модели позволяют транскрибировать приватные данные, а API — для быстрого прототипирования.
Ключевые выводы:
- Whisper — лучший open-source ASR для 100+ языков
- Whisper.cpp — быстрый локальный inference
- Real-time ASR возможен с правильным VAD
- Speaker diarization разделяет спикеров
- Fine-tuning улучшает для специфичных доменов
- API удобен для production