OpenAI Whisper: Audio and Video Transcription on a VPS

OpenAI Whisper: Audio and Video Transcription on a VPS

OpenAI Whisper is a powerful open-source speech recognition model that supports 99 languages, including Russian and Ukrainian. It runs entirely locally on a VPS without sending data to the cloud.

Installing Whisper

# Requirements: Python 3.8+, FFmpeg
apt install -y ffmpeg
pip install openai-whisper

Basic Usage (CLI)

# Transcribe an audio file
whisper audio.mp3 --language Russian --model medium

# Supported formats: mp3, mp4, wav, m4a, webm, ogg
# Models: tiny (39M), base (74M), small (244M), medium (769M), large (1.5G)

# Output in different formats
whisper audio.mp3 --output_format txt  # plain text
whisper audio.mp3 --output_format srt  # subtitles
whisper audio.mp3 --output_format vtt  # WebVTT

Python API

import whisper
import json

# Load the model (downloaded on first run)
model = whisper.load_model("medium")

# Transcription
result = model.transcribe("audio.mp3", language="ru", verbose=False)

print(result["text"])

# Get segments with timestamps
for segment in result["segments"]:
    start = round(segment["start"], 2)
    end   = round(segment["end"], 2)
    text  = segment["text"].strip()
    print(f"[{start}s - {end}s] {text}")

REST API Server with FastAPI

pip install fastapi uvicorn python-multipart

from fastapi import FastAPI, UploadFile, File
import whisper, tempfile, os

app = FastAPI()
model = whisper.load_model("small")

@app.post("/transcribe")
async def transcribe(file: UploadFile = File(...), language: str = "ru"):
    with tempfile.NamedTemporaryFile(delete=False, suffix=".mp3") as tmp:
        content = await file.read()
        tmp.write(content)
        tmp_path = tmp.name
    result = model.transcribe(tmp_path, language=language)
    os.unlink(tmp_path)
    return {"text": result["text"], "segments": result["segments"]}

uvicorn main:app --host 0.0.0.0 --port 8000

Faster-Whisper: 4x Speedup

pip install faster-whisper

from faster_whisper import WhisperModel

# Load with optimization (float16 for GPU, int8 for CPU)
model = WhisperModel("medium", device="cpu", compute_type="int8")

segments, info = model.transcribe("audio.mp3", language="ru")
for segment in segments:
    print(f"[{segment.start:.2f}s] {segment.text}")

Common Errors

The tricky part of Whisper is that it does not throw obvious errors on bad audio — it confidently "invents" text where it cannot make out the actual speech.

RuntimeError: CUDA out of memory When Loading the large Model

The large models are demanding on VRAM. If the server does not have enough video memory, use faster-whisper with compute_type="int8" on CPU, or pick a smaller model — medium gives close results for most speech transcription tasks at noticeably lower requirements.

Hallucinations and Looping on Silence and Music

During long pauses, background music, or silence, Whisper can start repeating the same phrase or "inventing" text that was never in the audio — this is a known trait of the model, not a bug in a specific setup. The risk is reduced by trimming silence before transcription (for example with ffmpeg and the silenceremove filter) and by using the condition_on_previous_text=False parameter.

The Trade-off Between Speed, Accuracy, and Memory Requirements

When Whisper Is Not a Good Fit

• You need real-time transcription with sub-second latency — base Whisper processes audio as a whole or in large chunks, and real-time use needs specialized streaming solutions.
• The audio is very noisy or recorded from several speakers at once without separated channels — recognition accuracy drops noticeably regardless of model size.
• You need "who is speaking" diarization out of the box — Whisper itself does not do this, and a separate library on top of transcription is required.
• The server has no GPU and limited CPU cores, and the volume of audio is large — processing will stretch out so much that an external API becomes the simpler option.

Verifying Transcription Quality

whisper sample.mp3 --model small --language Russian --output_format txt --verbose True

Run a short reference file with text you already know and check the result by hand. Pay attention to the --verbose True flag — it prints intermediate segments as they are recognized, and immediately shows where the model starts to drift: repeated phrases or sudden timestamp jumps point to a problem with a specific stretch of audio, not with the model configuration.