I tried building a live stream translator. The whole thing hinged on where to cut the audio.

There's a category of stream I keep landing on and can't follow: the game is legible, the chat is legible, and the person talking isn't. Japanese and Chinese ones mostly. Auto-generated captions on YouTube are hit and miss and don't exist at all on a Twitch live channel, so the options are "watch it as silent film" or "build something."

I built livetl. It points at a Twitch or YouTube URL and prints translated subtitles in the terminal, into a browser overlay, or into a text file OBS can read. Everything runs locally by default: Whisper for recognition, Marian or NLLB for translation, Google Cloud Translation available if you'd rather pay than wait.

The pipeline is short enough to fit on one line:

streamlink/yt-dlp -> ffmpeg -> 16kHz PCM -> Silero VAD -> Whisper -> MT -> sinks

I assumed the hard part would be model quality. It wasn't. The hard part was deciding where to cut the audio, and after that, a run of bugs that never announced themselves, because nothing anywhere in the pipeline errored.

Whisper is not a streaming model

The obvious first design is a timer. Buffer five seconds of audio, transcribe it, translate it, print it, repeat. It's about fifteen lines and it produces garbage.

Whisper takes a 30-second window and transcribes what's inside it. It has no concept of "this clip continues from the last one." So a fixed timer cuts mid-word, and the recognizer does its best with half a word at each boundary and no context for either. Then it hands the translator a sentence fragment, and machine translation on a fragment is worse than machine translation on a bad sentence, because now the model is also guessing at grammar that isn't there.

So the cut points have to come from the audio itself. Silero VAD scores each 32ms frame for speech probability, and the segmenter opens a buffer when the score crosses a threshold and closes it after enough trailing quiet:

if prob < cfg.threshold - 0.15:
    self._silence_frames += 1
else:
    self._silence_frames = 0

That - 0.15 is hysteresis, and it exists because a plain threshold splits utterances mid-word. Speech probability wobbles, especially across a fricative or a breath, and a single dipped frame shouldn't end a sentence. Getting into a segment takes a higher score than staying in one.

Two more rules sit on top of the basic gate. A hard cut at twelve seconds, for someone who genuinely never pauses, which replays 200ms of overlap into the next segment so a word straddling the cut still lands whole in one half. And a soft cut, which is the one that matters in practice:

needed_silence = (
    cfg.soft_silence_ms if duration >= cfg.soft_max_s else cfg.min_silence_ms
)

Past six seconds of continuous speech, a 240ms gap counts as a boundary instead of the usual 550ms. A comma instead of a period. Without it, a streamer on a monologue sits at ten seconds or more before a single caption appears, because latency here tracks utterance length: a sentence can't be translated until it's been spoken.

That's the floor, and it's structural. End-to-end lag is roughly utterance length plus 1.4 seconds, where the fixed part is pause detection plus ASR plus MT. The 1.4 is negotiable, by way of a smaller model or a faster backend. The rest of it is waiting for somebody to finish their sentence.

The ffmpeg flag that looked like a latency win

Early on I put -fflags nobuffer on the ffmpeg decode. It's in every low-latency ffmpeg incantation on the internet and this is a low-latency application, so in it went.

It discards already-demuxed packets. On a local test file I measured it eating 3.2 seconds of a 10.4 second clip, silently, off the front. Nothing errors, and unless you happen to be checking durations against the source you have no reason to look.

# NOT -fflags nobuffer: it discards already-demuxed packets and
# silently eats the first seconds of audio (measured: 3.2s of a
# 10.4s file). Live-edge latency is the puller's job, not ffmpeg's.
"-flags", "low_delay",

Live-edge latency belongs upstream anyway. For Twitch that's --twitch-low-latency on streamlink, which also gets --twitch-disable-ads so ad reads don't get transcribed as content.

The translation that never happened

This one cost the most time and produced the least evidence.

Captions were appearing, timings looked fine, and the Spanish was coming out as Spanish. Not badly translated. Identical, character for character.

mlx_whisper.transcribe() decodes in the correct language. It also reports "en" in its result dict, because that's the DecodingOptions default and the field never gets updated with what was actually detected. I was reading that field and passing it to the translator as the source language. The translator has a reasonable shortcut:

if src == tgt:
    return text

Source en, target en, nothing to do. Every stage reported success, and the timing output showed a translation happening on every single utterance, because one had been dispatched and had returned.

The fix is to detect the language explicitly rather than trust the result dict, and it has two traps of its own:

# Share transcribe()'s cached instance. Calling load_model() directly
# would hold a second full copy of the weights, at a different dtype.
if self._model is None:
    self._model = ModelHolder.get_model(self.repo, mx.float16)
# Pad the *audio* to a full 30s window, not the mel: zero-padding a
# log-mel produces values that never occur in a real spectrogram.
mel = log_mel_spectrogram(audio, n_mels=self._model.dims.n_mels, padding=N_SAMPLES)
dtype = self._model.encoder.conv1.weight.dtype
mel = pad_or_trim(mel, N_FRAMES, axis=-2).astype(dtype)

Zero-padding a log-mel spectrogram is the subtler of the two. It runs, it returns a language, and the values you padded with are ones no real audio ever produces, so you're asking the model to classify something outside its world. Pad the waveform and compute the mel from that.

The same shape of bug showed up again in transformers. from_pretrained's torch_dtype was renamed to dtype in 4.56, and because it takes **kwargs, passing the old spelling raises nothing at all. It lands in the config and the model loads at full precision, quietly doubling its memory footprint. Colab still ships 4.x, so I check the version and pick the spelling rather than forcing an upgrade.

When the translator gets stuck in a loop

ASR output is disfluent by nature: no punctuation where you'd want it, repeated filler words, the occasional half-word. Feed that into a small seq2seq model with greedy decoding and it will sometimes fall into a cycle it can't leave. A live Japanese caption came back as "a year to a year to a year..." at 457 characters.

GEN_KWARGS = dict(
    num_beams=1,
    max_new_tokens=256,
    no_repeat_ngram_size=4,
    repetition_penalty=1.15,
)

no_repeat_ngram_size is the one that does the work. The repetition penalty just discourages the loop earlier. That same caption came out at 56 characters afterwards.

Whisper has its own version of this, and a second habit besides. Over silence and music it hallucinates confident text, and it hallucinates the same text, because the training data was subtitle files: "Thank you for watching," "please subscribe," "ご視聴ありがとうございました," and a good number of subtitle-site credits. On a stream with a background music bed, that's most of what you'd be translating. So there's a blocklist, plus a regex for decoder repetition loops:

_REPEAT = re.compile(r"^(.{1,20}?)\1{3,}$", re.S)

The filtered count shows up in the exit summary, so it's visible rather than magic. It's a blunt instrument and I'm fine with that. The alternative is captioning applause.

Falling behind live is the one unrecoverable state

Every other failure here is local. A dropped caption is a dropped caption. But if ASR takes longer than realtime, the backlog grows every second and never shrinks, so the captions drift further behind the stream for as long as it keeps running, with no mechanism anywhere to catch back up.

So the queues between stages are bounded, and full means shed, not wait:

def _offer(self, s: Segment) -> None:
    """Enqueue a segment, shedding load instead of falling behind live."""
    try:
        self._seg_q.put_nowait(s)
        return
    except queue.Full:
        pass
    if not s.final:
        self.stats.dropped += 1
        return

Partials go first, since they're previews of something that's coming anyway. If it's a final and the queue is still full, the oldest item gets dropped, not the newest. Dropping the newest would keep the queue just as tidy while leaving you even further behind live, which is the thing the whole arrangement exists to prevent.

Two other things in that file exist because of bugs, and both are ordering problems rather than logic problems.

warmup() runs the models on a second of silence and a throwaway sentence before the stream opens. Without it, the first real utterance pays the model load and graph construction, measured at twelve seconds for NLLB, by which point the queues have already started shedding the beginning of the broadcast. The warmup call also has to use a source language that differs from the target, or translate() hits that same src == tgt shortcut and warms nothing.

And the workers end on a sentinel object, never on the stop event:

if item is _STOP:
    break

The version that checked self._stop inside the loop discarded everything still queued the instant capture ended. With partials on, the backlog is deep enough that this lost every caption from the tail of a stream. An interrupted run is bounded by a join timeout in run() instead, because that case is different: a user pressing Ctrl-C wants the process gone, not another twenty seconds of backlog.

Whisper can translate on its own, and shouldn't

Whisper has a task="translate" mode that goes straight to English without a separate MT stage. One model instead of two, and no language pair to resolve. It sounds like it should delete half the project.

In testing, whisper-small in fp16 ignored the instruction outright and returned untranslated Spanish. The q4 build did try, and rendered "bienvenidos al directo de hoy" as "Welcome to the Diary of the Huy."

It's a soft instruction that small models drop, and on the occasions they do honour it, the output is English-only and worse than an actual MT model would give you. --mt whisper still exists as an explicit flag, on the theory that a bigger model might hold the instruction better than small does. --mt auto will never pick it.

A GPU doesn't make this faster in the way you'd expect

I put this on a Colab T4 assuming VRAM would be the constraint, and it isn't remotely. large-v3 in float16 is about 3 GB out of 15.

The real constraint is that this pipeline is latency-bound, not throughput-bound. It handles one utterance, then waits for a human to say the next one. A T4 running large-v3-turbo sits idle most of the time. Using the card fully doesn't mean pushing more audio through it, it means spending the idle time on accuracy: the full large-v3 instead of turbo, --beam-size 5 instead of greedy, the 1.3B NLLB instead of the 600M.

What you can't do is stack all of it plus --partials. I measured that combination on a T4 and caption lag climbed past twenty seconds and kept climbing. Partials fire every 1.2 seconds of speech, so they multiply the number of ASR calls no matter how cheap each call is, and paying beam search for output that gets discarded within a second starves the finals people are actually reading. Partials now always decode greedily regardless of --beam-size:

beam_size=1 if quick else self.beam_size,
temperature=0.0 if quick else [0.0, 0.2, 0.4],

Which leaves two profiles rather than one dial: accuracy, or responsiveness. The README documents both and says plainly that you pick one.

Partials had a second problem I didn't anticipate. They're an in-place preview, redrawn by rich's Live as the utterance grows, and that needs a terminal. In a Colab cell, a pipe, or a log file there's nothing to redraw, so every partial got appended as its own line and hundreds of near-identical fragments buried the finished captions they were previewing. They're now suppressed with a one-line note when stdout isn't a terminal.

Colab breaks most of the assumptions

Colab got a separate launcher rather than flags on the main one, because almost every assumption changes.

No venv, deliberately. Colab ships torch built against its exact driver, and creating a venv pulls a second multi-gigabyte torch that may not match. LD_LIBRARY_PATH has to be exported for CTranslate2, which dlopens cuBLAS and cuDNN at runtime and on Colab finds them inside pip's nvidia-* packages rather than a system CUDA install. Skip that and it reports zero GPUs and silently runs on CPU, which cannot keep up with a live stream.

The one I didn't see coming: pip install --upgrade is the hazard on Colab, not the safeguard. Upgrading pulled rich and websockets past ceilings that several preinstalled packages pin (google-adk and langgraph-sdk want websockets<16, bigframes wants rich<14, pyiceberg rich<15), and left pip printing a wall of resolver conflicts in a runtime that had been fine a minute earlier. Dropping --upgrade and pinning both into the ranges Colab's stack accepts fixed it, and re-running setup now repairs a runtime the earlier version broke.

Where it's rough

Per-utterance language detection is unreliable on short clips, because Whisper decides from a single 30-second window and a 1.5 second utterance doesn't fill one. Synthetic speech is especially bad: both backends independently called macOS say Spanish "English" at 0.9 confidence, while real speech detects fine. Pass --source whenever you know it, which is nearly always.

Chinese drifts between Traditional and Simplified, sometimes inside one stream, because each utterance is recognised independently and the model picks a script per window. --initial-prompt with a sentence in the script you want anchors it, and that's a workaround, not a fix.

8 GB is tight and NLLB is what makes it tight. Whisper small plus NLLB fits on paper, but under memory pressure I measured ASR degrading from RTF 0.08x to 1.24x, which is slower than realtime, at which point the load shedding above starts earning its keep. On 8 GB the honest configuration is --source xx --mt marian.

The overlay has no authentication at all. --overlay-host 0.0.0.0 serves it to anything that can reach the port, so it's behind a firewall, behind a reverse proxy, or tunnelled over SSH, and there's nothing in the code to stop you from doing none of those.

And the quality ceiling is real: this is machine translation stacked on machine transcription, so errors compound across two stages, and streamer slang and proper nouns are exactly what neither model has good coverage of. It's good enough to follow what's happening on a stream. It is not good enough to quote anyone.

What I'd keep if I started over is the segmenter. I never sat down and A/B tested this properly, so treat it as an impression rather than a measurement, but every later upgrade I made, bigger Whisper models, beam search, the 1.3B NLLB, moved quality by an amount I had to look for. Cutting on speech pauses instead of a timer was the difference between output I'd read and output I'd close the terminal on. It's also the cheapest thing in the pipeline: a 2.2 MB model scoring one frame every 32 milliseconds, deciding nothing more complicated than whether the person is still talking.