Understanding Whisper Hallucinations: A Guide to Improving Transcription Accuracy

Whisper, OpenAI's automatic speech recognition (ASR) model, has been making waves lately. Recently, a user shared an experience that highlights a significant issue with this technology. They received feedback indicating they had said, "Thanks for watching," which they never actually uttered. This discrepancy is a result of what are known as Whisper hallucinations, where the model generates text during silent moments in audio recordings.

What Are Whisper Hallucinations?

Whisper hallucinations occur when the model fills silence with phrases it has learned from its training data. In this case, the user paused for about four seconds before stopping their response. Whisper, however, interpreted that silence as an opportunity to generate text, resulting in the inaccurate feedback.

In an analysis of 2,000 voice clips, it was found that 9.2% contained hallucinated text. This means that nearly one in eleven clips had phrases that were not actually spoken. The issue is particularly pronounced in clips with more than three seconds of trailing silence, where the hallucination rate jumped to 23%.

Why Do These Hallucinations Happen?

The underlying mechanism is tied to how Whisper's decoder works. It was trained on a vast amount of web audio data, where phrases like "Thanks for watching" are common during quiet moments, such as video outros. When the model encounters silence, it defaults to generating these familiar phrases, leading to inaccuracies.

The model operates on 30-second windows, meaning it looks for patterns in audio. When it finds stretches of silence, it may produce text that fits the context, even if it’s not accurate. Most of the time, it emits nothing or a no-speech token, but sometimes it confidently generates a phrase that sounds plausible.

Limitations of Built-in Thresholds

One might think that the no_speech_threshold parameter could help in detecting silence and preventing hallucinations. However, this is not always effective. The model can still produce text even when it is not confident about the silence. In fact, the analysis showed that many hallucinations occurred despite a high no-speech probability.

How to Fix the Issue

To tackle the hallucination problem, a few adjustments can be made. The key is to stop sending silence to the model altogether. A built-in filter called Silero VAD (Voice Activity Detection) can help strip out non-speech regions before decoding. Here’s a simple configuration that can be used:

from faster_whisper import WhisperModel
model = WhisperModel("large-v3", device="cuda", compute_type="float16")

def transcribe_answer(path: str) -> str:
    segments, _ = model.transcribe(
        path,
        vad_filter=True,
        vad_parameters=dict(min_silence_duration_ms=500),
        condition_on_previous_text=False,
    )
    kept = []
    for seg in segments:
        if seg.no_speech_prob > 0.5:
            continue  # Skip confident ghosts
        kept.append(seg.text.strip())
    return " ".join(kept)

This configuration can significantly reduce the number of hallucinations while still capturing real speech.

Conclusion

Whisper's hallucination issue is a notable challenge for users relying on accurate transcriptions. By understanding how these hallucinations occur and implementing strategies to mitigate them, we can improve the reliability of automatic speech recognition systems.

Merits

• Improved accuracy in transcriptions by reducing hallucinations.
• Enhanced user experience in applications relying on speech recognition.
• Better understanding of the limitations of ASR models.

Demerits

• Requires technical knowledge to implement fixes.
• May still encounter some limitations in very quiet audio.
• Adjustments could lead to loss of some real speech if not configured carefully.

Caution

This article is intended for educational purposes. Any placeholder values in the code examples must be replaced with actual data for practical use. Readers should verify claims against the original source before relying on them.

Frequently asked questions

• What are Whisper hallucinations? — They are instances where the Whisper model generates text during silent moments in audio recordings.
• How often do these hallucinations occur? — In a recent analysis, 9.2% of audio clips exhibited hallucinated text.
• What causes these hallucinations? — They occur because the model fills silence with common phrases it learned during training.
• Can the hallucination issue be fixed? — Yes, by using a Voice Activity Detection filter to remove silence before processing.
• What is the no_speech_threshold? — It is a parameter intended to detect silence, but it may not effectively prevent hallucinations.
• How can I improve transcription accuracy? — Implementing filters and adjusting model configurations can help reduce errors.

Tags

#ai #machinelearning #whisper #speechrecognition #hallucinations #transcription #technology #OpenAI #ASR #VAD