speech-recognition in Action: Making Agents Understand Speech, Not Just Transcribe

Why You Need This Skill

Here's what happened. Last Friday afternoon, Franky sent a voice message to the work group from his car: "List me the articles scheduled for next week."

Normally, that message would just sink into the chat, ignored. But with the speech-recognition skill, it became this:

Voice message → auto-transcribed → Agent recognizes intent → pulls schedule from CMS → replies with the list.

Total time: 4 seconds. Franky didn't even have to pull over.

That's the real value of speech-recognition in OpenClaw—not just "turning speech into text," but enabling Agents to respond to voice input, breaking through the last barrier of human-computer interaction.

Installation and Basic Configuration

Installation is straightforward—one command:

clawhub install speech-recognition

After installation, add this to your OpenClaw config:

skills:
  speech-recognition:
    provider: whisper
    model: base
    language: auto

Here's a critical choice: local Whisper or cloud API?

My direct advice: if you're processing Chinese speech, use the large-v3 model. The base model's Chinese recognition rate is around 65%, while large-v3 hits 93%+. The trade-off is memory—large-v3 needs about 3GB of VRAM/RAM. Our MS01 server runs large-v3 without issues, but if your machine is older, the small model is a reasonable compromise.

Real-World Scenario: Auto-Processing Voice Messages

This is our actual usage at SFD Lab.

When OpenClaw's Telegram bot receives a voice message, the speech-recognition skill automatically intercepts it, converts speech to text, then passes it to the Agent's decision layer.

Detail 1: Audio format conversion. Telegram sends voice as OGG, which Whisper natively supports. But if you're using other platforms (WeChat, Slack), you may need to convert to WAV or MP3 first. We added a format detection layer in the skill that auto-determines whether ffmpeg conversion is needed.

Detail 2: Long audio segmentation. Whisper has limits on single audio segment length. For voice over 30 seconds, we auto-split into 25-second chunks and process separately, then stitch the results. The catch: Whisper's output at segment boundaries may have duplicate words. Our solution: compare the last 5 words of one segment with the first 5 of the next, and deduplicate overlaps.

def merge_segments(segments):
    result = segments[0]
    for i in range(1, len(segments)):
        overlap = find_overlap(result[-50:], segments[i][:50])
        result += segments[i][overlap:]
    return result

Detail 3: Speaker diarization. If multiple people speak in one audio segment, Whisper can't separate speakers by itself. We pair it with pyannote.audio for speaker labeling, but this significantly increases processing time. Skip it if you only need transcription.

Pitfall Log

Three real-world pitfalls:

Pitfall 1: Background noise kills recognition rates. Franky once sent a voice message from a café—recognition dropped to 40%. Solution: add a noise reduction preprocessing step using the noisereduce library before feeding audio to Whisper. This brought accuracy back to 85%, with a ~1.5 second processing overhead.

Pitfall 2: Mixed Chinese-English speech gets garbled. This is the most common scenario for Chinese users. Whisper large-v3 handles this decently, but there's a trick: specify the primary language in the prompt parameter, like prompt="The following is primarily Chinese speech". The model will prioritize Chinese grammar structure, treating English terms as loanwords.

Pitfall 3: Memory leaks. If you're using local Whisper for high-frequency speech processing, don't reinitialize the model every time. We made the model instance a global singleton, loading it once at startup. Large-v3 takes 10-15 seconds per load—without caching, user experience falls apart.

Combining with edge-tts

speech-recognition + edge-tts is our most-used combo. speech-recognition handles "listening," edge-tts handles "speaking." Together, the Agent gets full voice interaction capability.

Real workflow: user sends voice on Telegram → speech-recognition transcribes → Agent understands and responds → edge-tts converts reply to voice and sends it back. Zero typing required.

Pairing with smart-web-scraper

Another practical combo: voice commands + web scraping. Say "check today's trending Python projects on GitHub," speech-recognition transcribes it, the Agent calls smart-web-scraper to fetch GitHub Trending, then returns results.

This "voice command → auto-execute → return results" pattern is where speech recognition truly delivers value.

SFD Editor's Note

While testing speech-recognition today, the little parrot asked: "If an Agent can understand speech, does that mean it has ears?"

I thought about it and answered: "Not really. Ears are just hardware—understanding is what matters. Whisper's strength isn't mechanically converting sound waves to text. It actually 'understands' the semantics of speech."