Mixed Chinese–English speech recognition: our benchmark

We ran the speech engines that Vocco ships, in the exact builds the app downloads, on 300 utterances of real Mandarin–English conversation. Here are the numbers, how we got them, and where every engine still fails.

Updated October 28, 2026 · Written by the developer of Vocco

Results

ASCEND test set, first 300 utterances. Lower is better for error rates. RTF is processing time divided by audio length on an Apple M5 Pro with 4 CPU threads; below 1 means faster than real time.

95% bootstrap intervals for MER (10,000 resamples): Qwen3 0.134–0.182, Paraformer 0.146–0.195, SenseVoice 0.161–0.212, Apple 0.179–0.236. The 0.013 gap between Qwen3 and Paraformer is not statistically significant; Qwen3's lead over SenseVoice is. *Whisper rows come from our July 2026 run on the same 300 utterances (WhisperKit, Neural Engine) and were not re-run. Vocco runs Whisper with a fixed language that follows the interface language, so with a Chinese interface the second Whisper row is the one that applies. Our July write-up quoted 0.152 for Qwen3; that was the full-precision reference model, and the int8 build scores 0.157 (0.158 with the libraries in Vocco 1.0.5 and later, see Method).

By type of sentence

The 300 utterances split into 155 Chinese-only, 83 English-only and 62 mixed. MER per subset:

Qwen3's lead comes mostly from English: on English-only sentences it is ahead of Paraformer by 0.089, a significant gap. On the sentences that mix languages Paraformer is ahead (not significant), partly because Qwen3 now and then translates a whole mixed sentence into English or answers in the wrong script, as in the first examples below. Apple's recognizer is as good as anything here on pure Chinese and the weakest on English.

Where the engines go wrong

Real utterances from the test set. The reference is what was said; the outputs are unedited.

Method

Test set

ASCEND (HKUST, LREC 2022) is a corpus of spontaneous Mandarin–English conversation: 10.6 hours from 23 speakers, with many short turns and fillers. We used the first 300 utterances of the test split in the corpus's own order, not a hand-picked sample: 2,070 reference tokens, of which 1,526 are Chinese characters and 544 English words. ASCEND is licensed CC BY-SA 4.0.

Metric

Mixed error rate (MER): Chinese is scored per character and English per word, in one edit-distance alignment. The same alignment gives a Chinese character error rate (CER) and an English word error rate (WER), so you can see which half of the sentence an engine loses. Before scoring, reference and output are both normalized the same way: Traditional characters to Simplified, full-width to half-width, lowercase, punctuation removed. Spelling style is not penalized; recognition errors are. Numbers are not normalized, so "3" against "three" counts as an error for every engine.

Engines and builds

Every model file's SHA-256 was checked against its repository. Re-running Qwen3 against the libraries inside an installed copy of Vocco 1.0.4 gave the same text on all 300 utterances. Through the PyPI sherpa-onnx wheel instead, Qwen3 scores 0.159: int8 output shifts slightly between runtime builds, while SenseVoice and Paraformer do not change.

Vocco 1.0.5 and later embed onnxruntime 1.27.0, which fixes a memory leak in speaker diarization. We re-ran the same 300 utterances with those libraries on October 27, 2026: Paraformer and SenseVoice gave identical text on all 300; Qwen3 int8 changed on 73 and scored 0.158 (Chinese CER 0.113, English WER 0.285), 0.0014 worse than the table above, with a 95% interval of −0.009 to +0.013, so not a real difference. The ranking is the same. The tables and examples on this page, and the files in the repository, are from the October 22 run.

The Apple row is a baseline, not one of Vocco's engines as run in the app. Vocco's Apple Speech option uses the older SFSpeechRecognizer API, which a command-line tool cannot get permission for without a prompt. SpeechTranscriber is Apple's newest on-device speech-to-text API; it is not the Dictation feature itself.

Hardware and software

October 22, 2026. Apple M5 Pro (18 cores), 48 GB memory, macOS 26.6.2, on AC power. Each utterance was decoded whole, one at a time, after one untimed warm-up. Scoring used the same mer.py and normalize.py as our July run (self-test 16 of 16). The 300 utterance IDs, the reference texts, every engine's output and the scripts are public at github.com/leemingynng-create/vocco-asr-benchmark; one command reproduces the run.

Limits of this test

• One corpus. ASCEND speakers were recorded in Hong Kong and the conversations are casual, so absolute error rates are higher than for clear, read speech. Compare engines with each other rather than reading the numbers as what you will see on your own voice.
• All 300 utterances come from one ASCEND session with two speakers, 77 of them are one- or two-character fillers such as 嗯 or 好的, and 7 references contain an unknown-word marker that every engine gets wrong. That is enough to rank the engines, but the intervals above understate how much the numbers would move on other speech; differences of one or two points between close engines are within noise.
• Each utterance was decoded whole. In the app, live and file transcription cut audio into windows of up to about 11 seconds; that path was not measured. Utterances here are at most 10.4 seconds long.
• We tested the engines Vocco ships, not other apps. Cloud apps cannot be run on a fixed offline test set the same way.
• Single-language results are separate (August 2026, reference model weights rather than the app's int8 build): on 60 utterances each from the FLEURS test set, Qwen3 ASR scored 9.6% character error on Japanese, 4.0% character error on Korean and 8.5% word error on French, while Whisper large-v3-turbo scored 5.4%, 3.6% and 5.9%. Whisper is better on single languages; Qwen3 is better when the language changes inside a sentence.

Questions

What is MER?

Mixed error rate. The reference and the output are split into Chinese characters and English words, aligned once, and the substitutions, deletions and insertions are divided by the number of reference tokens. It is the usual metric for Mandarin–English code-switching.

Did you test Wispr Flow, Typeless or macOS Dictation?

No. Wispr Flow and Typeless transcribe in the cloud, and neither they nor macOS Dictation can be pointed at a fixed set of audio files, so the comparison would not be like for like. We included Apple's on-device recognizer, through its developer API, as the closest built-in baseline.

Can I reproduce this?

Yes. The scripts, utterance IDs, reference texts and every engine's output are at github.com/leemingynng-create/vocco-asr-benchmark. One command downloads ASCEND and the models, builds the runner and scores everything. Text results should match exactly; timings depend on your Mac.

How this page was made: all numbers come from our own runs, dated above, with the scripts and outputs described in the method. Text drafted with AI assistance and checked against the run outputs.