A Cantonese OCR benchmark

01 · Motivation

Written Cantonese uses characters mainstream OCR never learned.

Gaps in existing resources

• No image dataset. There are twenty-odd Cantonese datasets on HuggingFace, all of them text or audio. We did not find one that provides Cantonese OCR data as images.
• Existing benchmarks measure language understanding. Recent Cantonese benchmarks (CantoNLU, HKCanto-Eval) evaluate comprehension and cultural knowledge. We did not find a public benchmark for visual recognition.
• Some Cantonese characters are missing from fonts entirely. Across the 33 open-licensed font instances we checked, 𠮶 (U+20BB6) is supported by none. A character no font carries is one a model never had a chance to learn.

The harder problem is not the rare characters, but the ones that look like common ones. 冇 (“to not have”) differs from 有 (“to have”) by two strokes and means the opposite. Misread one character and the sentence inverts.

What this project contributes

• A reproducible Cantonese OCR dataset. Cantonese characters, words and sentences are all rendered as images for model evaluation and training. Three tiers (character, word, sentence) covering 154 Cantonese-specific characters, with a fixed random seed so the same corpus produces bit-identical output.
• A design that separates two kinds of failure. The same character appears in both the character tier and the sentence tier, which distinguishes “could not see it” from “knew the shape but was overridden by language habit”.
• An openly licensed generation pipeline. Every corpus and font has an explicit licence, so the resulting dataset can be distributed publicly.
• Baseline scores for two vision-language models as a starting point for later comparison.

02 · Method

Three corpora, synthetic rendering, fully reproducible.

• Corpora: Cantonese Wikipedia, the Hong Kong Cantonese Corpus (HKCanCor), and the rime-cantonese lexicon. All three are cleanly licensed.
• Character set: derived by subtracting the mainland, Taiwan and Hong Kong standard character lists from corpus character frequencies, then reviewed by hand.
• Fonts: SIL OFL only (Chiron Hei HK, Chiron Sung HK, Noto CJK, Hong Kong Character Set), 33 instances in total.
• Augmentation: rotation, perspective, elastic distortion, stroke weight, blur, noise and JPEG artefacts, across four difficulty levels.
• Quality gates: the font is checked for the glyph before rendering and legibility is checked after, which avoids the classic failure where a blank box is rendered while the label still claims a character.
• Reproducible: a fixed seed means the same corpus snapshot produces bit-identical output.

Full methodology, parameters and validation data are in the GitHub README and the HuggingFace dataset card.

03 · Dataset

Three tiers, with the same character appearing across them.

Minimal pairs

The dataset deliberately includes the matching Standard Chinese characters (有, 既, 左, 但 and so on) as a baseline. Without that baseline there is no way to tell whether reading 冇 as 有 is a visual problem or a linguistic one. Some of the pairs the design targets:

Composition

Extension-block characters make up 14% of the character tier, far above their natural corpus frequency. This is deliberate long-tail flattening: sampling by frequency would fill the benchmark with common characters and never show the rare ones at all.

04 · Samples

What the data actually looks like.

Cantonese-specific characters are marked in red. Filter by tier and difficulty, or show only Unicode Extension characters and minimal pairs.

05 · Results

How the models do on Cantonese characters.

Two 4B-parameter vision-language models, run over all 4,930 images on an NVIDIA L40S with identical prompts and decoding parameters. Metrics follow OCR convention: CER (character error rate, lower is better), ACC (exact sequence match), 1−NED (normalised edit similarity).

CER

Character error rate. Compares the model's answer to the correct one character by character; CER is the share that are wrong. 30 wrong out of 100 characters is 30%. Lower is better.

ACC

Exact match. The whole line has to match exactly, character for character. Miss even one character and the line counts as wrong, so ACC is usually much stricter than "1 minus CER" would suggest.

1−NED

Normalised edit similarity. Similar idea to CER, but corrected for text length so short and long samples are compared fairly. This is the convention used in ICDAR competitions. Higher is better.

ACC by tier

The same exact-match accuracy, broken out by what kind of sample it is — a single character, a short word, a full sentence, or a rare Unicode Extension character — to see where each model actually falls apart.

All 4,930 images × 2 models, 9,860 runs total, completed with zero failures.

Observations

• Both models score 0% on the 168 extension-block samples — they cannot read the glyph shapes at all. Common Cantonese characters split into two patterns. 冇 scores 0/7 at the character tier for both models and stays near zero in sentences too (InternVL3.5 0/99, Qwen3-VL 7/99): context does not rescue it, a pure visual failure. 嘅 also scores 0/7 in isolation, but recovers to roughly a third correct in sentences (InternVL3.5) — the model cannot read the shape, yet can guess a common function word from language memory. Whether the language prior compensates for a visual gap turns out to depend on the character.
• The two models also fail differently. Asked to read 冇 in isolation, Qwen3-VL gives a different wrong answer almost every time (渡, 浸, 侵…) — closer to guessing at random. InternVL3.5 almost always returns the same visually similar character instead — a consistent, wrong preference rather than noise. A consistent error is arguably easier to fix with targeted training data than a random one.
• Exact-match accuracy does not simply fall as text gets longer. For InternVL3.5, character-tier ACC is 25.6%, word-tier climbs to 41.1%, then sentence-tier drops back to 20.9% — the word tier is the highest of the three. A plausible reason: common 2–4 character combinations give the model a recognisable pattern to latch onto that an isolated character does not, but a longer sentence accumulates more chances to go wrong, pulling ACC back down.

06 · Outlook

Synthetic data is only the starting point.

• A real-image evaluation set. Synthetic data cannot stand in for real scenes. The next step is collecting Hong Kong shop signs, menus and newspapers and proofreading them into a small real benchmark.
• Training set and fine-tuning evidence. Scale up enough to fine-tune, then measure whether a model trained on this data actually improves on real Cantonese images. Without that step, the value of a synthetic dataset remains a claim.
• Vertical text and handwriting. Hong Kong signage is often vertical; this release is horizontal only. Handwriting is blocked on the near-total absence of Cantonese handwriting fonts.