AHMAD BILAL
← AHMAD BILAL / WRITING AI FOR GOOD
AI FOR GOODACCESSIBILITYPEER-REVIEWED

The grammar was the hard part

How I built an English/Urdu to Pakistan Sign Language translation system, why I wrote a grammar instead of training a model, and what the field still can’t measure.

Ahmad BilalAug 2026~8 minAI for Good
FIG. 01 · THE PIPELINE Six labeled pipeline cells from left to right: AUDIO, TEXT and PUNCT filled with progressively calmer speckle noise, then PARSE as a clean syntax tree, HAMNOSYS as notation glyphs, SIGML as tag chevrons. Noise resolving into notation.
Six stages from broadcast media to a signing avatar. The first half of the pipeline fights noise; the second half is linguistics written down formally enough for a parser to run.
95.58%accuracy, sentence structures
93.98%accuracy, tenses
~50,000sentences in the largest PSL parallel corpus
1 KB vs 1 MBa sign as notation vs as video

Pakistan Sign Language translation AI turns English or Urdu speech and video into PSL gestures a deaf viewer can read. My system does it in stages: clean the audio, transcribe it, punctuate and tokenize the transcript, parse it through a context-free grammar written for PSL word order, look each word up in a HamNoSys database, then hand SiGML tags to a signing avatar. It scored 95.58% on sentence structures and 93.98% on tenses, and the paper cleared peer review at IEEE ICICS 2024.

That is the only peer-reviewed paper I have. Here is how it got built, including the call most people argue with.

01Start with the barrier

Captions look like access. For a lot of deaf viewers they are not. If your first language is a sign language, written English or Urdu is a second language you learned through a keyhole, and PSL has its own word order with no written form of its own. Subtitles quietly hand the work back to the reader.

The scale is worth knowing before you touch any code. The WHO counts 430 million people who need rehabilitation for disabling hearing loss, and puts the 2050 figure past 700 million. In Pakistan, a Rapid Assistive Technology survey run by the Ministry of National Health Services Regulations and Coordination found that 4.8% of people have a hearing disability, and the 2023 digital census recorded 9.64% living with a disability, the first time the country counted this nationally. Whatever you build lands on top of that.

So the target was never a demo that signs “hello”. It was media, in the two languages people actually broadcast in, translated into gestures with the grammar intact.

02Why I wrote a grammar instead of training a model

The reflex answer in 2024 was an encoder-decoder. Sensible, when you have data. Before you reach for one, count what your language pair actually has.

For PSL the count is thin. The largest published PSL parallel corpus I could point at held roughly 50,000 sentences, from Farooq and colleagues, who reported BLEU 0.83 and word error rate 0.17 with a multi-stack RNN. Good work. Now weigh that against spoken languages, where a production model trains on millions of pairs. Then look at what has landed since: PSL SignBank, published March 2026, is a machine-readable PSL dictionary of 300 words. Deaf Reach has documented 7,500 signs, a lexicon rather than a parallel corpus. Those are different objects and mixing them up is how you convince yourself the data exists.

A context-free grammar changes what scarcity costs you. You write the rules for how PSL builds a sentence once, with signers, and the parser applies them the same way every time. Every output traces to a rule, the same property I ask of research systems in auditable AI research. When a line comes out wrong, you can find the rule, argue about it with a deaf reviewer, and change one line. A learned model with 50,000 pairs behind it will drop a “not” on a shape it has never seen, and tell you nothing about why.

The tradeoff is real and I would rather state it than sell around it. A grammar does not generalize past what you wrote down. Feed it something outside the rules and it fails, visibly, instead of guessing plausibly. For a language where a wrong sign in a hospital is a wrong dose, failing visibly is the safer default. That is the whole case. If your corpus is large, make the opposite call.

03The pipeline, stage by stage

Six stages. Each one has a specific thing that bites.

Audio preprocessing. Broadcast audio arrives with music beds, two speakers, and levels nobody normalized. Clean it here or every later stage inherits the mess.

Transcribing. I used the Google Speech and NLP APIs to get text out of English and Urdu media. Urdu is where the accuracy budget goes; treat the two languages as separate problems with separate error rates, because they are.

Restoring punctuation. Raw speech transcripts arrive as unbroken strings. A parser needs to know where one sentence stops, so a machine learning pass punctuates the text and breaks it up before anything else touches it. Skip this and your grammar module receives one 400-word sentence.

Parsing. Tokenize, then run the tokens through the English-to-PSL module, which applies the PSL rules to reorder and validate. Linguistics did the heavy lifting here, not engineering: work out how PSL builds a sentence, then write that down formally enough for a parser to run it.

Looking up signs. Every word of the parsed sentence matches against a HamNoSys database. HamNoSys, from Thomas Hanke’s work at Hamburg, writes signs phonetically: handshape, where the hand sits, which way it faces, how it moves, all as symbols. A sign becomes instructions rather than a video file.

Rendering. The system generates SiGML tags per symbol and exports them to a SiGML player, which animates the avatar on screen. SiGML is the XML layer that turns HamNoSys into something a player can perform.

Storing signs as symbols instead of clips is more than a detail. PSL SignBank measured the same tradeoff at roughly 1 KB per sign in HamNoSys against about 1 MB as video. On a phone, on a Pakistani mobile link, that decides whether you ship.

04What two accuracy numbers say, and what they do not

The published system scored 95.58% on sentence structures and 93.98% on tenses.

Reporting those separately is the useful part. One blended number would have hidden which half is weaker, and tense is the weaker half, which tells you exactly where to spend the next month. If you take one habit from this, take that one: split your accuracy by the grammatical category you are trying to get right, and publish the ugly half.

Now the limit, stated plainly. Both numbers score output against PSL grammar rules. Neither one tells you whether a deaf viewer watched the avatar and understood it. Those are different claims and the paper supports the first. A reviewed venue is good at forcing that split into the open, because a reviewer will ask what your accuracy is accuracy of, and the answer goes in the paper where anyone can check it against the DOI.

05What is still unsolved

The data ceiling. A machine-readable PSL dictionary published in 2026 holds 300 words. Until that number grows another zero, every PSL system is either rule-based or overfitting, and the real bottleneck is deaf signers annotating data, not architecture.

Faces. Sign languages carry grammar on the face. Eyebrows mark a question, the mouth disambiguates, gaze points at who you mean. An avatar with correct handshapes and a dead face is still hard to read, and these non-manual features are the weakest part of most notation-to-avatar pipelines including mine.

Regional spread. PSL is not uniform across Pakistan, which Ulrike Zeshan documented in her grammar of Indo-Pakistani Sign Language long before anyone tried to automate it. Rules written with signers in one city inherit that city.

How the field scores itself. Matching the rules is a proxy and everyone knows it. Recent work openly attacks the automatic metrics used in sign language translation, and researchers are still arguing about how to score a pose. If you are entering this area, read that argument first.

If you are building for a low-resource sign language, I want to know the number you started from: how many validated parallel sentences did you actually have before you picked your architecture? That count decided my whole design, and I have never seen anyone publish theirs alongside the model. The full paper and the rest of the research sit on the research page.

06References

07Further study

Pick the branch that matches what you are trying to do.

If you are choosing an approach.

If you are building the rendering layer.

If you are working on PSL specifically.

If you want to attack the weakest claim. Here it is: rule-conformance accuracy, mine included, does not establish that deaf viewers understand the output. The strongest objection to this entire class of system is that the field measures what is easy to measure and calls it translation quality. That objection is live in the literature right now, in critical work on automatic evaluation in sign language translation and on pose-based evaluation. If you think the grammar-first approach is wrong, this is where you would land the punch, and I would rather you did than that nobody checked.