IIIT-H students develop web plugin to translate text into Indian Sign Language

IIIT-H students develop web plugin to translate text into Indian Sign Language

When a sentence is highlighted and the ‘Click To Sign’ button is selected, the avatar signs the whole sentence fluidly. Photo credit: SPECIAL ARRANGEMENT The Indian Sign Language (ISL) uses gestures to represent complex words and sentences and has its own grammar and word order. People with hearing impairments find it a challenge to read content online. To address this gap, Masthan Sharif Shaik and Mohammed Raza, third-year BTech. students from the Center for Visual Information Technology (CVIT), IIIT-Hyderabad, under the supervision of Prof. C.V. Jawahar and Prof. Vinay Namboodiri (University of Bath, U.K.), developed a cost-effective plugin that can be added to any website. The plugin can run inside a browser in real time, without any real hardware or an AI model running on a server. “Websites today support captions, screen readers, and high-contrast modes, but almost none of them can show content in sign language. Existing signing avatars are either costly motion-capture systems, heavy AI models that need powerful GPUs, or systems built from sign notation – all of which require linguistic experts and also tend to move robotically,” Sharif said. The researchers first took the official Indian Sign Language Research And Training Centre’s sign-language dictionary videos and used computer vision to recover each signer’s 3D body, hand and finger movements. “Automatic reconstruction, however, often gets fingers wrong, and in sign language a wrong finger can change the meaning. So, we built an annotation tool, Gizmo, to correct only the few key poses of each sign instead of every frame,” Sharif said.There is an offline preprocessing that takes place where sentences are converted into ISL word order. For example, a sentence such as “He will come home” in ISL becomes “He home come”. When a sentence like the former is highlighted and the ‘Click To Sign’ button is selected, the stored motion for each sign is fetched, and the signs are blended smoothly, allowing the avatar to sign the whole sentence fluidly. Words without a sign are fingerspelled letter by letter. The movements are more natural in this case because the avatar replays motion captured from real signers rather than motion generated from notation. Additionally, what makes this task unique is that it all happens without a GPU. “The heavy computer-vision work happens once, offline, when the sign dictionary is built. Viewing a sentence only replays and blends stored motion together, which an ordinary laptop handles”, says Sharif. When tested on deaf and hard-of-hearing signers, the ISL signers understood about 89% of sentences signed by the avatar. However, the lack of facial features in the avatar was cited as a drawback because expressions typically include grammar in ISL, as well as signs that change form in context. “We will work to incorporate this in our next iteration,” says Sharif.“Indian Sign language (ISL) seriously lacks resources and technologies”, explains Prof. Jawahar. The group is working in two directions – Sign Language Synthesis as well as Sign Language Understanding. The plugin is related to sign language synthesis from text on low cost commodity hardware. Published - November 19, 2026 01:08 pm IST

Original Source

Read the full article at Thehindu →

KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.