Google’s new sign-language model lets users sign to search instead of typing

Google has developed a sign-language translation model that can convert complex body movements into text, using more than 100,000 hours of data across over 50 sign languages. The technology, called sign-language-to-text (SL2T), is now moving from research into consumer devices, starting with American Sign Language (ASL) on Pixel 11. The model powers new sign-to-text features in Gboard and Live Transcribe. Instead of typing, users can sign to search the web, write messages and documents, or interact with Google’s Gemini. In Live Transcribe, users can also sign responses during conversations. Unlike speech transcription, sign-language translation cannot simply map a sequence of sounds to words. Sign languages are independent languages with their own grammar and vocabulary. Meaning can also be conveyed simultaneously through hand movements, facial expressions, head position, arms, and torso. That makes sign-language translation a computer vision problem as well as a language translation problem. Google’s SL2T is designed to handle both by turning the signer’s movements into a structured representation before translating them into text. Body landmarks replace video Rather than sending a raw camera feed for translation, the system uses Google’s MediaPipe Holistic model to identify pose landmarks on the signer. These include geometric coordinates representing movements of the hands, face, and body. Only those coordinates are sent to the translation system, while the original video can be discarded immediately. This approach is designed to reduce the amount of sensitive visual information that leaves the device while still providing the model with information about how a person is signing. SL2T then translates the sequence of landmarks directly into text. It does not rely on intermediate “glosses”, simplified labels often used in sign-language research to represent individual signs. Google says this allows the model to better capture elements such as spatial relationships and non-manual signals. The system was trained on more than 100,000 hours of sign-language data spanning over 50 languages, with about one-quarter of the data coming from ASL. Google says training across languages, dialects, and different proficiency levels helped the model learn structures shared between sign languages. On the FLEURS-ASL benchmark, Google reports a zero-shot score of 70 BLEURT, which it says is substantially higher than previously reported results. The company also tested the model against practical challenges that could affect everyday use. Making translation work live The system is designed for streaming translation rather than processing a completed video. Google says its development included work on reducing latency, avoiding hallucinated output when someone is not signing, and improving performance for left-handed and one-handed signing. One-handed signing is particularly relevant for smartphone use because a person may need to hold the phone with one hand while signing with the other. Google also involved Deaf users and organizations throughout development, including testing, data collection and evaluation. The company says the technology was developed with input from Deaf experts and an advisory committee representing Deaf organizations. The initial rollout supports ASL-to-English translation on Gboard and Live Transcribe on Pixel 11, with Google saying additional devices and sign languages will follow.The work could eventually extend beyond translation, with Google also pointing to sign-language generation and other applications as future directions.Get the latest in engineering, tech, space & science - delivered daily to your inbox.With over a decade-long career in journalism, Neetika Walter has worked with The Economic Times, ANI, and Hindustan Times, covering politics, business, technology, and the clean energy sector. Passionate about contemporary culture, books, poetry, and storytelling, she brings depth and insight to her writing. When she isn’t chasing stories, she’s likely lost in a book or enjoying the company of her dogs.