ALS Stole His Voice—Then, a ‘Digital Twin’ Read His Thoughts And Started Speaking for Him

5 min readHere’s what you’ll learn when you read this story:A single brain implant let people with paralysis speak and gesture through one personalized digital avatar.As one user describes it, the brain-computer interface can reflect his intentions and personality and move how he wants to move.The virtual avatar study shows that human communication is neurologically more intertwined than the technology originally assumed.Did Walter want coffee? His virtual avatar responded, but it gave an unexpected answer. Instead of nodding or signaling yes with a thumbs-up, it waved goodbye. The mistake was funny, but it illustrated a serious problem: When technology communicates on someone’s behalf, an incorrect gesture can change the meaning of an entire exchange.Walter, who has amyotrophic lateral sclerosis, or ALS, knows how much can be lost when communication becomes difficult. He went from slightly slurred speech to being unintelligible within approximately six months. He eventually relied on emails, text-to-speech applications, and telephone relay services. ALS is a progressive neurological disease that damages the nerve cells responsible for voluntary muscle movement, gradually affecting a person’s ability to speak, swallow, move, and breathe. Although the disease weakens muscles, it does not necessarily impair thinking or understanding.“Somehow you try to keep a communication going while compromising to focus on the most essential content,” Walter says. “That doesn’t leave space for emotions and gestures at all.”Determined to find a more efficient way to communicate, Walter began researching brain-computer interfaces. After attending an ALS Association webinar about emerging technologies, he discovered that neurosurgeon Edward Chang, MD, at the University of California, San Francisco, was leading a study that could potentially change lives. He contacted the research team himself, undeterred by the prospect of the invasive implant it would require. With ALS shortening his life expectancy, he also saw participation as an opportunity to advance technology that might benefit future patients, even if he never lived to use a commercial version.The team began working with Walter on the virtual avatar in early summer 2025, gradually training the system to recognize his attempted speech and movements separately before combining them. “Every step and progress was fascinating and quite unbelievable at the same time,” he recalls. But the defining moment came when the avatar successfully decoded his speech and gestures together. “What a great milestone!” he says.The research team’s brain-computer interface (BCI) decodes attempted speech and gestures simultaneously through a single implant, translating those signals into words and movements performed by a personalized virtual avatar. Their study, published in Nature Neuroscience, offers a glimpse of technology that could eventually restore something beyond words: the expressive dimensions of human conversation.The process of getting your own virtual avatar started with electrocorticography (ECoG), which records electrical activity from electrodes positioned over the brain’s surface. The researchers’ implant covered a broad region of the sensorimotor cortex, which is involved in controlling speech and movement. Different areas are more strongly associated with different body movements, providing clues that machine-learning models can use to distinguish attempted speech from gestures.But the researchers encountered an unexpected complication.“What surprised us was that models trained on speech and gestures separately didn’t work as well when participants attempted both at the same time,” Samantha Brosler, a UCSF researcher and first co-author of the study, says.The problem involved overlapping neural activity. Some brain regions respond during both speech and gesture attempts, and their activity patterns can change when those actions occur together.Jennifer Collinger, PhD, a neuroprosthetics researcher at the University of Pittsburgh who was not involved in the research, describes this as context-dependent neural activity. “The neural activity associated with a particular movement, such as waving, may be different depending on whether or not you are speaking at the same time,” she explains.To address this, the UCSF team, led by neurosurgeon Edward Chang, MD, trained two parallel decoding models using recordings of participants attempting speech and gestures separately and together. A participant attempting to say “hello” while waving, for example, supplied neural activity that could train one model to recognize the phrase and another to recognize the movement. The combined training improved decoding performance.Encouragingly, the models could recognize some speech-and-gesture combinations they had not encountered during training. That matters because teaching a system every possible combination of human words and movements would be impractical.Chang Lab, UCSFThe system decodes neural activity recorded from a participant with paralysis in real time to control a personalized virtual avatar and enable speech and gesture communication.The study involved three participants with paralysis, although the main simultaneous speech-and-gesture experiments involved two. For a participant with ALS, the researchers tested 10 predefined phrases and 10 gestures. For one participant, the system recognized 10 conversational phrases, including “hello,” “thank you,” “nice to meet you,” and “how’s it going?,” alongside 10 gestures such as waving, nodding, shrugging, pointing, and shaking hands. The avatar could combine the two, allowing someone to say “hello” while waving or “nice to meet you” while extending a hand. Researchers tested all 100 possible phrase-and-gesture pairings, including combinations that would rarely occur in ordinary conversation, to assess how reliably the system could decode both actions together. During real-time testing, speech decoding reached 70 percent accuracy, and gesture decoding reached 66 percent. Offline testing for participants with paralysis following a brainstem stroke also produced valuable results.Collinger considers the findings encouraging but emphasizes their experimental boundaries, in part because this study was limited to a finite set of speech and gesture actions. The system wasn’t generating spontaneous conversation. It recognized predefined phrases, displayed decoded speech as text, and triggered corresponding avatar animations when gestures were detected.That distinction raises questions about user control. Walter’s accidental goodbye was amusing, but an incorrect gesture in another situation could misrepresent someone’s intentions. User feedback is essential to ensuring the technology communicates what someone actually intends, Collinger says.Future versions could allow users to confirm or correct decoded outputs, potentially through eye tracking, Brosler says.The system currently relies on a wired connection to external computers. An everyday user would need an overhaul that allows true freedom: wireless transmission, portability, and reduced communication delays. It must overcome multiple engineering challenges, Brosler says, such as miniaturizing parts, attaining reliable neural recordings, lowering power consumption, and reducing heat generation.Walter has his own priorities: the human dimension of the technology’s development. “It’s a two-sided learning curve for the avatar and the patient,” he explains.An ideal solution would decode a user’s neural cues, he says, adding that wireless mobility and less invasive technology could make such systems more attractive to users.During one successful experimental dialogue, the system decoded Walter’s attempted speech and gestures together without error. He remembers being overwhelmed. “I was extremely moved by the avatar’s response to my ‘thoughts,’ I couldn’t help laughing uncontrollably with pure joy.”Even with its limitations, Walter describes the personalized avatar as a digital twin whose movements reflect his intentions and personality.For Collinger, experiences like Walter’s demonstrate why accuracy alone cannot define success. “Restoring the ability to communicate is not just measured in words per minute,” she says. Independence, relationships, community participation, and personal expression matter, too.The UCSF experiment has not yet restored unrestricted, natural conversation. But it suggests that the future of brain-computer interfaces may depend on understanding communication as more than the transmission of words—and giving people greater control over everything those words cannot express.Download Pop Mech Digital IssuesTaayoo is an award-winning writer based in New York City. She regularly covers health topics, primarily inequity in healthcare and issues in aging. Her work has been published in Mayo Clinic Press, Yahoo, Essence, Cancer Today, New York Amsterdam News and many others. When she’s not writing, she enjoys reading and bingeing on crime dramas.