Microsoft Patents an AI That Can Literally Put Words in People’s Mouths in Real Time

Shutterstock / Futurism Microsoft just filed a patent for an AI product that would be able to transform live video in real time to alter speakers’ words and faces. The patent, as first flagged by patent-tracker Patentlyze, is described by its inventors as a “near real-time artificial intelligence (AI) enhancement of video or augmented reality (AR) presentations of live events.” Using AI agents, the system would identify and alter or remove certain words, while editing a speaker’s face and mouth to match. The agents would rely on set parameters or user profiles to make the changes, and would also edit the video and audio just before and after a removed or replaced word to allow for “smooth transitions.” In an era of media and the internet increasingly defined by AI-fueled deepfakes, disinformation, and information-muddling slop, the system described in this patent may represent a new world of filtering tools that subtly alter reality as it takes place, in this case on a highly-personalized level. Viewers, the patent suggests, could consume whatever live event they want to without hearing what they don’t want to hear. The potential use cases floated in the patent vary. One possible application is real-time translation, with the system allowing for broadcasts and other live footage to be both heard and seen in a viewer’s preferred language. The patent also posits that such a tool could be used to remove filler words like “um” or “ah” from live speech, or to erase stutters or speech impediments. Other possible use cases, though, go much further. According to the patent, the system could be used to cut down on “personal words or phrases” like “coined terms,” as well as “colloquial terms” like the term “y’all” or the phrase “bite the bullet.” The patent also claims that the system could eliminate “mumbled words, mispronounced words, misused words, misspoken words, offensive words, insensitive words, or inappropriate words.” Replacement words, it adds, may come from an “internet search.” In other words, live broadcasts may not be so live anymore after all. Microsoft didn’t immediately respond to a request for comment. Dubbing has existed for ages, and video footage is edited all the time, both for normal and for insidious reasons. Automatically editing live videos in this way with the help of AI agents, however, could change how audiences fundamentally engage with live footage. As it stands, if people don’t like what they hear, they can simply turn something off; the system outlined by Microsoft describes an environment where people can instead adjust someone else’s speech to their personal taste. Language translation aside, does using a tool that eliminates errors, or erases words and phrases associated with a speaker’s local dialect, move us further into personalized bubbles? What’s so bad about hearing someone from the South say “y’all?” To that end, the patent does mention that an event’s original live video and audio may be recorded and stored for the sake of “ground truth,” nodding to potential misuse and confusion that could arise from such a tool. The kinds of live footage that could be transformed by the system include lectures, presentations, a range of television productions — awards ceremonies, game and cooking shows — and live sports, among several others. More on Big Tech patents: Meta Patents Technology to Detect Your Mood by Constantly Analyzing Your Tone of Voice