Microsoft patents AI that can change people's words and faces in live videos

A person watching a live speech could soon have the option to hear something entirely different from what the speaker actually said. Microsoft is exploring an AI technology that could edit a person's words, remove unwanted expressions and even change their mouth movements and face while a video is still playing. The idea comes from a newly filed Microsoft patent that describes a system capable of making changes to live audio and video in near real time. Instead of simply filtering sound or adding subtitles, the proposed technology would modify the footage itself, making it appear as though the speaker had originally said the altered words.The system would use AI agents to detect specific words or phrases and decide whether they need to be removed or replaced. Once a change is made to the audio, the technology would also adjust the speaker's mouth movements and facial appearance to match. It would edit the surrounding audio and video as well so that viewers do not notice sudden cuts or interruptions. The changes could be based on predefined instructions or individual user profiles, allowing the same broadcast to appear differently to different viewers.Microsoft AI patent goes beyond translation and speech correctionOne of the more practical applications mentioned in the patent is live language translation. A viewer could potentially watch a speech in their preferred language, with the speaker's mouth movements edited to match the translated audio. The system could also make speakers sound more fluent by removing unnecessary pauses, filler words such as 'um' and 'ah', and stutters. It could correct mispronounced or incorrectly used words without requiring the speaker to repeat anything. But Microsoft's proposal does not stop at making speech easier to understand.The patent also describes removing expressions that viewers may not want to hear, including colloquial phrases, coined terms and local expressions. For example, someone using 'y'all' or 'bite the bullet' during a presentation could have those phrases replaced with different words. The system could also identify and remove language classified as offensive, insensitive or inappropriate. It may even use an internet search to find suitable replacement words. This would give viewers a level of control over live conversations that goes beyond conventional subtitles, translation and audio filters.The problem with editing someone's words in real timeThe technology also brings up an important question: when does improving a live broadcast become changing what actually happened? Removing a stutter or translating a speech can help audiences follow an event. However, replacing someone's expressions or removing words because they do not match a viewer's preferences is a different proposition. A speaker's choice of words, accent and local expressions are part of how they communicate. If AI starts removing these elements, viewers may end up watching a version of the speaker that does not accurately represent them. There is also the possibility of different audiences receiving different versions of the same speech without realising it. A statement could sound completely different depending on the changes applied to each viewer's feed. The patent does account for the need to preserve the original footage. It suggests that the unmodified audio and video could be recorded and stored as 'ground truth', providing a reference against which the edited version can be checked. The proposed system could be used for several kinds of live content, including lectures, business presentations, award ceremonies, television shows, cooking programmes and live sports.Microsoft has not announced plans to turn the patent into an actual product. As with any patent filing, it remains unclear whether the technology will ever reach consumers.- Ends