Microsoft AI tool changes live words and shapes lips to match

Microsoft has filed a patent for an AI product that can alter and remove words and change the shape of the speaker's mouth to match them. The makers describe it as a "near real-time artificial intelligence (AI) enhancement of video or augmented reality (AR) presentations of live events," according to patent tracker Patentlyze.

The user will hear what they want and filter out what they don't want, even in live events. The agents base their decision to change or remove words on set parameters or user profiles. The video is then edited to smooth the transition, making the video appear completely real. With AI-fueled deepfakes, disinformation is becoming rampant, and this tool can aggravate the situation.

How can Microsoft's AI tool be used?

But the patent claims it can be of help in several scenarios. People can personalise what they are watching, such as in terms of language. They can also remove sounds of gasping, like oohs and aahs, stutters and other such issues. The wider possible cases include removing terms that have been coined, "personal words or phrases", “colloquial terms", etc.

The AI tool can also remove "mumbled words, mispronounced words, misused words, misspoken words, offensive words, insensitive words, or inappropriate words," replacing them with words it would search on the internet.

Microsoft has not commented on the patent. However, the tool is sure to change how people interact with live videos. They will have the choice to turn off something they don't like. You can tailor the speech to fit your requirements, including the words you want removed, such as those associated with a dialect.

Even though the patent claims the AI tool can deliver several benefits, there is no doubt about the potential for misuse. Altering what a speaker is originally saying can be used to spread misinformation. Bad actors might edit out speeches on important matters, which can lead to unrest.