Meta discovers AI-generated deceptive content on Facebook and Instagram

Meta, the parent company of Facebook and Instagram, announced on Wednesday that it has uncovered and disrupted a disinformation campaign on its platforms that involved "likely AI-generated" content. The campaign featured statements supporting Israel's handling of the Gaza conflict, which were embedded in news items from global organisations and US congressmen.

Meta stated in its most recent quarterly security report that the campaign was tracked back to STOIC, a Tel Aviv-based political marketing agency. The accounts posed as Jewish students, African Americans, and other concerned citizens, with a focus on audiences in the United States and Canada. This is the first time Meta has revealed the usage of text-based generative AI technology in such operations since its launch in late 2022.

Researchers are concerned about the potential for generative AI to bolster disinformation campaigns and influence elections, as it can produce human-like text, pictures, and audio quickly and cheaply. Despite these concerns, Meta's security executives maintained that the novel AI technologies have not hampered their capacity to dismantle such influence networks.

"There are various examples of how these networks use generative AI technology to create content. Perhaps it allows them to do it faster or with more volume. But it has not had much of an impact on our capacity to detect them," said Mike Dvilyanski, Meta's head of threat investigations, during a press call.

The research identified six covert influence activities that Meta halted in the first quarter. Aside from the STOIC network, Meta also disassembled an Iran-based network dedicated to the Israel-Hamas conflict, albeit no generative AI was found in that campaign.

Meta and other technology companies are facing the difficulty of preventing the misuse of new AI technologies, particularly in the context of elections. There have been reports that image generators from firms such as OpenAI and Microsoft have produced photographs containing voting-related disinformation, despite restrictions prohibiting such content.

These firms have advocated digital tagging solutions to help identify AI-generated material as it is created. However, these technologies do not apply to text, and academics challenge their overall efficacy.

With the impending elections in the European Union in June and in the United States in November, Meta's ability to combat such disinformation tactics will be tested significantly.

(With inputs from Reuters)