Big companies, news outlets block OpenAI's web crawler that feeds on sites for information

Several big companies, including some prominent news outlets including Reuters, CNN, The New York Times, and the Australian Broadcasting Corporation (ABC), have blocked GPTBot over the recent announcement that the tool was using web crawlers.

GPTBot is OpenAI's web crawler and can be identified by the following user agent and string. OpenAI has launched it to improve future artificial intelligence models like GPT-4 and the future GPT-5.

OpenAI recently said that GPTBot was using the crawler for data that would feed its popular chatbot, ChatGPT.

The introduction of tools like ChatGPT received mixed responses, many hailed it as the "next big thing". ChatGPT is an AI-powered language model, which is capable of generating human-like text based on context and past conversations. However, some raised concerns over its usage as it threatens jobs.

GPTBot was being used to collect massive amounts of data from the internet to train ChatGPT. But it has been blocked by 70 of the top 1,000 websites in the world as of this week.

According to Originality.ai, a company that determines if content is AI-generated or plagiarised revealed that more than 15 per cent of the top 100 websites had chosen to restrict GPTBot in the last two weeks.

As quoted on it original site, the analysis said: "GPTBot launched 14 days ago and the percentage of Top 1,000 sites blocking it has been steadily increasing."

The key findings revealed that 9.2 per cent of the Top 1000 websites have blocked GPTBot in its first 14 days.

The first "Top 100" website to block GPTBot was Reuters.com. The top six biggest websites that are currently blocking GPTBot are: Amazon.com, Quora.com, NYTimes.com, Shutterstock.com, Wikihow.com, and CNN.com.

WATCH WION LIVE HERE