OpenAI revealed it had suspended some China-linked accounts that were allegedly using ChatGPT to “design promotional materials and project plans for a social media listening tool.” The company states that suspended accounts were using a VPN to connect to ChatGPT from China and were allegedly working for a government client.
The tool these accounts were working on is described as a social-media “probe” that helps operators monitor platforms such as X, Facebook, Instagram, Reddit, TikTok, and YouTube for content the operators define as extremist, or as related to ethnicity, religion, or politics.
OpenAI clarified that the banned accounts did not utilize its AI models for social media monitoring and that the company cannot independently verify whether the tool was used by any Chinese government entity.
OpenAI has also banned another account that used ChatGPT to draft a proposal for a so-called “High-Risk Uyghur-Related Inflow Warning Model.” The proposed system was allegedly intended to help operators track the movements of Uyghur people, cross-reference them with police records, and issue early warnings of travel activity. Over the past few years, human rights groups have repeatedly accused the Chinese government of violating the rights of Uyghur Muslims.
The threats report released by OpenAI also sheds more light on Chinese, Russian, and Korean-speaking groups that allegedly sought to use ChatGPT to develop and refine malware, including remote-access trojans, credential stealers, and tools designed to evade detection. OpenAI said it also disrupted scam networks based in Cambodia, Myanmar, and Nigeria that had attempted to misuse its models.
“We found no evidence of new tactics or that our models provided threat actors with novel offensive capabilities. In fact, our models consistently refused outright malicious requests,” OpenAI added.
Since February 2024, following the release of its public threat reports, OpenAI has disrupted and reported more than 40 networks that violated the company’s policies.