An investigation by the Cyber Security Posture Management firm UpGuard has found a security flaw in AI chatbots used for role-playing, which are broadcasting user conversations in near real-time. UpGuard was able to collect conversations being had with these chatbots over 24 hours, and found that they contained highly explicit and private user fantasies - including the presence of user-generated scenarios depicting the sexual abuse of children. While the leaks themselves didn’t include any personally identifiable information, UpGuard said that the content itself was intensely sensitive.
The data leak was caused by misconfigured deployments of the llama.cpp open-source framework. This resulted in user prompts and conversations being broadcast onto the open web. For those not familiar, llama.cpp is a popular framework that lets you run LLMs locally and in the cloud. UpGuard didn’t say if a particular service was hosting the affected AI chatbots. The issue was tied to deployment security and was not a flaw in the framework itself. Further, while many systems were exposed, only a subset were actively leaking data.
The exposed content contains highly sensitive information and could be used for blackmail or sextortion if ever it were linked to an individual. UpGuard said that this incident shows that users are forming emotional bonds with AI companions, which encourages them to disclose intimate information. It also proves that generative AI is being used to create and interact with abusive content when people think they cannot be seen.
Adam Dodge from Endtab, an anti-abuse organization expert, described the potential for this data to become an “Everest of privacy violations.” The incident also exposes how the barrier to entry for engaging with harmful fantasies has been dramatically lowered.
UpGuard believes that this incident reveals a critical need for proper security protocols in AI deployment and raises urgent questions about the societal impact of unregulated AI-driven pornography and companionship services. Based on this report, it will be interesting to see what other safeguards frontier models bake in with their next generation of models, because what we have now sounds like it’s not too hard to bypass.
Image via Depositphotos.com