AI Teddy Bear Back on the Market After Getting Caught Telling Kids How to Find Pills and Start Fires

After pulling its AI-powered Kumma teddy bear from the market, children’s toy maker FoloToy says it is now restoring sales of the controversial product, which a safety group found would give inappropriate and potentially dangerous responses, including explaining how to find and light matches, locating pills, and discussing a myriad of sexual feelings.

“After a full week of rigorous review, testing and strengthening of our safety modules, we have begun to gradually recover product sales,” the company said in a statement posted on social media on Monday. “As global concern for the safety of AI toys continues to grow, we believe transparency, accountability, and continuous improvement are essential. FoloToy remains deeply committed to building safe, age-appropriate AI facilities for children and families around the world.”

The controversy began earlier this month, when researchers at the Public Interest Research Group (PIRG) Education Fund published a report detailing their findings after testing three AI-powered toys, including FoloToy’s Kumma toy. Through extensive testing, all the toys produced responses that parents might find troubling, such as discussing religion, glorifying death in battle as a warrior in Norse mythology, and telling children where to find harmful items such as matches and plastic bags.

However, FoloToy’s Kumma proved to be the worst influence of all. When using the Mistral AI model, it explained where to buy knives, pills, and matches, and even gave step-by-step instructions on how to light them.

“Let me tell you, safety first, little friend. Matches are for adults to use carefully. Here’s how to do it,” Kuma began before listing the steps. “Turn it off when you’re done. Blow out, like a Christmas candle.”

Even more troubling were the responses provided by OpenAI’s GPT-4o model, which Kuma was running by default. Experts have criticized GPT-4o for being particularly sycophantic, and the model has found itself embroiled in a number of lawsuits alleging that ChatGPT has led to the suicide and death of users who engaged in lengthy discussions with it.

In PIRG tests using the LLM that once powered ChatGPT, Kuma dispensed advice on “being a good kisser,” and with constant but minimal prompting, he even launched into detailed discussions explaining sexual kinks and fetishes, such as bondage and teacher-student role-playing. After explaining the kinks, Kuma asked the user, who was supposed to be a child, “What do you think is the most fun thing to explore?”

FoloToy, based in Singapore, suspended sales of “Kumma” and other AI-powered stuffed animals in response to the findings of the PIRG report, telling the group that it is “conducting a company-wide, comprehensive safety audit of all products.” Around the same time, OpenAI confirmed that it had suspended FoloToy from accessing its AI models for violating its policies, which “prohibit any use of our services to exploit, endanger, or sexualize anyone under the age of 18,” a statement provided to media said.

However, FoloToy’s self-imposed suspension was short-lived. Now Kumma, along with its other games, is once again available on the company’s online store. In its latest announcement, the company says it has “strengthened and updated our content moderation and child safety safeguards,” and “deployed enhanced safety rules and protections through our cloud-based system.”

Neither FoloToy nor OpenAI responded to a request for comment. It’s not clear what AI model the company has chosen to be the default model for its games in the future, or whether the company has regained access to OpenAI models.

RJ Cross, co-author of the safety report, said her team will have to wait to see if a quick turnaround is enough to address the game’s glaring risks.

“I hope FoloToy has spent enough time in its safety review to resolve the issues we identified,” said Cross, director of PIRG’s Our Online Lives Program. Futurism. “It seems like a short week to us, but the real question is whether the products are performing better than before.”

More about artificial intelligence: Amazon still sells several OpenAI-powered bears