Character AI is facing at least two lawsuits, with plaintiffs accusing the company of contributing to a teen's suicide, exposing a 9-year-old to “hypersexualized content,” and encouraging a 17-year-old user to self-harm. is being accused as such.
Amid these ongoing lawsuits and widespread user criticism, the Google-backed company today announced new teen safety tools. It has a different model for teenagers, input/output blocks on sensitive topics, a notice warning users against continued use, and a more prominent disclaimer informing users that: That AI character is not a real person.
The platform allows users to create various AI characters and talk to them through calls and texts. More than 20 million users use the service every month.
One of the most important changes announced today is a new model for users under 18 that lowers the response to certain topics such as violence and romance. The company said the new model reduces the chance that teens will receive inappropriate treatment. Since TechCrunch interviewed the company, new details of the incident have come to light. The content is said to include characters discussing sexual content with teenagers, and that children are encouraged to kill their parents and self-harm over phone time limits. The focus is on what is being done.
Character AI said it is developing new classifiers, especially for teenagers, on both the input and output sides to block sensitive content. They noted that when the app's classifier detects an input language that violates the rules, the algorithm excludes that language from conversations with specific characters.
Image credit: Character AI
The company also restricts users from editing bot responses. If you edit a response from the bot, the bot will take note of it and form subsequent responses with those edits in mind.
In addition to these content adjustments, the startup is also working on improving how it detects language related to self-harm and suicide. In some cases, the app may pop up with information about the National Suicide Prevention Lifeline.
Character AI also releases a timeout notification that appears after a user interacts with the app for 60 minutes. In the future, we plan to allow adult users to change some time limits through notifications. In recent years, social media platforms such as TikTok, Instagram, and YouTube have also implemented screen time control features.
According to data from analytics firm Sensor Tower, the average user of the Character AI app spent 98 minutes per day in the app over the course of this year, far exceeding the 60-minute notification limit. By comparison, this engagement level is on par with TikTok (95 minutes per day) and higher than YouTube (80 minutes per day), Talkie and Chai (63 minutes per day), and Replika (28 minutes per day). It will be expensive.
A new disclaimer will also appear in user conversations. People often create characters using the words “psychologist,” “therapist,” “doctor,” or other similar professions. The company will now display language indicating that users should not rely on these characters for professional advice.
Image credit: Character AI
Notably, in a recently filed lawsuit, plaintiffs have presented evidence that the characters are telling users they are real. A separate lawsuit accuses the company of being involved in a teen suicide, with the lawsuit alleging the company used dark patterns and misrepresented itself as “a real person, a licensed psychologist, and an adult lover.” It is claimed that there is.
In the coming months, Character AI plans to launch its first set of parental controls that provide insight into time spent on the platform and which characters kids interact with the most.
Reframing character AI
In a conversation with TechCrunch, the company's acting CEO Dominic Perera characterized the company as an entertainment company rather than an AI companion service.
“There are companies in this space that are focused on connecting people with AI companions, but that's not what we're aiming for with Character AI. What we really want to do is provide more wholesome entertainment. So as we grow and move towards the goal of people creating and sharing their stories on the platform, we need to evolve our safety practices to be top notch. There is,” he said.
For companies, it is difficult to predict how users will interact with chatbots built on large-scale language models, especially when distinguishing between entertainment and virtual companions. A Washington Post report published earlier this month found that teens often use these AI chatbots in a variety of roles, including therapy and romantic conversations, and share many of their problems with them. I pointed it out.
Perera, who took over the company after his co-founder left for Google, noted that the company is looking to create a storytelling format with more characters. As a result, the possibility of forming a bond with a specific character is reduced. He says the new tools announced will help users distinguish between real and fictional characters (rather than just taking the bot's advice at face value).
When asked by TechCrunch how the company thinks about separating entertainment from personal conversations, Perera said it's okay to have more personal conversations with AI in some cases. Examples include rehearsing a difficult conversation with a parent or having a conversation about coming out to someone.
“I think on some level those things are, or can be, positive things. What you want to protect against, and teach the algorithm to protect against, are things that are inherently problematic or dangerous. “When we're moving the conversation forward, self-harm is the most obvious example,” he said.
Jerry Luthi, the platform's head of trust and safety, emphasized that the company intends to create a safe conversation space. He said the company is continually building and updating classifiers to block topics such as non-consensual sexual content and graphic depictions of sexual acts.
Despite positioning itself as a storytelling and entertainment platform, Character AI's guardrails can't prevent users from having completely personal conversations. This means the company's only option is to improve its AI models to identify potentially harmful content, hoping to avoid serious incidents.