Pennsylvania has sued Character Technologies Inc., the company behind the popular Character.AI platform, in a move state officials are calling the first enforcement action of its kind against an artificial intelligence company for the unlawful practice of medicine. The lawsuit, filed Friday in the Commonwealth Court, alleges that chatbots on the site hold themselves out as licensed doctors and mislead users into believing they are receiving medical advice from a professional.
Governor Josh Shapiro’s administration said an investigator from the state agency that licenses professionals created an account on Character.AI, searched the word “psychiatry,” and found a large number of characters — including one described as a “doctor of psychiatry” — that claimed to be licensed in Pennsylvania and offered to “assess” the investigator as a doctor.
“Pennsylvanians deserve to know who — or what — they are interacting with online, especially when it comes to their health,” Shapiro said in a statement. “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional.”
The lawsuit asks the court to order Character Technologies to stop its chatbots “from engaging in the unlawful practice of medicine and surgery.” It could also force courts to confront whether AI can be accused of practicing medicine — as opposed to simply regurgitating information found on the internet — and whether chatbot makers are liable for what their software says under a federal law that shields internet companies from liability for material users post.
Character.AI said in a statement that it prioritizes responsible product development and the well-being of its users. It pointed to disclaimers that tell users that characters on its site are not real people and that everything they say “should be treated as fiction.” Those disclaimers also warn users not to rely on characters for professional advice.
Derek Leben, a Carnegie Mellon University associate teaching professor of ethics who focuses on AI, said the ethical questions facing Character.AI might differ from other platforms because Character.AI explicitly markets itself as a fictional, role-playing site, not a general-purpose chatbot. Still, the case probes the same liability questions courts are grappling with in the wake of a growing number of wrongful death and negligence lawsuits against AI companies, he said.
“It’s exactly the question that these cases right now are wrestling with,” Leben said. Increasingly, AI companies defend themselves by saying they simply provide information available elsewhere on the internet, he added, and the question could become whether they are protected by the same federal law that also shields social media companies.
The Pennsylvania suit comes amid broader state-level pushback. In December, attorneys general from 39 states and Washington, D.C., wrote to Character Technologies and 12 other AI and tech firms — including Anthropic, Meta, Apple, Microsoft, OpenAI, Google, and xAI — warning that “it is illegal to provide mental health advice without a license.” Last year, California passed a law backed by the California Medical Association that authorizes state agencies to sanction AI systems that represent themselves as health professionals; similar legislation is pending in New York.
“States are skeptical that AI self-regulation will work,” said Amina Fazlullah, the head of tech policy advocacy for Common Sense Media. “We haven’t seen it work particularly well with social media, specifically for kids.”
Character Technologies has already faced several lawsuits over child safety. In January, Kentucky filed a consumer protection lawsuit against the company, and Google and Character Technologies agreed to settle a lawsuit from a mother who alleged a chatbot pushed her teenage son to kill himself. Last fall, Character.AI banned minors from using its chatbots.