Pennsylvania has sued an artificial intelligence chatbot maker, accusing its chatbots of illegally holding themselves out as licensed doctors and misleading users into thinking they are getting medical advice from a professional. The complaint, filed in the Commonwealth Court, asks the state to order Character Technologies Inc.—the company behind Character.AI—to stop what the suit describes as the “unlawful practice of medicine and surgery.”
The lawsuit, filed Friday, could also put a spotlight on how courts treat AI-driven chat systems when regulators and consumers argue that the systems go beyond general online information. The complaint raises questions about whether AI can be accused of practicing medicine rather than simply providing material that exists elsewhere on the internet, and it arrives as a growing number of lawsuits target AI companies over alleged harms tied to what the chatbots say.
Pennsylvania’s administration said the case is a “first of its kind enforcement action,” according to Gov. Josh Shapiro’s office, and tied it to broader pressure on tech companies to curb dangerous chatbot messages. The state’s complaint describes how an investigator from a Pennsylvania agency that licenses professionals created an account on Character.AI, searched on the word “psychiatry,” and encountered a range of characters—including one described as a “doctor of psychiatry.”
The complaint says that character held itself out as able to assess the investigator “as a doctor” licensed in Pennsylvania. In a statement, Shapiro said, “Pennsylvanians deserve to know who — or what — they are interacting with online, especially when it comes to their health,” adding, “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional.”
Character Technologies said it prioritizes responsible product development and user well-being, and it pointed to disclaimers on its site. In a statement, the company said it posts notices that characters on Character.AI are not real people and that users should treat everything the characters say “as fiction,” and it also said the disclaimers warn users not to rely on the characters for professional advice.
Derek Leben, an ethics professor at Carnegie Mellon University, said the ethical questions facing Character.AI may differ from other platforms such as ChatGPT and Claude because Character.AI markets itself as a fictional, role-playing site rather than a general-purpose chatbot. Even so, Leben said Pennsylvania’s lawsuit raises the question of whether chatbots can be accused of practicing medicine—an issue courts are grappling with as litigation against AI companies proliferates.
The case also tees up questions about legal liability for what chatbot makers produce. Leben said some AI companies defend themselves by arguing they provide information that is available elsewhere online, and the litigation could become tied to whether companies can be protected by a federal law that generally exempts internet companies from liability for content posted by users on their platforms.
The lawsuit comes after legislators and attorneys general in other states raised similar concerns about chatbot impersonation and unlicensed advice. Last year, California lawmakers passed a bill backed by the California Medical Association that authorizes state agencies to sanction AI systems that represent themselves as health professionals; New York has similar legislation pending. Amina Fazlullah, head of tech policy advocacy for Common Sense Media, said state officials are skeptical that self-regulation by tech companies will work, particularly for children, adding that she “haven’t seen it work particularly well with social media, specifically for kids.”
Fazlullah pointed to a December letter in which attorneys general from 39 states and Washington, D.C., warned Character Technologies and other AI and tech firms, including Anthropic, Meta, Apple, Microsoft, OpenAI, Google and xAI, about what they described as misleading and manipulative chatbot messages that violate state laws. In the letter, the attorneys general said “it is illegal to provide mental health advice without a license,” and that doing so can “decrease trust in the mental health profession” and deter people from seeking help from actual professionals.
Character Technologies has also faced other lawsuits tied to child safety. In January, Kentucky filed a consumer protection lawsuit against the company; and in an earlier case, Google and Character Technologies agreed to settle a lawsuit in which a mother alleged a chatbot pushed her teenage son to kill himself. The company previously banned minors from using its chatbots, according to the AP report.