Google on Friday sued a group of cybercriminals it described as one of the most prolific bad actors in the spamosphere, alleging they used the company’s Gemini AI model to generate hundreds of fake website templates for phishing scams.
The lawsuit, filed in federal court in New York, targets a group that law enforcement and Google call “Outsider.” According to court filings, the group has been sending text messages that appear to come from mobile phone carriers, telling recipients they have reward points that must be redeemed before they expire. The messages include links to websites designed to mimic the carriers’ official login pages. The ultimate goal, Google said, is to steal credit card numbers that can be resold or used to purchase gift cards and luxury items.
The Federal Bureau of Investigation said Outsider has stolen an estimated 3.87 million credit card numbers through more than 8,000 phishing websites since July 2023, resulting in $1.9 billion in losses. The FBI and telecom companies are working with Google on the case.
“Criminals increasingly use AI to make fraud like this more convincing and harder to detect,” Brett Leatherman, assistant director of the FBI’s cyber division, said.
The addition of AI supercharged the group’s operations, Google said. The scammers developed a guide for using Gemini to generate computer code for fake websites, one of the most popular applications of large language models. That allowed them to circulate hundreds of different website templates and rapidly adapt their methods. Because of AI, the total number of websites that could be created in a scheme like Outsider’s is effectively unlimited, the company said.
Google’s general counsel, Halimah DeLaine Prado, said the company’s content filters did not flag the group’s requests because they were generic — the type of coding assistant queries millions of users make daily. “A content filter isn’t necessarily going to block a person’s ability to create code for a website,” she said. “That’s a fairly innocuous ask.”
Telecom-focused phishing has surged over the past year, according to the threat-intelligence firm WMC Global. Phishing messages targeting one U.S. telecommunications company increased tenfold in the month ending in early June, the firm said.
Google said it received roughly 55,000 reports of suspicious messages on Google Messages, the default text platform for Android users, in the two-week period ending June 1, including many attributed to Outsider.
The case is the first where Google has sued a defendant accused of using its Gemini AI model. The company is asking the court to halt the activity so that the websites and communication channels can be taken down. Google argues that the use of its products and logos damages its public image.
The lawsuit is part of a broader wave of action against AI-powered fraud. Last month, the online intelligence firm DarkTower identified a group of West African scammers using an AI-powered system to send fake invoices to corporations in what is known as business email compromise, according to Gary Warner, the firm’s director of threat intelligence. That tool conducted research on target organizations, discovered executive email accounts, built phishing attacks to steal login credentials, analyzed compromised accounts for payment-related messages, and wrote the fraudulent invoices — all managed by an AI system, Warner told the Journal.
AI companies are under pressure to police bad actors using their platforms amid a rise in scams and harmful chatbot interactions, part of a growing backlash against the technology.