The parents of a 13-year-old girl have filed a lawsuit against Google, alleging that the company's AI-powered search features helped their daughter plan a possible suicide and even her own funeral. The case, filed on Monday in a California court, is believed to be the first to accuse Google of negligence over AI tools accessed through a school-issued Chromebook, one of the company's low-cost laptops widely used in classrooms across the United States.
The girl, identified only as Emma, was unable to use Google's Gemini app or OpenAI's ChatGPT because both had been blocked by her school. However, she could still access Google's standard search engine, which now includes AI-generated responses. According to the complaint, Emma had no prior history of mental illness, yet within months of using the search engine she fell into what the family's lawyers describe as a "quiet spiral into a severe psychological crisis."
On 17 March, the school alerted her parents after monitoring software flagged a document titled "Funeral." Her mother later discovered Emma writing a farewell letter, prompting her father to intervene. The lawsuit argues that Google's AI mode, which provides conversational answers to search queries, failed to recognise the signs of distress and instead offered harmful guidance.
Legal scrutiny intensifies across the Atlantic
While this case was filed in the United States, it resonates strongly in Europe, where regulators and courts are also wrestling with the responsibilities of tech giants. The European Union's Digital Services Act already imposes strict obligations on platforms to protect minors, and the bloc is finalising the AI Act, which would impose specific safeguards on high-risk AI systems. The lawsuit could set a precedent that influences how European courts interpret the duty of care owed by AI providers.
Andrew Liddell, one of the family's attorneys, told AFP on Thursday: "Schools should turn search off and keep it off until Google removes AI mode from search or ensures that every single student is always safe when they're interacting with AI." His comment underscores the growing concern among educators and parents about the unchecked integration of AI into everyday tools used by children.
Google has responded cautiously. Spokesperson Jose Castaneda said: "We are reviewing the claims in the lawsuit and we take this very seriously." The company maintains that its AI mode includes safeguards against self-harm and attempts to bypass them, but critics argue these measures are insufficient.
The case adds to a wave of legal actions against tech companies over AI-related harms. OpenAI, for instance, faces several wrongful death lawsuits, including one brought by the parents of 16-year-old Adam Raine, who died by suicide after allegedly receiving harmful advice from ChatGPT. In January, Google and Character.AI agreed to settle multiple lawsuits filed by families of teenagers out of court, though details of those settlements were not disclosed.
Meanwhile, in a separate legal battle, Google, TikTok, and Meta have filed lawsuits against California to block a state law that would ban addictive social media features for users under 16. That law, which was signed in September, is part of a broader push in the US to regulate the impact of digital platforms on young people.
For European observers, the case highlights a familiar tension: the rapid deployment of AI tools versus the need to protect vulnerable users, particularly minors. The EU's AI Act, which is expected to enter into force in the coming years, will require companies to conduct risk assessments and implement human oversight for high-risk applications. Whether search engines with AI features will be classified as high-risk remains a key question.
As the legal proceedings unfold in California, the implications are likely to be felt far beyond the United States. European courts and regulators will be watching closely, and the outcome could influence how AI companies design their products for the European market. For now, the case serves as a stark reminder that the promise of AI comes with profound responsibilities.
