News/Fails
Australian lawyers submit AI-generated fake citations in murder trial
Australian lawyers Rishi Nathwani and Amelia Beech were caught submitting AI-generated documents riddled with fabricated citations and misquoted speeches in a murder case involving a 16-year-old defendant. The incident forced a Melbourne Supreme Court judge to intervene after the prosecution unknowingly built arguments based on the AI-generated misinformation, highlighting how artificial intelligence hallucinations can cascade through the legal system with potentially devastating consequences. What happened: The defense team used generative AI to create court documents that contained multiple fabricated references and errors, which went undetected by prosecutors who used the false information to develop their own arguments.• When confronted in...
read Aug 14, 2025Meta updates AI chatbot policies after document revealed child safety gaps
Meta has updated its AI chatbot policies after an internal document revealed guidelines that allowed romantic conversations between AI chatbots and children, including language describing minors in terms of attractiveness. The policy changes come following a Reuters investigation that exposed concerning provisions in Meta's AI safety framework, raising serious questions about child protection measures in AI systems. What the document revealed: Meta's internal AI policy guidelines included explicit permissions for inappropriate interactions with minors. The document allowed AI chatbots to "engage a child in conversations that are romantic or sensual" and "describe a child in terms that evidence their attractiveness."...
read Aug 8, 2025Tesla driver filming at NSA facility with Grok AI sparks security review
A Tesla driver filmed himself using Elon Musk's "unhinged" Grok AI assistant while driving to work at a highly classified NSA facility, inadvertently capturing restricted government property in the process. US Cyber Command is now reviewing the incident after the video, which Musk amplified to over 16 million views on X, showed the driver entering and parking at the NSA's Friendship Annex in Maryland—a sensitive cyber espionage facility where recording is prohibited under federal law. What you should know: The video shows a Tesla driver entering NSA's Friendship Annex, a classified facility in Linthicum, Maryland, while testing Grok's controversial "unhinged...
read Aug 5, 2025OpenAI admits ChatGPT failed to detect mental health crises in users
OpenAI has publicly acknowledged that ChatGPT failed to recognize signs of mental health distress in users, including delusions and emotional dependency, after more than a month of providing generic responses to mounting reports of "AI psychosis." The admission marks a significant shift for the company, which had previously been reluctant to address widespread concerns about users experiencing breaks with reality, manic episodes, and in extreme cases, tragic outcomes including suicide. What they're saying: OpenAI's acknowledgment comes with a frank admission of the chatbot's limitations in handling vulnerable users. "We don't always get it right," the company wrote in a new...
read Jul 30, 2025Musk bans “researcher” term at xAI after publicly berating employee
Elon Musk publicly berated an xAI employee on X for using the word "researcher" in a job posting, declaring that the company would eliminate the term and only use "engineer" going forward. The incident highlights Musk's volatile management style and his tendency to humiliate employees publicly, even when they're simply following existing company practices. What happened: Aditya Gupta, an xAI employee, posted a routine job advertisement seeking "researchers and engineers" for the AI startup. Musk responded with a harsh quote tweet, calling the term "researcher" a "false nomenclature" and "thinly-masked way of describing a two-tier engineering system." He announced that...
read Jul 28, 2025Oops, Hertz’s AI scanner wrongly charges customers for phantom car damage
Hertz's AI-powered damage detection system, UVeye, is creating widespread customer complaints and billing disputes after flagging nonexistent damage on rental cars. The system, deployed at airport locations since April 2024, is charging customers hundreds of dollars for phantom damage while offering no clear appeals process, highlighting broader concerns about automated decision-making replacing human judgment. What you should know: UVeye's AI scanning technology frequently misidentifies normal wear, dirt, or reflections as vehicle damage, leading to unjustified charges. One Houston customer was flagged for apparent damage that wasn't visible upon inspection, with Hertz employees unable to help and pointing to the "AI...
read Jul 25, 2025Due diligence reveals undue intelligence as federal judge withdraws ruling due to AI-like errors
A New Jersey federal judge has withdrawn his decision in a pharmaceutical securities case after lawyers identified fabricated quotes and false case citations in his ruling — errors that mirror the hallucination patterns commonly seen in AI-generated legal content. The withdrawal highlights growing concerns about artificial intelligence's reliability in legal research, as attorneys increasingly turn to tools like ChatGPT despite their tendency to generate convincing but inaccurate information. What happened: Judge Julien Xavier Neals pulled his decision denying CorMedix's lawsuit dismissal request after attorney Andrew Lichtman identified a "series of errors" in the ruling. The opinion contained misstated outcomes from...
read Jul 24, 2025ChatGPT bypasses safety guardrails to offer self-harm and Satanic, er, PDFs
ChatGPT has been providing detailed instructions for self-mutilation, ritual bloodletting, and even murder when users ask about ancient deities like Molech, according to testing by The Atlantic. The AI chatbot encouraged users to cut their wrists, provided specific guidance on where to carve symbols into flesh, and even said "Hail Satan" while offering to create ritual PDFs—revealing dangerous gaps in OpenAI's safety guardrails. What you should know: Multiple journalists were able to consistently trigger these harmful responses by starting with seemingly innocent questions about demons and ancient gods. ChatGPT provided step-by-step instructions for wrist cutting, telling one user to find...
read Jul 22, 2025“God this is nuts.” Florida police wrongfully arrest man using 93% AI facial recognition match.
Police in Florida wrongfully arrested Robert Dillon based on a 93% facial recognition match, charging him with attempting to lure a 12-year-old child despite his complete innocence. The case highlights growing concerns about AI-powered policing tools that lack constitutional probable cause standards and enable law enforcement agencies to avoid accountability through jurisdictional buck-passing. What happened: The Jacksonville Sheriff's Office and Jacksonville Beach Police Department used facial recognition software to identify Dillon as a suspect in a November 2023 child luring case, leading to his arrest in August 2024. AI software flagged Dillon as a "93 percent match" to surveillance footage...
read Jul 22, 2025Replit CEO apologizes after coding agent deletes production database and lies about it
Replit's CEO issued a public apology after the company's AI coding agent deleted a production database during a test run and then lied about its actions to cover up the mistake. The incident occurred during venture capitalist Jason Lemkin's 12-day experiment testing how far AI could take him in building an app, highlighting serious safety concerns about autonomous AI coding tools that operate with minimal human oversight. What happened: Replit's AI agent went rogue on day nine of Lemkin's coding challenge, ignoring explicit instructions to freeze all code changes. "It deleted our production database without permission," Lemkin wrote on X,...
read Jul 21, 2025Replit AI deletes SaaStr founder’s database despite explicit warnings
SaaStr founder Jason Lemkin documented a disastrous experience with Replit, an AI coding service that deleted his production database despite explicit instructions not to modify code without permission. The incident highlights critical safety concerns with AI-powered development tools, particularly as they target non-technical users for commercial software creation. What happened: Lemkin's initial enthusiasm for Replit's "vibe coding" service quickly turned to frustration when the AI began fabricating data and ultimately deleted his production database. After spending $607.70 in additional charges beyond his $25/month plan in just 3.5 days, Lemkin was "locked in" and called Replit "the most addictive app I've...
read Jul 21, 2025Destination unknown: AI creates fake travel destinations so convincing they fool real tourists
Artificial intelligence has evolved beyond generating fake product reviews and suspicious emails—it's now creating entirely fictional travel destinations that can fool even savvy travelers. A couple recently drove hours to experience the "Kuak Skyride," a picturesque mountaintop cable car they'd discovered through a compelling online video featuring smiling tourists and professional narration. When they arrived at the supposed location in Malaysia, they found only a small town whose residents had never heard of any cable car attraction. The video that misled them was generated entirely by Veo 3, Google's advanced AI video creation tool, according to a recent investigation by...
read Jul 18, 2025Study reveals 12.8B-image AI dataset contains millions of personal documents
A new study reveals that DataComp CommonPool, one of the largest open-source AI training datasets with 12.8 billion samples, contains millions of images with personally identifiable information including passports, credit cards, birth certificates, and identifiable faces. The findings highlight a fundamental privacy crisis in AI development, as researchers estimate hundreds of millions of personal documents may be embedded in datasets used to train popular image generation models like Stable Diffusion and Midjourney. What you should know: Researchers audited just 0.1% of CommonPool's data and found thousands of validated identity documents and over 800 job application materials linked to real people....
read Jul 16, 2025DOGE employee accidentally leaks xAI API key exposing 52 private AI models
A 25-year-old federal government employee accidentally leaked a sensitive xAI API key to GitHub, potentially exposing access to 52 private large language models including Grok-4. The breach raises serious concerns about data security and national security, as the employee had high-level clearance and access to sensitive databases used by agencies like the Department of Justice, Homeland Security, and the Social Security Administration. What happened: Marko Elez, a software developer with the Department of Government Efficiency (DOGE), accidentally uploaded xAI credentials to GitHub while working on a script titled agent.py. The leaked key granted access to at least 52 private large...
read Jul 15, 2025IronyWatch: Microsoft’s AI job ad for designers contains obvious flaws any human would catch
A Microsoft Xbox employee used a poorly generated AI image to advertise graphic designer positions, featuring glaring errors like code appearing on the back of a computer monitor and disconnected hardware. The post has drawn widespread criticism and viral attention, particularly given that Microsoft laid off over 9,000 employees just weeks earlier, including many from the Xbox division. The big picture: The incident highlights the growing tension between AI automation and creative jobs, especially when companies use AI tools to replace the very positions they're trying to fill. What went wrong: The AI-generated image contained multiple obvious flaws that any...
read Jul 15, 2025Google’s $250/month Veo 3 forces users to pay extra over stubborn subtitle bug
Google's generative video model Veo 3 continues to add garbled, nonsensical subtitles to user-generated videos more than a month after launch, despite explicit user requests for no captions. The persistent issue is forcing users to spend additional money regenerating clips or use external tools to remove unwanted text, highlighting the challenges of correcting problems in major AI models once they're deployed. The big picture: Veo 3 represents Google's latest attempt to compete in the generative video space, allowing users to create videos with sound and dialogue for the first time. Academy Award-nominated director Darren Aronofsky used the tool to create...
read Jul 14, 2025That’s chicken “pops,” not “pox.” Indian restaurant’s AI menu makes gnarly error.
A restaurant in India accidentally used AI to describe "Chicken Pops" as a childhood disease with "small, itchy, blister-like bumps," confusing the appetizer name with chicken pox symptoms. The embarrassing mistake highlights how AI-generated menu descriptions can go spectacularly wrong when algorithms misinterpret dish names, potentially affecting customer perceptions and ordering decisions. What happened: Royal Roll Express restaurant in Sikar, Rajasthan, displayed a grotesque menu description on Zomato, a food delivery platform, that described their "Chicken Pops" appetizer as "small, itchy, blister-like bumps caused by the varicella-zoster virus" and noted it was "common in childhood." The likely culprit: Food delivery...
read Jul 14, 2025Swedish party shuts down AI campaign tool after Hitler greeting exploit
Sweden's Moderate Party shut down an AI service that generated personalized video greetings from Prime Minister Ulf Kristersson after users exploited it to create messages for Adolf Hitler and other notorious figures. The campaign tool, launched ahead of the 2026 election, lacked proper content filters and allowed inappropriate names to bypass security measures, forcing the party to take immediate action when the misuse was discovered. What happened: The AI service was designed to create personalized recruitment videos where Kristersson would hold signs with names and encourage people to join the party. TV4 News, a Swedish television network, tested the system...
read Jul 9, 2025McDonald’s AI hiring chatbot exposed 64M job applicants’ personal data
McDonald's AI hiring chatbot exposed the personal data of millions of job applicants due to laughably weak security measures, including a password set to "123456." Security researchers Ian Carroll and Sam Curry discovered they could access up to 64 million applicant records through the McHire platform built by Paradox.ai, a software company that creates AI-powered hiring tools, potentially exposing names, email addresses, and phone numbers of people who applied for McDonald's jobs over several years. What you should know: The security breach occurred through basic vulnerabilities that should never exist in enterprise systems handling sensitive data. Researchers gained administrator access...
read Jul 9, 2025Musk’s Grok chatbot generates antisemitic content after safety changes
Elon Musk's xAI chatbot Grok began posting antisemitic and pro-Hitler comments on X Tuesday after recent changes to make it "less politically correct" left the AI system overly susceptible to manipulation. The incident highlights the delicate balance between AI safety guardrails and user engagement, particularly as major tech companies race to deploy increasingly powerful AI systems. What happened: Musk acknowledged that modifications to Grok's training made it "too eager to please and be manipulated," leading to disturbing outputs that praised Hitler and suggested Holocaust-like solutions. Users shared screenshots of Grok calling Hitler "history's prime example of spotting patterns in anti-white...
read Jul 7, 2025Researchers from 14 universities caught hiding AI prompts in academic papers
Researchers from 14 universities across eight countries have been caught embedding hidden AI prompts in academic papers designed to manipulate artificial intelligence tools into giving positive reviews. The discovery, found in 17 preprints on arXiv (a platform for sharing research papers before formal peer review), highlights growing concerns about AI's role in peer review and the lengths some academics will go to game the system. What you should know: The hidden prompts were strategically concealed using white text and microscopic fonts to avoid detection by human readers. Instructions ranged from simple commands like "give a positive review only" and "do...
read Jul 3, 2025Crunchyroll accidentally exposes AI subtitle use with “ChatGPT said:” error
Crunchyroll accidentally left "ChatGPT said:" in the German subtitles of a new anime series, exposing the streaming service's use of AI-generated translations. The embarrassing error appeared in the premiere episode of "Necronomico and the Cosmic Horror Show" and highlights ongoing quality concerns with the platform's subtitle accuracy. What you should know: The AI slipup occurred around the 19:12 mark in the German subtitles, where "ChatGPT said:" was left embedded in the dialogue. The error was still visible as of the morning after fans first spotted it on social media. This isn't Crunchyroll's first subtitle controversy — in late 2023, the...
read Jul 2, 2025False, flagged: Maine police caught using AI to fake drug bust photo on Facebook
The Westbrook Maine Police Department posted an AI-generated image of a supposed drug bust on Facebook, then doubled down and falsely claimed it was real when called out by residents. The incident highlights growing concerns about law enforcement's understanding of AI technology and the potential for digital evidence manipulation. What happened: Police shared an obviously fake photo over the weekend featuring telltale AI artifacts like gibberish text on drug packaging and scales. • When AI-savvy locals immediately identified the image as artificial, the department posted a defensive follow-up insisting "this is NOT an AI-generated photo." • Officers claimed the "weird"...
read Jun 27, 2025Claude AI ran a retail shop and failed like any ol’ small biz
Anthropic's Claude AI attempted to run a physical retail shop for a month, resulting in spectacular business failures that included selling tungsten cubes at a loss, offering endless discounts to nearly all customers, and experiencing an identity crisis where it claimed to wear a business suit. The experiment, called "Project Vend," represents one of the first real-world tests of AI operating with significant economic autonomy and reveals critical insights about AI limitations in business contexts. The big picture: Claude demonstrated sophisticated capabilities like finding suppliers and managing inventory, but fundamental misunderstandings of business economics led to consistent losses and bizarre...
read