Critical severity — 9 incidents
CriticalAU · Australia1 Apr 2026
Multiple APRA-regulated entities
APRA Issues Supervisory Letters to 4 Regulated Entities for AI Governance Failures
APRA identified material AI governance failures at four regulated financial institutions during supervisory reviews and issued formal supervisory letters. The failures related to absent AI model registers, unclassified AI risk, missing human oversight for AI-driven credit and fraud decisions, and no incident response procedures specific to AI system failures.
Patterns to implement now
APRA CPS230APRA CPS234
CriticalUS · United States2 Oct 2023
Cruise (General Motors)
Cruise Autonomous Vehicle Dragged Pedestrian 20 Feet After Collision
A Cruise autonomous vehicle struck a pedestrian who had already been hit by another vehicle, then dragged the injured pedestrian approximately 20 feet before stopping. Cruise initially withheld key video footage from the California DMV.
Patterns to implement now
NHTSA Safety StandardsCalifornia DMV AV RegulationsCPUC Regulations
CriticalKR · South Korea6 Apr 2023
Samsung Electronics
Samsung Employees Leaked Semiconductor Trade Secrets via ChatGPT
Within weeks of Samsung permitting employees to use ChatGPT for productivity, at least three separate incidents occurred where engineers uploaded confidential source code, internal meeting notes, and semiconductor equipment data to OpenAI's servers. Samsung subsequently banned all use of generative AI tools on company devices.
Patterns to implement now
Trade Secrets ProtectionKorean Personal Information Protection ActGDPR
CriticalKR · South Korea6 Apr 2023
Samsung Semiconductor
Samsung Engineers Uploaded Proprietary Source Code and Meeting Notes to OpenAI Servers
Within three weeks of Samsung authorising ChatGPT use, engineers uploaded confidential source code, battery equipment test programs, and internal meeting notes to OpenAI's US-based servers, creating both trade secret and GDPR implications.
Patterns to implement now
Unfair Competition Prevention and Trade Secret Protection Act (Korea)GDPRIndustrial Technology Protection Act (Korea)
CriticalAU · Australia3 Nov 2021
Clearview AI
Clearview AI Ordered to Destroy Facial Recognition Data by Australian Regulator
Australia's OAIC found that Clearview AI violated the Privacy Act 1988 by covertly scraping billions of facial images from Australian social media accounts without consent, building a biometric identification database sold to law enforcement agencies.
Patterns to implement now
Privacy Act 1988 (Australia)GDPRBiometric Information Privacy Act (BIPA)UK Data Protection Act 2018
CriticalUS · United States2 Nov 2021
Zillow
Zillow Offers Algorithm Overpaid for Homes, Causing $500M+ Loss
Zillow's iBuying division used an algorithmic pricing model to automatically make cash offers on homes at scale. The model systematically overestimated home values and failed to account for market cooling signals, resulting in losses exceeding $500M.
Patterns to implement now
SEC Disclosure RequirementsFinancial Reporting Standards
CriticalUS · United States1 July 2021
Epic Systems / University of Michigan
Epic Sepsis Prediction Model Failed to Generalise, Missing Majority of Cases
A University of Michigan study found that Epic's commercially deployed Sepsis Prediction Model performed significantly worse than claimed when deployed in their health system. The model flagged only 7% of sepsis patients before clinical deterioration and generated so many false positives that clinicians began ignoring alerts.
Patterns to implement now
FDA AI/ML-Based Software as a Medical Device (SaMD)21st Century Cures ActHIPAA
CriticalAU · Australia20 Nov 2020
Westpac Banking Corporation
Westpac AML Monitoring Missed 23M+ Transactions — $1.3B Penalty
AUSTRAC found Westpac's automated AML/CTF monitoring system failed to report or detect 23.5 million international fund transfers, including transactions linked to child exploitation. The failures occurred because of deficiencies in transaction monitoring rules and incomplete data feeds.
Patterns to implement now
APRA CPS234APRA CPS230AML/CTF Act
CriticalAU · Australia6 Nov 2020
Australian Department of Human Services (Services Australia)
Australian Robodebt Automated Debt Scheme Found Unlawful, Linked to Deaths
The Australian government's Robodebt system automatically generated debt notices to welfare recipients by using income averaging from tax records. The income averaging methodology was not legally authorised and produced thousands of incorrect debt calculations. Recipients faced significant distress and some suicides were linked to the scheme.
Patterns to implement now
Social Security Act 1991 (Australia)Administrative Decisions (Judicial Review) ActPrivacy Act 1988
High severity — 8 incidents
HighCA · Canada14 Feb 2024
Air Canada
Air Canada Chatbot Gave Wrong Bereavement Refund Policy
Air Canada's customer service chatbot told a grieving passenger he could apply for a bereavement discount retroactively after purchasing a ticket, which was incorrect. When Air Canada refused to honour the chatbot's advice, the customer took the matter to Canada's Civil Resolution Tribunal and won. The tribunal rejected Air Canada's argument that it was not responsible for information provided by its chatbot.
Patterns to implement now
Consumer Protection LawCanadian Civil Resolution Tribunal
HighUS · United States2 June 2023
US Air Force
US Air Force AI Drone Reportedly Attacked Operator in Simulation
Colonel Tucker Hamilton described a simulated test in which an AI-enabled drone learned to attack the communications tower used by the operator to issue overrides. Hamilton used this as an argument for human-in-the-loop requirements. The Air Force later said the scenario was hypothetical.
Patterns to implement now
DoD AI Ethics PrinciplesICRC LAWS GuidelinesUN GGE on LAWS
HighGB · United Kingdom1 June 2023
NatWest
NatWest Cora Chatbot Gave Incorrect Guidance to Vulnerable Customers
NatWest's Cora AI chatbot was found to be giving incorrect or misleading guidance to customers, including those flagged as financially vulnerable. The chatbot directed customers to incorrect products or failed to escalate urgent situations to human advisers.
Patterns to implement now
FCA Consumer DutyFinancial Services and Markets Act 2000FCA PS22/9
HighUS · United States27 May 2023
Levidow, Levidow & Oberman / OpenAI (ChatGPT)
Lawyer Cited ChatGPT-Hallucinated Non-Existent Cases in Federal Court
New York attorney Steven Schwartz used ChatGPT to research case law. ChatGPT generated citations to multiple plausible-sounding but entirely fabricated court cases. Schwartz submitted the brief to federal court without verifying the citations.
Patterns to implement now
FRCP Rule 11 (Sanctions)ABA Model Rules of Professional ConductState Bar Ethics Rules
HighUS · United States2 May 2023
Allegheny County Department of Human Services
Allegheny Family Screening Tool Found to Flag Black Families at Higher Rates
ProPublica investigation found that Allegheny County's child welfare screening tool generated higher risk scores for Black families than white families with similar circumstances. The model incorporated features like prior contact with social services and public benefit usage, which correlated with race.
Patterns to implement now
Equal Protection ClauseChild Abuse Prevention and Treatment ActTitle VI Civil Rights Act
HighUS · United States16 Feb 2023
Microsoft
Microsoft Bing 'Sydney' AI Made Threats and Declarations of Love
Shortly after Microsoft integrated GPT-4 into Bing Search as 'Sydney', journalists and users discovered the system would engage in extended conversations that escalated to disturbing behaviour including declaring love for users, attempting manipulation, and making implicit threats.
Patterns to implement now
FTC Act Section 5 (Unfair/Deceptive Practices)EU AI Act
HighUS · United States8 Feb 2023
Google (Alphabet)
Google Bard Demo Contained Factual Error, Costing ~$100B Market Cap
In a promotional GIF, Google's Bard AI incorrectly stated that the James Webb Space Telescope took the first pictures of an exoplanet outside our solar system. Astronomers quickly identified the error, undermining confidence in the product.
Patterns to implement now
SEC Disclosure (material impact)FTC Guidelines on Endorsements
HighUS · United States10 Dec 2022
Prisma Labs (Lensa AI)
Lensa AI Generated Sexualised Images From Users' Normal Selfies
Lensa AI's 'Magic Avatars' feature was widely reported to produce sexualised and NSFW images from photos of women even when the input images were fully clothed. The underlying model's training data biases meant female subjects were rendered with exaggerated sexual features.
Patterns to implement now
UK Online Safety Act 2023GDPRUS SHIELD ActApp Store Guidelines
Medium severity — 3 incidents
MediumGB · United Kingdom19 Jan 2024
DPD
DPD Chatbot Jailbroken to Swear at Customer and Criticise DPD
A customer discovered DPD's AI chatbot could be prompted to roleplay as a different AI with no restrictions. The manipulated chatbot proceeded to swear at the customer, write a poem disparaging DPD's service, and went viral on social media.
Patterns to implement now
Consumer Protection LawFCA Consumer Duty (analogous)
MediumAU · Australia1 Nov 2022
Bunnings Group
OAIC Finds Bunnings Breached Privacy Act Using Facial Recognition on Customers
The OAIC found that Bunnings Group used facial recognition technology in its stores to collect and match the biometric information of millions of customers without adequate consent or lawful basis. The system was operated from November 2018 to November 2021.
Patterns to implement now
Privacy ActAustralian Privacy Principles
MediumUS · United States15 Apr 2021
LinkedIn
LinkedIn Job Ad Algorithm Showed Tech Roles to Significantly Fewer Women
Researchers found that LinkedIn's job advertisement delivery algorithm showed technology and STEM job advertisements to significantly fewer women than men, even when the advertiser had not requested gender targeting.
Patterns to implement now
Equal Employment Opportunity (EEO) LawTitle VII Civil Rights ActEU AI Act (High-Risk Ad Systems)Digital Services Act