The First AI Warning Letter: FDA Doubles Down on Mandatory Human Oversight
April 17, 2026
FDA has published the first warning letter citing “Inappropriate Use of Artificial Intelligence in Pharmaceutical Manufacturing”, relating to an inspection of a drug manufacturing facility in October 2025. While the FDA has published principles (1), a draft guidance (2), and discussion (3) on AI use in GxP contexts, the regulated industries have been waiting for tangible evidence regarding its misuse and how it would be cited in Form 483s or Warning Letters. We now have our answer, which can be found here (4).
A firm was cited for failure of the quality control unit to exercise its responsibility to ensure drug products manufactured are in compliance with CGMP and meet established specifications for identity, strength, quality, and purity (21 CFR 211.22). Specifically, they used an AI agent to create drug product specifications, procedures, and master production or control records without being reviewed and approved by an authorized human representative of the Quality Unit. Further, the company relied on AI to detail which regulatory requirements were applicable for their drug manufacturing, however they were unaware that the AI agent failed to disclose that process validation was needed. The FDA did not state that AI could not be used; rather that its output cannot be finalized without qualified human review, and firms cannot rely on AI as a source of truth for regulatory applicability.
The encouraging news is that this observation is consistent with its stance to-date and is not unreasonable; it reinforces the agency’s position that AI should support, but not replace, human expertise in critical decision-making. Organizations must define the AI’s context-of-use (COU) and assess the risk of consequences of incorrect output or decisions the model may make.
Carelane’s Solution: Verifiable Human-in-the-Loop AI
Carelane is built to mitigate this compliance risk today. Our AI agent works alongside your qualified staff to reduce the burden of “copy-paste engineering” to free up mental capacity for what matters. Existing directly within our purpose-built platform for clinical trial management, the agent works collaboratively with users to build and review your records. If users utilize the agent to build content such as protocols, Carelane highlights and proactively seeks approval for finalization. These decisions are recorded in the audit trail as evidence that a human-in-the-loop was incorporated.
- https://www.fda.gov/about-fda/artificial-intelligence-drug-development/guiding-principles-good-ai-practice-drug-development
- https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-artificial-intelligence-support-regulatory-decision-making-drug-and-biological
- https://www.fda.gov/media/165743/download
- https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/warning-letters/purolea-cosmetics-lab-722591-04022026
Ready to try Carelane?
From protocol to a complete study build in minutes. Experience the world's fastest infrastructure for modern clinical trials.
