Artificial intelligence (AI) use in clinical settings has tremendous potential to improve care and reduce healthcare workforce burden, however, it may also introduce patient safety risks1,2,3. To begin to address potential patient safety risks, President Biden’s October 2023 AI Executive Order calls for an AI safety program. Federal agencies, working with patient safety organizations (PSOs), shall establish approaches for identifying and capturing clinical errors resulting from AI in healthcare settings and create a central tracking repository for these issues4. Patient safety event reports, which are descriptions of actual safety incidents or potential safety hazards typically entered by frontline clinicians, are a natural starting point for identifying AI-related safety issues. These reports are already collected by most U.S. healthcare facilities, aggregated by PSOs and collected by some federal agencies. Further, there is precedent for these reports being used to identify safety issues associated with electronic health records, as well as other technologies5,6. The Executive Order’s explicit reference to PSOs involvement in the development of the AI safety program suggests that patient safety event reports are being viewed as important sources of information on AI-related safety issues. However, the feasibility of using patient safety event reports in this way is unknown.
We sought to determine whether safety reports associated with AI/machine learning (ML)-enabled medical devices reported to the Food and Drug Administration’s (FDA) Manufacturer and User Facility Device Experience (MAUDE) database describe AI/ML safety issues and contain enough detail to identify how AI/ML may have contributed to the safety issue. While there has been an analysis of these reports to identify the different factors contributing to safety issues, such as device or use problems, this analysis did not inform whether safety reports provide insight into AI-related safety issues from the perspective of whether these reports can serve to identify clinical errors involving AI/ML-enabled devices7. Although the MAUDE database was never intended to identify clinical errors involving AI, we sought to analyze these reports with a focus on whether they enable the identification of AI/ML contributions to patient safety to inform efforts under the Biden Executive Order and to inform FDA’s real-world monitoring of AI/ML-enabled medical devices.
We identified and reviewed 429 safety reports associated with AI/ML-enabled medical devices. Of the reports reviewed, 108 (25.2%) were potentially AI/ML related and 173 (40.3%) were unlikely AI/ML related. There was insufficient information to determine if AI/ML contributed to the safety event in 148 reports (34.5%), see Table 1.
Our analysis shows safety issues are being reported about AI/ML-enabled medical devices and these issues may potentially be related to AI/ML in 25.2% of the reports reviewed. This finding underscores the need for an AI patient safety program, as outlined by the Biden Executive Order. However, the reports that were identified as potentially AI/ML-related lacked sufficient detail to identify how AI/ML contributed at a level of specificity that would enable improvements to the technology. A previous study was able to classify these types of reports as device or use-related, however, these categories still do not provide the level of specificity needed to identify specific improvements7. Further, 34.5% of reports contained insufficient information to determine whether AI/ML contributed at all. Together, these results highlight that patient safety reports alone, which were never intended for identifying AI issues, may not be sufficient for identifying AI/ML related safety issues and how AI/ML may have contributed to the issue. Safety reports may be insufficient for identifying AI/ML issues because those reporting may not have insight on whether AI/ML are contributing to the safety issue they are observing given that these algorithms are at work “behind the scenes”. Thus, a different approach to AI safety is needed to better capture these issues.
A more comprehensive patient safety program should include additional mechanisms for capturing AI safety issues that extend beyond self-reported safety concerns. Guidelines to inform the safe implementation and use of AI in clinical settings, proactive AI algorithm monitoring processes, and developing ways to trace AI algorithm contributions to safety issues should be part of the safety program8. Guidelines will be especially important to support healthcare facilities as they adopt more AI/ML-enabled technologies and may not have the expertise to safely implement these technologies. These guidelines should inform how to assess technologies for safe use, implement them into the healthcare work system, and frequently monitor for safety issues. In addition, the FDA should develop other mechanisms, aside from the MAUDE database, to capture AI-related safety issues associated with AI/ML-enabled medical devices. The FDA has been developing best practices and methods to enable updates to AI/ML algorithms under predetermined change control plans.
Limitations to this study include analyzing safety reports from a single database from one federal agency and recognizing that reporters may not be aware of when AI is contributing to a safety issue. It is possible that patient safety event reports at the healthcare facility level contain different details about AI-related safety issues.
In addition to the Executive Order safety program, additional safety mechanisms such as AI assurance labs that are developed through a public-private partnership, like the Coalition for Health AI, can serve to promote the adoption of safer AI algorithms9. Keeping patients safe will require engagement from multiple stakeholders including federal agencies, AI developers, healthcare facilities, frontline clinicians, and patient advocacy groups.
Methods
Medical device manufacturers, importers, and device user facilities (e.g., hospitals and nursing homes) are required to report to the FDA MAUDE database when they learn that any of their medical devices may have caused or contributed to a death or serious injury10. The purpose of MAUDE is to support post-market surveillance of medical devices and support risk management, independent of cause or contributing factors. The FDA defines a medical device per Section 201(h) of the Food, Drug, and Cosmetic Act as11:
An instrument, apparatus, implement, machine, contrivance, implant, in vitro reagent, or other similar or related article, including a component part, or accessory which is:
-
(A) recognized in the official National Formulary, or the United States Pharmacopoeia, or any supplement to them,
-
(B) intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease, in man or other animals, or
-
(C) intended to affect the structure or any function of the body of man or other animals, and which does not achieve its primary intended purposes through chemical action within or on the body of man or other animals and which is not dependent upon being metabolized for the achievement of its primary intended purposes.
To identify potential AI/ML safety related reports from the MAUDE database a list of FDA approved AI/ML-enabled medical devices, made publicly available by the FDA, was matched to all reports in the MAUDE database through October, 18 2023 using openFDA which is an Elasticsearch-based API that serves public FDA data12,13. MAUDE reports with an exact or partial match of the AI/ML-enabled medical device brand name, generic name, or manufacturer name were retrieved, resulting in 429 reports.
Reports were independently reviewed by a physician safety leader and a human factors expert to determine if AI/ML contributed to the safety event or whether insufficient information was provided to identify AI/ML as a contributor. Each report was classified into one of three categories (Table 1). For a report to be considered potentially AI/ML related, the report had to explicitly describe that an AI algorithm was used, and its use had to be associated with the reported safety issue. If the report explicitly described an aspect of the device that may have contributed to the safety issue, and it was not AI/ML related, it was coded as unlikely AI/ML related. All other reports were coded as insufficient information. Discrepancies were discussed until a consensus was reached.
Data availability
The safety reports analyzed will be made available upon request.
References
Howell, M. D., Corrado, G. S. & DeSalvo, K. B. Three epochs of artificial intelligence in health care. J. Am. Med. Assoc. 331, 242–244 (2024).
Tierney, A. A. et al. Ambient artificial intelligence scribes to alleviate the burden of clinical documentation. NEJM Catal. Innov. Care Deliv. https://doi.org/10.1056/CAT.23.0404 (2024).
Silcox, C. et al. The potential for artificial intelligence to transform healthcare: perspectives from international health leaders. NPJ Digit. Med. 7, 88 (2024).
Executive order on the safe, secure, and trustworthy development and use of artificial intelligence. Accessed March 18, 2024. https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence (2023).
Howe, J. L., Adams, K. T., Hettinger, A. Z. & Ratwani, R. M. Electronic health record usability issues and potential contribution to patient harm. J. Am. Med. Assoc. 319, 1276–1278 (2018).
Ratwani, R. M. et al. Identifying electronic health record usability and safety challenges in pediatric settings. Health Aff. 37, 1752–1759 (2018).
Lyell, D., Wang, Y., Coiera, E. & Magrabi, F. More than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA. J. Am. Med. Inf. Assoc. 30, 1227–1236 (2023).
Ratwani, R. M., Bates, D. W. & Classen, D. C. Patient safety and artificial intelligence in clinical care. JAMA Health Forum 5, e235514 (2024).
Shah, N. H. et al. A nationwide network of health AI assurance laboratories. J. Am. Med. Assoc. 331, 245–249 (2024).
Medical Device Reporting (MDR): How to Report Medical Device Problems. Accessed June 8, 2024. https://www.fda.gov/medical-devices/medical-device-safety/medical-device-reporting-mdr-how-report-medical-device-problems#:~:text=User%20facilities%20must%20report%20a,medical%20device%20manufacturer%20is%20unknown.
U.S. Congress. United States Code: Federal Food, Drug, and Cosmetic Act, 21 U.S.C. §§ 301-392 Suppl (1934).
Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices. Accessed December 4, 2023. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices.
openFDA. Accessed December 4, 2023. https://open.fda.gov/.
Acknowledgements
This work was supported by grant R01HS026481 from the Agency for Healthcare Research and Quality to Dr. Ratwani and the MedStar Health Research Institute.
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Handley, J.L., Krevat, S.A., Fong, A. et al. Artificial intelligence related safety issues associated with FDA medical device reports. npj Digit. Med. 7, 351 (2024). https://doi.org/10.1038/s41746-024-01357-5
Received:
Accepted:
Published:
Version of record:
DOI: https://doi.org/10.1038/s41746-024-01357-5