The Regulation of Large Language Models & AI in Healthcare: The Promise and Peril

The rapid advancements in artificial intelligence (AI) have ushered in the era of sophisticated Large Language Models (LLMs), such as OpenAI’s GPT-4 and Google’s Bard. Notably, ChatGPT, GPT-4’s predecessor released in November 2022, garnered an astonishing 100 million users within two months of its launch, marking an unprecedented adoption rate for any emerging technology.

This meteoric rise of AI-driven conversational models has triggered global discussions on their potential role in healthcare and the practice of medicine. LLMs boast diverse applications, ranging from aiding in clinical documentation, generating discharge summaries, and crafting clinic, operation, and procedure notes to obtaining insurance pre-authorization, summarising research papers, and even functioning as chatbots to address patient queries with personalised responses based on their data and concerns.

Moreover, LLMs have the capability to assist physicians in diagnosing medical conditions by analysing vast datasets, including medical records, images, and laboratory results, while also suggesting appropriate treatment plans. In principle, patients stand to gain greater autonomy by receiving personalised assessments of their health data, symptoms, and queries, a leap beyond traditional search methods.

Systematic reviews further highlight the potential benefits of LLMs, including enhancing scientific writing, promoting research equity, streamlining healthcare workflows, reducing costs, and enriching personalised medical education.

Given the far-reaching implications for patient outcomes and public health, the imperative to regulate these AI-based tools becomes evident. Regulating LLMs in healthcare without stifling their promising progress presents a timely and critical challenge. 

The overarching goal is to ensure safety, uphold ethical standards, preempt biases and unfairness, and safeguard patient privacy. This is especially pertinent considering that the complexities and implications of LLMs amplify any concerns associated with AI.

1. Benefits of LLMs in Healthcare

The integration of Large Language Models (LLMs) into healthcare is revolutionising the sector. These advanced AI tools are proving to be invaluable in various aspects of healthcare, from clinical decision-making to patient engagement and research. Here, we explore how LLMs are enhancing healthcare delivery and research, drawing on real-world applications and innovations.

Clinical Decision Support: LLMs offer clinicians real-time access to medical literature and patient data, aiding in informed decision-making. For example, the Clinical Digital Assistant developed by Oracle, expected to launch by the end of 2024, leverages LLMs for administrative tasks and simplifies clinical note-taking, allowing doctors to focus more on patient care.

Telemedicine and Remote Monitoring: In telemedicine, LLMs enhance communication between healthcare providers and patients. Hippocratic AI has pioneered the use of LLMs in creating virtual nurses. These AI-driven nurses help in chronic care management, providing reminders for medication, follow-up appointments, and assisting in navigating care-access issues.

Drug Discovery and Research: LLMs accelerate the process of drug discovery by analysing large datasets. This is exemplified in the work of Amber Simpson at Queens University, where LLMs are used to predict metastatic cancer, aiding in the formulation of targeted treatment strategies and precision medicine.

Patient Engagement: LLMs play a crucial role in engaging patients in their healthcare journey. They can provide personalised health information, respond to queries, and assist in managing well-being. The use of LLMs in analysing social determinants of health, as researched by Maxim Topaz at Columbia University, highlights their role in identifying patients’ non-medical needs, which significantly impacts health outcomes.

Administrative Efficiency: LLMs streamline administrative processes within healthcare institutions. The utilisation of LLMs for tasks like email categorisation and response, as practiced by healthcare professionals using systems like Oracle’s Clinical Digital Assistant, showcases how LLMs can reduce the administrative burden, thereby enhancing overall healthcare delivery.

Conversational AI Diagnostics: LLMs are also making strides in the field of diagnostic imaging. Greg Corrado from Google AI discusses the integration of LLMs with medical imaging systems to improve diagnostic accuracy, particularly in mammography. This AI-driven approach enhances the interaction between clinicians and diagnostic systems, offering a more nuanced and accurate analysis of medical images.

2. Challenges and Risks in LLMs

LLMs, like GPT-4, can generate outputs that are not based on factual information or input data, a phenomenon often referred to as “hallucinating” results. This misinformation, especially when related to diagnoses, treatments, or tests, can mislead healthcare providers and patients, potentially leading to dangerous consequences. Such errors can arise from incomplete or biased training data, the probabilistic nature of LLMs, or lack of context.

The challenge of mitigating bias in LLMs is crucial, as biases in training data can significantly impact clinical decision-making, patient outcomes, and healthcare equity. Biases may arise from underrepresentation of certain demographic groups, overemphasis on specific treatments, or reliance on outdated medical practices. These biases, if not addressed, can lead to incorrect diagnoses or suboptimal treatment recommendations, potentially causing harm or delaying appropriate care for patients.

The deployment of LLMs like GPT-4 in healthcare settings raises critical ethical concerns that necessitate a robust regulatory framework. Key issues include transparency, accountability, and fairness. Healthcare professionals and patients should be informed about the AI’s involvement in decision-making processes and provided with explanations for AI-generated recommendations. Regulations must also ensure the confidentiality and security of patient information, including guidelines for data anonymisation, encryption, and secure storage, as well as measures to prevent unauthorised access or misuse of data.

The global release of LLMs, with no country-specific iterations, calls for a global regulatory approach. It’s unclear what technical category LLMs will fall into from a regulatory perspective. A new regulatory category might be needed to address LLM-specific challenges and risks. Regulations for LLMs are necessary if they are used, modified, or directed toward medical purposes. This includes LLMs specifically trained on medical data and databases, which are likely to fall under medical regulatory scrutiny. 

These challenges necessitate that regulatory bodies not only start regulating LLMs as they are deployed but also approach them differently than current AI technologies. The unique nature of LLMs requires a tailored regulatory response to ensure their safe and ethical use in healthcare settings. This includes establishing new regulatory categories and guidelines that address the specific challenges posed by LLMs, different from those applied to other AI technologies.

3. The Role of Patients in Regulation

The integration of Large Language Models (LLMs) in healthcare necessitates a patient-centered approach in regulatory processes. Recognising patients as key stakeholders, their involvement is essential in shaping regulations that govern the use of LLMs in medical settings. This section highlights the critical role of patients in ensuring that regulations for LLMs are grounded in the realities of patient care, encompassing aspects of patient-centric regulation, transparency, informed consent, and ethical considerations. By placing patients at the forefront, regulators aim to create a regulatory framework that is not only effective but also empathetic and responsive to the needs of those it serves.

Involving patients in the regulatory process is crucial to ensure that Large Language Models (LLMs) align with real-world patient needs. This involvement includes patient participation in advisory groups or panels, providing feedback on LLM design and implementation. Patients, with their unique experiences and perspectives, can offer invaluable insights into the usability, effectiveness, and ethical considerations of LLMs in healthcare. Their input helps to ensure that these technologies are user-friendly, meet actual healthcare needs, and address potential ethical dilemmas in a manner that respects patient values and preferences.

Patients should be empowered with clear and transparent information regarding the use of LLMs in their healthcare. This includes educating patients about the capabilities, limitations, and the extent to which these AI-driven tools influence their care. Informed consent processes should be robust and comprehensive, including detailed explanations of the role and implications of AI-driven tools in diagnosis, treatment planning, and prognosis. Ensuring that patients understand how their data is used and the extent of AI’s involvement in their care is fundamental to maintaining trust and autonomy in patient-provider relationships.

Patients play a pivotal role in shaping the ethical guidelines governing LLMs. These guidelines should prioritize patient-centric values such as fairness, transparency, and privacy. Ensuring that AI systems are free from biases and respect the diverse backgrounds and needs of patients is crucial. Patients’ insights are valuable in developing guidelines that address concerns about data security, consent, and the equitable distribution of AI benefits. Engaging patients in these discussions ensures that ethical guidelines are not only technically sound but also aligned with societal values and patient expectations, ultimately ensuring that AI in healthcare benefits all stakeholders equitably.

4. Recent Developments in LLM Regulation

The landscape for regulating Large Language Models (LLMs) is rapidly evolving, reflecting their increasing integration into various sectors, including healthcare and digital communication. Recent actions by countries and the involvement of global regulatory bodies and industry leaders highlight the complexity and urgency of establishing effective regulatory frameworks. 

These developments underscore the need for adaptable and comprehensive regulations to address the unique challenges posed by LLMs. This section explores recent pivotal events and trends that are influencing the regulatory discourse around LLMs, providing insights into the current state and future direction of LLM regulation.

Italy’s data protection authority made a significant move by ordering OpenAI to stop processing people’s data locally, citing concerns that ChatGPT might be breaching the European Union’s General Data Protection Regulation (GDPR). This action was driven by issues such as the unlawful processing of people’s data and the lack of any system to prevent minors from accessing the technology. This decision is a prominent example of national regulatory bodies taking steps to manage the risks associated with LLMs.

On the global front, the U.S. Food and Drug Administration (FDA) has been a leader in discussions on regulatory oversight of emerging technologies, including AI and machine learning in medical devices. The FDA’s approach involves regulating Software as a Medical Device (SaMD) and adapting its regulatory framework to address AI/ML technologies in medical devices. Their approach includes a total product lifecycle (TPLC) approach to regulating AI/ML-based SaMD, focusing on continuous monitoring and improvement of these technologies. This global leadership is indicative of the growing trend towards establishing harmonised standards for LLMs and AI technologies.

In the healthcare sector, companies are actively integrating LLMs into their services. Microsoft-owned Nuance has added GPT-4 AI to its medical note-taking tool, and the French startup Nabla has built a tool using GPT-3 to assist physicians with paperwork. These examples demonstrate the growing interest and implementation of LLMs in healthcare, signalling a shift in industry practices and influencing the direction of regulatory frameworks.

These developments in Italy, the actions of the FDA, and industry involvement highlight the dynamic nature of LLM regulation. They illustrate the need for regulatory bodies to not only start regulating LLMs as these models are deployed but also to consider regulating them differently from current AI technologies. This approach is crucial to address the unique challenges and potential risks associated with LLMs while fostering their responsible and beneficial use.

5.Differentiated Regulatory 

The rapid advancement of Large Language Models (LLMs) in healthcare necessitates a differentiated regulatory approach. As LLMs are increasingly implemented in medical settings, their applications bring forth unique challenges and opportunities, requiring regulatory frameworks that are both specific and adaptable. Below we examines the current state of regulatory frameworks for LLMs, emphasising the need for distinct regulations for medical versus general-purpose LLMs, continuous monitoring post-deployment, and the importance of collaborative efforts in developing these frameworks.

The differentiation between LLMs for medical use and general-purpose LLMs is becoming increasingly important. The FDA’s regulation of Software as a Medical Device (SaMD) is a prime example of this approach, focusing on software solutions used in medical contexts. This regulation is part of the FDA’s broader strategy to adapt its regulatory framework to AI and machine learning technologies in medical devices, emphasising a Total Product Lifecycle approach.

Continuous monitoring of LLMs is crucial for ensuring ongoing compliance with safety and performance standards. The FDA’s proposed framework for AI/ML-based SaMD underscores the importance of real-world performance monitoring and transparency throughout the lifespan of these technologies. This approach reflects a shift towards continuous evaluation and adaptation of regulatory practices in response to evolving capabilities of LLMs.

Collaborative efforts among stakeholders are essential for developing comprehensive regulatory frameworks. Several innovative applications of LLMs in healthcare illustrate the need for such collaboration:

1. Virtual Nurses: Hippocratic AI is using LLMs to create virtual nurses for chronic care, providing administrative support to healthcare professionals. This includes reminding patients to take medication, scheduling appointments, and navigating care-access issues.

2. Clinical Note-Taking: Oracle Computer announced an AI-powered Clinical Digital Assistant for administrative tasks, showing how LLMs can create efficiencies for clinicians by managing emails and patient records.

3. Adverse-Event Detection: Vivek Rudrapatna at the University of California, San Francisco, has worked with the FDA on using LLMs for detecting adverse events from clinical notes within electronic health record systems.

4. Predicting Cancer Metastasis: Amber Simpson at Queens University collaborates with Memorial Sloan Kettering Cancer Center to use LLMs for predicting metastatic cancer and assisting in clinical treatment responses.

5. Social Determinants of Health: Maxim Topaz at Columbia University is developing LLM-based methods to deliver information on social determinants of health, crucial for clinical decision-making.

6. Conversational AI Diagnostics: Google AI’s research into integrating LLMs with medical imaging systems for diagnostic accuracy further illustrates the potential of LLMs in healthcare.

These few examples underscore the varied applications of LLMs in healthcare and the necessity for regulatory frameworks that can adapt to their diverse uses while ensuring patient safety and ethical use.

6. Navigating AI Regulations and Commercialisation

AI regulation across major global markets is complex and rapidly evolving, particularly in the United States, European Union, and the United Kingdom. Here we take a look at the most recent regulatory developments, initiatives, and policies that are shaping the integration, use, and commercialisation of AI technologies in healthcare. It underscores the diverse approaches and strategies adopted by these regions, balancing the pursuit of technological innovation with the critical demands of patient safety, data security, and ethical governance in the AI healthcare domain.

6.1 United States

Health Data, Technology, and Interoperability (HTI-1) Rule: The U.S. Department of Health and Human Services (HHS) finalized the HTI-1 rule in 2023, focusing on algorithm transparency in certified health IT. This rule establishes first-of-its-kind requirements for AI and predictive algorithms, ensuring they are assessed for fairness, appropriateness, validity, effectiveness, and safety.

USCDI Version 3: The United States Core Data for Interoperability (USCDI) Version 3 is set to be the new standard within the ONC Health IT Certification Program by 2026. It aims to enhance patient characteristics data for equity, reduce disparities, and support public health data interoperability. 

Enhanced Information Blocking Requirements: The HTI-1 rule revises certain definitions and exceptions related to information blocking, promoting efficient and secure exchange of electronic health information.

Industry and Leadership Involvement: In July 2023, leaders of seven top AI companies made voluntary commitments to support AI safety, transparency, and anti-discrimination. There’s a push for U.S. government rules for AI, advocating that AI companies set the terms for regulation

6.2 European Union

AI Act Proposal: The European Commission proposed a comprehensive legislative framework for AI, known as the AI Act. This framework categorizes AI systems based on risk levels and requires conformity assessments for high-risk AI systems, such as those used in healthcare. This is moving forward at pace and looks to be enacted in early Q2 2024. 

Territorial Applicability Concerns: There are concerns about the broad scope of the AI Act and its potential for overregulation due to a wide definition of AI, which might require compliance from entities not previously covered.

6.3 United Kingdom

National AI Strategy: The UK government published the National AI Strategy in September 2021, aiming to drive AI development over the next ten years. The focus is on creating a “proportionate, light-touch, and forward-looking” regulatory framework that adapts to new opportunities and risks.

Regulatory Approach: The UK’s regulatory approach involves defining the core characteristics of AI and allowing individual regulators to build on this definition as appropriate for their domains. The aim is to regulate the application of AI according to its use or sector.

MHRA Guidance on AI as a Medical Device: The MHRA outlined that AI as a Medical Device (AIaMD) will be treated as a subcategory of Software as a Medical Device (SaMD), with robust guidance provided but not separate from the guidance for software.

NHS AI Lab and National Strategy for AI in Health and Social Care: The NHS AI Lab is developing a National Strategy for AI in Health and Social Care, setting the direction for AI up to 2030. This includes efforts to engage the public and clinicians and ensure accessible guidance and legislation.

7: Deviceology – Your Regulatory Partner for AI in Healthcare

In the rapidly advancing field of AI in healthcare, Deviceology acts as a trusted partner for medical device companies looking to navigate the complex landscape of AI regulation and market entry. Leveraging our expertise and a comprehensive suite of services, we offer specialised support for AI as a Medical Device (AIaMD) and broader health tech innovations. Our services include:

1. Regulatory Compliance and Certification: We provide guidance on meeting the requirements of key regulatory bodies, including the EU’s Medical Device Regulation (MDR), the In Vitro Diagnostic Regulation (IVDR), U.S. FDA regulations, Brazil’s ANVISA regulations, and UK-specific regulations. This encompasses assistance with CE marking, UKCA marking, FDA 510(k) submissions, and ANVISA registration, ensuring your product complies with the General Safety and Performance Requirements as well as specific compliance standards in these regions. Our expertise covers the full spectrum of regulatory pathways to facilitate successful market entry in the EU, U.S., Brazil, and the UK.

2. Quality Management Systems: Our team assists in implementing and auditing Quality Management Systems compliant with ISO 13485, a harmonised and internationally recognized standard for medical devices. Additionally, we support adherence to U.S. FDA Quality System Regulations (QSR), ensuring compliance with both FDA’s QMS requirements and ISO standards. We guide companies through the Medical Device Single Audit Program (MDSAP), facilitating a single audit that satisfies multiple regulatory jurisdictions, including the U.S. FDA, Health Canada, and others. Our approach is designed to streamline the process of achieving compliance across various global regulatory environments.

3. Product Lifecycle Support: From concept to post-market, we guide medical device companies through every stage of the product lifecycle. Our comprehensive support includes initial risk analysis, regulatory strategy formulation, defining clinical trials protocols, design validation, market plan development, devising reimbursement strategies, product launch, and post-market surveillance. Our tailored approach ensures that every aspect of your product’s journey, from early development to patient delivery and beyond, is strategically managed to align with regulatory requirements and market expectations.

4. AI-Specific Regulatory Insight: Our team offers specialised knowledge in the application of AI in medical devices, closely reviewing emerging regulations as they are drafted and strategically planning how to assist our clients in meeting their obligations. We ensure compliance with current and evolving regulatory requirements in this dynamic field, staying ahead of industry trends and updates. In addition to providing guidance on existing standards, we actively engage in industry discussions and regulatory development processes, offering insights on the implications of new regulations and preparing our clients for future changes, ensuring they are well-equipped to navigate the regulatory landscape of AI in medical devices.

5. Market Access Strategies: We offer tailored strategies for market-specific adaptation and compliance, including navigating different geographical regulatory landscapes and reimbursement models. This is vital for companies looking to introduce AI-driven medical devices into new markets

6. Risk Management and Technical Documentation: Assistance in drafting technical documentation and managing risks in line with ISO 14971 standards, ensuring a comprehensive and compliant approach to medical device safety and performance

7. Conformity Assessments and Audits: We conduct comprehensive conformity assessments under relevant regulations and legislation, encompassing thorough audits of the manufacturer’s quality system and technical documentation reviews in preparation for external auditors. Our services extend to providing internal audit services and readiness reviews for clients prior to certification, ensuring they are fully prepared for the evaluation process. Moreover, we offer representation during certification audits, utilising our expertise to facilitate a smooth certification journey for our clients. 

8. Database and Regulatory Monitoring: Utilizing resources like the European Database on Medical Devices (EUDAMED), U.S. FDA databases, and Brazil’s ANVISA system, we offer a comprehensive view of the medical device lifecycle and regulatory changes within the EU, U.S., and Brazil. This includes monitoring updates and providing insights on evolving regulations and compliance requirements across these key markets. Our goal is to ensure our clients are always informed and ahead of regulatory changes, facilitating proactive adaptation and compliance in their global operations.

At Deviceology, our goal is to ensure your AI-driven medical devices not only achieve regulatory compliance but also succeed in the global market. Our expert team, equipped with extensive experience and a deep understanding of the evolving regulatory landscape, is ready to support your journey in bringing innovative health tech solutions to patients worldwide. Please contact us at to discuss your requirements and see how we can help!