--- title: "Human in the Loop, Accountable on the Record: Governing AI in Clinical Trial Safety" description: "Learn how sponsors and CROs can govern AI in SAE and SUSAR management through human oversight, explainability, validation, and regulatory compliance." url: https://qbdgroup.com/en/blog/human-in-the-loop-ai-clinical-trial-safety type: "Blog post" language: en published: 2026-07-08 author: "Anam Ahmad" category: "Pharmacovigilance" publisher: "QbD Group" citation: "QbD Group, \"Human in the Loop, Accountable on the Record: Governing AI in Clinical Trial Safety\", https://qbdgroup.com/en/blog/human-in-the-loop-ai-clinical-trial-safety" --- # Human in the Loop, Accountable on the Record: Governing AI in Clinical Trial Safety > Learn how sponsors and CROs can govern AI in SAE and SUSAR management through human oversight, explainability, validation, and regulatory compliance. The conversation around artificial intelligence in pharmacovigilance has evolved. For sponsors and CROs, the question is no longer whether AI can support safety operations. The question is **how to deploy AI responsibly within existing regulatory frameworks while maintaining compliance, oversight, and accountability.** Regulators are already providing direction. The European Medicines Agency (EMA) has acknowledged the potential role of AI in adverse event management, signal detection, and other pharmacovigilance activities. At the same time, regulators consistently emphasise transparency, governance, explainability, and human oversight. Importantly, the regulatory frameworks that already apply to clinical trial safety, including Regulation (EU) No 536/2014, ICH E6(R3), ICH E2A, and GDPR, do not disappear simply because an AI model is performing part of the workflow. The challenge, therefore, is not AI adoption itself. **The challenge is ensuring that AI remains governed, explainable, and accountable within safety-critical processes.**

In This Blog Post

- Why human oversight remains essential in AI-supported pharmacovigilance - What a human-in-the-loop governance model looks like in practice - Why explainability is becoming a regulatory expectation - How validation and inspection readiness apply to AI-enabled workflows - What upcoming UK and EU developments mean for sponsors and CROs
## Why Human Oversight Is Not Optional Current UK and EU frameworks do not permit sponsors to transfer pharmacovigilance accountability to AI systems. Under ICH E6(R3), sponsors remain responsible for: - Trial oversight - Data governance - Quality management - Safety-related decision-making This accountability remains unchanged regardless of how much automation supports the process. **The legal responsibility for expedited reporting decisions continues to sit with the sponsor, not with the software provider, CRO, or AI model.** The practical consequence is straightforward. Medically qualified review remains essential. AI outputs require human verification, and pharmacovigilance expertise must remain central to safety-critical decisions. **AI should augment professional judgement, not replace it.** ## What Human-in-the-Loop Governance Looks Like in Practice Many organisations are now converging on a human-in-the-loop governance model. While the phrase is widely used, effective implementation requires specific operational controls. In a well-designed governance framework: - AI supports prioritisation rather than final determination - Predefined escalation thresholds trigger mandatory review - Confidence scoring influences oversight levels - Medically qualified professionals retain final decision-making authority This approach aligns far more closely with current regulatory expectations than models that aim for fully autonomous safety assessment. This is particularly important because core pharmacovigilance decisions remain highly dependent on clinical judgement. Causality assessments, expectedness evaluations, and SUSAR determinations frequently involve: - Complex medical histories - Disease progression that mimics adverse reactions - Oncology combination therapies - Incomplete temporal associations - Limited safety datasets In these situations, human interpretation remains essential. Fully automated SUSAR determination is, in most environments today, neither operationally reliable nor regulatorily defensible. ## Explainability: Being Able to Show Your Working Human oversight is only credible if it can be demonstrated. The EMA reflection paper on AI within the medicinal product lifecycle repeatedly highlights the importance of transparency, explainability, and appropriate governance. This creates a significant challenge for black-box models. Inspectors may reasonably ask: - Why a case was prioritised or deprioritised - How seriousness was classified - Which factors influenced expectedness assessments - How model performance is monitored over time Explainable AI methodologies exist precisely to make these questions answerable. In practical terms, explainability means reviewers can understand *why* the system reached a conclusion, not simply *what* conclusion it reached. For example, when a case is classified as low priority, reviewers should be able to see: - Which adverse event term was coded - Whether the event appears in the RSI - What causality assessment was assigned - Which data elements influenced the recommendation An opaque confidence score without supporting rationale creates a governance problem for reviewers and a documentation problem during inspections. ## Inspection Readiness Starts With Documentation Dedicated AI validation standards for pharmacovigilance remain limited. However, the underlying principles already exist within established GxP frameworks. Relevant expectations can be found across: - EudraLex Volume 4 Annex 11 - GAMP guidance - Computerised system validation (CSV) frameworks - Risk-based quality management approaches Organisations implementing AI within SAE and SUSAR workflows should maintain documented evidence of: - Intended use definitions - Validation strategies - Performance qualification activities - Change management controls - Version control processes - Audit trail functionality - Periodic review procedures - Vendor qualification activities - Business continuity planning Validation should always be proportionate to patient safety impact and regulatory risk. A narrative drafting tool may require a different validation strategy than a system influencing expedited reporting decisions. ## Regulatory Expectations Are Continuing to Evolve The governance landscape is becoming more complex. Two developments deserve particular attention from safety leaders. ### EudraLex Volume 4 Annex 11 Revision The forthcoming revision extends computerised systems principles more explicitly to artificial intelligence and machine learning applications. This signals increasing regulatory focus on governance and lifecycle management for AI-enabled systems. ### EU AI Act Implementation The August 2026 deadline for high-risk AI obligations introduces additional requirements around: - Conformity assessments - Technical documentation - Ongoing monitoring - Lifecycle oversight For sponsors operating across both the EU and UK, governance becomes even more nuanced. The UK MHRA has signalled a more principles-based approach that relies primarily on existing GxP frameworks rather than introducing dedicated AI legislation. As a result, a validation strategy designed solely around EU AI Act requirements may not automatically satisfy UK expectations. Many organisations will therefore benefit from maintaining jurisdiction-specific governance annexes within their AI quality documentation. ## Competitive Advantage Comes From Governance, Not Technology There is a tendency to view AI adoption primarily as a technology challenge. In reality, the strongest differentiator is governance maturity. Competitive advantage does not come from deploying the most sophisticated model. It comes from building the quality systems, oversight structures, validation practices, and organisational culture required to use AI responsibly. The organisations that will derive the greatest long-term value from AI are not treating implementation as a procurement project. They are treating it as a quality management initiative. --- Source: https://qbdgroup.com/en/blog/human-in-the-loop-ai-clinical-trial-safety — © QbD Group. Quote freely with attribution and a link back.