--- title: "Six Ways LLMs Can Trip You Up in GMP" description: "Discover six risks of using LLMs in GMP, from hallucinations and inconsistent outputs to data leakage, and how to manage them responsibly." url: https://qbdgroup.com/en/blog/llm-pitfalls-gmp-generative-ai-risks type: "Blog post" language: en published: 2026-10-14 author: "Elias Muys & Jan Appelmans" category: "Software Solutions & Services" publisher: "QbD Group" citation: "QbD Group, \"Six Ways LLMs Can Trip You Up in GMP\", https://qbdgroup.com/en/blog/llm-pitfalls-gmp-generative-ai-risks" --- # Six Ways LLMs Can Trip You Up in GMP > Discover six risks of using LLMs in GMP, from hallucinations and inconsistent outputs to data leakage, and how to manage them responsibly. Generative AI is powerful, but it behaves very differently from the rule-based systems GMP teams know. Large Language Models (LLMs) can help draft documents, summarise information and support routine quality activities. But their outputs are not always predictable, accurate or consistent, and the risks are not necessarily obvious to the people using them. In a GMP environment, where decisions need to be justified, documented and traceable, those differences matter. **The challenge is not simply to introduce AI. It is to understand how it can fail and design appropriate controls around those failures.** Here are six pitfalls GMP teams should be aware of, plus one that becomes increasingly important as AI gets better.

In This Blog Post:

## 1. Same Question, Different Answer: LLM Outputs Are Not Always Consistent Traditional rule-based systems generally follow predefined logic. Given the same inputs and conditions, they are designed to produce consistent results. LLMs work differently. They generate responses by predicting the next token in a sequence. A parameter called **temperature** influences how much randomness is introduced during generation. A lower temperature generally makes outputs more repeatable, while a higher temperature allows for greater variation. But there is a complication. **The vendor often controls that setting, not the user.** Even when temperature is low, identical prompts do not necessarily guarantee identical responses. ### Why Does This Matter in GMP? Imagine using an LLM to support the initial assessment of a deviation. If the same description produces different interpretations across repeated runs, how do you demonstrate that the process is sufficiently consistent for its intended use? That does not automatically make the technology unsuitable. It means repeatability and output variability need to be considered during the risk assessment.

Depending on the application, appropriate controls might include:

The important point is that **repeatability should be demonstrated where it matters, not simply assumed**. ## 2. Confident When Wrong: LLMs Can Hallucinate One of the most familiar risks of generative AI is also one of the most dangerous in regulated environments. LLMs can produce incorrect information, fabricated references or flawed reasoning while presenting the answer with complete confidence. Unlike a human expert, an LLM has no reliable understanding of when it knows something and when it is guessing. And the response may look perfectly credible. ### Why Does This Matter in GMP? Consider an LLM drafting part of a deviation investigation. It might produce a convincing explanation of a potential root cause, supported by what appears to be a relevant regulatory reference. But what if that reference does not exist? Or the reasoning overlooks an important piece of evidence? The danger is not necessarily an obviously nonsensical response. **It is an incorrect response that looks sufficiently professional to be accepted without further scrutiny.** For GMP applications, this makes verification essential.

Useful controls may include:

AI can accelerate the production of a first draft. It cannot replace the need to establish whether that draft is correct. ## 3. It Agrees With You: The Risk of AI Sycophancy LLMs can be surprisingly eager to agree with the person asking the question. This behaviour is often called *sycophancy*. Consider two ways of asking about a deviation. **Leading question:** "This is a Category 2 deviation, right?" **Neutral question:** "What category is this deviation, based on the available information?" The first question already suggests the expected answer. An LLM may be more likely to reinforce that assumption rather than challenge it. That is a problem when AI is being used to support decisions that require independent assessment. ### How Can GMP Teams Reduce This Risk? Start by asking neutral questions. Rather than asking an LLM to confirm your interpretation, ask it to assess the available information against clearly defined criteria. You can also challenge the initial response.

For example:

These techniques do not eliminate the risk of sycophancy, but they can help expose weaknesses in the initial response. **The goal is not to make AI agree with your judgement. It is to use AI to help examine that judgement critically.** And ultimately, the decision remains with the qualified professional. ## 4. It Forgets: More Information Does Not Always Mean Better Answers LLMs can process large amounts of information, but that does not mean they use every piece of it equally well. Upload too many batch records, and the model may overlook relevant information buried in the middle of the material. It may then produce a confident answer without making it clear that something important was missed. This is particularly relevant when organisations use LLMs to summarise lengthy documentation or compare information across multiple records. ### Why Does This Matter in GMP? Imagine asking an LLM to identify recurring deviations across a large collection of batch records. If it overlooks several relevant entries, the resulting summary could suggest that a pattern does not exist when it actually does. The output may still be well written. But the underlying assessment is incomplete. ### How Can You Manage This Risk? The author's recommendation is straightforward: **Feed the model less information, and structure it better.**

Where appropriate, consider:

Structured input can improve processing, but it is not a guarantee of completeness. For GMP-relevant applications, the ability to identify and verify the source information remains important. ## 5. It Changes Without Telling You: Vendor Updates Can Affect Performance With traditional validated systems, changes are generally managed through defined change-control processes. LLM-based services introduce a different challenge. The underlying technology may change without the organisation directly initiating that change.

For example:

These changes can influence outputs even when the organisation continues using the same interface. ### Why Does This Matter in GMP? Suppose an organisation has assessed an LLM-based application for a specific documentation task. The application performs acceptably during initial testing. But several months later, the provider changes the underlying model. Can the organisation still rely on the original assessment? Not necessarily. **If the technology changes, its performance against the intended use may need to be reassessed.** That is why vendor transparency matters.

For GMP-relevant applications, organisations should consider:

The level of control should be proportionate to the intended use and associated GMP risk. But relying on an external AI provider does not remove the organisation's responsibility for understanding and managing that risk. ## 6. It Gets Used Regardless: Uncontrolled AI Creates New Risks There is another risk that is easy to underestimate. Employees may use generative AI whether or not their organisation has formally approved it. If companies do not provide a safe internal AI environment, people may turn to personal accounts or publicly available tools. And that can create risks that are difficult to detect or control. ### Why Does This Matter in GMP? An employee might upload part of a deviation report, an internal SOP or confidential manufacturing information to an external AI service to obtain a quick summary. The intention may be entirely practical.

But the consequences can include:

Simply prohibiting AI does not necessarily prevent this behaviour. It may instead push it outside the organisation's controlled environment. ### What Should Organisations Do? A more practical approach is to define where and how AI can be used safely.

That may include:

**The objective is not unrestricted AI adoption. It is controlled adoption that reduces the incentive for uncontrolled use.** ## One More Trap: The Illusion of Human Oversight There is one final risk that deserves particular attention. As AI models improve, their mistakes may become less frequent. That sounds like good news. And in many ways, it is. But there is a paradox. **A draft that is 97% accurate may be easier to rubber-stamp than one that is only 50% accurate.** When an AI system produces consistently poor outputs, reviewers quickly learn to question everything. When it produces consistently good outputs, reviewers may become less vigilant. The remaining mistakes can become harder to spot precisely because the surrounding content appears so reliable. This creates an illusion of effective human oversight. A person may still review and approve the output, but that does not necessarily mean the review is sufficiently critical. ### Human in the Loop Means Expert in the Loop For GMP-relevant applications, meaningful human oversight requires more than someone clicking an approval button.

The reviewer needs the expertise to:

As AI performance improves, these capabilities become more important, not less. **The better the model, the more your reviewer's expertise matters.** That is perhaps the most important lesson of all. ## Managing LLM Risks in GMP Starts With Understanding Their Limitations Generative AI can bring real value to GMP environments. But its risks are not identical to those of traditional software. Inconsistent outputs, hallucinations, sycophancy, incomplete information processing, vendor-controlled changes and uncontrolled employee use all require consideration. The answer is not necessarily to avoid AI. Nor is it to assume that conventional controls will automatically address every new risk. The more useful approach is to understand the technology's limitations, assess the risks associated with each intended use and design proportionate controls. Above all, organisations need to ensure that qualified people remain capable of critically evaluating the outputs on which GMP-relevant decisions depend. **AI can support expert judgement. It should not quietly replace it.** ## Build a Risk-Based Approach to AI in GMP Introducing LLMs into a GMP environment requires more than selecting the right technology. Organisations need to understand how AI behaves, assess its intended use and establish appropriate governance, validation, data integrity and human oversight controls. QbD Group helps life sciences organisations navigate the use of emerging technologies in regulated environments, combining expertise in GMP, quality management, computerised system validation and risk-based compliance. --- Source: https://qbdgroup.com/en/blog/llm-pitfalls-gmp-generative-ai-risks — © QbD Group. Quote freely with attribution and a link back.