Beyond data integrity: model drift, shadow AI and defending AI decisions in GxP

Proving an AI-generated record is accurate is now only half the job. You also have to defend the decision the model helped shape: that a competent person made it, using a system still inside its validated state, on data whose lineage you can produce. I call that second obligation decision integrity.

In Part 1, on translating ALCOA+ for AI, I argued that AI leaves the data integrity principles standing and raises the evidence bar underneath them. This is the harder half of the problem.

The bigger risk sits beyond the record: in the model's changing behaviour, in the data it learned from, in the commercial platform that updates it without telling you, in the colleague quietly using an unapproved chatbot, and in the reviewer who accepts a decision without understanding how it was reached. As AI starts to influence deviation assessments, batch record reviews, risk classifications and release decisions, you have to show the model was still working inside its validated state, and that the decision it helped shape was sound and defensible.

Like Part 1, this came out of a long discussion with the regulatory and validation team at BGO Software. I've kept the questions they pressed hardest on.

Key takeaways

• Decision integrity is the new layer. On top of proving the record accurate, you have to be able to defend the decision the AI helped shape.
• Model drift is a data integrity risk. A model behaving differently from the day you validated it has left its validated state, and nobody signed for the change.
• Shadow AI is already inside GxP records, with no attribution, no model version and no audit trail.
• Training data is a GxP input. If the model supports GxP decisions, its training data needs lineage, governance and impact assessment.
• Vendor models carry silent-change risk, because embedded AI can be updated without the regulated company being told.
• Inspection readiness means the full story of the decision, from the input to the accountable signature, with no gap in the chain.

Why is model drift a data integrity risk?

Because a model that behaves differently today from the day you validated it has quietly left its validated state, and nobody signed for that change. I treat drift the way I'd treat an unapproved change to any validated system.

Traditional validation assumes behaviour stays stable until someone introduces a controlled change. AI breaks that assumption, because the change can happen on its own. So decide before go-live what still counts as the validated model, and be clear about 3 things:

• the kind of data you expect coming in,
• the performance you'll hold the model to, and
• the points at which something should raise its hand and alert you.

Then watch it for the life of the system. Validation now runs across the whole lifecycle and doesn't end neatly at go-live.

An inspector won't stop at asking whether you validated the model. The sharper question, and the one I'd prepare for, is how you know it still does today what it did on the day you validated it, and what would make you go back and look again. I've set out the rest of that conversation in what an inspector asks about your AI.

The trap I see most often is treating retraining as quiet housekeeping. Retraining is a change. It deserves the same control as any other change, with a record of what the model looked like before and after.

If the model behaves differently today than it did the day you validated it, your validated state has quietly expired, and nobody signed for the change.

Are organisations watching the right AI risks?

Mostly no. Teams watch the model closely and almost everything around it hardly at all. I understand why: the model is the part the vendor demonstrates, so that's where attention goes. In my experience the risk usually sits in the plumbing around it.

These are the blind spots I'd name.

1. Training data, which very few teams treat as the GxP input it is.
2. Life after go-live, including drift and retraining.
3. Shadow AI, the unofficial use of tools like ChatGPT and Copilot that's almost certainly happening in the building already.
4. Decision integrity, the biggest of them and the one I keep coming back to.

We still pour our energy into proving the record is accurate. The harder question is whether the decision the AI helped shape was sound, and whether you could defend it. That's where I expect the next wave of inspection findings to land.

How serious is shadow AI for data integrity?

Serious, because it's invisible. In most organisations unvalidated AI is already sitting inside GxP documents, and Quality usually can't see it.

Data integrity is already one of the most cited categories in FDA drug GMP warning letters, roughly 61% of the 2021 letters by one European Pharmaceutical Review analysis. Shadow AI pours untraceable content into the very area inspectors look at hardest.

An SOP or investigation summary a model drafted carries no attribution, no version and no audit trail, and it reads as though a person sat down and wrote it. The scenario that worries me is a regulator finding it before the company does, inside a controlled document whose origin nobody can account for.

Banning it doesn't work. I've watched bans push the behaviour underground, where you can't see it at all. The better path is a sanctioned, governed tool for the low-risk drafting people are clearly already doing, and a very clear line around the work AI must never touch without review.

Picture an analyst dropping a messy investigation into a public chatbot, getting back a clean summary and pasting it into the report. It reads well. The source data has left the building, and no sentence can be traced back to where it came from. That's a data integrity gap hiding inside good writing.

Does training data become a GxP record?

Yes. If a model supports GxP decisions, the data it learned from is a GxP input and should be governed like one. I think it's the most under-discussed risk in the field right now.

Governing it means knowing where the data came from, knowing it's sound, knowing whether it represents what you now use the model for, and being able to show all of that when you're asked.

If the data later turns out to be flawed or biased, the problem reaches back across every decision the model has touched. The logic matches a contaminated reference standard: you assess the impact backwards, because fixing things from today onward leaves every earlier decision exposed.

Most firms couldn't produce the lineage of their training data this afternoon if you asked. That gap is a finding waiting to happen.

Picture a model that learned what a good batch looks like from years of history drawn mostly from one site. Put it to work across the network and it will quietly mark the other sites down. The model works exactly as built. The data was never representative, and nobody kept its lineage.

A validated model trained on unvalidated data is a contaminated system, and the contamination stays invisible until the inspection.

What are the risks of AI embedded in commercial platforms?

The central risk is silent change. The vendor updates the model behind the scenes, and your validated behaviour shifts with no change record on your side. Supplier oversight has to reach all the way to the model, and your audit trail has to take account of the vendor's.

Before I'd trust the output, I'd want to see:

• how the model was trained and validated,
• how and when it gets updated,
• whether you're told before a change lands,
• how it performs for your particular use, as distinct from in general, and
• how the vendor will stand behind you during an inspection.

Notification of change belongs in the contract, where goodwill can't quietly lapse. And accountability stays with you. The regulator holds you responsible for the decision, whoever's model produced it, and “the vendor takes care of that” was never going to be a defence.

Here's how it plays out. A QMS vendor improves its deviation classifier in a routine release. Overnight, borderline events start being categorised differently. You validated the old behaviour, you're now running the new one, and you found out from a trend in your own data instead of a change notice. If you build these platforms, my advisory for quality-tech and AI vendors covers how to answer these questions before a customer asks them.

What does AI inspection readiness actually require?

The whole story of the decision, end to end, told well enough that a regulator could rebuild it without me standing in the room. The evidence pack has to carry 6 things.

1. The intended use and risk assessment, meaning what the model is for and what could go wrong.
2. What was validated, and the acceptance criteria you held it to.
3. The decision record: the input or prompt, the model and its version, the output, and the person who accepted it.
4. Monitoring evidence that the model was inside its validated state at that moment.
5. The change and retraining history since.
6. Evidence of human oversight: who reviewed it, what they were checking, and that they had the standing to overrule it.

Put as simply as I can, I want to show that a competent person, using a controlled system performing within limits we'd defined, made an accountable decision, and that I can prove every link in that chain.

It's close to a batch record, for a decision in place of a product. You have the inputs, the equipment in the shape of the model and its version, the in-spec evidence from your monitoring, and the signature at the end.

Do regulators need a different inspection model for AI?

Not yet, and I think applying the existing frameworks is the right call, because they stretch further than people give them credit for. ALCOA+, Annex 11, 21 CFR Part 11, GAMP 5 and ICH Q9 between them cover most of what AI needs, as long as you read them honestly. The gap is in specific guidance, and that's arriving through the draft EU GMP Annex 22, the EU AI Act, the ISPE GAMP Guide: Artificial Intelligence and the NIST AI Risk Management Framework.

Where the current model will strain is with systems that keep learning and changing in production. An annual snapshot inspection suits a system that holds still. I expect a move towards continuous evidence and monitoring expectations, inside the framework we already have.

Think of inspecting a process that retunes itself every shift. A single annual photograph tells you very little. The direction of travel is inspectors reading your monitoring evidence as well as your binders.

What is the industry still underestimating?

Decision integrity, and close behind it, the fact that validation is now a continuous obligation. We spent years getting data integrity right, and AI lifts the bar to a harder question: whether the decision itself was sound, on top of whether the record of it was accurate.

People still treat AI validation as something you complete and file away. The system keeps moving, so the obligation keeps going too. And training data as a GxP record in its own right is still barely on anyone's radar.

A perfectly accurate, fully attributable record of a bad AI recommendation still passes a data integrity check, and still leads to a wrong release.

That gap, between a clean record and a sound decision, is the space decision integrity exists to fill.

What question should quality leaders ask about AI?

“If this AI were wrong, how would we know, and how far back would the damage reach?”

Almost everyone asks whether the model is accurate. Far fewer ask how far a quiet failure would spread, or how long it might run before anyone noticed.

The follow-up hardly anyone asks is where AI is already sitting inside our GxP records without ever having been governed. Most leaders assume the answer is nowhere. In my experience it almost never is.

Sit a quality team down and ask what the first sign of drift would be, and who in the building would see it. The silence that follows is usually the real finding.

If you remember one thing

AI leaves ALCOA+ standing. It raises the bar on what it takes to satisfy it, and it adds one layer on top: you now have to defend the decision as well as the record.

The firms that come through this well will be the ones who can still answer the oldest question in our field, “can you prove it?”, for a system that reaches its own conclusions. 20 years ago that question meant “can you prove this number is real?” Today it's growing into “can you prove this judgement was sound?” The instinct behind it hasn't changed. The stakes have gone up.

Sachin Bhandari runs TrustBridge Compliance, a senior-led advisory for AI in GxP governance, CSV to CSA and digital validation in pharma, biotech and CDMOs. He sits on the Global GAMP Steering Committee, contributed to the EU GMP Annex 22 (AI) industry review, and wrote Validating AI in GxP Environments: A Practitioner's Handbook. The monitoring and decision-record layers are set out in the AI Governance Stack. If you want a senior read on your own AI evidence before an inspector asks, that's what the Inspection Readiness Review is for, and the free AI Readiness Check takes about 7 minutes.

A version of this article appears on BGO Software's blog as Beyond Data Integrity: Model Drift, Shadow AI, and Defending AI Decisions in GxP. My thanks to BGO Software's regulatory and validation team for the conversation behind it.

Sources: European Pharmaceutical Review on FDA warning letters · FDA draft guidance on AI to support regulatory decision-making (January 2025) · NIST AI RMF 1.0 · European Commission consultation on the Annex 22, Annex 11 and Chapter 4 drafts · ISPE GAMP Guide: Artificial Intelligence · ICH Q9(R1).

Questions leaders ask

What is decision integrity in GxP?

Decision integrity is the obligation to show that a decision an AI system influenced was sound and defensible, on top of showing that the record of it was accurate. It sits as a layer above data integrity.

Is model drift a GMP change?

In effect, yes. A model behaving differently from its validated baseline has left its validated state without a change record. Retraining is a change and should go through change control, with a documented before and after.

Is training data a GxP record?

Yes, where the model supports GxP decisions. Training data is a GxP input that needs lineage, governance and evidence that it represents the intended use. Flawed or biased training data triggers a backwards impact assessment across every decision the model has touched.

How do you control shadow AI in a regulated environment?

Give people a sanctioned, governed tool for low-risk drafting and draw a clear line around the work AI must never touch without review. Bans tend to push the behaviour out of sight.

Who is accountable when a vendor's embedded AI produces a wrong output?

The regulated company. Accountability stays with the company that made the decision, so supplier oversight has to reach the model itself, with notification of model changes written into the contract.

What goes in an AI decision evidence pack for an inspection?

The intended use and risk assessment; what was validated and against which criteria; the decision record, meaning the input, model, version, output and the person who accepted it; monitoring evidence for that moment; the change and retraining history; and evidence of meaningful human oversight.

Do regulators need a new inspection model for AI?

Not fundamentally. ALCOA+, Annex 11, 21 CFR Part 11, GAMP 5 and ICH Q9 cover most of it when read honestly. The gap is specific guidance, which is arriving through the draft EU GMP Annex 22, the EU AI Act, and the ISPE and NIST frameworks. Expect a move towards continuous monitoring evidence for systems that change in use.