AI governance for LLM training

What does AI governance for LLMs cover?

It covers every point where company data meets a model: what you train or fine-tune on, what you let the model retrieve at answer time, which vendors see the data, who can query the result, and how you check it before and after launch. Responsible AI principles, such as fairness, accountability and explainability, sit on top of the same controls.

NIST's AI Risk Management Framework (AI RMF 1.0) is a useful spine. It is intended for voluntary use and is organized around four functions: Govern, Map, Measure and Manage.^[1] NIST also publishes a Generative AI Profile, NIST AI 600-1, for risks specific to generative models.^[1]

Which data can go into training?

Less than most teams assume. Once data shapes model weights, you cannot simply delete one record from the model, and models can repeat rare training text. OWASP lists Sensitive Information Disclosure as LLM02 and Data and Model Poisoning as LLM04 in its 2025 Top 10 for LLM applications.^[2]

Classify sources first, then decide the route:

Retrieval keeps sensitive records in a store you control, where you can enforce permissions and remove a document the day a legal hold or deletion request arrives. Our guide to enterprise RAG pipelines covers that setup, and fine-tuning open-weight models covers when tuning is still worth it.

What do vendor terms say on training?

Read the current page for each product you use, because terms differ by product and change. OpenAI's platform docs say that since 1 March 2023, data sent to the OpenAI API is not used to train or improve its models by default, unless the customer opts in.^[3] Anthropic says that by default it will not use inputs or outputs from its commercial products to train its models.^[4]

Consumer apps and free tiers can work differently, which is why a staff policy on approved tools matters as much as the API contract. Record each vendor's training, retention and sub-processor terms in one register and review it at renewal.

Which controls protect IP and PII?

The same controls you use for any sensitive system, applied to models and their data stores:

• Least-privilege access. Training data, vector stores and fine-tuned weights each get named owners and role-based access. Retrieval must check the user's permissions before a chunk reaches the prompt; OWASP lists Vector and Embedding Weaknesses as LLM08.^[2]
• Minimization and redaction. Strip or mask personal data before it enters a training set or index.
• Lineage. Tag every document with source, owner, classification and version, so you can trace and remove it later.
• Logging. Log prompts, retrieved sources and outputs for audit, and treat those logs as sensitive data themselves.
• Input and output guards. Prompt Injection is LLM01 and System Prompt Leakage is LLM07 on the OWASP list.^[2] See our guide to prompt injection security gateways.

How do you test and document a model?

Test before launch and keep testing. Red-team exercises should try to make the model reveal training data, system prompts or records the user should not see, and every finding should feed a fix and a retest. Add accuracy checks for answers that matter; our hallucination verification guide covers that.

Where a model informs decisions about people, such as hiring or credit, add a pre-launch impact assessment, bias testing across the groups affected, and a named human owner who can explain and overturn an output.

Document each model with a short model card: intended use, data sources and classes, known limits, test results, owner and approval date. Then write an incident process that names who can switch a model or index off, who investigates, and who informs customers or regulators when personal data is involved.

What does the EU AI Act require now?

The AI Act, Regulation (EU) 2024/1689, entered into force on 1 August 2024.^[5][6] Prohibited AI practices and AI literacy obligations applied from 2 February 2025, governance rules and obligations for general-purpose AI models from 2 August 2025, and the Act became applicable on 2 August 2026 with some exceptions.^[5]

The timing changed. The Commission proposed the Digital Omnibus on AI on 19 November 2025.^[5] It was adopted as Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026, and entered into force on 27 July 2026.^[7] It moves high-risk obligations to 2 December 2027 for Annex III systems and to 2 August 2028 for AI in products covered by the EU product laws listed in Annex I.^[7] It also rewrites AI literacy as a duty for providers and deployers to "take measures to support the development of AI literacy of their staff".^[7]

Model providers can also follow the General-Purpose AI Code of Practice, published on 10 July 2025, a voluntary tool with chapters on transparency, copyright, and safety and security.^[8] Our EU AI Act guide goes further. This is not legal advice: confirm your obligations with your own legal adviser.

Where should a team start?

1. List every AI tool and model in use, with owner, vendor and data classes.
2. Classify data sources and decide training versus retrieval for each.
3. Record vendor training and retention terms in one register.
4. Add access checks, redaction and logging to every retrieval path.
5. Red-team, write a model card and agree an incident owner before launch.

Our AI governance service covers these steps as part of our AI consulting work.