Normalizing unstructured data for LLMs

What does normalization mean for LLMs?

It means rewriting raw files into one consistent form so that the same content always looks the same to the model and the search index. For a retrieval (RAG) system or a fine-tuning set, that covers how text is extracted, which characters it uses, what is removed, how it is split and which metadata travels with it.

Skipping it shows up later as missed search results, duplicate answers, broken tables and personal data in places it should not be.

Which data should you inventory first?

Start with a list, not a parser. For each source, record the system it lives in, its owner, its format, how often it changes, who may see it and whether it holds personal data. Typical B2B sources are policy and product documents, contracts, support tickets, CRM notes, wikis and shared drives.

Separate structured from unstructured data. Records in a CRM, ERP or billing database already have fields and keys; query them directly or turn the fields you need into short text records. Documents need the full pipeline below. Where one customer or product appears in several systems, agree which system is the source of truth before you index anything, or the model will retrieve conflicting answers.

How do you extract text from files?

Use a parser that keeps structure. Headings, lists and tables carry meaning: a price table flattened into one line of text loses which number belongs to which row. Converting each document to Markdown or HTML with headings and tables intact gives later steps something to split on.

• Born-digital PDFs and Office files: extract text with layout, and keep page numbers for citations.
• Web pages: strip navigation, footers and cookie banners, keep the main content and its headings.
• Scans and images: run OCR, then spot-check pages with tables, stamps or handwriting, where OCR errors cluster.

Then clean: fix broken encodings, normalize line endings and repeated whitespace, and remove repeated headers, footers and boilerplate.

Which Unicode form should you use?

NFC for stored text, in most cases. The Unicode standard defines four normalization forms: NFD and NFC rely on canonical equivalence, while NFKD and NFKC add compatibility equivalence.^[1] Canonical equivalents represent the same abstract character, so NFC makes "é" one consistent sequence whether it arrived precomposed or as "e" plus an accent.^[1]

Compatibility forms go further, folding characters such as ligatures and full-width letters. Unicode warns that forms KC and KD "must not be blindly applied to arbitrary text" because they can remove distinctions that matter to meaning.^[1] A practical rule: store NFC, and use NFKC only for derived keys such as search terms or deduplication hashes, after checking samples in each language you hold.

How do you remove duplicates and PII?

Deduplicate at two levels. Exact duplicates go by hash; near-duplicates, such as the same policy saved in three versions, need similarity matching and a rule for which copy wins. For training data, research by Lee and colleagues found that common datasets contain many near-duplicates, and that deduplicated models emit memorized text ten times less frequently.^[2] For retrieval, duplicates crowd out other results.

For personal data, NIST's guidance says PII "should be protected from inappropriate access, use, and disclosure".^[3] Detect names, emails, phone numbers and IDs, then mask, redact or exclude the document. Detection tools help, but Presidio's own documentation says there is no guarantee it will find all sensitive information.^[4] Keep the most sensitive sources out of the index entirely.

How should you chunk and tag content?

Split along the document's structure: sections, clauses and table rows, not arbitrary character counts. Defaults vary by tool; OpenAI's file search, for example, uses chunks of 800 tokens with a 400-token overlap unless you change them.^[5] Test a few sizes on your own questions, as our guide to RAG chunking and reranking explains.

Attach metadata to every chunk: source, title, section, owner, version, date and access group. Metadata powers filters, and OpenAI's vector store files, for example, accept up to 16 key-value attributes for filtering.^[5] Access tags matter for security too: OWASP lists Vector and Embedding Weaknesses as LLM08 in its 2025 Top 10.^[6] Our vector database comparison covers where these chunks live.

How do you keep the index current?

Treat preparation as a pipeline that runs again, not a one-off migration. Give every document a stable ID and a version, re-process only what changed, and delete old chunks when a document is replaced or withdrawn. Without deletion, the model can quote a policy you retired. Log each run so you can trace any answer back to the file and version it came from.

How do you know the data is ready?

Test retrieval before you connect a model. Write a set of real questions with known answers and source documents, then check that the right chunks come back near the top. Re-run the same set after every pipeline change. Our guides to enterprise RAG pipelines and hallucination checks cover the next steps.

Our AI data infrastructure service covers this preparation as part of our AI consulting work.