Enterprise RAG architecture: the data pipeline

What is enterprise RAG architecture?

Retrieval-augmented generation (RAG) gives a language model a second source of knowledge: an index of documents it can search at answer time. The original paper by Lewis and colleagues combined a pre-trained sequence-to-sequence model with "a dense vector index of Wikipedia", accessed through a neural retriever.^[1] Enterprise RAG swaps Wikipedia for your own policies, product docs, tickets and contracts.

That makes the architecture two systems, not one:

• The retrieval layer: embed the question, search the index, filter, optionally rerank, and pass the best passages to the model.
• The data pipeline: collect content from source systems, parse it, chunk it, embed it, attach metadata and permissions, and keep all of that current.

Why do RAG projects fail on data?

Because retrieval can only return what the pipeline stored. If a table in a PDF was flattened into noise, a policy was indexed twice in two versions, or a deleted page still sits in the index, the model receives that context and answers from it. A better model does not fix a wrong passage.

A demo on a few clean files hides these problems. A production system pulls from many source systems, file formats and owners, and each one adds its own parsing, freshness and permission rules. Our guide to data preprocessing and normalization for LLMs covers the cleaning step in detail.

What does the ingestion pipeline do?

It turns each source document into records the vector store can serve. A workable sequence:

1. Connect to each source and record its document ID, owner, last-modified time and access list.

2. Parse the content into clean text, keeping headings, tables and page references where they matter for answers.

3. Chunk the text and give each chunk a stable ID that points back to its parent.

4. Embed each chunk and store the model name and version with it.

5. Validate a sample before it goes live: readable text, sensible chunk sizes, no empty or duplicate records.

Vector databases do not chunk for you. Pinecone's data modeling guide tells you to chunk in your application before upserting, store each chunk as its own record, and add metadata such as a document ID that ties chunks back to the parent document.^[2]

How should you chunk and embed?

Split on the structure the content already has. Headings in a manual, clauses in a contract and question and answer pairs in a support article are natural units. Fixed-size windows are a reasonable starting point, but test them against real questions before you commit. Our article on reranking and chunking for advanced RAG compares the options.

For embeddings, a few facts shape the design. An embedding is a list of numbers, and the distance between two embeddings measures how related they are.^[3] Models differ in output size: OpenAI's text-embedding-3-small returns 1,536 dimensions by default and text-embedding-3-large returns 3,072, and both accept a dimensions parameter to shorten the vector.^[3] That size has to fit your database's index limits. In pgvector, for example, the standard vector type can be indexed up to 2,000 dimensions and the half-precision halfvec type up to 4,000.^[4] Record the model with each chunk, because you need to re-embed everything with the new model if you switch.

Semantic search can miss exact terms such as part numbers or error codes. Our guide to hybrid search with dense and sparse embeddings explains how to add a keyword signal.

How do you keep the index fresh?

Treat every document as a unit that can change or disappear. When a source document changes, replace its chunks; when it is deleted, delete them. Pinecone recommends that when the number or order of chunks changes, you first delete all of a document's chunks with a metadata filter, then upsert the new ones.^[2]

That only works if the pipeline knows what changed. Use change events or modified-time checks from each source, run them on a schedule your users can live with, and alert when a source stops sending updates. Show the source and its date next to each answer so users can spot a stale passage.

Who can see what the model retrieves?

Anyone the index lets through, unless you stop it. OWASP's 2025 list for LLM applications includes vector and embedding weaknesses, with examples such as unauthorized access through weak access controls, leaks between users or tenants sharing one vector database, and data poisoning.^[5]

Its guidance maps directly onto the pipeline:^[5]

• Permission-aware stores: copy each document's access list into chunk metadata and filter on the user's groups at query time.

• Validated sources: index only from approved systems, and check content before it goes live.

• Logs: keep a record of what was retrieved, for whom, and when.

Filtering has a performance side. In pgvector, filters on an approximate index are applied after the index is scanned, so a narrow filter can return fewer results than requested; version 0.8.0 added iterative index scans to fetch more until enough rows match.^[4] Test permission filters with your narrowest real user groups. Checking answers against the retrieved sources is covered in AI hallucination verification pipelines.

Where should a team start?

1. Pick one question set with a clear owner, such as support answers or sales policy questions.

2. List the source systems behind it and who may see what.

3. Build the pipeline for those sources first: parse, chunk, embed, tag and delete.

4. Write a set of real questions with known answers and test retrieval before the model is involved.

5. Choose the store last. Pinecone vs pgvector compares two common options.

Our data infrastructure for AI service covers these steps as part of our AI consulting work.