RAG for SAP Project Documents: Build a Doc Assistant That Cites Its Sources

How to build an AI assistant that answers questions from SAP specs, change requests and cutover plans, with a citation for every answer and no guessing.

Wed Nov 18, 2026

Every SAP project generates a mountain of documents: functional specs, technical specs, change requests, test scripts, cutover plans, RAID logs, meeting minutes. Six months in, nobody remembers which document decided what, and the answer to "what's the effort on that interstate e-way bill CR?" is buried in a folder three levels deep.

The obvious idea is to "just ask ChatGPT". The problem is that a general chatbot doesn't know your project, and when it doesn't know, it often guesses with confidence. On a delivery project, a confident wrong answer is worse than no answer.

Retrieval-augmented generation (RAG) fixes this. This post explains how it works on SAP project documents, and what makes the difference between a demo and something a delivery team can trust.

What RAG actually does

RAG has a simple idea at its core: don't let the model answer from memory. Make it answer from your documents.

The pipeline has four steps:

  1. Chunk. Split each document into small, meaningful sections (a requirement, a paragraph of a cutover plan).
  2. Embed. Turn each chunk into a vector, a list of numbers that captures its meaning, using an embedding model.
  3. Search. When someone asks a question, embed the question too, and find the chunks with the closest meaning. This is semantic search, so "how long will the e-way bill change take?" finds a chunk that says "effort: 5 person-days", even though the words differ.
  4. Answer with citations. Give only those chunks to the model, and instruct it to answer from them, cite the document and section, and say "I don't know" if the answer isn't there.

The three things that make it trustworthy

1. Citations on every answer

An answer like "5 person-days (ORION-CR-020, section 2)" can be checked in ten seconds. An answer with no source has to be trusted blindly. Make citations mandatory, and treat an uncited answer as a failure.

2. Refusing when the documents don't say

The test that separates a real assistant from a chatbot: ask it something that isn't in the documents. "What's the CFO's favourite colour?" should get "That isn't in your project documents", not a creative guess. Build this behaviour in and test it.

3. Project scoping enforced in code

Most SI teams run more than one project. A consultant on Project ORION shouldn't get answers from Project VEGA's documents.

The common mistake is to put "only answer about the user's project" in the prompt. A prompt is a request, not a control. The reliable way is to filter the search itself: the agent's search tool only returns chunks from the user's own project, applied in code before the model sees anything. Then even a cleverly worded question can't pull another project's data.

A practical architecture on Google Cloud

A setup that works well for a delivery team:

  • Cloud Storage (private bucket) for the source documents.
  • Cloud SQL for PostgreSQL with pgvector to store chunks and their embeddings, with a read-only database user for the assistant.
  • Gemini embeddings for semantic search.
  • MCP Toolbox for Databases exposing a small set of safe tools (like "search project documents") to the agent, on a private Cloud Run service.
  • An Agent Development Kit (ADK) agent on Gemini that calls those tools, applies project scoping, and writes cited answers.

No part of this needs direct access to your SAP system. It works on the documents your project already produces.

Don't skip evaluation

Teams often build the pipeline, try three questions, and call it done. Instead, keep a small evaluation set: 10 to 20 real questions with known answers and the document each answer should come from. Run it after every change.

When a question fails, check why:

  • Retrieval failure: the right chunk wasn't found. Fix chunking or add synonyms.
  • Phrasing failure: the right chunk was found but the answer was poorly written. Adjust instructions.

That split tells you exactly where to fix, instead of endlessly tweaking prompts.

Where this helps on real projects

  • Onboarding: new team members ask questions instead of reading 200 documents.
  • Change requests: find previous decisions, effort estimates and impacted specs in seconds.
  • Audits and steering committees: answer "when did we decide this, and where is it written?" with a citation.
  • Hypercare: pull the cutover and support procedures quickly when something breaks at 2 am.

Build one yourself

Our Lab 2: Grounded Doc Assistant for SAP Projects walks you through the whole build on your own Google Cloud project. It uses 36 synthetic project documents across two fictional rollouts, and covers:

  • pgvector on Cloud SQL
  • Gemini embeddings
  • MCP Toolbox with a read-only user
  • an ADK agent that cites every answer and only sees the user's own project

You get a 21-page guide, an auto-checker for every module, a 12-question evaluation set and a capstone where you add a new change request and query it.

→ Lab 2 on Brainix Tech

Not affiliated with or endorsed by SAP SE or Google. All documents, companies and people in the lab are synthetic.