Chatbot providers for own documents

Distinguish answer search from RAG

A maintained answer search finds approved entries and reproduces their wording. Retrieval-Augmented Generation, RAG for short, combines the search for suitable content with a language model that formulates an answer from it. For the selection it matters which of these scopes a provider actually describes.

RAG needs suitable search results and a controlled passing on of content. Technical integration alone does not yet decide whether the answer is usable for your specialist area. Check the result and its source location together.

Basics: Microsoft Learn: RAG, information retrieval and access control

Source maintenance is part of the product scope

Name the files and systems that are to be used. A one-off document import is something different from a connection that is maintained on an ongoing basis. Ask when changes are picked up, how deleted content disappears and how conflicting versions are handled.

Have the process shown with your own documents: one document is changed, a second is replaced and a third is blocked for a user group. Then check the same questions again. This makes updating and access rights visible as usable processes.

Test with different user accounts

A login does not yet prove that the rights from every connected system are retained. Check separately who may search a source, open a source location and see its content in an answer. Also include the withdrawal of a permission in the test.

The specific implementation depends on sources and architecture. A reference to a particular platform therefore does not replace a demonstration of your own permissions situation.

Basics: Microsoft Learn: RAG, information retrieval and access control

Build a small specialist test set

• Questions with a clear, current source.
• Questions that need information from several sections.
• Questions on outdated or contradictory documents.
• Questions without sufficient information.
• Questions about content that the respective test account should not be able to access.

Shape the knowledge base and its maintenance with T-NEX · Clarify data protection questions for the specific use