What’s the use case?
Companies are often required to fill out questionnaires about their products and services. Yet the answers needed to complete the questionnaires are usually spread across many documents and in the minds of department experts.
What if AI could bring all that knowledge together? The vision: to build an app that can answer questions about a company, its products, and services, using sets of documents as reference. At its core, this approach applies Retrieval Augmented Generation (RAG), a method that allows AI to combine the precision of factual retrieval with the fluency of natural language generation. The goal is to create a RAG-based system that works reliably – with built-in checks and approvals during the creation process.
What technology did we choose?
One option to bring this idea to life is Microsoft Copilot Studio. It works seamlessly with different document formats and integrates naturally into the Microsoft ecosystem, where most users already collaborate every day. By deploying the app in Microsoft Teams, the familiar chat interface becomes the bridge between people and AI.
Behind the scenes, some setup involving deployment and billing is essential to ensure reliability and scalability. This includes:
- 1. App authors need a MS 365 Copilot or Copilot Studio license inside the admin center assigned to their user
- 2. A MS Power Platform admin must establish a Pay-As-You-Go environment with MS Dataverse and Generative AI functions enabled. Apps created within the default environment would not be available for Pay-As-You-Go billing, once the default token limit is reached, the app would lose access to the models backing the app
What does the workflow look like?
Rather than simply uploading the required documents and starting a chat building process, we recommend taking a more controlled and structured approach.
In theory, one could create an agent in MS Copilot Studio, upload documents and a prompt, and deploy the app directly. From our experience, this will produce a basic demo, but not a dependable and real solution. The quality of the results depends heavily on the complexity of the source documents, and the ability of the model to comprehend the content. A more deliberate setup ensures that AI becomes a reliable partner that reflects the depth, accuracy, and trust your organization stands for.
The approach is this:
1. Building the foundation
Start by creating an intermediate agent in Copilot Studio. All documents belonging to a specific department or topic area (for example, data privacy or application documentation) can be uploaded into its knowledge base with clear, unique names, e.g. A.01-xxx, A.02-xxx.
Next, a prompt needs to instruct the agent to:
- read a single document starting with the prefix entered by the user, e.g. A-01.
- generate a list of questions and answers derived from that single document
- display the findings in the chat window
- convert the questions and answers into a format that we could download per document, CSV or JSON format.
This procedure needs to be repeated for each document. The result: a set of structured question–answer documents that capture the essence of the organization’s knowledge.
While it might be tempting to create another agent to compile these results or to process all documents simultaneously, our experience shows that such approaches led to lower-quality output.
Instead, we recommend compiling the extracted questions and answers into one source document (an Excel spreadsheet for example) and to repeat this process for all documents belonging to a single department and expertise group. Then repeat this process to generate another source document with information stemming from another department, until all of the resource documents from all departments are collected.
2. Human review and validation
Before any AI-generated content becomes “official,” it must pass through expert hands. Each source document or spreadsheet should therefore be shared with the responsible persons to review the generated content and correct any errors or inconsistencies. Once reviewed, the sheets can become trusted input material and the basis for the actual agent. This step adds an important layer of human quality assurance to the workflow as the results combine the efficiency of AI with the knowledge of human experts.
3. Building the actual “Questionnaire Expert”
With verified material in place, a new agent can be built. The validated source spreadsheets can be uploaded to its knowledge base and marked as ‘Official’ to indicate higher trust and priority for these documents. Original source documents can also be included as references but unofficial ones.
The prompt for this agent should then define clear rules:
- Answers should be provided from the official documents first.
- If no answer was found, search in an adjacent official document for a related question.
- If still unresolved, search in the source documentation.
This hierarchy introduces structure and trust into the model’s reasoning — elevating it beyond a basic RAG setup into a reliable, explainable assistant.
4. Deployment
Finally, the agent could be deployed as a MS Teams app to everyone who needs to fill in questionnaires.
Such an agent can save valuable time when completing complex questionnaires while ensuring that every answer is reliable, consistent, and traceable. The human level of control remains a crucial part of the AI workflow.
The technology behind the incredible models we use daily may seem like magic, but trust in AI should never be blind. By combining intelligence of machines with insight and knowledge from your company and experts, organizations can unlock the full potential of their data and achieve results that are both efficient and trustworthy.
Next time we will venture outside the boundaries of the Microsoft ecosystem. Stay tuned.
Jerry Amadon
Senior Advisor
