“Hey Copilot, can you write an article about Copilot?” – “Of course! I’d be happy to help you.” This is the response from Microsoft’s AI assistant when you ask it for help for writing an article. But Copilot doesn’t just promise support in the creative field. Copilot for Microsoft 365 is integrated into all Office apps and is intended to become a permanent assistant in everyday working life – when writing emails or creating presentations as well as summarizing meetings and much more. But can Copilot really help in a professional context? The uncertainty is justifiably great.
We tested Copilot for Microsoft 365 in our day-to-day work and talk about our experiences in this blog article.
No easy task - the expectations of an AI assistant are high
- …perform tasks reliably, efficiently, accurately and consistently
- …be easy to use and yet be able to process complex requests – without compromising data protection or data security
- …have a strong natural language understanding
- …be able to put things into context
- …learn and adapt over time.
First AI solutions show where the journey is heading
Certainly, many of us have already made their first attempts with AI tools such as ChatGPT and made both fun and serious requests. Meanwhile, there are many areas in which AI tools are involved. They give an idea of the potential of AI assistants.
What is our experience with Copilot?
Whether it’s communicating by email, chat or meeting or creating a presentation or Excel spreadsheet: There are many situations in which an AI assistant could be incredibly useful. Can Copilot for Microsoft 365 add value in such situations?
Creating PowerPoint presentations
Task and test result: We asked Copilot to create a presentation about phishing. We provided the bot with key questions – without success. Although the content was certainly suitable for the topic, it was not suitable for us and our services. The slides were also of little use as a starting point. Only with the help of a company-specific article on the topic, the AI was able to achieve reasonably useful results: Copilot distributed the content appropriately on slides. However, we had to make improvements in a few places, especially with the wording, and the result was still far away from being a presentation that corresponded to our corporate design. Referring Copilot to our EBF template didn’t help either.
Conclusion: In addition to the necessary adjustments to the content, there was still a lot of work to be done to create a uniform presentation that matched our brand.
Emails in Outlook
Task and test result: We asked Copilot to provide positive feedback on a cooperation request. The tool created a text that was largely appropriate in terms of content, but did not manage to find the right style. Initially, the text was written far too stiffly and – after a new instruction – ended up using language that was too casual. Copilot was also unable to incorporate the overall context – neither with this request nor with the request to create an agenda for a customer event. We almost always had to revise the content.
However, Copilot was valuable when it came to summarizing long emails. This worked well even for conversations that consisted of several emails. Links were provided to jump to the respective emails to read further details.
Conclusion: While Copilot offers little support for writing emails, the summaries are indeed helpful. However, it is questionable whether you can and want to rely on them 100%.
Functionality in Teams
Task and test result: We instructed the AI assistant to create a summary of the topics that had come up in a chat over the past week. The result was surprisingly good and also contained references to the relevant messages. This worked for a chat as well as for a meeting accompanied by AI, and in the latter case Copilot was able to summarize the discussion in real time. The tool was also able to compile tasks resulting from chats or meetings – although not every to-do made it onto the list.
Conclusion: All in all, Copilot can certainly provide support in the complex chat and meeting landscape and allows employees to avoid having to attend every meeting if it doesn’t fit in with their schedule. The AI assistant can also help to keep track of topics, maintain an overview and provide a summary at the end of a meeting. Nevertheless, you can’t rely on it completely – and the assistant certainly can’t replace personal contact.
Cross-app request
Task and test result: In the Copilot web interface, we instructed the AI to summarize which topics were discussed with a certain person in a defined period of time. Similar to Outlook and Teams, the results were surprisingly good. However, the summaries were often incomplete and the AI was not always able to correctly classify when a person was only cc’d in an email – meaning without direct interaction.
Conclusion: Copilot’s summary serves as a rough guide and can provide good points of reference in a hurry. But here, too, we were left with the feeling that we cannot rely on it 100%.
What are four key success factors for Copilot?
- Use Cases: The first step for companies is to find out whether Copilot offers added value, for whom and for which tasks. If no use case is found, companies should regularly check whether new options may arise in the future.
- Training: If there are suitable use cases for Copilot, it is important to train the employees who are supposed to work with the AI assistant. After all, they will only do so successfully if they know what they can use the bot for.
- „Prompting“: “Prompting” needs to play a key role in the training. After all, Copilot needs precise instructions to deliver the desired results. Since this is not so easy, companies should train employees on how to give prompts and how they should follow up if the feedback is not appropriate.
- Transparency: Companies that use AI should be transparent about it. A dedicated AI guideline can help here. It can provide clarity both internally and externally and also take into account the issue of data protection.
Our conclusion on Copilot
The benefits of Copilot depend heavily on the respective use case and the users’ prompting skills. When used correctly, Copilot can certainly save time and provide a certain level of orientation for Teams and Outlook by providing summaries. When it comes to personal communication and creative tasks, however, the results are less satisfactory – and the overall impression remains that we cannot rely on the tool in many cases so far.
In addition, we should not forget how important personal exchange is and that behind all the optimization there are sometimes completely different problems – such as too many meetings.
In any case, the decision to implement Copilot and the selection of the user group should be carefully considered due to the not inconsiderable costs and the introduction should be well supported. The tool does not yet appear to be suitable for mass use – and it will probably be some time before AI assistants like Copilot are really reliable, understand the context and match us and our style.
That’s why Copilot didn’t come into play for this article either: after the tool had created an unusable draft article, we replied: “I’ve changed my mind and would rather write the article myself.” – “Ok. If you have any questions, I’m available.” Maybe next time. Because one thing is certain: it’s worth continuing to follow the developments.
Copilot test as whitepaper
Download our Copilot whitepaper to receive the practical test in PDF format – with further exciting statistics and possible applications for AI. (German language)
