Soon, all software will come with an intelligent copilot assistant.
This is the word from Kevin Scott, Microsoft’s chief technology officer, who explains that a copilot is an application that uses modern AI and large language models to assist you with a complex cognitive task – from writing a sales pitch or catching up on a missed meeting to generating images for a presentation or planning a themed dinner party.
Microsoft introduced the concept of a copilot nearly two years ago with GitHub Copilot, an AI pair programmer that assists developers with writing code. This year, Microsoft rolled out copilot experiences across its core products and services, from the AI-powered chat in Bing that’s changing how people search the internet to Microsoft 365 Copilot, GitHub Copilot X, Dynamics 365 Copilot, Copilot in Microsoft Viva and Microsoft Security Copilot.
This week at its Microsoft Build developers conference, Microsoft announced that it has expanded this ecosystem of Microsoft Copilots to include Copilot in Power BI and Copilot in Power Pages in preview, Copilot in Microsoft Fabric, available in preview soon, and Windows Copilot, which will start to become available for preview in June.
The company also introduced new features that will help developers build their own copilots and next-generation AI applications. This includes new tools called plugins that make copilots more useful by allowing them to interact with other software and services.
“You may look at Bing Chat and think this is some super magical complicated thing, but Microsoft is giving developers everything they need to get started to go build a copilot of their own,” Scott says. “I think over the coming years, this will become an expectation for how all software works.”
Expanding the plugin ecosystem
OpenAI introduced ChatGPT plugins in March. Microsoft announced plugins for Bing earlier this month. That technology allows ChatGPT and Bing Chat to help you find and book a restaurant reservation using an OpenTable plugin, for example.
Now, Microsoft is adopting the same open plugin standard that OpenAI introduced for ChatGPT, enabling interoperability across ChatGPT and the breadth of Microsoft’s copilot offerings. That means developers can now use one platform to build plugins that work across both business and consumer surfaces, including ChatGPT, Bing, Dynamics 365 Copilot, Microsoft 365 Copilot and Windows Copilot. Microsoft also announced it is bringing Bing to ChatGPT as the default search experience.
As part of this shared plugin platform, Bing is adding to its support for plugins. In addition to the ones previously announced for OpenTable and Wolfram Alpha, it will also have Expedia, Instacart, Kayak, Klarna, Redfin, TripAdvisor and Zillow among many others in the Bing ecosystem.
In addition, developers will now be able to extend Microsoft 365 Copilot with plugins. Plugins for Microsoft 365 include ChatGPT and Bing plugins, as well as Teams message extensions and Power Platform connectors – enabling developers to leverage their existing investments. And developers will be able to easily build new plugins with the Microsoft Teams Toolkit for Visual Studio Code and Visual Studio.
Expanding the plugin ecosystem
Developers will also be able to create, test and deploy their own plugins in a number of ways – to eventually deploy for use with Microsoft Copilots and to augment the capabilities of their own applications built with generative AI technology.
Microsoft is releasing a set of capabilities to facilitate the creation of plugins that work across its copilot surfaces. Visual Studio Code, GitHub Copilot and GitHub Codespaces will make it simple for developers to create, debug and deploy new plugins, for example, and Azure AI will add capabilities to run and test plugins on private enterprise data. Once created, these plugins will work across Microsoft’s Copilot experiences.
John Montgomery, Microsoft corporate vice-president of program management for the AI platform, notes that the addition of plugins will accelerate the pace at which Microsoft customers can develop innovative applications with Azure OpenAI Service, which provides access to OpenAI’s generative AI models including GPT-4 and is currently serving more than 4 500 companies.
New tools for building your own copilot
While Microsoft has deployed generative AI technology into the company’s products and services, developers will eventually build most of the copilots in the world, according to Scott.
“They will understand a particular thing that they or their users are trying to accomplish, and they will use this AI software development pattern to go build those things for those users,” he says.
Developers can use tools such as the Semantic Kernel software development kit that Microsoft released to open source in March to integrate large language models with conventional programming languages. The kit includes tools for memory and orchestration and support for plugins, which accelerates the development of copilots, according to Montgomery.
At Microsoft Build, the company also introduced new tools to simplify the development of copilots. For example, with the new Azure AI Studio, developers can now more easily ground conversational AI models on their private data. With a new capability called Azure OpenAI Service on your data, developers can uncover organization-specific insights from data, text and images using natural language-based application interfaces. To further extend the capabilities of large language models, Microsoft also announced support for plugins with Azure OpenAI Service. Azure AI also now supports Azure Machine Learning prompt flow, a sophisticated prompt engineering tool, notes Montgomery.
“We’ve built a bunch of these copilots. We’ve looked at a bunch of the different architectures out there for how you load other information in, how you create reproducible content, how you feed the model the right stuff so that it gives the right answers,” he said. “Build is going to be a celebration of all the things we’ve learned, the tooling we’ve created and bring that out to the world.”
Responsible AI updates
Importantly, developers also need to ensure the copilot returns the intended results and avoids outputs that are biased, sexist, racist, hateful, violent or prompt self-harm, notes Sarah Bird, a partner group product manager at Microsoft who leads responsible AI for foundational technologies.
At Microsoft Build, the company announced that Azure AI Content Safety is in preview. This new Azure AI service helps developers create safer online environments and communities with models that are designed to detect inappropriate content across images and text. The models assign a severity score to flagged content, indicating to human moderators what content requires urgent action.
“It’s the safety system powering GitHub Copilot, it’s part of the safety system that’s powering the new Bing. We’re now launching it as a product that third-party customers can use,” says Bird.
Azure AI Content Safety is integrated into Azure OpenAI Service, providing customers of generative AI seamless access to it. The service can also be applied to non-AI systems, such as online communities and gaming platforms, and the filters can be fine-tuned for context.
In addition, Microsoft announced new media provenance capabilities coming to Microsoft Designer and Bing Image Creator that will enable users to verify whether an image or video was generated by AI. The technology uses cryptographic methods to mark and sign AI-generated content with metadata about its origin.
A new platform for development
About four years ago, Scott and his colleagues hypothesised that generative AI systems based on large language models would become platforms. They would allow developers to build new applications and services with relatively little effort compared to more classical forms of machine learning that require getting data, choosing a model, training it, testing it, deploying it and so on.
“The point where we are today is just fantastic. You can take a large language model like GPT-4 and start using that to build applications,” Scott says. “We’ve established this new application platform called copilot.”
Microsoft believes copilot represents both a new paradigm in AI-powered software and a profound shift in the way software is built – from imagining new product scenarios, to the user experience, the architecture, the services that it uses, and how to think about safety and security. The Microsoft ecosystem, Scott added, is the place for developers to build copilots end-to-end.
“We have everything you need on Azure for making a copilot,” he said. “And those things work super well together, so trying your idea and iterating quickly will be easier to do on top of Azure than it will be any other way.”