A week after announcing a series of updates to its enterprise suite of Copilot AI-powered products, Microsoft is rolling out updates to all users, including tools that help them understand what they see on their screens and answer questions. We're releasing new Copilot features on Windows.
The updated Copilot apps for iOS, Android, Windows, and Web are rolling out today, and all Copilot now has a “warmer” and “distinct” style, as Microsoft describes it. Microsoft is also bringing a chatbot to WhatsApp, allowing users to chat with Copilot via DMs. This is similar to the experience you get with other bots on Meta's messaging platform.
co-pilot vision
Copilot Vision displays what you see on your PC. Specifically, lenses will appear on sites you visit in Microsoft Edge. Copilot Vision is gated behind Copilot Labs, a new Copilot Pro-only opt-in program for experimental Copilot features that analyzes text and images on web pages and queries (e.g. “What's the food in this photo? What's the recipe?'' they.
Vision, which appears when you type @copilot in Edge's address bar, isn't exactly a technological marvel. Google offers similar search technology on Android and recently brought some of that technology to Chrome as well.
But Microsoft suggests Copilot Vision is more powerful and privacy-friendly than previous screen analysis features.
“Copilot Vision…suggests next steps, answers your questions, helps you navigate what you want to do, and assists you with tasks just by speaking to it in natural language,” Microsoft shared with TechCrunch. I wrote about it in a blog post. “Imagine you're trying to furnish your new apartment. Copilot Vision helps you find furniture, find the right color palette, consider every option from rugs to throws, and help you find the right color palette.” It helps us make suggestions on how to place things.”
Use Copilot Vision to ask questions about photos on the web. Image credit: Microsoft
No doubt they want to avoid another round of bad publicity from AI privacy failures. Microsoft emphasizes that Copilot Vision is designed to delete data immediately after a conversation. The company claims that processed audio, images, and text are not saved, at least in this preview version, and are not used to train models.
Copilot Vision is also limited in the types of websites it can interpret. For now, Microsoft is blocking the feature from working with paywalled “sensitive” content and limiting Vision's use to a pre-approved list of “popular” web properties.
What exactly does “sensitive” content mean? Pornography? violence? Microsoft won't say anything at this point.
Microsoft has recently found itself in legal trouble over accusations that it used AI tools to circumvent paywalls. In an ongoing lawsuit, the New York Times alleged that Microsoft allowed users to bypass paywalls by serving New York Times articles through a Copilot chatbot on Bing. According to the Times, when prompted in a certain way, Copilot, powered by OpenAI's model, which has a close partnership with Microsoft, will provide verbatim (or near-verbatim) snippets of paid articles. He says he will.
Microsoft said Copilot Vision is currently only available in the US and will respect the site's “machine-readable AI controls,” including rules that prohibit bots from scraping data for AI training. However, the company did not say exactly which management Vision would honor. Some are used. We've asked Microsoft for clarification.
Many large publishers have chosen to block AI tools from trawling their websites, not only for fear of their data being used without permission, but also to prevent AI tools from driving up server costs. If current trends continue, Copilot Vision may stop working on some of the web's top news sites.
Microsoft said it is committed to “accepting feedback” to allay concerns.
“Before rolling out broadly, we will continue to improve our safety measures and keep privacy and responsibility at the center of everything we do,” Microsoft said in a blog post. “No special processing will be performed on the content of the website you are viewing.” [with Copilot]There is also no AI training. Copilot Vision simply works with you to read and interpret images and text that appear on a page for the first time. ”
think deeper
Like Vision, Copilot's new Think Deeper feature is an attempt to make Microsoft's Assistant more versatile.
According to Microsoft, Think Deeper gives Copilot the ability to reason about more complex problems. This is thanks to a “reasoning model” that takes time to return a step-by-step answer.
Which inference model? When I asked, Microsoft was a little stubborn, only saying that Think Deeper uses “OpenAI's latest model fine-tuned by Microsoft.” Reading between the lines, there is no doubt that these are customized versions of OpenAI's o1 model.
“We designed Think Deeper to help with all kinds of practical, everyday challenges, such as comparing two complex options side-by-side,” Microsoft said in a blog post. “Think Deeper can help with everything from solving difficult math problems to comparing the costs of managing home projects.”
Microsoft has touted Think Deeper's potential quite a bit in its press materials. But assuming the underlying model is o1, it will definitely fall short in some areas. I'll be interested to see what enhancements Microsoft has made to the base model, and how future Think Deepers will address its limitations.
Think Deeper is available starting today to a limited number of Copilot Labs users in Australia, Canada, New Zealand, the United States, and the United Kingdom.
co-pilot's voice
The new Copilot feature that is now generally available is Copilot Voice (not to be confused with GitHub's Copilot Voice). Voice launches in English in New Zealand, Canada, Australia, the UK, and the US, and adds four synthetic voices that let you speak to Copilot and hear its responses aloud.
Image credit: Microsoft
Similar to ChatGPT's OpenAI advanced voice mode, Copilot Voice can pick up your tone during a conversation and respond accordingly, and you can interject at any time while Copilot Voice is responding. . A Microsoft spokesperson said the mode uses “the latest voice technology with new models fine-tuned for the Copilot app.” What technology? Which model? Mom's word for details.
There is one thing to note. Copilot Voice has time-based usage limits. Copilot Pro subscribers will have access to more time, but that number will “fluctuate” based on demand, Microsoft said.
personalization
Microsoft says Copilot will soon be able to be customized to your tastes and preferences thanks to new personalization settings.
When this setting is enabled, Copilot uses your past actions and history, as well as your interactions with other Microsoft apps and services (Microsoft doesn't say which apps or services), to Recommended usage.
“This provides both useful Copilot features and conversation starters to help you get your work done,” Microsoft wrote in a blog post.
Copilot personalization can be turned off in the Windows Copilot settings menu, but this is not planned for the foreseeable future in the UK or EU. However, users in other locations should start seeing this setting this afternoon.
Microsoft and the EU have a rocky relationship over the rollout of the company's AI products. In May, the EU could fine Microsoft up to 1% of its annual global revenue under the EU's online governance regime, the Digital Services Act, for failing to respond to requests for information focused on generative AI. Microsoft has been warned that there is. tool.
Many tech giants other than Microsoft, including Apple and Meta, are wary of launching AI tools in the EU, wary of violating EU laws governing data privacy and model deployment. .
A Microsoft spokesperson told TechCrunch: “We are evaluating options before offering this level of Copilot personalization to users in the European Economic Area (EEA) and a limited number of other countries.” he said. “Some features will not be available in the EEA until a later date.”