Microsoft's AI can now read your screen, and therefore the websites you're viewing.
The company on Thursday began rolling out a limited, U.S.-only preview of Copilot Vision, a tool that lets you use Microsoft Edge to understand and answer questions about the sites you're visiting. Gated behind Copilot Labs, an opt-in program for experimental AI capabilities, Copilot Vision analyzes text and images on web pages to answer questions like “What's the recipe for this lasagna?” I can answer.
Note that Copilot Labs requires a subscription to Microsoft's Copilot Pro plan, which costs $20 per month.
In addition to answering questions, Copilot Vision can handle tasks such as summarizing and translating text and spotlighting discounted products in your store catalog. It also acts as a gaming assistant, providing pointers during matches on Chess.com.
“When you choose to enable Copilot Vision, Copilot Vision can see the pages you're on, read them aloud to you, and discuss any issues you're facing together,” Microsoft said on TechCrunch he said in a blog post he shared. “It’s a new way to invite AI as you navigate the web, and it’s nestled neatly at the bottom of the Edge browser whenever you want to ask for help.”
Image credit: Microsoft
No doubt hoping to avoid further bad press due to AI privacy issues, Microsoft emphasizes that Copilot Vision deletes data after each session. The company says that, at least in this preview release, processed audio, images, and text are not stored or used to train models.
The types of websites that Copilot Vision can display are also limited. For the time being, Microsoft has blocked the feature from working for “sensitive” content protected by paywalls, and Vision is now available on a pre-approved list of “popular” sites. Limited.
What exactly does “sensitive” mean? Porn? Graphic violence? Microsoft won't tell.
“The list of allowed websites is determined on a case-by-case basis for each category,” a Microsoft spokesperson said. “We are starting with a small list of thoroughly tested sites, but we plan to add more sites over time.”
Microsoft's cautious approach is in part the product of legal disputes with news organizations. In one ongoing lawsuit, the New York Times alleges that Microsoft allowed users to bypass paywalls by serving NY Times articles through its Copilot chatbot on Bing.
Many large publishers have chosen to block AI tools from trawling their websites, not only for fear of their data being used without permission, but also to prevent AI tools from driving up server costs. Microsoft said Copilot Vision respects the site's “machine-readable AI controls,” including rules that prohibit bots from scraping data for AI training. However, the company did not say exactly which management Vision would honor. Some are used.
If current trends continue, Copilot Vision may stop working on some of the web's top news sites. But Microsoft said it is committed to “accepting feedback” to allay publishers' concerns.
“Some of the companies we work with are third-party publishers who are looking to understand how they can use Vision to help people better engage and make decisions on their pages. ” the company said in a blog post. “This information helps us design Vision to be more useful when you interact with web pages.”