Close Menu
TechBrunchTechBrunch
  • Home
  • AI
  • Apps
  • Crypto
  • Security
  • Startups
  • TechCrunch
  • Venture

Subscribe to Updates

Subscribe to our newsletter and never miss our latest news

Subscribe my Newsletter for New Posts & tips Let's stay updated!

What's Hot

Confusing accused of scraping websites that explicitly blocked AI scraping

August 4, 2025

Openai says ChatGpt is on track to reach 700m-a-week users

August 4, 2025

Elon Musk says he's reclaiming Vine's archives

August 4, 2025
Facebook X (Twitter) Instagram
TechBrunchTechBrunch
  • Home
  • AI

    OpenAI seeks to extend human lifespans with the help of longevity startups

    January 17, 2025

    Farewell to the $200 million woolly mammoth and TikTok

    January 17, 2025

    Nord Security founder launches Nexos.ai to help enterprises move AI projects from pilot to production

    January 17, 2025

    Data proves it remains difficult for startups to raise capital, even though VCs invested $75 billion in the fourth quarter

    January 16, 2025

    Apple suspends AI notification summaries for news after generating false alerts

    January 16, 2025
  • Apps

    Openai says ChatGpt is on track to reach 700m-a-week users

    August 4, 2025

    Elon Musk says he's reclaiming Vine's archives

    August 4, 2025

    Chariture.ai adds social feeds to its app

    August 4, 2025

    TrueCaller's call recording function will not work on iPhones from September 30th

    August 1, 2025

    TrueCaller's call recording function will not work on iPhone from September 30th

    August 1, 2025
  • Crypto

    North Korean spies pretending to be remote workers have invaded hundreds of businesses, CloudStrike says

    August 4, 2025

    Telegram's Crypto Wallet will be released in the US

    July 22, 2025

    Indian Crypto ExchangeCoindCX confirms $44 million stolen during hack

    July 21, 2025

    North Korean hackers blamed record-breaking spikes in 2025

    July 17, 2025

    Bitcoin surpasses $118K at the second highest high in 24 hours

    July 11, 2025
  • Security

    Confusing accused of scraping websites that explicitly blocked AI scraping

    August 4, 2025

    Commerce Dept.'s backlog reportedly has suspended Nvidia's H20 chip license

    August 1, 2025

    The Commerce Department's backlog reportedly stalling Nvidia's H20 chip license

    August 1, 2025

    Sex toy maker Ravens threatens legal action after fixing security flaws that published user data

    August 1, 2025

    Authorities seize servers of black suit ransomware gang

    August 1, 2025
  • Startups

    7 days left: Founders and VCs save over $300 on all stage passes

    March 24, 2025

    AI chip startup Furiosaai reportedly rejecting $800 million acquisition offer from Meta

    March 24, 2025

    20 Hottest Open Source Startups of 2024

    March 22, 2025

    Andrill may build a weapons factory in the UK

    March 21, 2025

    Startup Weekly: Wiz bets paid off at M&A Rich Week

    March 21, 2025
  • TechCrunch

    OpenSea takes a long-term view with a focus on UX despite NFT sales remaining low

    February 8, 2024

    AI will save software companies' growth dreams

    February 8, 2024

    B2B and B2C are not about who buys, but how you sell

    February 5, 2024

    It's time for venture capital to break away from fast fashion

    February 3, 2024

    a16z's Chris Dixon believes it's time to focus on blockchain use cases rather than speculation

    February 2, 2024
  • Venture

    What should the founder think about if they are trying to raise the series c?

    August 2, 2025

    Venture company CRV raises $750 million and returns capital to investors, then downsizing

    August 1, 2025

    Kleiner Perkins has a really good week

    August 1, 2025

    Details will be revealed about how Windsurf's VCS and Founders were paid from Google transactions

    August 1, 2025

    Basic Research Lab nabs over $30 million to build AI agents vertically

    August 1, 2025
TechBrunchTechBrunch

Confusing accused of scraping websites that explicitly blocked AI scraping

TechBrunchBy TechBrunchAugust 4, 20253 Mins Read
Facebook Twitter Pinterest Telegram LinkedIn Tumblr WhatsApp Email
Share
Facebook Twitter LinkedIn Pinterest Telegram Email


According to the Internet Infrastructure Provider CloudFlare, AI startups' stumps are raw scraping content from websites that explicitly indicate they don't want to be scraped away.

On Monday, CloudFlare published a survey that found that AI startups ignored blocks and observing their raw or scraping activities. The Network Infrastructure giant has accused them of obscuring their identity when trying to scrape web pages “to avoid website preferences,” CloudFlare researchers wrote.

AI products like Prplexity offer rely on gobbling large amounts of data from the Internet, and AI startups have repeatedly scraped text, images and videos from the Internet without permission to make the product work. Recently, the website has tried to fight back using the Web Standard Robots.txt file. It tries to tell search engines and AI companies whether they can index their efforts that they have seen a wide range of results.

The confusion seems to be willing to bypass these blocks by changing the bot “user agent”. According to CloudFlare, it is a number that not only changes an autonomous system network, or ASN, but also essentially identifies a large network on the Internet.

“This activity was observed across tens of thousands of domains and millions of requests per day. We were able to fingerprint this crawler using a combination of machine learning and network signals,” read CloudFlare's post.

Perplexity spokesman Jesse Dwyer dismissed the CloudFlare blog post as “sales pitch” and added an email to TechCrunch that said it “indicates that the content was not accessed.” In a follow-up email, Dwyer insisted on the CloudFlare blog a bot named “Not us.”

CloudFlare said the action was first noticed after customers complained that they were baffled and raw and rubbed the site in distress, especially to block known bots in Prplexity. CloudFlare then ran tests to check and confirmed that the confusion was avoiding these blocks.

TechCrunch Events

San Francisco | October 27-29, 2025

“Perplexity observed that it uses not only declared user agents, but also a common browser that impersonates Google Chrome on MacOS when declared crawlers are blocked,” CloudFlare said.

The company also said it has created Perplexity bots from its verified list and added new techniques to block them.

CloudFlare has recently taken a public stance against AI Crawlers. Last month, CloudFlare announced the launch of a market that will allow website owners and publishers to claim AI scrapers to visit their sites. CloudFlare CEO Matthew Prince sounded the alarm at the time, saying that AI was breaking the internet, particularly the publisher's business model. Last year, CloudFlare launched a free tool to prevent bots from shaking websites to train AI.

This is not the first time that confusion has been accused of rubbing without permission.

Last year, news outlets such as wired claim that confusion was plagiarizing their content. A few weeks later, Perplexity CEO Aravind Srinivas was unable to answer immediately when asked to provide a definition of plagiarism in an interview with Devin Coldewey of The TechCrunch at The Disrupt 2024 Conference.



Source link

Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

Commerce Dept.'s backlog reportedly has suspended Nvidia's H20 chip license

August 1, 2025

The Commerce Department's backlog reportedly stalling Nvidia's H20 chip license

August 1, 2025

Sex toy maker Ravens threatens legal action after fixing security flaws that published user data

August 1, 2025

Authorities seize servers of black suit ransomware gang

August 1, 2025

Google is experimenting with machine learning power age estimation technology in the US

July 31, 2025

Palo Alto Networks agrees to buy Cyberark for $25 billion

July 30, 2025

Leave A Reply Cancel Reply

Top Reviews
Editors Picks

7 days left: Founders and VCs save over $300 on all stage passes

March 24, 2025

AI chip startup Furiosaai reportedly rejecting $800 million acquisition offer from Meta

March 24, 2025

20 Hottest Open Source Startups of 2024

March 22, 2025

Andrill may build a weapons factory in the UK

March 21, 2025
About Us
About Us

Welcome to Tech Brunch, your go-to destination for cutting-edge insights, news, and analysis in the fields of Artificial Intelligence (AI), Cryptocurrency, Technology, and Startups. At Tech Brunch, we are passionate about exploring the latest trends, innovations, and developments shaping the future of these dynamic industries.

Our Picks

Confusing accused of scraping websites that explicitly blocked AI scraping

August 4, 2025

Openai says ChatGpt is on track to reach 700m-a-week users

August 4, 2025

Elon Musk says he's reclaiming Vine's archives

August 4, 2025

Subscribe to Updates

Subscribe to our newsletter and never miss our latest news

Subscribe my Newsletter for New Posts & tips Let's stay updated!

© 2025 TechBrunch. Designed by TechBrunch.
  • Home
  • About Tech Brunch
  • Advertise with Tech Brunch
  • Contact us
  • DMCA Notice
  • Privacy Policy
  • Terms of Use

Type above and press Enter to search. Press Esc to cancel.