Publishers and sites increasingly block AI agents to block ai crawlers
Major websites and publishers are restricting AI agents, which can reduce AI-driven referrals and citations for businesses.
Multiple outlets report that, since mid‑October 2026, a growing number of publishers and websites are actively blocking or restricting AI agents and crawlers. The restrictions affect agents from companies including Perplexity, OpenAI, Meta and others and can limit their ability to access site content.
What happened
On 17 October 2026 The Verge published a report titled "Websites are closing their doors to AI agents" describing a rising trend of sites refusing or limiting access to AI agents and agentic features. The article names companies whose agent technology promises tasks such as booking reservations, cleaning inboxes and buying products.
Outlets and sites cited in subsequent reports include Amazon, which banned Perplexity’s shopping agent and later entered litigation that briefly limited agent shopping on its site, and Delta Air Lines, which prevents agents from booking tickets on behalf of users. Yelp has said it only permits AI agents that join its paid data licensing programme.
Financial firms and other large services have also moved to block agents for security reasons: JPMorgan Chase and Charles Schwab have confirmed they block agent access to accounts. Users report that travel and classifieds sites such as Amtrak and Cars.com no longer work with some agents.
Technical and platform controls are part of the trend. Cloudflare introduced a one‑click option in 2025 to block AI crawlers and began blocking crawlers that scrape simultaneously for search indexing and for AI training. Some retailers have taken the opposite approach by partnering with agent providers: Meta has announced retail partnerships to let its Muse agent shop on partner sites and Shopify has opened storefronts to agents, according to The Verge.
Why it matters for businesses
When a website decides to block ai crawlers or agent access, AI assistants can no longer read or interact with that site’s content. That reduces the chance the business will be cited, summarised or used by assistants as a source when serving user queries.
Loss of agent access can reduce referral traffic and discovery that would otherwise come from assistants that surface links or direct users to sites. It also changes how products, booking options and account actions are presented to users who expect agents to carry out tasks on their behalf.
Decisions are often made for security or commercial reasons. Banks and airlines have cited security when blocking agents from account access or bookings, while some platforms require data licensing fees or partnership agreements to allow agent access. That means access is uneven: some merchants have opened their sites to agents via partnerships, while others close them off.
For marketers and site owners this alters the balance between protecting data and participation in AI‑driven discovery. Blocking agents can lower unwanted scraping and protect systems, but it may also reduce AI referrals and the visibility that comes from being cited by assistants.
What the AI assistants say today
We checked major assistants for whether they reflect the recent reporting. None fully reflect all of the specific examples and legal details in the news; most give general advice about impacts and mitigations but omit many named companies and lawsuits from the October coverage.
Our assistant check found that ChatGPT, Gemini, Perplexity and Google AI Overviews each reflected the broad trend partly. Perplexity's output included links to several third‑party tools and sites, while the others offered general guidance without citing the specific companies or legal actions named in The Verge and other reports.
| Assistant | Reflects the news? | What it said | Sites it cited |
|---|---|---|---|
| ChatGPT | Partly | The answer correctly explains that blocking AI agents can reduce AI-driven citations, referrals, and server load and that different bots serve different purposes, but it does not mention recent reports that many publishers and sites (including Amazon, Delta, Yelp, JPMorgan, Cars.com, Amtrak, etc.) are actively blocking or restricting agents or the specific legal actions and company responses described in the news. | none |
| Gemini | Partly | The answer correctly states that sites are blocking AI agents and explains impacts and mitigations, but it does not mention the specific companies, legal actions, or examples (Amazon, Perplexity, Delta, Yelp, JPMorgan, etc.) cited in the news. | none |
| Perplexity | Partly | The answer discusses how blocking AI agents reduces AI-driven discovery and gives mitigation steps, which aligns with the news that sites are restricting agents, but it does not mention specific recent examples, companies, or legal actions named in the news items. | ai-visibility.org.uk, ai.testmuai.com, crawlytics.app, llmpulse.ai, maxaeo.ai, scrunch.com |
| Google AI Overviews | Partly | The answer correctly describes that blocking AI agents reduces discovery, referrals, and prevents scraping, but it does not mention recent reports that many major publishers and specific companies (e.g., Yelp, Delta, JPMorgan, Amazon/Perplexity) are actively blocking or restricting agents as described in the news. | augusta.edu, cloudflare.com, instagram.com, marketplace.org, timesofai.com, youtube.com |
What to do now
Decide whether to block ai crawlers or permit controlled access based on your priorities: protecting accounts and content, or maintaining visibility via AI assistants. Practical steps to follow:
- Audit what agents and crawlers are currently accessing your site by reviewing server logs and bot traffic patterns.
- If you decide to restrict access, implement clear bot rules via robots.txt, CAPTCHAs for risky actions and Cloudflare or similar controls that can block specific agent crawlers.
- If you want safe participation, offer an API or data licensing option that provides structured, rate‑limited access and explicit terms for commercial use.
- For account‑sensitive areas, require authentication and explicit user consent rather than allowing agent‑initiated actions.
- Document your policy publicly so partners and users know whether you allow agent access, and monitor for legal or platform changes that affect that policy.
- Optionally, get an independent check of what AI assistants currently say about your business; Vasukai offers that as one option for checking how assistants cite or summarise your site.
Questions
Which major companies are reported to be blocking AI agents?
Reports name several large companies and sectors: Amazon has banned certain agent shopping features and pursued legal action; Delta Air Lines blocks agents from booking tickets; Yelp allows access only through a paid data licensing programme; and banks such as JPMorgan Chase and Charles Schwab block agent access for security reasons.
Are all retailers blocking AI agents?
No. Some retailers have partnered with agent providers. The Verge reports that Meta has retail partners for its Muse agent and that Shopify opened its storefront network to agents, while other merchants and marketplaces have restricted agent access.
What technical controls are being used to block AI crawlers?
Platforms and hosting services are offering controls. For example, Cloudflare introduced a one‑click block for AI crawlers in 2025 and has blocked crawlers that scrape simultaneously for indexing and model training; sites also use robots.txt, CAPTCHAs and rate limits.
Will blocking AI agents protect my users' accounts?
Blocking agent access can reduce the risk of automated account actions and scraping, and banks and airlines cite security as a reason for blocking agents. For sensitive account areas, requiring authentication and explicit user consent is still a recommended control.
Sources
- Websites are closing their doors to AI agents, theverge.com
- Websites block AI agents, limiting online shopping and bookings, mezha.net
- Yelp, Delta and banks restrict AI agents on their services — The Verge, ua.news
- Amazon Won’t Let Your AI Agent Shop. Its Ad Business Explains Why, businessmodelanalyst.com
Facts in this report come from the sources above; figures were checked against them. Drafted with AI and checked by the Kael Tripton editorial team. How we report.