Only 18,5% of the world’s most-visited landing pages are operating without any technical barrier to automated access, according to a new study by Web data extraction specialist Zyte.
This means that less than one in five of all the most-visited sites are fully open and easy for automated tools to access without being blocked or challenged.
Zyte’s State of Web Access 2026 report – which covers nearly 25 000 websites across 230 countries & dependencies – highlights numerous key findings, including:
* 81,5% of websites deploy at least one technical barrier, such as Web Application Firewalls (WAFs), dynamic rate-limiting, or bot management systems to govern incoming automated traffic.
* Only 18,5% operate without detectable barriers, suggesting a shift in the “Open Web”.
* 50% of websites use two or more barriers, creating complex technical hurdles at both the network and application transport layer.
* Despite barriers, 88% of landing pages still sit at simple or easy accessibility – the two lowest tiers.
While CAPTCHAs are the most visible barrier to human users, invisible back-end security dominates automated access control – led by WAFs on over nine in 10 sites, and dynamic rate-limiting deployed on over half.
Apparel & Fashion, Adult, Furniture, and Gambling are the hardest gated industries, while Airline, Mass Media and Construction sectors remain significantly more open, maintaining low-barrier access.
Over half of all websites are now stacking two or more defensive technologies on top of each other, attempting to stop AI crawlers and data collection at the protocol level long before they reach the actual page.
Three in four websites are also using robots.txt files to disallow access by the new wave of AI training bots. Newspapers, media, and publishing sites have the most sophisticated robots.txt files, featuring a curated list of allowed and disallowed agents. Retail and online services follow behind due to high perceived value of product data.
Industries are facing a strategic decision: allow open access to maximise visibility in AI search, or erect technical barriers to protect their content.
“The data confirms what our customers have been telling us for years – the Web keeps getting more systematically defended, not easier to access,” says Shane Evans, CEO of Zyte. “But this research also shows that harder doesn’t mean inaccessible. The sites that matter most to our customers are well within reach with the right platform.
“State of Web Access 2026 gives data teams the clearest picture yet of exactly what they’re up against and what it takes to operate reliably at scale,” Evans adds. “For example: despite barriers, 88% of landing pages are nevertheless Simple or Easy to access through Zyte API, which can access the vast majority of pages using only level-1 or level-2 capabilities from its five-tier set of tooling, ensuring the price of access is kept low.”
Iain Lennon, chief product officer at Zyte, adds: “What strikes me most in this data is the precision of the response to AI crawlers. Sites aren’t just tightening access broadly – they’re making deliberate distinctions between which automated visitors they’ll tolerate and which they won’t. That level of intentionality is new, and it’s accelerating. For anyone building systems that depend on Web data – whether for AI training, price intelligence, or market research – understanding these access patterns isn’t optional anymore. It has to be built into the architecture from day one.”