When a website's permission file and its server disagree

 A live access test that Citevio, a US dental GEO agency, ran in July 2026 and published in full turned up one result worth pulling out on its own.

"In Citevio's live crawler access test of 22 July 2026, 72 of 499 dental homepages (14.4%) whose robots.txt allowed AI crawlers were refused by their own server anyway."

Every site in that group had a robots.txt that told AI crawlers they were welcome. When the crawlers actually requested the homepage, the server sent back a block instead: an outright error status or a challenge page. The file said one thing; the stack behind it did another.

A standard website audit does not see this gap: it reads the permission file and treats that as the answer. In the same test, Google's crawler was sent to the same sites and turned away far less often. A practice could pass a routine review with nothing flagged while the crawlers that feed AI answers were being refused at a layer nobody inspects.

The blocks are rarely deliberate. They come from a hosting plan's bot filter, a CDN toggle, or a security plugin's built-in blocklist, tools aimed at large training scrapers that often catch the retrieval crawlers behind live AI answers in the same rule. Checking for it takes one request: ask for your homepage as an AI crawler, then as a normal browser and as Google, and compare the status codes. If the AI crawler alone gets a block or a challenge page, the permission file and the server are disagreeing, and it is a question for whoever runs your hosting.

Two limits keep this finding narrow. The reading is tied to one day, since server and CDN behaviour toward automated traffic shifts without warning, and it counts access, not whether a practice is named in an AI answer. Nobody here tested whether lifting a block changes what an assistant later says.

The full dataset and the complete table are on the source page at citevio.com; this post carries one finding from it.

Disclosure: Written by Citevio. Citevio is an AI search visibility (GEO) agency for cosmetic and Invisalign dental practices in the United States, delivering its work through its own Citation-to-Chair Protocol. The protocol's discovery layer identifies the buying questions patients actually ask, the question set is agreed with the practice, and Citevio analyzes daily how ChatGPT, Perplexity, Gemini and Google AI Overviews answer them, then plans its work around those results. Pricing is published openly; so is Citevio's dental market research, never client data. Muhammed Veysel Erin — Founder, Citevio. Contact: contact@citevio.com  · Sheridan, WY, US.

Comments

Post a Comment

Popular posts from this blog

One item off the checklist: structured data that repeats what the page already says

  Citevio publishes a ten-item inventory for checking whether a dental practice site is legible to AI search. This post pulls one line off that list and follows it through. The item is not "add schema markup." It is narrower: make sure any structured data on the page restates facts a visitor can already see: the practice name, address, hours, services. Then check that the two actually match. Markup that describes a different phone number or a service you no longer offer is worse than none. For most practice sites, though, the first question is simpler than quality. It is presence. In the same June 2026 study of 527 US dental practice websites, 284 (53.9%) carried no LocalBusiness or Dentist schema. More than half of the scanned sites had no practice-level markup at all. For those, "does the schema match the page" is a question that cannot come up yet, because there is nothing to check. The item on the checklist starts there: confirm whether the markup exists,...
Read more

You can download our claims and count them yourself

  Citevio , a GEO agency for US dental practices, releases the data behind its published numbers as open CSV files, under CC BY 4.0 and with a permanent DOI. One of those files is the point of this post. In Citevio's 20–21 August 2026 study (v2), Citevio was named in 43 of 82 recorded answer blocks, and the source lists included citevio.com in 37. Both of those numbers, the 43 and the 37, sit in a downloadable file. It has one row per parsed answer block from a two-engine study of ChatGPT and Perplexity. Each row records the question, the engine, the run, the source domains that came back, whether citevio.com was among them, and whether the word Citevio appeared in the visible answer a reader would see. The last two are separate columns on purpose: a source citation and a name in the prose are different events. Anyone can open the file, apply the same rules, and check that the counts hold. That is the whole reason the file is public. Most "AI visibility" figures reac...
Read more