How to Check Whether AI Tools Can Actually Read Your Website At All

A site that looked fine and answered for nobody

A local specialist retailer's site looked perfectly normal in a browser — clean design, working navigation, fast enough. But when the owner asked several AI assistants about the shop, none of them could describe it accurately, several claiming to have no information at all. Nothing was technically broken in the sense a human visitor would notice. The problem turned out to be a combination of an overly aggressive bot-blocking security plugin and a homepage that rendered its actual content entirely through JavaScript after the initial page load — both invisible to a person clicking around, both a wall for machines trying to read the page directly.

Why 'it looks fine to me' isn't a real test

A browser executes JavaScript, respects cookies, and renders a fully interactive page. Many crawlers and retrieval bots do a much lighter-weight fetch — grabbing the raw HTML response and extracting what text is directly present, without necessarily executing scripts the way a browser does. A site that relies on client-side JavaScript to inject its core content — product descriptions, service details, pricing — can look complete to a human and be nearly empty to a bot that only sees the initial HTML payload.

The audit sequence

1. View the page's raw HTML source directly (not the browser's rendered DOM inspector) and confirm the actual text content — service descriptions, hours, pricing — is present in that raw source, not injected afterward by JavaScript
2. Check robots.txt for any Disallow rules that might be unintentionally blocking major crawlers, including a blanket Disallow: / left over from a staging environment
3. Check security or firewall plugin settings for overly broad bot-blocking rules that might be catching legitimate crawlers alongside malicious ones
4. Test with a simple command-line request using a tool like curl, setting the User-Agent header to a known crawler string, and confirm a normal response comes back rather than a block page or empty response
5. Ask a few different AI assistants directly about the business and compare their answers against reality, noting any that claim no information at all

• Common culprits: content rendered entirely client-side with no server-rendered fallback
• A security plugin's bot-blocking rules configured too broadly
• A leftover staging robots.txt accidentally deployed to production
• A CDN or firewall rule blocking based on user-agent or request rate in a way that catches legitimate crawlers

A site that a browser renders beautifully and a crawler can't read at all isn't invisible by accident — something specific is blocking it, and it's almost always findable.

The server-side rendering fix

For sites that do rely on JavaScript for meaningful content, the most durable fix is ensuring critical text — anything a customer would ask about — is present in the initial server-rendered HTML rather than injected afterward, even if the interactive polish layers on top of that afterward. This doesn't mean abandoning JavaScript entirely; it means the core facts shouldn't depend on it being executed.

A common misconception: a fast, modern-looking site is automatically crawlable

There's a natural assumption that a fast, modern, visually polished site must also be easy for machines to read, since both seem like signs of technical quality. The two are actually independent — a site can load quickly and look great specifically because it defers content rendering to client-side JavaScript, which is the exact pattern that makes a page invisible to a lighter-weight crawler. Visual polish and machine-readability are solved by different parts of the build, and neither one guarantees the other.

This gap is worth taking seriously precisely because it's invisible from the usual vantage points — a human visitor sees a fine site, an analytics dashboard shows real traffic, and everything looks healthy right up until someone specifically checks whether AI assistants can describe the business accurately, at which point the gap becomes obvious.

How this risk differs across business types

• A financial services or legal services site often loses the most from this gap, since these are exactly the businesses where an AI assistant claiming "no information available" instead of surfacing accurate details is most damaging to trust
• An automotive dealer or repair shop with inventory or service listings rendered dynamically through JavaScript is a common candidate for this issue, since inventory pages are frequently built as single-page apps
• A local specialist retailer with a simpler, more traditional site build is often in better shape here by accident, since simpler sites more often render their content directly in the HTML without a JavaScript framework in between

The general pattern holds across industries: the more a site's build relies on client-side rendering for its core content, the more valuable this specific audit becomes, and the more directly a fix translates into machines being able to describe the business accurately.

Making this a recurring check

Security plugins update their rules, hosting providers change firewall defaults, and a well-intentioned 'block bad bots' setting can silently start catching more than intended after an update — so this audit is worth repeating periodically, not treating as a one-time fix. NetWebMedia includes this kind of accessibility check as part of its standard site audits; run one at https://netwebmedia.com/free-audit.html to see whether a site's actual content is reaching the crawlers that matter.

← Back to all articles