Why Your Site Can Rank on Google and Still Be Invisible to AI

Two things to check on your own site before you spend a dollar on Answer Engine Optimisation.

Open your robots.txt. If someone added AI crawler rules in the last two years, there is a fair chance they blocked the bot that would have cited you and left the one that trains on your content alone.

Then read what Google actually says about AI Overviews. There is no special markup. No AEO schema. No file to upload. The eligibility rule is that the page is indexed and can be shown with a snippet.

Neither of those is what AEO gets sold as, and both are free to check this afternoon.

The bot you blocked is probably not the one that cites you

This is the part that costs real money and almost nobody checks. OpenAI and Anthropic each run several crawlers, and they do different jobs. Blocking them all with one rule is a decision about your pipeline, and it usually gets made by whoever was worried about training data.

OpenAI documents three.

User agent	What it does	What blocking it costs you
GPTBot	Crawls content that may be used to train OpenAI's foundation models	Nothing in ChatGPT search. This is the training crawler.
OAI-SearchBot	Shows websites in ChatGPT's search results	Your pages will not be shown in ChatGPT search answers, though they can still appear as navigational links
ChatGPT-User	Fetches a page when a person asks ChatGPT to read it	OpenAI notes that because a person starts these, robots.txt rules may not apply

Anthropic splits it the same way. ClaudeBot collects public web content that may be used to train its models, Claude-SearchBot crawls to improve the quality of Claude's search results, and Claude-User fetches a page when a Claude user asks a question that needs one. All three honour robots.txt.

So the useful position for most Australian businesses is to decide separately about training and about being quoted. Plenty of companies are happy to be named in an answer and much less happy about being training data, and that is two rules in robots.txt, one line each. What you should not do is block everything with AI in the name and then wonder why ChatGPT never mentions you.

Go and read the file. Your domain, slash robots dot txt, it takes a minute. Then check what your CMS and your firewall are doing on top of it, because a WAF rule blocking AI user agents does the same damage and does not show up in robots.txt at all.

Google says there is no AEO markup, and it means it

Google's documentation on AI features is unusually blunt. To appear as a supporting link in AI Overviews or AI Mode, a page has to be indexed and eligible to be shown in Google Search with a snippet, meeting the ordinary Search technical requirements. Google says there are no additional requirements beyond that, and no special optimisations necessary. You do not need to create new machine readable files, AI text files or markup, and there is no special schema.org structured data to add.

That kills a few things people are being sold right now.

• llms.txt. Google says you do not need one. It takes an hour and breaks nothing, so publish one if you want to, but do not build a plan around it.
• FAQ schema as the AEO lever. Structured data is still worth having, for rich results in ordinary Search. It is not the thing that gets you into an AI Overview.
• Anything sold as an AI Overviews ranking factor. The requirement is being indexed and snippet eligible. If your best commercial page carries a nosnippet tag, that is your problem and no report will tell you.

The controls Google does honour all run the other way. nosnippet, data-nosnippet, max-snippet and noindex limit what can be shown, and Google-Extended governs training use in Google's other systems. Those work exactly as documented. Which means a well meaning tag a developer added in 2024 can be the reason your best page never appears in an AI answer.

What actually gets a page quoted

With the technical myths out of the way, what is left is mostly writing.

Answer engines do not only use what they learned in training. Ask a current question and they search the live web and build the answer out of what they find. Retrieval augmented generation, if you want the term for it. The practical version is that your page has to be fetchable today and easy to lift a clean answer out of.

Answer in the first two sentences

The inverted pyramid, and the highest return change in this whole piece.

1. The direct answer. One or two sentences at the top of the section, answering the question the heading asks.
2. The evidence. Bullets, numbers, or the steps.
3. The context. The business reasoning, for the reader who clicks through.

A 2,000 word guide that reaches the answer in paragraph nine is a page a machine skips and a person bounces off.

Headings shaped like the question

People ask answer engines whole questions, so make your headings whole questions. Read your own HubSpot chat transcripts for the phrasing your buyers actually use. It is free research and it beats guessing at long tail keywords.

Say what you cover, in the same words every time

Answer engines work on entities: people, companies, concepts and the relationships between them. Building a topic cluster with HubSpot's SEO tools is a way of stating plainly that you cover this subject and everything next to it. Your HubSpot content does that job when the internal links agree with each other and does the opposite when they contradict.

Be current, and be checkable

Freshness counts, and so does Google's Experience, Expertise, Authoritativeness and Trustworthiness framing. Links from places people genuinely reference still do work. So do customer stories with names and numbers attached, which is why it pays to be collecting testimonials and case studies automatically in HubSpot instead of chasing them once a year.

Do not make the page slow

A page that cannot be fetched cannot be quoted. Fast page loads on HubSpot CMS and clean semantic HTML out of your modules matter here for the dull reason that everything downstream depends on the crawl working.

The Australian part, and there is less of it than you would like

No Australian law or regulator governs AEO. There is no standard to meet and no code of practice to comply with, so anyone selling you compliance work in AEO is making it up.

What is real is a lopsided starting position. These models build their picture of the world from the public web, and the public web is not spread evenly. Bigger English language markets have more published articles, more reviews, more coverage and more discussion, so the models know those brands best. An Australian business can lead its category here and still be missing from the answer, because the model is reflecting how much has been written about you and mistaking that for how good you are. We wrote up why Australian brands are invisible in AI search, and the fix on its own, and the fix is mostly geography: be explicit about where you operate and who you serve, then compete on the questions only an Australian business would answer well.

The other real number is adoption. The ABS reported that around 12 per cent of Australian businesses used AI in 2024-25, up from 1 per cent in 2021-22. Among medium sized businesses it was 22 per cent, against 3 per cent three years earlier. That is business use of AI at work, so do not read it as a purchasing statistic. Read it as speed. A behaviour went from one in a hundred to one in eight in three years, and medium sized companies moved faster than the average.

On measurement, the honest position is that numbers on AI referred traffic are still thin, and anyone quoting you a precise share of buyers who start in ChatGPT is guessing. What you can do is check where you stand today and watch it move. We wrote up how to actually measure whether your AEO is working, and it comes down to asking the engines about your category on a schedule and recording what comes back.

You already paid for this content

A company of 50 to 500 people has usually spent years on blog posts, guides and case studies that answer the questions a buyer asks before the first call. If the buyer now asks an answer engine those questions and gets your competitor's version, that spend is still on the books and it has stopped working.

Two things follow. First, the pages worth fixing are the ones that answer buying questions, not the ones with the most traffic. Second, when a prospect does arrive having read an answer that quoted you, they arrive further along, which changes what the first call is for. Same pattern as zero click discovery calls: the research happened without you, so the conversation has to start somewhere else.

In HubSpot, attribution reports are where you will see it, imperfectly. A closed deal whose first interaction was a direct visit to one specific FAQ page is usually an answer engine having sent someone your way. Not proof, but worth watching.

Questions we get asked

What is the difference between SEO and AEO?

SEO is about ranking a page in a list of results. AEO is about being the source an answer engine uses when it writes the answer instead of showing the list. The work overlaps heavily, because an answer engine mostly picks from pages that are already indexed and already good.

Does AEO replace SEO?

No. Google's rule is that a page has to be eligible in ordinary Search before it can appear in an AI Overview, so technical health, relevance and authority are still the entry ticket.

Do I need schema markup or an llms.txt file to appear in AI Overviews?

No. Google states there is no special schema.org structured data required and no AI text file to create. Structured data is still worth having for rich results in ordinary Search.

Should we block GPTBot?

That is a business decision about training data and it is separate from being quoted. Blocking GPTBot does not remove you from ChatGPT search. Blocking OAI-SearchBot does. Decide the two independently, and write both rules deliberately.

Will AEO reduce our website traffic?

Probably at the top of the funnel. People who only wanted a definition will get it without clicking. The clicks you keep tend to come from people further along, which is a better trade for a B2B pipeline than it looks in an analytics chart. Track qualified conversations instead of sessions.

How does HubSpot help with any of this?

The CMS and blog tools make it easy to publish clean semantic HTML, structured data and question shaped headings, and the SEO tools handle topic clusters. HubSpot's blog research agent helps find the questions worth answering. None of it does the thinking about which of those questions matter to your buyers.

Where to start

In this order, because the first two are free and the third is not.

1. Read your robots.txt, and any firewall rules sitting over it. Decide about training crawlers and search crawlers separately.
2. Check your best commercial pages for nosnippet, max-snippet and noindex tags nobody meant to leave behind.
3. Rewrite the top of your ten most commercially useful pages so the answer is in the first two sentences.

That is a week of work and none of it needs a new tool. If you want the acronyms sorted out first, we have written up the difference between AEO, GEO and SEO. If the site itself is the problem and the writing is fine, that is our Operating Systems and Websites work.

One thing to try before you leave

Go and ask ChatGPT the question your best buyer asks before they call you. If the answer names three companies and none of them is you, you now know something useful, because there are only two reasons for it: nobody wrote the page, or something on your site is stopping the answer engine from reading it.

Which one is it on your site?