Can AI companies train on your website content?

On 6 October 2026, at a public hearing of the federal parliament's Joint Select Committee on Artificial Intelligence, Google, Microsoft, OpenAI and Anthropic were among the AI companies that, AAP reported, told the inquiry it is too hard to get permission for each piece of Australian content they want to train on. The ABC told the same hearing current copyright law is "completely adequate". The committee reports by 30 November 2026.

So can AI companies train on your website? Under Australian law as it stands at the time of writing, 16 October 2026, there is no text and data mining exception, and the government ruled one out in October 2025. In practice, the bigger question for most businesses is technical: your pages, case studies and public PDFs can already be read by AI crawlers, and the only control most sites use is a robots.txt file the crawler chooses to respect.

What to decide this week: which AI bots may train on your content, which may read it to answer questions, and which files should not be public at all. For a 50 to 250 person B2B services business, our default is to let the search and answer bots in, decide training bots on purpose, and move anything you charge for or would not hand a competitor behind a login.

What happened at the inquiry?

Parliament set up the Joint Select Committee on Artificial Intelligence on 20 August 2026. Its terms of reference include "the interaction of AI with existing intellectual property and copyright laws, including the use of Australian creative, cultural and media content in the training of AI models". Submissions closed on 14 September and 563 have been published.

Reuters reported on 22 September that Anthropic's submission said the government "could permit AI model training through a narrow form of conditional approval", and that it was open to conditions such as investment supporting Australian creators. OpenAI's own submission asks for "a balanced copyright framework that allows models to learn from publicly available information", and ties copyright settings to where AI infrastructure gets built.

Other outlets reported that Anthropic's proposal would let rights holders opt out by blocking crawlers in robots.txt, and that SBS replied that robots.txt is routinely ignored or bypassed. We could not read Anthropic's submission ourselves, so treat that detail as reported.

On the other side, Reuters reported on 6 October that the ABC's Kate Gilchrist told the hearing an opt-out model "places the burden on rights holders" and that the ABC believes its content has likely already been scraped. The same report said media reports had the government considering an opt-out clause. We found no government statement confirming that.

1. 26 October 2025The Attorney-General rules out a text and data mining exception for AI training.
2. 20 August 2026Parliament sets up the Joint Select Committee on Artificial Intelligence, with copyright in its terms of reference.
3. 14 September 2026Submissions close. 563 have been published.
4. 22 September 2026Reuters reports Anthropic and OpenAI asking Australia to allow AI training, Anthropic through a narrow form of conditional approval.
5. 6 October 2026At the public hearing, AI companies say permission for each piece of content is too hard. The ABC says current law is "completely adequate".
6. 30 November 2026The committee reports. Any change to the law would still need legislation.

Sources: Attorney-General, 26 October 2025; Parliament of Australia; Reuters, 22 September and 6 October 2026; AAP, 6 October 2026.

What does Australian law say right now?

This is general information, not legal advice.

• Your website copy is protected. The Attorney-General's Department lists articles, photographs, diagrams and software code among the material copyright covers. Copying a substantial part without permission is infringement unless an exception applies.
• There is no exception for AI training. On 26 October 2025 the Attorney-General said the government "is not considering a Text and Data Mining Exception". That stands as at 16 October 2026.
• The government is looking at licensing. The same release set the Copyright and AI Reference Group three areas to work on, including whether to create a paid collective licensing scheme for AI or keep voluntary licensing. The department's submission to the inquiry says those consultations are still running.
• Nothing has changed yet. The committee recommends; it does not change the law. Any opt-out or licensing scheme would need legislation first.

What the law cannot do is reach back and remove your content from a model that has already been trained. Anthropic's help page says blocking its training crawler excludes your site's "future materials". Read every vendor's opt-out the same way: it works from the day the bot next reads your robots.txt.

Training bots and answer bots are different things

This is where most businesses trip. An "AI crawler" does one of three jobs, and each vendor gives each job its own name in robots.txt:

• Training crawlers collect pages that may be used to train future models.
• Search crawlers build the index an assistant searches when it answers a question and cites sources.
• User fetchers open a page in the moment because someone asked the assistant to.

Block a training crawler and you may stay out of the next model. Block a search crawler or user fetcher and you disappear from AI answers today, which is the opposite of what a services business wants. OpenAI's documentation says its training and search settings are independent: you can allow OAI-SearchBot and disallow GPTBot. Our earlier post, Is your website actually letting the AI crawlers in?, walks through checking your current file line by line.

Sources: OpenAI, Anthropic and Perplexity crawler documentation, as at 16 October 2026. The bot names are examples from each kind.

A robots.txt policy for a B2B services business

Here is the table we start from, using the names each vendor documents as at 16 October 2026. The right column is our default for a business whose website exists to win work. A publisher, or a business selling its content, would make different calls.

Google says Google-Extended has no effect on whether your site appears in Google Search or how it ranks. Blocking it will not move your rankings, but it can take you out of Gemini answers that rely on grounding, so do it only on purpose.

The training call. If your site is marketing, case studies and how-to articles, being part of what models know about your field is mostly an advantage: it is how an assistant learns that a Brisbane firm does this work. Block training bots when the content is the product, for example paid research, course material or a pricing method you would not hand a competitor. You can also block training on one folder only.

A sample group that allows answers and blocks training on a research folder:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /research/

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

What robots.txt cannot do

• It is a request. Reputable vendors document that they follow it. Bots that do not care will not, which is the SBS point.
• User fetchers may skip it. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page.
• It works per host. Anthropic's help page says to add the rule on each subdomain. If your resources sit on a separate HubSpot or docs subdomain, that host needs its own file.
• It is not the past. As above, it only affects future crawls.

An llms.txt file does not change any of this. It is a guide for AI tools, with no access control in it. Our post Does your business need an llms.txt file? covers when it is worth having.

Your PDFs and gated content are probably public

This is the one we find most often on audits. A business puts a form in front of its pricing guide or whitepaper and assumes it is protected. The form protects the landing page. The PDF itself usually sits on a file URL that anyone, or any bot, can open.

In HubSpot, files in the file manager are Public by default, which HubSpot describes as "publicly accessible to anyone who has or guesses the URL", and indexable by search engines. There are two other settings: Public - noindex, which asks search engines not to index the file, and Private, which needs a HubSpot login or a temporary link. Only Private actually keeps a crawler out.

Three checks for this week:

• List what sits behind your forms. For each file, decide: happy for it to be read and cited, or not.
• Set the visibility to match. Marketing PDFs can stay public. Anything with your pricing method, client data or internal process goes Private, or off the website entirely.
• Remember the old copies. Switching to noindex does not remove a file a search engine already holds. Google Search Console has a removal request for that.

What to decide this week

1. Open yourdomain.com.au/robots.txt and write down which of the bots in the table it names. Silence means allowed.
2. Agree the training call with whoever owns marketing and whoever owns your intellectual property. One sentence each: allow, block, or block one folder.
3. Allow the search and user bots unless you have a specific reason. Australian brands already struggle to show up in AI answers; our post on why Australian brands are invisible in AI search explains why.
4. Audit your file manager for public PDFs that should be private.
5. Diary 30 November 2026. Read the committee's recommendations when they come out, and look again if the government moves on opt-out or licensing.

None of this needs a developer. It is about an hour of decisions and a ten-line text file.

Frequently asked questions

Is it legal for AI companies to train on Australian websites? Australia has no text and data mining exception, and on 26 October 2025 the government said it would not introduce one. Copying a substantial part of copyright material without permission is infringement unless an exception applies. For your specific situation, ask a lawyer.

Does blocking GPTBot remove me from ChatGPT? No. GPTBot is OpenAI's training crawler, and OpenAI says it is independent of OAI-SearchBot, which decides whether your site appears in ChatGPT search. You can block one and allow the other.

Can I get my content out of a model that is already trained? Generally no. Vendor opt-outs such as Anthropic's apply to future crawls, so blocking a training bot today stops new collection without removing anything already collected.

Are PDFs behind a HubSpot form protected from AI crawlers? Usually not. HubSpot files are Public by default, so anyone with the file URL can open them. Set sensitive files to Private in the file manager.

When will the AI inquiry report? The Joint Select Committee on Artificial Intelligence is due to report by 30 November 2026. Its recommendations would need legislation before the law changes.

Sources

• Joint Select Committee on Artificial Intelligence, Parliament of Australia, for the appointment date, terms of reference, submissions and 30 November reporting date.
• Submission 11, OpenAI, Parliament of Australia, for OpenAI's copyright position.
• Submission 4, Attorney-General's Department, Parliament of Australia, for the current consultations on copyright and AI.
• Albanese Government to ensure Australia is prepared for future copyright challenges emerging from AI, Attorney-General, 26 October 2025, for the ruling out of a text and data mining exception and the reference group's work.
• Copyright basics, Attorney-General's Department, for what copyright protects and what counts as infringement.
• Anthropic, OpenAI call for Australia to relax ban on training of AI models, Reuters via The Star, 22 September 2026, for Anthropic's submission.
• Australia's ABC rejects AI copyright carveout, believes already been scraped, Reuters via Y94, 6 October 2026, for the ABC's evidence and the reported opt-out option.
• AI giants fly in as safety, copyright debates heat up, AAP via The Canberra Times, 6 October 2026, for the companies' evidence at the hearing.
• ABC and SBS want AI firms to pay for news, as Anthropic seeks opt-out, The Next Web, for the reported robots.txt opt-out and the SBS response.
• Overview of OpenAI crawlers, OpenAI, for GPTBot, OAI-SearchBot and ChatGPT-User.
• Does Anthropic crawl data from the web?, Anthropic, for ClaudeBot, Claude-SearchBot and Claude-User.
• Google's common crawlers, Google, for Google-Extended.
• Perplexity crawlers, Perplexity, for PerplexityBot and Perplexity-User.
• Common Crawl FAQ, Common Crawl, for CCBot and robots.txt.
• Organize, edit, and delete files, HubSpot Knowledge Base, for file URL visibility settings.

---

Want a clear AI content policy without a month of meetings? See our AI consulting work, or talk to us.