Websites and search
Decide how AI can use your public website content
OpenAI plans to support a shared AI training signal. Understand how training preferences differ from search visibility and what to review on your business website.
You publish project photographs and useful advice so prospective customers can understand your work. You may also have a view about whether AI companies should use that material to train their models. For a small business, those decisions can get lost inside a single request to 'block AI'. It helps to write down what you actually want before anyone changes the website.
What changed this week?
On 30 October, OpenAI announced plans to honour a common signal for website operators to express AI training preferences. We reviewed the announcement and its linked documentation on 31 October. The useful detail is the implementation status: this is a commitment to future support.
OpenAI says implementation is planned for the coming months
OpenAI says the voluntary signal would let website operators communicate the same training preference to multiple participating providers through robots.txt. It aims to implement support in the coming months and publish further details. The company also says its existing crawler controls will continue to apply.
Our reading is that business owners can use this announcement to clarify their preferences now, while asking their developer to check actual provider support before relying on the shared signal. The announcement does not establish that every AI provider already honours it.
OpenAI: Making it easier for websites to express AI training preferencesPublished 30 October 2026; reviewed 31 October 2026
The linked proposal is still an Internet-Draft
The linked document proposes a machine-readable training preference using a policy token in robots.txt. It distinguishes content-use preferences from instructions about whether a crawler may retrieve a page.
The IETF Datatracker identifies it as an individual Internet-Draft, not an endorsed IETF standard. Treat it as work in progress when discussing implementation, rather than describing it as a finished universal rule.
IETF Datatracker: Machine-Readable Training Preference SignalDraft dated 30 October 2026; reviewed 31 October 2026
What could this mean for a small business?
Consider an illustrative cabinetmaking business. Its public website contains finished kitchens, explanations of materials and advice about planning a renovation. The owner wants people comparing local cabinetmakers to find those pages. They might still prefer that the material is not used for model training.
Start with that plain-language brief. It gives the developer something more precise than a request to block every automated visitor. A different business may choose to permit training use. The important step is to make the choice deliberately and record what each setting is intended to do.
OpenAI already separates search and training controls
OpenAI's crawler documentation says OAI-SearchBot supports its search features, while GPTBot retrieves content that may be used for training generative foundation models. It describes the settings as independent: a website can allow the search crawler while disallowing the training crawler.
The documentation separately describes ChatGPT-User requests initiated by users, for which robots.txt rules may not apply. That is another reason to check the specific purpose of a control rather than assuming one setting covers all AI access.
For the cabinetmaker, our suggested starting point is to review the existing rules against the owner's stated preferences. Keeping a crawler permitted does not guarantee that a project page will appear in an answer or bring an enquiry.
OpenAI: Overview of OpenAI CrawlersExisting documentation; reviewed 31 October 2026
Keep a record of the choices behind your settings
A short record can list the content involved, the desired use, the relevant provider setting, who approved it and when it was checked. Keep public project pages and genuinely private documents distinct in that record. Review it when your website platform changes or a provider announces support for a new control.
Ask your developer to retain the previous configuration and explain how to reverse a change. After an update, check representative public pages, the sitemap and the services that normally read your website. A successful request shows that a page is reachable; it does not prove how a model provider subsequently uses its contents.
The shared proposal may eventually reduce the number of separate preferences a business maintains across participating providers. That is a possible operational benefit, not something to assume before support and behaviour are documented.
Private documents need access controls
Google Search Central distinguishes password protection for confidential content from noindex, which keeps content out of Google Search but still allows people with the link to visit it. Its guidance was updated on 21 October.
In our cabinetmaking example, customer drawings, addresses and detailed quotes belong in an appropriately restricted workflow. A preference about training use should not be treated as the mechanism that makes those files private.
Google Search Central: Control what you share with Google SearchUpdated 21 October 2026; reviewed 31 October 2026
What can you do this week?
Use a small sample of your website to prepare a clear brief. You do not need to rewrite every page or choose a new platform to begin.
- Choose three examples: a service page, an original project or advice page, and a document intended only for a customer. Record who should be able to access each.
- Write down your preferences for public discovery and AI training use separately. Be specific about the material and providers involved.
- Ask your developer to review the current robots.txt file and any hosting controls against that brief. Request an explanation of which settings are supported today.
- Agree on a limited change and a rollback plan if a mismatch is found. Recheck public page access, sitemap access and the tools your business uses after deployment.
- Set a review point for the shared signal. Revisit the provider's implementation announcement before treating planned support as available.
How Context Aware can help
We can review how your website's crawler settings, page metadata and hosting controls fit the way you want your content used. We can document the purpose of each proposed change and test that important public pages remain accessible.
If public project pages and private customer files have become mixed together, we can also help plan a clearer workflow. Bring us a few example pages and your preferences, and we can scope the changes needed without assuming that your whole website needs rebuilding.
A note on the sources
Topic selection included Hacker News, Apple, OpenAI, Anthropic and Google Search Central, within our Western-provider research scope. This article uses OpenAI's announcement and crawler documentation, the linked Internet-Draft and Google's guidance. Direct X requests were blocked, and targeted searches returned no verifiable current discussion for this topic. We have not relied on X reactions. The cabinetmaking business is an illustrative example.
Prepared by Context Aware with AI assistance and linked primary sources. Examples are illustrative. How we write our insights.