Website

Set a robots.txt policy for AI crawlers and publish it safely

Helps you decide which AI crawlers the site should be open to by separating search, user fetch and training roles. Previews a new robots.txt that keeps your existing rules, writes it to your hosted site or WordPress after approval and verifies it live.

When to use it

You want your site to be cited in AI search but want to decide separately about model training, or you do not know what your robots.txt tells AI crawlers today.

What you get

A before and after decision table for every AI crawler, a new robots.txt that keeps your existing private-area rules, the file published after your approval and verified live, and the version you can roll back to.

Does it change anything in my account?

Only with your approval. The assistant first shows a preview of every change and waits for your go-ahead.

Works with

SEO Intelligence, WordPress

Fill in before sending

[site URL][Opus Growth hosting or WordPress]
Rework the robots.txt of my site [site URL] around a deliberate policy for AI crawlers. Site type: [Opus Growth hosting or WordPress]. My preference: [A: open to all / B: search open, training closed / C: custom, explain]. 1. Read the current state. Read [site URL]/robots.txt with site_read_url. If the site is on Opus Growth hosting, also read robots.txt with site_read_file and confirm it matches what is live. If it is WordPress, find out which SEO plugin runs it with wordpress_seo and check with wordpress_snippet action='list' whether any snippet already touches robots.txt. List every existing rule group, every Disallow line and every Sitemap line. 2. Explain the roles. Put the crawlers into three groups and tell me in one sentence each what I lose or gain: 1. Search and citation: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot, Applebot. Closing these stops the site from being cited as a source in that assistant's answers. 2. Live fetch on behalf of a user: ChatGPT-User, Claude-User, Perplexity-User. These arrive when a user shares my link or the assistant opens a page to answer. According to the providers' documentation, robots.txt may not apply in every case to these user-initiated requests, so say that a rule here is not a guarantee. 3. Model training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent. Closing these limits use of the content in future training, it does not remove content from models already trained. Closing Google-Extended does not affect Google Search or AI Overviews. 3. Write the draft. Prepare the new robots.txt for my preference and follow these rules: 1. Once you write a group under a crawler's own name, that crawler no longer reads the `User-agent: *` group. So copy every Disallow line from the * group (admin area, cart, internal search results and so on) into every named group, otherwise you will open private areas without noticing. 2. Do not block CSS, JavaScript or image files, because crawlers need them to render the page. 3. Keep the existing Sitemap lines, and add the full sitemap URL if it is missing. 4. Do not close any search-role crawler unless I explicitly ask for it. 5. Leave competitor SEO crawlers (such as AhrefsBot and SemrushBot) alone unless I ask, they are outside the scope of this job. 4. Show a preview. Compare the old and new file line by line, and show a table with the before and after decision for each crawler: crawler, role, old decision, new decision. Wait for my approval. 5. Apply after approval. 1. Hosted site: write robots.txt with site_write_file. The platform takes a version snapshot before each write, so tell me which version from site_versions I can return to. 2. WordPress: when no physical file exists on the server, WordPress generates robots.txt itself and the `robots_txt` filter can change it. Create a PHP snippet with wordpress_snippet that uses this filter, location 'everywhere'. The snippet is created inactive, so show me the code and turn it on only after I approve. If a cache plugin is active, clear the cache with wordpress_purge_cache. 6. Read it back. Read robots.txt again with site_read_url and show that the new content is live. On WordPress, if the content did not change, there is a physical robots.txt file on the server and the filter cannot reach it. In that case turn the snippet off and walk me through changing the file in the SEO plugin's file editor (Yoast: Tools, File editor; Rank Math: General Settings, Edit robots.txt) or in the hosting panel. End with a two-sentence note: the change takes effect when crawlers re-read the file (usually within a day), and if a CDN or firewall rule blocks by crawler name, robots.txt alone is not enough.

Example result

An example of the output this prompt produces. Its structure comes from the tools' real output; numbers and names are fictional.

Example output (based on a fictional site, not real customer data)

The current file has two groups: User-agent: * (Disallow: /panel/, /cart/) and User-agent: GPTBot (Disallow: /). Your preference is B: search open, training closed.

CrawlerRoleOld decisionNew decision
OAI-SearchBotSearchAllowedAllowed
Claude-SearchBotSearchAllowedAllowed
PerplexityBotSearchAllowedAllowed
ChatGPT-UserUser fetchAllowedAllowed
GPTBotTrainingBlockedBlocked
ClaudeBotTrainingAllowedBlocked
Google-ExtendedTrainingAllowedBlocked
CCBotTrainingAllowedBlocked

Note: the old GPTBot group did not repeat the /panel/ and /cart/ lines. That was harmless there because the group closed the whole site, but in the new file both lines are written into every named group.

After your approval robots.txt was written with site_write_file. site_read_url read it live, it returned 200 and shows the new content. To roll back, the 14:02 version in site_versions holds the old file.

Note: crawlers usually re-read the file within a day. If a firewall rule blocks by crawler name, robots.txt alone is not enough.

Related searches

These prompts work best when your AI assistant is connected to your accounts with Opus Growth.

Connect your accounts free

More prompts

WebsiteChanges after your approvalLive demo

Build an editorial expert site with a guides section and author schema

Has your assistant build an expert's site with a magazine-cover home page, long-form guides and an author profile. Serif typography, Article and Person schema and dated, sourced articles produce a publication people read and AI answers can cite.

When to use itWhen you are a consultant, specialist or one-person knowledge business and want your site to read like a publication that shows your expertise through writing, not like a brochure.

SEO Intelligence, Search Console
WebsiteChanges after your approvalLive demo

Build a heritage family business site with a classic look and modern standards

Has your assistant build a site in the spirit of an old label and shop sign for a producer in its third generation, framed and with a paper texture. The look is old, the build is new: mobile first, JavaScript-free, self-hosted fonts, WebP, Product and HowTo schema.

When to use itWhen you sell a craft passed down through generations, such as Turkish delight, olive oil, cheese or copperware, and want a classic look that carries that history without giving up speed or accessibility.

SEO Intelligence, Search Console
WebsiteChanges after your approvalLive demo

Build a Swiss-minimal law or accounting firm site with strong trust signals

Has your assistant build a still firm site on a strict grid, with hierarchy carried by a single type family. It puts the trust signals a YMYL field needs on the page: practitioner profiles, a dated and sourced guidance note, wording that respects professional advertising rules, and Person schema.

When to use itWhen a law, accounting or similar professional practice needs a site that looks serious and plain, stays within professional advertising rules and proves its expertise.

SEO Intelligence, Search Console
WebsiteChanges after your approvalLive demo

Build a JavaScript-free multi-page corporate site and deliver it at 100

Has your assistant build a still, fast and accessible site for corporate work such as architecture, engineering or consulting. Menu and footer are static HTML, fonts and CSS are self-hosted, and every page is measured before hand-over.

When to use itWhen a corporate business needs a multi-page site where speed and accessibility matter more than decoration, with no animation and no JavaScript.

SEO Intelligence, Search Console