Fill in before sending
[site URL][Opus Growth hosting or WordPress]
Rework the robots.txt of my site [site URL] around a deliberate policy for AI crawlers. Site type: [Opus Growth hosting or WordPress]. My preference: [A: open to all / B: search open, training closed / C: custom, explain].
1. Read the current state. Read [site URL]/robots.txt with site_read_url. If the site is on Opus Growth hosting, also read robots.txt with site_read_file and confirm it matches what is live. If it is WordPress, find out which SEO plugin runs it with wordpress_seo and check with wordpress_snippet action='list' whether any snippet already touches robots.txt. List every existing rule group, every Disallow line and every Sitemap line.
2. Explain the roles. Put the crawlers into three groups and tell me in one sentence each what I lose or gain:
1. Search and citation: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot, Applebot. Closing these stops the site from being cited as a source in that assistant's answers.
2. Live fetch on behalf of a user: ChatGPT-User, Claude-User, Perplexity-User. These arrive when a user shares my link or the assistant opens a page to answer. According to the providers' documentation, robots.txt may not apply in every case to these user-initiated requests, so say that a rule here is not a guarantee.
3. Model training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent. Closing these limits use of the content in future training, it does not remove content from models already trained. Closing Google-Extended does not affect Google Search or AI Overviews.
3. Write the draft. Prepare the new robots.txt for my preference and follow these rules:
1. Once you write a group under a crawler's own name, that crawler no longer reads the `User-agent: *` group. So copy every Disallow line from the * group (admin area, cart, internal search results and so on) into every named group, otherwise you will open private areas without noticing.
2. Do not block CSS, JavaScript or image files, because crawlers need them to render the page.
3. Keep the existing Sitemap lines, and add the full sitemap URL if it is missing.
4. Do not close any search-role crawler unless I explicitly ask for it.
5. Leave competitor SEO crawlers (such as AhrefsBot and SemrushBot) alone unless I ask, they are outside the scope of this job.
4. Show a preview. Compare the old and new file line by line, and show a table with the before and after decision for each crawler: crawler, role, old decision, new decision. Wait for my approval.
5. Apply after approval.
1. Hosted site: write robots.txt with site_write_file. The platform takes a version snapshot before each write, so tell me which version from site_versions I can return to.
2. WordPress: when no physical file exists on the server, WordPress generates robots.txt itself and the `robots_txt` filter can change it. Create a PHP snippet with wordpress_snippet that uses this filter, location 'everywhere'. The snippet is created inactive, so show me the code and turn it on only after I approve. If a cache plugin is active, clear the cache with wordpress_purge_cache.
6. Read it back. Read robots.txt again with site_read_url and show that the new content is live. On WordPress, if the content did not change, there is a physical robots.txt file on the server and the filter cannot reach it. In that case turn the snippet off and walk me through changing the file in the SEO plugin's file editor (Yoast: Tools, File editor; Rank Math: General Settings, Edit robots.txt) or in the hosting panel.
End with a two-sentence note: the change takes effect when crawlers re-read the file (usually within a day), and if a CDN or firewall rule blocks by crawler name, robots.txt alone is not enough.
Example result
An example of the output this prompt produces. Its structure comes from the tools' real output; numbers and names are fictional.
Example output (based on a fictional site, not real customer data)
The current file has two groups: User-agent: * (Disallow: /panel/, /cart/) and User-agent: GPTBot (Disallow: /). Your preference is B: search open, training closed.
| Crawler | Role | Old decision | New decision |
|---|
| OAI-SearchBot | Search | Allowed | Allowed |
| Claude-SearchBot | Search | Allowed | Allowed |
| PerplexityBot | Search | Allowed | Allowed |
| ChatGPT-User | User fetch | Allowed | Allowed |
| GPTBot | Training | Blocked | Blocked |
| ClaudeBot | Training | Allowed | Blocked |
| Google-Extended | Training | Allowed | Blocked |
| CCBot | Training | Allowed | Blocked |
Note: the old GPTBot group did not repeat the /panel/ and /cart/ lines. That was harmless there because the group closed the whole site, but in the new file both lines are written into every named group.
After your approval robots.txt was written with site_write_file. site_read_url read it live, it returned 200 and shows the new content. To roll back, the 14:02 version in site_versions holds the old file.
Note: crawlers usually re-read the file within a day. If a firewall rule blocks by crawler name, robots.txt alone is not enough.