Guide · WordPress

How to optimize a WordPress site for AI search

Updated 2026-11-09 · kagentx

Short answer

Four things to change on WordPress: robots.txt (Yoast: Tools → File editor; Rank Math: General Settings → Edit robots.txt), sitemap, structured data and llms.txt (both Yoast and Rank Math have an official switch). The first two decide whether crawlers can get in, so do them first; llms.txt comes last, and Rank Math itself says it is only a proposal. Below is a command you can run as is; five lines check everything.

Which four things do I change on a WordPress site for AI search?

Four: robots.txt, sitemap, structured data and llms.txt. The first two decide whether crawlers can get in and find your pages; the last two are extras. Below are the menu locations from the official docs of the Yoast SEO and Rank Math plugins (checked 2026-11-09):

What to changeYoast SEORank Math
robots.txtYoast SEO → Tools → File editorRank Math SEO → General Settings → Edit robots.txt (only visible in Advanced Mode)
sitemapsitemap_index.xmlThe Sitemap line in the default content is also sitemap_index.xml; WordPress core uses wp-sitemap.xml
Structured dataThe official page only gives an example where a post is Article plus author; the full list is unverifiedNo official list found: unverified
llms.txtSettings → Site features → AI tools → LLMS.txt, switch it on, then click Save changesLLMS Txt module; choose post types; default limit 100

Menu names may differ between versions; each section below states what the official docs say and their limits.

How do I edit robots.txt in WordPress?

When there is no real robots.txt in the site root, WordPress generates a virtual one. Both Yoast and Rank Math can edit it from the dashboard:

  • Yoast: Tools → File editor. The menu does not appear if file editing is disabled on the site; use FTP instead. If no physical file exists there is a button to create one. When Yoast generates robots.txt it adds a sitemap link automatically.
  • Rank Math: General Settings → Edit robots.txt. Rank Math says a real robots.txt in the site root blocks editing, and you must delete it over FTP or cPanel. If the menu is missing, check whether you are in Easy Mode.

To let AI search crawlers in, you only need to make sure nothing blocks the whole site; you can also name them as below (full rules in How to write robots.txt for AI crawlers, crawler list in AI crawler user agents: a side-by-side table):

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

Rank Math says changes usually take about 24 hours to apply. Bot blocking in security plugins, hosts or CDNs ignores robots.txt, so check those too; see How to set up Cloudflare's AI crawler controls.

What should I watch for with the sitemap?

Make sure the address in the robots.txt Sitemap line opens and that only one sitemap is being served. Yoast's page notes that to use WordPress's own wp-sitemap.xml instead of Yoast's sitemap_index.xml, you must first switch off Yoast's XML sitemap and llms.txt, then switch llms.txt back on after about 5 minutes. For how to write lastmod, see Does sitemap lastmod matter to AI crawlers?.

How do I handle structured data and llms.txt?

Structured data: both plugins output some by default, but we did not find an official full list, so use the self-check command below to count the JSON-LD blocks, then check the content in Google's or schema.org's validator. Which types are worth writing by hand: Does structured data help AI search?; FAQ markup: Is FAQ structured data still useful in AI search?.

llms.txt: Yoast's docs say it is not supported on multisite, and if you see "couldn't be auto-generated" check file permissions on the server. Rank Math's module lets you choose post types, posts set to noindex stay out of the file, and Rank Math itself says this is still only a proposal. Whether AI reads this file is covered in What is llms.txt?; do not skip the earlier items for it.

How do I check my own site? A command you can run as is

Replace SITE with your address and run it in a terminal. We ran it on kagentx.com: sitemap.xml returned 200, the other addresses returned 404, the home page had 1 JSON-LD block, and robots.txt had a Sitemap line.

SITE=https://example.com   # replace with your site

# 1. robots.txt: what is served, and is there a Sitemap line?
curl -s $SITE/robots.txt
# 2. Which sitemap and llms.txt addresses answer? (200 = exists)
for p in sitemap_index.xml wp-sitemap.xml sitemap.xml llms.txt; do printf "%s " $p; curl -s -o /dev/null -w "%{http_code}\n" $SITE/$p; done
# 3. Structured data blocks on the home page
curl -s $SITE/ | grep -c 'application/ld+json'
# 4. Is the home page marked noindex?
curl -s $SITE/ | grep -io '<meta name="robots"[^>]*>'
# 5. Does an AI crawler's user agent get the page? (200 = yes)
curl -s -A "OAI-SearchBot" -o /dev/null -w "%{http_code}\n" $SITE/
ResultMeaningWhat to do
robots.txt has no Sitemap lineAI crawlers may not find the sitemapAdd a line in the dashboard robots.txt editor
Both sitemap addresses return 200Two sitemaps are being servedKeep one and switch the other off
JSON-LD block count is 0The page has no structured dataCheck plugin settings, or whether the theme or a cache plugin strips it
noindex appearsThe page is marked as not to be indexedCheck "Discourage search engines from indexing this site" in Reading settings and the per-page setting in the plugin
The last line is not 200This User-Agent is being blockedCheck security plugins, the host's firewall and the CDN

Also check whether the body text is written directly in the HTML: fetch the page with curl and grep for one sentence. Some page-builder or cache plugins push the body into scripts; see Do AI crawlers render JavaScript? for why that matters. To confirm crawlers have visited, check server logs; see How to check your server logs for AI crawlers.

After the changes, measure again two to four weeks later with the same set of questions, using the method in How do you measure AI visibility? Four metrics and a table you can follow.

If you are a plugin author in the WordPress.org plugin directory, see How do WordPress plugins get recommended by AI? for the readme.txt fields and a test of the directory page.

If your site is built with Webflow or Framer, the comparison with WordPress is in Can AI search read your Webflow or Framer site?.

FAQ

Where do I edit robots.txt in WordPress?

By default WordPress serves a virtual robots.txt. In Yoast SEO edit it under Tools → File editor; in Rank Math under General Settings → Edit robots.txt. Rank Math says a real robots.txt file in the site root blocks editing, so delete it first.

Can both Yoast and Rank Math generate llms.txt?

Both have it in their official docs. In Yoast, switch it on under Settings → Site features → AI tools → LLMS.txt; Rank Math has an LLMS Txt module. Rank Math itself says llms.txt is currently a proposal, not an official standard, with limited adoption, so it comes after robots.txt, the sitemap and having the body text in the HTML.

Is an SEO plugin enough?

No. Plugins manage files and markup such as robots.txt, the sitemap, titles and structured data. Whether the body text is written directly in the HTML, and whether a CDN or security plugin blocks crawlers, you have to check yourself with the commands below.

What structured data do Yoast and Rank Math output?

Yoast's official page gives just one example: with Yoast alone, a post is an Article with an author. We did not find an official list of everything either plugin outputs, so it is unverified. Use the self-check command to count the JSON-LD blocks on your own pages.

Which address is the sitemap?

Yoast's is sitemap_index.xml, the default content in Rank Math also says sitemap_index.xml, and WordPress core uses wp-sitemap.xml. Use the self-check command to see which returns 200, and confirm the Sitemap line in robots.txt points to it.

Sources

kagentx

Do AI answers
mention you?

The evaluation checks whether crawlers can get in and whether the text is served directly, then asks ChatGPT, Perplexity and Claude (with web search on) your buyers' questions and sees whether you are mentioned. The Basic plan is $1, and results usually take 2–3 minutes.