Guide · Newsletters

How do I get my newsletter cited by AI? Substack, beehiiv and Ghost compared

Updated 2026-11-07 · kagentx

Short answer

In the robots.txt we read on 2026-11-07, the public posts on Substack, beehiiv and Ghost do not singly block the main AI crawlers; the difference is who can edit robots.txt and whether there is a network-level blocking switch. Ghost can override it with a robots.txt in the theme, beehiiv's default file is customizable, and Substack's help page could not be read so it is marked not verified. To be cited, first read your own publication's robots.txt, then confirm the archive page's HTML directly contains post links.

Can AI crawlers fetch posts on Substack, beehiiv and Ghost?

In the robots.txt we read on 2026-11-07, none of the three platforms singles out the main AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) for blocking. The real differences are who can edit robots.txt and whether there is a network-level blocking switch. Every cell below comes from our curl that day or from an official page.

Platformrobots.txt as read on 2026-11-07Can you edit robots.txt?
Substack (platformer.substack.com and others)Disallows only BLEXBot; the general group disallows /action/, /publish, /sign-in, /subscribe, /p/*/comment/*, /embed and more; no AI crawler is listed; Sitemap points to sitemap.xml and news_sitemap.xmlNot verified: the help page could not be read (403); the robots.txt of the several publications we read were identical, with no sign of customization
beehiiv (the default file of aiweekly.beehiiv.com)Amazonbot is disallowed site-wide; Googlebot has its own group; the general group disallows /polls/*/results, /unsubscribe/, /login and more; SEO tools such as AhrefsBot have Crawl-delay: 10; GPTBot, ClaudeBot and the like are not listedThe file's header says the default is used when custom content is left empty; you can customize it or upload your own robots.txt
Ghost (demo.ghost.io, the official demo site)The general group disallows /ghost/, /email/, /members/api/comments/counts/, /r/, /webmentions/receive/ and /.ghost/analytics/api/; has a Sitemap; no AI crawler is listedYes: the documentation says a robots.txt in the theme overrides the default

This is a same-day snapshot of three specific sites, not every publication. Platforms can change defaults at any time and may have switches you only see in your dashboard, so the last step is always to read your own publication's robots.txt. What each AI crawler does is in AI crawler user agents: a side-by-side table, and general writing is in How to write robots.txt for AI crawlers.

# 1) Read the platform's robots.txt
curl -s https://YOURNAME.substack.com/robots.txt | head -40
# 2) Does the archive list posts in the raw HTML (not only after JavaScript)?
curl -s https://YOURNAME.substack.com/archive | grep -o 'href="[^"]*/p/[^"]*"' | sort -u | wc -l
# 3) Is a post readable with an AI crawler's User-Agent string? (local simulation, not the real crawler)
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; GPTBot)" https://YOURNAME.substack.com/archive

Will a platform's built-in "block AI" switch affect being cited?

It can, so first know what you turned on. Substack has a "Block AI training" switch; we could not read the original text of Substack's help page (403) and have only seen claims about it in third-party articles, so what exactly it changes and whether it affects search-type crawlers is not verified.

We read beehiiv's official note on AI Crawl Control: it says it is available on Pro and Enterprise plans, needs the Website Builder and a custom domain, and default subdomains such as yourpublication.beehiiv.com are out of scope; it tracks 22 crawlers in four categories (training, AI search, AI assistant, search engine); everything is accessible by default and a toggle turned on means blocking; it blocks at the network level through Cloudflare and does not depend on crawlers voluntarily following robots.txt. The same note says the "Discoverable on the web" switch changes robots.txt. A search-result summary gave the plan as Max, which disagrees with the original text we read, so go by your own dashboard.

So to be cited by AI search: do not turn the "AI search" and "AI assistant" categories to blocking; if you only want to keep content out of training, block only the "training" category. The similar switch on the Cloudflare side is in Does Cloudflare block AI crawlers by default? How to check and allow them.

Your own domain or the platform's subdomain?

We have no data showing that an own domain is cited more easily, so we draw no such conclusion. Three facts we can confirm:

  1. 1. A Substack subdomain redirects to the custom domain

    On 2026-11-07 a request to astralcodexten.substack.com returned a 301 to www.astralcodexten.com; the robots.txt on that domain matches the platform's, and its Sitemap points to its own domain. So once you attach a domain, the address that gets cited should be your own domain.

  2. 2. beehiiv's network-level blocking switch is only for custom domains

    That is the scope the official note states. If you need fine control over crawlers, you need your own domain.

  3. 3. When you change platforms, your own domain keeps article addresses unchanged

    If the address changes, existing citations and links have to be caught by 301 redirects; see How do you keep search and AI citations when you change domain or URLs? for how.

What to do about the archive page, sitemap and author pages?

Crawlers follow links. If the archive page lists post links in the raw HTML, a crawler can find them all from one entry point. We read platformer.substack.com/archive: it returns 200, its canonical points to itself, the HTML directly contains 24 /p/ post links, and sitemap.xml also lists /archive, /about and each post.

ItemSubstackGhost
Archive page has post links in the raw HTMLYes (platformer.substack.com/archive, 24 links)Not checked separately; Ghost serves server-rendered theme pages
sitemapsitemap.xml contains /archive, /about and every post; there is also a news_sitemap.xmlSplit into pages, posts, authors and tags
Author pages in the sitemapNot verifiedYes (sitemap-authors.xml lists /author/john/ and others)

For what author pages do for AI, see Author pages and E-E-A-T; for how posts on social platforms get cited, see How social posts show up in AI answers; for how to write the articles themselves so they are easier to cite, see How to write content AI will quote.

# Ghost theme: add robots.txt at the theme root to override Ghost's default
User-agent: *
Allow: /
Disallow: /ghost/
Disallow: /email/
Sitemap: https://your-domain.com/sitemap.xml

If you also run a podcast, see How do podcasts get cited in AI answers? for how to write show notes and transcript pages.

FAQ

Can ChatGPT, Claude and Perplexity crawlers fetch posts on Substack?

In the robots.txt we read on 2026-11-07, Substack lists no rules for these AI crawlers, so they fall under the general group and post pages are not disallowed. Requests to the archive page with these crawlers' User-Agent strings all returned 200, but that is a local simulation, not the real crawlers. How Substack's "Block AI training" switch affects robots.txt is not verified because the help page could not be read.

Which is cited more easily by AI, the platform subdomain or your own domain?

We have no comparison data, so we draw no conclusion. What we can confirm: a Substack subdomain 301-redirects to the custom domain, and robots.txt and sitemap follow; beehiiv's AI Crawl Control is only for custom domains. The benefit of your own domain is that addresses stay the same if you change platforms later.

Does a beehiiv newsletter block AI crawlers by default?

In the default robots.txt of aiweekly.beehiiv.com (read 2026-11-07), only Amazonbot is disallowed site-wide; GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot are not listed and fall under the general group. The official note says AI Crawl Control lets all crawlers in by default and only blocks when you turn it on.

How do I change robots.txt on Ghost?

Ghost's documentation says a robots.txt in the theme overrides Ghost's default one. Whether a Ghost(Pro) hosted site lets you edit the theme directly depends on your plan, which we did not verify.

Do archive and author pages help AI citation?

There is no official data. An archive page lets a crawler find all posts from one entry point, as long as the HTML directly contains post links. Ghost author pages go into the sitemap; whether Substack and beehiiv author pages go into the sitemap separately is not verified.

Sources

kagentx

Can crawlers read your pages?
Let the evaluation tell you

The evaluation checks whether crawlers can get in and whether the text is served directly, then asks ChatGPT, Perplexity and Claude (with web search on) your buyers' questions and sees whether you are mentioned. The Basic plan is $1, and results usually take 2–3 minutes.