Do the robots.txt files of X and LinkedIn let AI crawlers read posts?
Going by robots.txt, mostly no. On 2026-11-07 we read both files: x.com does not name GPTBot, ClaudeBot or PerplexityBot, so they fall under the “User-agent: *” group, which blocks the whole site; linkedin.com names each of them and blocks the whole site. The exception is that LinkedIn does not block OAI-SearchBot and Claude-SearchBot site-wide. A robots.txt file is only a site’s request to crawlers and says nothing about how many posts AI answers actually cite; we have not verified that.
| Crawler | x.com | linkedin.com |
|---|---|---|
| GPTBot (OpenAI training) | Not named; the “*” group applies: whole site blocked | Disallow: / (whole site blocked) |
| OAI-SearchBot (ChatGPT search) | Not named; the “*” group applies: whole site blocked | Its own group, not blocked site-wide; a long list of blocked paths such as /public-profile/, /people/search/, /search* and /settings/, with no rule for /posts or /in/ |
| ChatGPT-User (a user asks ChatGPT to open a page) | Not named; the “*” group applies: whole site blocked | Disallow: / (whole site blocked) |
| ClaudeBot (Anthropic training) | Not named; the “*” group applies: whole site blocked | Disallow: / (whole site blocked) |
| Claude-SearchBot, Claude-User | Not named; the “*” group applies: whole site blocked | Claude-SearchBot has exactly the same rules as OAI-SearchBot and is not blocked site-wide; Claude-User is Disallow: / |
| PerplexityBot, Perplexity-User | Not named; the “*” group applies: whole site blocked | Both are Disallow: / (whole site blocked) |
| Google-Extended | Disallow: * (whole site blocked) | Disallow: / (whole site blocked) |
| Googlebot, Bingbot | Googlebot and Bingbot share a group; most URLs are crawlable, with /*/followers, /*/likes, /*/media and a few others blocked | Googlebot has a long list of blocked paths, including /feed/update/; LinkedInBot is Allow: / |
| Other | “*” group: Disallow: /, Crawl-delay: 1; plus the __usage_ai-training signal: Disallow: / | “*” group: Disallow: /; plus the __usage_ai-training signal: Disallow: /; the file opens by stating that automated access without permission is prohibited and that you can apply for an allowlist |
Three things to keep in mind when reading the table:
- The names GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot do not appear in the x.com file; we infer that the “*” group applies to them from how robots.txt is read, not because the file mentions them.
- LinkedIn does not block OAI-SearchBot and Claude-SearchBot site-wide, but we did not test whether those crawlers can actually fetch post pages; login walls and similar factors are unverified.
- robots.txt is a request, not a technical block. Partnerships, licenses or APIs would not be written in this file, and we have not verified whether X or LinkedIn have any.
We could not connect to the robots.txt files of Threads and Bluesky from our machine this time, so we do not state their rules.
So do AI answers actually cite posts from X and LinkedIn?
We found no official statement and no evidence of actual citations, so this is unverified. All we can confirm is the table above: the robots.txt files of both platforms are closed to most AI crawlers. Some articles publish how often each platform is cited; we did not check their original data, so we do not repeat any share. To find out who AI cites in your field, the most reliable way is to ask it yourself and read the sources; see How do you measure AI visibility? Four metrics and a table you can follow.
How can what an indie developer posts on social platforms get cited?
A social post is where you tell the readers you already have, not where you get AI to cite you. Put the content you want cited on your own website: a page with a stable URL, body text rendered on the server, and a robots.txt you control. Link to that page from the post, and keep the post itself short.
| Content | Where to put it | Why |
|---|---|---|
| What the product is, who it is for, price, what it supports | Your own home page, pricing page and about page | You control the crawler rules and the text; see the guide on writing content AI will quote |
| What changed in this version, data, tutorials | A page on your own site whose URL stays the same | A post sinks within days; a page at a fixed URL can be kept up to date |
| How it differs from similar products | A comparison page on your own site, with a source and check date for every row | A post that says “faster than X” has no source, which makes it hard for AI to cite |
| Launch announcements, small updates, talking with readers | Social platforms | This is what social platforms are good at; link back to your own page |
| What other people say about you, discussions | Forums and Q&A communities | See the Reddit guide for taking part honestly; do not pose as a bystander |
Reddit and YouTube are different from X and LinkedIn, and each has its own guide: Why do AI answers love to cite Reddit? What product teams can and shouldn't do; Are YouTube videos cited in AI answers?.
How can a launch post be written?
Here is an example (the name and URL are placeholders). The post itself is one or two complete sentences: what you did and one concrete number. The last line is the full URL of the matching page on your own site, which is where the definition, requirements and comparison go.
Just shipped ExampleTool 2.4: incremental builds. A 2,000-note rebuild now takes seconds instead of minutes. What it is, requirements and the comparison table: https://example.com/exampletool/
How do I fix it, step by step?
1. Write the page you want cited on your own website first
Product pages, comparison pages and tutorials, one question per page, with the answer up front. See How to write content AI will quote: before and after.
2. Check that your own site does not block AI crawlers
Read your own robots.txt; the rules are in How do you write robots.txt for AI crawlers?, and the crawlers are listed in AI crawler user agents: a side-by-side table.
3. Keep the post short and end with a link
Write one concrete fact in the post and put the full URL of the matching page on the last line. Do not publish the only explanation in the post alone.
4. Put your website URL in your profile
Both X and LinkedIn profiles have a website field; fill it with your own website rather than another social account.
5. Keep the URLs of your posts
If AI ever cites a post, you will be able to match it. If a post is deleted or the account is renamed, the link breaks.
6. Ask AI and check whether social domains appear in the sources
Following How do you measure AI visibility? Four metrics and a table you can follow, ask each question three times and note how often x.com and linkedin.com appear among the sources, and how often your own site does.
7. Get mentioned on pages other people write
AI more often cites third-party articles, lists and forum threads that mention you; see Do brand mentions matter more than backlinks for AI search? What two studies say.
Quora's robots.txt rules for AI crawlers: see Does ChatGPT use Quora content?.
For posts on newsletter platforms (Substack, beehiiv, Ghost), see How do I get my newsletter cited by AI? Substack, beehiiv and Ghost compared.
FAQ
Will ChatGPT cite my posts on X?
We found no official statement and no evidence of actual citations, so this is unverified. What we can confirm is that the x.com/robots.txt we read on 2026-11-07 does not name GPTBot, OAI-SearchBot or ChatGPT-User, so the “*” group applies to them and blocks the whole site.
Does LinkedIn allow AI crawlers to read posts?
In the robots.txt of 2026-11-07, GPTBot, ClaudeBot, ChatGPT-User, Claude-User, PerplexityBot, Perplexity-User and Google-Extended are all Disallow: /; OAI-SearchBot and Claude-SearchBot each have a group of blocked paths and are not blocked site-wide. The file also opens by stating that automated access without permission is prohibited.
What about Threads and Bluesky?
We could not reach their robots.txt files from our machine this time, so we state no rules.
Should I still post on social platforms?
Yes. Social platforms are where you tell the readers you already have and get feedback; just do not treat them as the place for AI to cite you. Link to your own site from posts and keep the content you want cited there.
Sources
- X · x.com/robots.txt (Read 2026-11-07)
- LinkedIn · www.linkedin.com/robots.txt (Read 2026-11-07)
- kagentx evaluations: the data on the case study pages; each example gives its date.