Which web pages does Baidu ERNIE's answer use?
Baidu does not say. We read the announcements and help pages on the Baidu Search Resource Platform and found no page explaining how Wenxin Yiyan, Wenxiaoyan or Baidu AI Search choose sources or which crawler they use. One thing is confirmed: on 2026-09-15 the platform announced "AI traffic analysis", saying it will provide data such as AI citations, impressions and clicks of site content in traditional search, but "the data service has not been formally enabled yet and complete data cannot be shown" (read 2026-11-12).
This shows Baidu treats "content being cited by AI" as data site owners should see, and that citation sits in the same system as the sites registered on the platform. As for which web pages ERNIE's answers draw from, Baidu does not say; claims in third-party articles (for example that Baidu Baike and Baijiahao are often cited) have no first-hand source we could find, so they are unverified. We also did not check Baidu's own pages for how the three names (Wenxin Yiyan, Wenxiaoyan, Baidu AI Search) relate, so we list them side by side.
Which crawlers has Baidu officially published?
| Name | Official statement | Source (viewed 2026-11-12) |
|---|---|---|
| Baiduspider/2.0 | The User-Agent for both PC and mobile crawling is unified as Baiduspider/2.0, with http://www.baidu.com/search/spider.html in the UA | Resource Platform announcement, 2015-08-11 |
| Baiduspider-render/2.0 | Rendering crawl: fetches a site's CSS, JavaScript and images to understand pages more accurately; please avoid blocking this UA | Resource Platform announcement, 2017-03-24 |
| Baiduspider-image | Image crawling; Baidu's own robots.txt has a separate group for it, and the help page says to block image crawling with path rules | Baidu Help Center, www.baidu.com/robots.txt |
| Baiduspider-news, -cpro, -ads | Seen only on third-party registry sites | Not verified, no first-party document |
| A dedicated crawler for ERNIE, Wenxiaoyan or AI Search | No official statement found | Not verified |
These announcements date from 2015 and 2017 and we found no newer official UA list, so go by your server logs for current UAs. For a name comparison see AI crawler user agents: a side-by-side table.
What should I know about Baidu's robots.txt rules?
Baidu's Help Center says the spider checks the root robots.txt before visiting a site; pages blocked by robots are not crawled, indexed or shown by Baidu, but if another site links to the page it can still appear in results, with a snippet taken from the other site's description.
| Rule | What the Baidu help page says |
|---|---|
| User-agent | A robots.txt may have only one User-agent: * record; if you write User-agent: Baiduspider, the rules under that group apply |
| Order | The order of Allow and Disallow matters: the spider matches from top to bottom and acts on the first match |
| Wildcards | * and $ are supported |
| No robots.txt | No file or an empty file means the whole site is open to all search engines |
| Tool | The Resource Platform's Robots tool can validate and update it and show how it takes effect at Baidu; it supports content checks up to 48k |
The help page has no date and its footer says 2011. The ordering rule differs from other engines (Google uses longest match), so the same robots.txt may behave differently at Baidu than at Google; check it once with the Robots tool.
How are our own four sites' robots.txt files set for Baiduspider?
| Site | robots.txt rules (2026-11-12 curl, all 200) | For Baiduspider |
|---|---|---|
| kagentx.com | User-agent * Allow: /; only /admin.html, /candidates.html and /api/ are blocked | Allowed |
| mochistamp.com | The * group blocks /api/ (with Allow /api/content.js and /api/strings/ afterwards), /files/ and /admin.html | Home page allowed; /api/content.js unverified (see below) |
| gammaslide.com | * group blocks /d/, /decks, /account and /admin | Allowed |
| pagent.kagentx.com | * group blocks only /i18n/ | Allowed |
None of the four sites names Baiduspider. Evaluated by text with Python's robots.txt parser, Baiduspider and Baiduspider-image are allowed on all four home pages. This is not a real Baidu spider visit, and we did not check whether Baiduspider appears in our server logs.
One thing about mochistamp.com deserves a separate look: it writes Disallow: /api/ first and Allow: /api/content.js afterwards. Under the ordering rule on Baidu's help page, the later Allow may not take effect for Baiduspider, while pages reference /api/content.js. We did not verify this with the Robots tool; unverified at the time of writing.
How do I hand URLs to Baidu? How do submission methods and quotas work?
| Tool | Docs | Sources |
|---|---|---|
| General submission | Actively pushes data to Baidu to shorten the time the crawler needs to find links; three ways (API, manual, sitemap); inclusion is not guaranteed | General-submission page; Search College article (2020-08-06) |
| Fast crawl | Manual or API submission, crawled with priority, with no guarantee of inclusion or display; submitting uses up quota | Search College article (2020-08-06); 2024-04-24 announcement: replaces "Fast Indexing" from 26 April |
| sitemap | Submission has a quota that differs by site; index-type sitemaps are not processed, and while one exists new files cannot be submitted | Search College article (2020-08-06) |
| Quota tightening | Sitemap submission was switched off for sites under non-real-name accounts and for low-quality sites, and the daily API push quota was adjusted | Resource Platform announcement, 2023-09-22 |
| Crawl diagnosis | Shows what Baiduspider actually fetched; 70 uses per week, showing only the first 200KB | Search College article (2020-08-06), numbers not checked one by one |
On how to obtain fast-crawl entitlement, the 2020 article says it is granted automatically to VIP partners of the Resource Platform, or earned through platform events; we did not sign in to check current conditions; unverified.
What can an independent developer do now?
1. First confirm Baiduspider is not blocked
Check robots.txt and your CDN's bot rules. Do not write a rule that lets in only Google. A reference version:
# Let Baiduspider in; keep private paths out User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Sitemap: https://your-site.com/sitemap.xml
For details on writing it see How do you write robots.txt for AI crawlers?.
2. Add your site on the Baidu Search Resource Platform and check with the Robots tool
If you have rules where Allow comes after Disallow, use this tool to confirm how Baidu parses them.
3. Submit a sitemap, as a single file
The official article says index-type sitemaps are not processed, so list URLs in a single sitemap; when you publish new pages, push them with the general-submission API. Inclusion is not guaranteed.
4. Put the body text directly in the HTML
Baidu says only that rendering crawls fetch CSS and JavaScript, not how far execution goes, so price, features and the one-line introduction should be visible without scripts too; the reasons are in Do AI crawlers render JavaScript?.
5. If your target users ask in Chinese, build Chinese pages with their own URLs
The Chinese page needs its own URL, a Chinese title and description; for multilingual writing see How do you write hreflang on a multilingual site? How does AI search treat it?. This is inference: Baidu does not say how ERNIE handles multilingual pages.
6. Ask in Baidu AI Search and Wenxiaoyan yourself, three times
Ask the same question three times, note whether you are mentioned and which pages are cited; the method is in How do you measure AI visibility? Four metrics and a table you can follow.
For other Chinese models see How do I get Doubao and Kimi to recommend my product? and How do I get Qwen (通义千问) to recommend my product?; for the matching guide on the Korean market see How do I get Naver (네이버) to recommend my product?.
FAQ
Does Baidu Wenxin Yiyan's answer use Baidu Search pages?
Baidu does not say. The Resource Platform's "AI traffic analysis" announcement mentions citations and impressions of site content in AI, but does not explain how ERNIE fetches pages; unverified.
Does ERNIE have its own crawler?
We found no officially published name for a dedicated crawler; unverified. What Baidu publishes is Baiduspider/2.0 and Baiduspider-render/2.0.
What happens if robots.txt blocks Baiduspider?
The Baidu help page says blocked pages are not crawled, indexed or shown, but can still appear when other sites link to them, with a snippet from the other site's description.
Does submitting a sitemap guarantee inclusion?
No. Baidu says inclusion is not guaranteed; sitemaps also have a quota, and index-type sitemaps are not processed.
Are Baidu's robots.txt rules the same as Google's?
Not exactly. The Baidu help page says Allow and Disallow are matched from top to bottom and the first match is applied; check it with the Robots tool.
Can I use "AI traffic analysis" now?
The 2026-09-15 announcement says it is opening in stages and the data service is not formally enabled, so complete data cannot be shown. We did not sign in to check the current state; unverified.
Should I make Chinese pages specifically for Baidu?
If your target users ask in Chinese, build Chinese pages with their own URLs. This is inference; Baidu does not say how it handles multilingual pages.
How do I know whether ERNIE recommends me?
Turn on web search, ask the same question three times, and note whether you are mentioned and which pages are cited; Baidu does not publish its ranking method, and answers differ each time.
Sources
- Baidu Search Resource Platform · AI流量分析功能上线公告(2026-09-15) (Read in full 2026-11-12)
- Baidu Search Resource Platform · 新版Baiduspider移动ua升级公告(2015-08-11) (Read in full 2026-11-12)
- Baidu Search Resource Platform · 百度Spider新增渲染抓取UA公告(2017-03-24) (Read in full 2026-11-12)
- Baidu Search Resource Platform · 关于升级平台「快速收录」工具的通知(2024-04-24) (Read in full 2026-11-12)
- Baidu Search Resource Platform · 关于回收网站提交配额的通知(2023-09-22) (Read in full 2026-11-12)
- Baidu Search Resource Platform · 详解平台工具的使用和常见问题(2020-08-06) (Read in full 2026-11-12, numbers may have changed)
- Baidu Help Center · Baiduspider Help Center: How to block the crawling (Read in full 2026-11-12, page has no update date)
- Baidu Search Resource Platform · Robots 工具说明(未登录可见部分) (checked 2026-11-12)
- kagentx · robots.txt (Fetched 2026-11-12)
- MochiStamp · robots.txt (Fetched 2026-11-12)
- Gamma AI Slide · robots.txt (Fetched 2026-11-12)
- Pagent · robots.txt (Fetched 2026-11-12)
- kagentx evaluations: the data on the case study pages; each example gives its date.