How does Qwen's web search get its sources?
Very little is official. Alibaba Cloud Model Studio's “web search agent” documentation says it fetches real-time information from many online sources, automatically filters authoritative content and extracts the key points; a developer can specify up to 100 target domains, the system fetches from those first, and runs a general web search when they have no matching information. This is from the Alibaba Cloud Model Studio docs, original text read 2026-11-11.
This page documents an API for developers; it does not mention crawler names, robots.txt, or how a site gets included. Whether the Qwen app uses the same search, the docs do not say; not verified. We also did not find an official Alibaba page describing the Qwen app's web search.
Which crawlers has Alibaba officially published?
None found. The sites that list QwenBot, TongyiBot, Qwen-User and AliyunSecBot in search results are all third-party registries, and they themselves mark them “unverified” or “robots.txt compliance not confirmed”. So the table below is “third-party claims, not verified”, not an official Alibaba list.
| Name | What third-party registries say | Status (2026-11-11) |
|---|---|---|
| QwenBot | An Alibaba Cloud crawler related to Qwen | Not verified, no first-party document |
| TongyiBot | Fetches web content for the Tongyi Qianwen assistant | Not verified, no first-party document |
| Qwen-User | User-triggered fetches | Not verified, no first-party document |
| AliyunSecBot | Alibaba Cloud's generic crawler, purpose unclear | Not verified, no first-party document |
Whatever the names turn out to be, the safest approach is not to write a rule that “blocks all unknown crawlers”. A name comparison is in AI crawler user agents: a side-by-side table.
How do our four sites set robots.txt for them today?
| Site | robots.txt rules (curl 2026-11-11, all 200) | For the four names above |
|---|---|---|
| kagentx.com | User-agent * Allow: /; only /admin.html, /candidates.html and /api/ are blocked | All allowed |
| mochistamp.com | * group blocks /api/ (except /api/content.js and /api/strings/), /files/ and /admin.html | All allowed |
| gammaslide.com | * group blocks /d/, /decks, /account and /admin | All allowed |
| pagent.kagentx.com | * group blocks only /i18n/ | All allowed |
None of the four sites names an AI crawler; computed by text with Python's robots.txt parser, the home page is allowed for all four names. This is not verification with real crawlers, and we did not check whether our server logs contain visits from these crawlers.
How is Qwen different from DeepSeek, Doubao and Kimi?
| Qwen | DeepSeek | Kimi | Doubao | |
|---|---|---|---|---|
| Official crawler names published | Not found | Not found | Yes: KimiBot, Kimi-User, Kimi-SearchBot | Not found |
| Official description of how sources are fetched | The Model Studio API docs describe how sources are fetched and domain targeting | API docs say in-house search engine | The API docs have a web search description | No official description found |
| Whether the app and the API share one search | Not stated | Not stated | Not stated | — |
These comparisons come from our two published guides; details in How do I get DeepSeek to recommend my product? and How do I get Doubao and Kimi to recommend my product?.
What can an independent developer do now?
1. Do not block crawlers you do not recognize
Check robots.txt and your CDN's bot rules, and do not block unknown crawlers as a group. How to write it: How do you write robots.txt for AI crawlers?. A reference:
# Do not block unknown crawlers as a group; keep private paths out User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Sitemap: https://your-site.com/sitemap.xml
2. Put the body text directly in the HTML
Prices, features and the one-line description must be visible even without scripts; the reason is in Do AI crawlers render JavaScript?. Whether Qwen runs scripts, the docs do not say, so take the conservative approach.
3. Build a Chinese page with its own URL
If your target users ask in Chinese, the Chinese page needs its own URL, a Chinese title and a Chinese description, rather than relying on browser translation. This is an inference: Alibaba does not say how it treats multilingual pages.
4. Check that Chinese search engines have indexed you
Alibaba does not say which search engine Qwen uses, so do not look only at Google. Search your product name on Baidu and Bing once each and see whether it is indexed. This is also an inference.
5. Ask three times in Qwen yourself
Turn on web search, ask the same question three times, and note whether you are mentioned and which pages are cited; the method is in How do you measure AI visibility? Four metrics and a table you can follow.
FAQ
Which search engine does Qwen's web search use?
The Alibaba Cloud Model Studio docs only say it fetches from many online sources and do not name the underlying search engine (original text read 2026-11-11). Whether the Qwen app uses the same one is not verified.
What is Qwen's crawler called and should I allow it?
Alibaba has not published it. Third-party registries list QwenBot, TongyiBot, Qwen-User and AliyunSecBot, all unverified. The conservative approach is not to block unknown crawlers as a group.
Can I tell Qwen to fetch only from my site?
The Model Studio API lets developers specify up to 100 target domains and the system fetches from those first. This is a setting a developer makes when calling the API, not a feature site owners can submit themselves.
Do I need a Chinese page?
If your target users ask in Chinese, a Chinese page with its own URL is recommended. This is an inference; Alibaba does not say how it treats multilingual pages.
Does submitting my URL to Alibaba speed up indexing?
We found no official submission channel: unverified.
Can I be guaranteed a Qwen recommendation?
No. The ranking is not published and answers differ each time; you can only watch how often you appear across repeated questions.
Does my robots.txt allow Qwen?
With the reference setup on this page, anything not named in a block is allowed. By the text, our four sites all allow them, but this is not verification with real crawlers.
Sources
- Alibaba Cloud Model Studio · Qwen Web Search Agent overview (original text read 2026-11-11, the page has no update date)
- kagentx · robots.txt (fetched 2026-11-11)
- MochiStamp · robots.txt (fetched 2026-11-11)
- Gamma AI Slide · robots.txt (fetched 2026-11-11)
- Pagent · robots.txt (fetched 2026-11-11)
- kagentx evaluations: the data on the case study pages; each example gives its date.