Does Hugging Face block AI crawlers? Is the page body in the raw HTML?
No. On 2026-11-09 we read huggingface.co/robots.txt with curl: it has one group, User-agent: * with Allow: /, and names no GPTBot, ClaudeBot or PerplexityBot. We then fetched one model page and one Space page: the model card body is directly in the model page HTML, while the Space page HTML is almost only the title bar and the app itself runs in an iframe.
| Page | What we saw (2026-11-09) | What it means for you |
|---|---|---|
| robots.txt | User-agent: * / Allow: /, plus a sitemap; no AI crawler is named | Read literally, all three crawlers may fetch |
| 模型页 openai-community/gpt2 | HTTP 200, about 243 KB; the body contains the first paragraph of the model card, the tags (Text Generation, English, License: mit) and usage code; the meta description is an automatic pattern: “Text Generation model with 0.1B parameters. Download, explore files and run it on Hugging Face.” | Crawlers can read the model card text, but the meta description is not the first paragraph you wrote |
| 模型页 google-bert/bert-base-uncased | Also in the HTML; the meta description follows the same pattern: “Fill-Mask model with 0.1B parameters…” | The automatic pattern contains only the task and parameter count |
| Space 页 gradio/hello_world | HTTP 200, about 34 KB; with scripts removed the body is only about 142 characters; the meta description is the generic “Discover amazing ML apps made by the community”; it has WebApplication JSON-LD | The Space page itself does not tell a crawler what it does unless you wrote a short_description |
| Space 页 HuggingFaceH4/zephyr-chat | The meta description is a specific sentence, but there is no short_description in its README header and we did not find where it comes from; the body says the Space is paused | A paused Space page is left as an empty shell; where the description comes from is not verified |
This is only what robots.txt says and what our own requests returned, not what an AI crawler really fetches. For whether crawlers run scripts, see Do AI crawlers render JavaScript?; for crawler rules, see How do you write robots.txt for AI crawlers?.
How should you write the first paragraph of a model card?
The official docs (read on 2026-11-09) say model cards are essential for discoverability, reproducibility and sharing, and should describe the model, its intended uses and limitations (including bias), training parameters, training datasets and evaluation results. The first paragraph is the first text anyone reads on the page; we suggest facts only: what the model is, what task it does, how big, which language, what it was fine-tuned on, and one metric that can be checked.
| Weak | Better | |
|---|---|---|
| 第一段 | A state-of-the-art, powerful model that revolutionizes customer support. | A fine-tuned DistilBERT model that sorts English customer-support tickets into five categories. 66M parameters; accuracy 91.4% on the held-out set in this repo. |
| 标题 | MyModel-v2-final | Say what the model is: Ticket Triage DistilBERT |
| 指标 | “More accurate than the rest” | Give the dataset, the metric name and the number, and say you measured it yourself |
| 局限 | Leave it out | State known cases where it does not apply: English only, the time range of the training data |
| 许可证 | Only in a file | Also put it in the license field of the YAML header; the page shows it and users can filter by it |
--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: text-classification base_model: distilbert/distilbert-base-uncased datasets: - example/support-tickets tags: - customer-support --- # Ticket Triage DistilBERT A fine-tuned DistilBERT model that sorts English customer-support tickets into five categories (billing, bug, how-to, account, other). 66M parameters; runs on CPU in about 20 ms per ticket. Accuracy 91.4% on the held-out set in this repo.
After the first paragraph, we suggest in order: how to use it (a few lines of code), training data and process, an evaluation table, limits and bias. The page's title and meta description are generated by the Hub, so to get your own sentence read elsewhere, put it in the first paragraph of the model card and on your own site.
Which metadata fields show up on the page and in search?
The official docs say the YAML metadata at the top of a model card supports discovery and use: it can be used to filter models on huggingface.co/models and is shown on the model page. Below are the fields the docs mention, and what we saw on the gpt2 page.
| Field | What the official docs say it does | How to write it |
|---|---|---|
| license | Shown on the model page and filterable by license; for a custom license use license: other plus license_name and license_link | Always set it, using a standard identifier |
| language | Language identifiers, as listed on the Hub's Languages page | List the languages it really supports |
| pipeline_tag | Task type, shown on the model page and filterable; for transformers models it is inferred from config.json and can be overridden | Set it yourself for models that are not transformers models |
| library_name | Names the library; the docs recommend setting it explicitly, discourage auto-detection, and say repositories created after August 2024 are no longer assumed to be transformers | Set it, for example transformers |
| tags | Open-ended tags, shown on the page | Use the task and domain words users search for, not unrelated ones |
| datasets | The page shows “Datasets used to train:” and links to the datasets on the Hub | Set it if you trained on Hub datasets |
| base_model | Shows the base model and can be filtered by it; the Hub infers whether the relation is finetune, adapter, quantized or merge | Required for fine-tuned, quantized and merged models |
In the raw HTML of the gpt2 page we saw the tags that correspond to this metadata: Text Generation, Transformers, English, License: mit. That repository's YAML only sets language, tags and license; Text Generation was inferred by the Hub. Whether AI answers use these fields has no official statement and is not verified.
How should you write a Space description?
The raw HTML of a Space page does not contain the app (it runs in an iframe), so the description depends on fields in the README header. The official docs (Spaces Configuration Reference) say short_description is a short description of the Space shown in its thumbnail, and the models field lists the models used and is parsed from your code if left out. In the Spaces we saw without a short_description, the meta description was a generic sentence.
--- title: Ticket Triage Demo emoji: 🎫 sdk: gradio app_file: app.py short_description: Paste a support ticket and get its category from Ticket Triage DistilBERT. models: - example/ticket-triage-distilbert ---
1. Write short_description
One sentence: what to paste in, what you get back, and which model is used.
2. Set models in the README header
Let the Space page and the model page link to each other.
3. Link back to the Space from the model page
In the model card, write “Try it online: Space link”.
4. Keep the Space running
The zephyr-chat page we sampled says Paused, and only an empty shell remains when you open it.
Should you write a page about the model on your own site?
Yes. A Hugging Face page is the platform's template: the meta description is generated and a Space page has no body text. A page on your own site is the version you fully control, and it gives an AI one more source outside Hugging Face.
1. Make the title a question
For example “Is there an open model that sorts support tickets automatically?”. Answer in the first paragraph: Ticket Triage DistilBERT, 66M parameters, English, with the accuracy and the test set.
2. Add a spec table
Task, parameter count, language, license, training data, metrics and hardware requirements.
3. Add usage code people can copy
A few lines of Python, written as text, not screenshots.
4. Link to the model page, the Space and GitHub
And link back to your site from the model page and README, so all three say the same sentence.
5. State the version and date
Update it whenever the model changes.
For how AI tools in general get recommended, see How do AI tool products get recommended by ChatGPT?; for how to write a GitHub README, see Will AI search cite my GitHub README?; for places beyond the README, see How do open source projects get recommended by AI search? Six places beyond the README.
FAQ
Does Hugging Face block AI crawlers?
The robots.txt we read on 2026-11-09 has only User-agent: * with Allow: / and names no GPTBot, ClaudeBot or PerplexityBot. This is only a judgement from the robots.txt text.
Does the first paragraph of a model card become the page description?
Not in what we saw: the meta description of a model page is an automatic pattern (task plus parameter count), and the first paragraph is in the page body.
Which metadata matters most?
The official docs say metadata supports discovery and filtering. At least set license, language, pipeline_tag (for non-transformers models), library_name and tags, and set datasets and base_model if you have training data and a base model.
What can a crawler read on a Space page?
On the Space pages we sampled, the body had only a few hundred characters once scripts were removed, and the app is in an iframe. To give the page a description, write short_description in the README header.
Will an AI cite Hugging Face pages?
There is no official statement and we have no evidence of actual citations, so this is not verified.
Do you need another page on your own site?
Yes. The platform page's meta description is automatic, while the page on your own site is the version you control and can carry a spec table, usage code and dates.
Sources
- Hugging Face · Model Cards (checked 2026-11-09)
- Hugging Face · Spaces Configuration Reference (checked 2026-11-09)
- Hugging Face · huggingface.co/robots.txt (Read 2026-11-09)
- Hugging Face · openai-community/gpt2 (Read 2026-11-09)
- Hugging Face · spaces/gradio/hello_world (Read 2026-11-09)
- kagentx evaluations: the data on the case study pages; each example gives its date.