Guide · Hugging Face

How do models and Spaces on Hugging Face get recommended by AI?

Updated 2026-11-09 · kagentx

Short answer

Write the first paragraph of the model card as one checkable fact (which model, what it does, how big, trained on what, one metric), fill in metadata such as license, pipeline_tag, language and base_model in the YAML header, write a short_description for a Space, and put the same sentence on your own site. Tested on 2026-11-09: huggingface.co's robots.txt allows every crawler, the model card body is directly in the raw HTML of a model page but the meta description is an automatic pattern, and a Space page has almost no body text in its HTML. Whether AI answers cite Hugging Face pages has no official statement and is not verified.

Does Hugging Face block AI crawlers? Is the page body in the raw HTML?

No. On 2026-11-09 we read huggingface.co/robots.txt with curl: it has one group, User-agent: * with Allow: /, and names no GPTBot, ClaudeBot or PerplexityBot. We then fetched one model page and one Space page: the model card body is directly in the model page HTML, while the Space page HTML is almost only the title bar and the app itself runs in an iframe.

PageWhat we saw (2026-11-09)What it means for you
robots.txtUser-agent: * / Allow: /, plus a sitemap; no AI crawler is namedRead literally, all three crawlers may fetch
模型页 openai-community/gpt2HTTP 200, about 243 KB; the body contains the first paragraph of the model card, the tags (Text Generation, English, License: mit) and usage code; the meta description is an automatic pattern: “Text Generation model with 0.1B parameters. Download, explore files and run it on Hugging Face.”Crawlers can read the model card text, but the meta description is not the first paragraph you wrote
模型页 google-bert/bert-base-uncasedAlso in the HTML; the meta description follows the same pattern: “Fill-Mask model with 0.1B parameters…”The automatic pattern contains only the task and parameter count
Space 页 gradio/hello_worldHTTP 200, about 34 KB; with scripts removed the body is only about 142 characters; the meta description is the generic “Discover amazing ML apps made by the community”; it has WebApplication JSON-LDThe Space page itself does not tell a crawler what it does unless you wrote a short_description
Space 页 HuggingFaceH4/zephyr-chatThe meta description is a specific sentence, but there is no short_description in its README header and we did not find where it comes from; the body says the Space is pausedA paused Space page is left as an empty shell; where the description comes from is not verified

This is only what robots.txt says and what our own requests returned, not what an AI crawler really fetches. For whether crawlers run scripts, see Do AI crawlers render JavaScript?; for crawler rules, see How do you write robots.txt for AI crawlers?.

How should you write the first paragraph of a model card?

The official docs (read on 2026-11-09) say model cards are essential for discoverability, reproducibility and sharing, and should describe the model, its intended uses and limitations (including bias), training parameters, training datasets and evaluation results. The first paragraph is the first text anyone reads on the page; we suggest facts only: what the model is, what task it does, how big, which language, what it was fine-tuned on, and one metric that can be checked.

WeakBetter
第一段A state-of-the-art, powerful model that revolutionizes customer support.A fine-tuned DistilBERT model that sorts English customer-support tickets into five categories. 66M parameters; accuracy 91.4% on the held-out set in this repo.
标题MyModel-v2-finalSay what the model is: Ticket Triage DistilBERT
指标“More accurate than the rest”Give the dataset, the metric name and the number, and say you measured it yourself
局限Leave it outState known cases where it does not apply: English only, the time range of the training data
许可证Only in a fileAlso put it in the license field of the YAML header; the page shows it and users can filter by it
---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-classification
base_model: distilbert/distilbert-base-uncased
datasets:
- example/support-tickets
tags:
- customer-support
---

# Ticket Triage DistilBERT

A fine-tuned DistilBERT model that sorts English customer-support tickets into
five categories (billing, bug, how-to, account, other). 66M parameters; runs
on CPU in about 20 ms per ticket. Accuracy 91.4% on the held-out set in this repo.

After the first paragraph, we suggest in order: how to use it (a few lines of code), training data and process, an evaluation table, limits and bias. The page's title and meta description are generated by the Hub, so to get your own sentence read elsewhere, put it in the first paragraph of the model card and on your own site.

Which metadata fields show up on the page and in search?

The official docs say the YAML metadata at the top of a model card supports discovery and use: it can be used to filter models on huggingface.co/models and is shown on the model page. Below are the fields the docs mention, and what we saw on the gpt2 page.

FieldWhat the official docs say it doesHow to write it
licenseShown on the model page and filterable by license; for a custom license use license: other plus license_name and license_linkAlways set it, using a standard identifier
languageLanguage identifiers, as listed on the Hub's Languages pageList the languages it really supports
pipeline_tagTask type, shown on the model page and filterable; for transformers models it is inferred from config.json and can be overriddenSet it yourself for models that are not transformers models
library_nameNames the library; the docs recommend setting it explicitly, discourage auto-detection, and say repositories created after August 2024 are no longer assumed to be transformersSet it, for example transformers
tagsOpen-ended tags, shown on the pageUse the task and domain words users search for, not unrelated ones
datasetsThe page shows “Datasets used to train:” and links to the datasets on the HubSet it if you trained on Hub datasets
base_modelShows the base model and can be filtered by it; the Hub infers whether the relation is finetune, adapter, quantized or mergeRequired for fine-tuned, quantized and merged models

In the raw HTML of the gpt2 page we saw the tags that correspond to this metadata: Text Generation, Transformers, English, License: mit. That repository's YAML only sets language, tags and license; Text Generation was inferred by the Hub. Whether AI answers use these fields has no official statement and is not verified.

How should you write a Space description?

The raw HTML of a Space page does not contain the app (it runs in an iframe), so the description depends on fields in the README header. The official docs (Spaces Configuration Reference) say short_description is a short description of the Space shown in its thumbnail, and the models field lists the models used and is parsed from your code if left out. In the Spaces we saw without a short_description, the meta description was a generic sentence.

---
title: Ticket Triage Demo
emoji: 🎫
sdk: gradio
app_file: app.py
short_description: Paste a support ticket and get its category from Ticket Triage DistilBERT.
models:
- example/ticket-triage-distilbert
---
  1. 1. Write short_description

    One sentence: what to paste in, what you get back, and which model is used.

  2. 2. Set models in the README header

    Let the Space page and the model page link to each other.

  3. 3. Link back to the Space from the model page

    In the model card, write “Try it online: Space link”.

  4. 4. Keep the Space running

    The zephyr-chat page we sampled says Paused, and only an empty shell remains when you open it.

Should you write a page about the model on your own site?

Yes. A Hugging Face page is the platform's template: the meta description is generated and a Space page has no body text. A page on your own site is the version you fully control, and it gives an AI one more source outside Hugging Face.

  1. 1. Make the title a question

    For example “Is there an open model that sorts support tickets automatically?”. Answer in the first paragraph: Ticket Triage DistilBERT, 66M parameters, English, with the accuracy and the test set.

  2. 2. Add a spec table

    Task, parameter count, language, license, training data, metrics and hardware requirements.

  3. 3. Add usage code people can copy

    A few lines of Python, written as text, not screenshots.

  4. 4. Link to the model page, the Space and GitHub

    And link back to your site from the model page and README, so all three say the same sentence.

  5. 5. State the version and date

    Update it whenever the model changes.

For how AI tools in general get recommended, see How do AI tool products get recommended by ChatGPT?; for how to write a GitHub README, see Will AI search cite my GitHub README?; for places beyond the README, see How do open source projects get recommended by AI search? Six places beyond the README.

FAQ

Does Hugging Face block AI crawlers?

The robots.txt we read on 2026-11-09 has only User-agent: * with Allow: / and names no GPTBot, ClaudeBot or PerplexityBot. This is only a judgement from the robots.txt text.

Does the first paragraph of a model card become the page description?

Not in what we saw: the meta description of a model page is an automatic pattern (task plus parameter count), and the first paragraph is in the page body.

Which metadata matters most?

The official docs say metadata supports discovery and filtering. At least set license, language, pipeline_tag (for non-transformers models), library_name and tags, and set datasets and base_model if you have training data and a base model.

What can a crawler read on a Space page?

On the Space pages we sampled, the body had only a few hundred characters once scripts were removed, and the app is in an iframe. To give the page a description, write short_description in the README header.

Will an AI cite Hugging Face pages?

There is no official statement and we have no evidence of actual citations, so this is not verified.

Do you need another page on your own site?

Yes. The platform page's meta description is automatic, while the page on your own site is the version you control and can carry a spec table, usage code and dates.

Sources

kagentx

Can crawlers read your pages?
Let the evaluation tell you

The evaluation checks whether crawlers can get in and whether the text is served directly, then asks ChatGPT, Perplexity and Claude (with web search on) your buyers' questions and sees whether you are mentioned. The Basic plan is $1, and results usually take 2–3 minutes.