Use Applebot when you want Apple Search to crawl and render your site. Use Applebot-Extended when you want to control whether Apple uses content crawled by Applebot for foundation-model training. Use noindex to keep a page out of Apple’s index and nosnippet to keep page data out of broad AI-generated answers. A page-level isAccessibleForFree: false paywall signal has a different job again. Apple’s About Applebot documentation says Applebot-Extended is not a crawler and that its site rules are not considered in Apple Search ranking. These controls have scoped documented effects; they do not promise rankings or traffic.
Choose the outcome before editing a directive
These controls sit at different layers. A robots.txt group addresses whether Applebot may fetch a path. An indexing directive addresses whether Applebot stores and surfaces a page. nosnippet addresses the description or web-answer context Apple can create from a page. Structured data describes the access state of a paywalled page. The Applebot-Extended choice addresses training use of data fetched by Applebot; it does not replace the Applebot crawl rule.
| Desired outcome | Primary control | Documented effect | It does not do |
|---|---|---|---|
| Keep a path open to Apple Search | User-agent: Applebot in robots.txt | Allows the general Applebot search crawl when the rest of the site permits it. | Guarantee indexing, a result, a ranking or traffic. |
| Keep Search access while opting out of training use | User-agent: Applebot-Extended with a scoped Disallow | Controls how Apple uses data crawled by Applebot for foundation-model training. | Block the Applebot crawler or remove a page from Search. |
| Keep a page out of Apple indexing surfaces | noindex in HTML | Apple says the page will not be indexed or appear in Spotlight or Siri Suggestions. | Opt the page out of foundation-model training or replace a crawl rule. |
| Keep page data out of broad AI answer context | nosnippet meta or response header | Apple will not generate a description or web answer from the page; suggestions may retain its title. | Opt out of training or guarantee that Applebot will not crawl the page. |
| Describe a paid, metered or subscription page | Page-level JSON-LD isAccessibleForFree: false | The page can remain eligible in search while Apple does not use its content as additional AI-answer context. | Control training use or apply a section-level paywall rule. |
The examples below are illustrative configurations. They are not executed by this article. Apply them to the smallest path or page set that matches the policy decision, then verify the response that visitors and Applebot actually receive.
Keep Apple Search crawling and rendering separate
Applebot is Apple’s general web crawler for Search-related experiences, including features integrated with Spotlight, Siri and Safari. Apple says it respects standard robots.txt directives targeted at Applebot. A path-scoped group can keep a crawler from requesting a path while leaving public content available. robots.txt is not an access-control boundary, so private data still needs authentication and server-side access controls:
User-agent: Applebot
Allow: /
Disallow: /private/
User-agent: *
Disallow: /not-allowed/This source-checked pattern means that Applebot may crawl the public paths but should not request /private/. If the entire site should be excluded from the Applebot crawl, the Applebot group would need a site-wide Disallow: /; that is a much broader choice than blocking one directory. When auditing the served file, inspect the Applebot-specific group and any generic or Googlebot group separately. An explicit Applebot group is not an additive override of the generic group; when no Applebot group is present, Apple says Applebot follows Googlebot instructions.
Apple says that if a file does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. It does not follow crawl-delay. It may also render pages in a browser, so blocking the JavaScript, CSS or XHR resources needed to present the useful content can reduce what Applebot can understand. Check the rendered page and the resource rules together. For the wider crawl, render and index sequence, use the technical SEO guide.
Use Applebot-Extended for a training preference
Applebot-Extended is a secondary user agent that lets publishers control whether website content is used to train Apple’s general-purpose foundation models, including models used across Apple Intelligence, Services and Developer Tools. Applebot-Extended does not crawl webpages. Apple uses the rule to decide how to use data fetched by Applebot.
User-agent: Applebot-Extended
Disallow: /private/The example opts the /private/ path out of the Applebot-Extended use described by Apple’s documentation. If the intention is to opt out the whole site, the scope would be Disallow: / under the Applebot-Extended group. Keep the ordinary Applebot group aligned with the separate Search decision. Disallowing Applebot-Extended does not itself prevent Applebot from crawling or prevent a page from remaining discoverable through Apple Search experiences.
Apple also states that site rules for Applebot-Extended are not considered in Search ranking. Treat that as a boundary on the control, not as evidence of a ranking benefit or penalty. A training preference is a data-use decision; it is not a ranking lever.
Keep noindex and nosnippet in different buckets
Use noindex when the page should not be indexed
Apple supports robots meta tags in the HTML <head>. Its documented noindex behavior is that Applebot will not index the page and the page will not appear in Spotlight or Siri Suggestions. This answers an indexing-surface question. It is not a training opt-out and it is not a substitute for a crawl-access decision. The Applebot-specific form below scopes the example to Apple; a generic name="robots" tag can also affect other engines.
<meta name="applebot" content="noindex" />Use the directive when the page should stay out of those Apple surfaces. Do not add it merely because the business wants to keep content out of foundation-model training; use the Applebot-Extended group for that separate decision. The directive also has to be delivered in a response Applebot can access, otherwise an upstream crawl block can prevent the crawler from seeing the instruction.
Use nosnippet when the page may remain searchable but should not feed broad answers
Apple documents nosnippet as a way to prevent a page description or web answer and to keep the page data from being used as additional context and up-to-date content when Apple models generate output. Suggestions may include the page title. That is narrower than noindex: the page can remain eligible for discovery while the content is withheld from this answer-context use. For broader answer-engine context, see the technical SEO for AEO guide.
For HTML, use an Applebot-scoped meta tag:
<meta name="applebot" content="nosnippet" />For a response header on HTML or a non-HTML resource, use the server-delivered form:
X-Robots-Tag: applebot: nosnippetThe header form is useful for PDFs, images or a server-wide rule. Neither form is the Applebot-Extended training control. If a page must be excluded from indexing as well as answer context, record both decisions explicitly and test the combined response.
Mark paywalled pages at page level
For a page behind a paywall, meter or subscription, Apple’s documentation supports the Schema.org isAccessibleForFree property in page-level JSON-LD:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"isAccessibleForFree": false
}
</script>Apple says a page marked isAccessibleForFree: false remains eligible to appear in search, but Applebot will not use that content as additional context when AI models generate output for Apple products and services. This signal applies to the page. Section-level markup using hasPart is not supported for this purpose.
This is an access-state description, not a training preference. A publisher can need both the page-level paywall signal and an Applebot-Extended rule, but they answer different questions. Apply the structured data to each page whose access state is actually paid or metered; do not copy it onto a free page as a generic AI-control switch.
Verify the control you actually changed
- Read the served
robots.txt. Confirm the Applebot and Applebot-Extended groups, path scope and any generic fallback. Do not infer the live rule from a local template or a remembered screenshot. - Inspect the delivered page response. For HTML, check the final
<head>fornoindexor Applebot-specificnosnippet. For PDFs and images, inspect the response headers. For paywalled pages, inspect the page-level JSON-LD and its actual access state. - Identify real crawler requests. Apple’s page says Applebot traffic can be checked by reverse DNS in the
*.applebot.apple.comdomain or against Apple’s published IP CIDRs. The reverse result should end in.applebot.apple.com, and a forward lookup of that hostname should return the original IP. These illustrative checks show the two-way DNS test described by Apple:
host 17.58.101.179
host 17-58-101-179.applebot.apple.com- Check rendering dependencies. Review access logs and the rendered output for the resources Applebot needs. A robots rule that blocks required CSS, JavaScript or XHR can make a page render incompletely even when the main HTML is crawlable.
- Separate observation from proof. A verified Applebot request shows that the crawler reached a resource. It does not prove indexing, ranking, model training use or an AI answer. Search appearance and logs are separate observations to record after the relevant recrawl window.
An illustrative decision for one product page
A public product page may need to remain available to Apple Search while the publisher opts out of training use and broad AI answer context. The operator can keep the Applebot group open, disallow that path in the Applebot-Extended group, and add nosnippet. If the page should also stay out of Spotlight or Siri Suggestions, add noindex and accept that it changes the indexing outcome. If the page is paid or metered, add isAccessibleForFree: false at page level. The illustration shows how the controls combine; it is not a claim about a specific site’s Apple Search result or model behavior.
Sources and scope
The primary source is Apple’s About Applebot documentation, published September 4, 2026 and checked for this guide on September 7, 2026. Search Engine Roundtable’s report is included as independent discovery context. Apple’s documentation controls the operational claims here; the independent report does not establish a ranking, training or traffic effect.
