Tools / robots.txt & llms.txt Monitor: AI Crawler Block Checker

robots.txt & llms.txt Monitor: AI Crawler Block Checker

Bulk-check which AI crawlers (GPTBot, ClaudeBot, etc.) each site blocks in robots.txt, plus llms.txt presence. Input: domains. Output (JSON/CSV): one row per item with domain, robotsFound, aiBotsBlockedCount, aiBotsBlocked, blocksAllAiTraining, llmsTxtFound, llmsFullTxtFound, changed, changes.

Run it on Apify Store →

Example input

{
 "domains": [
  "nytimes.com",
  "stripe.com",
  "anthropic.com",
  "example.com"
 ],
 "monitorMode": true
}

Example output

{
  "domain": "example.com",
  "robotsStatus": 404,
  "robotsFound": false,
  "robotsHash": null,
  "robotsBytes": 0,
  "sitemaps": [],
  "userAgentGroups": 0,
  "aiBotPolicy": {
    "GPTBot": "NO_ROBOTS",
    "ChatGPT-User": "NO_ROBOTS",
    "OAI-SearchBot": "NO_ROBOTS",
    "ClaudeBot": "NO_ROBOTS",
    "Claude-Web": "NO_ROBOTS",
    "anthropic-ai": "NO_ROBOTS",
    "Google-Extended": "NO_ROBOTS",
    "PerplexityBot": "NO_ROBOTS",
    "Perplexity-User": "NO_ROBOTS",
    "CCBot": "NO_ROBOTS",
    "Bytespider": "NO_ROBOTS",
    "Applebot-Extended": "NO_ROBOTS",
    "Meta-ExternalAgent": "NO_ROBOTS",
    "FacebookBot": "NO_ROBOTS",
    "Amazonbot": "NO_ROBOTS",
    "cohere-ai": "NO_ROBOTS",
    "Diffbot": "NO_ROBOTS",
    "YouBot": "NO_ROBOTS"
  },
  "aiBotsBlocked": [],
  "aiBotsBlockedCount": 0,
  "blocksAllAiTraining": false,
  "llmsTxtFound": false,
  "llmsTxtStatus": 404,
  "llmsTxtHash": null,
  "llmsTxtTitle": null,
  "llmsTxtLinkCount": 0,
  "llmsFullTxtFound": false,
  "llmsFullTxtBytes": 0
}

Input reference

Field	Type	Notes
`domains` required	array	Domains or URLs (homepage host is used).
`includeContent`	boolean	Add full robots.txt and llms.txt text to each record.
`maxConcurrency`	integer	Domains in parallel (3 small requests each).
`monitorMode`	boolean	Remember the last result between runs (named key-value store) and flag changes. Schedule the Actor to get alerts.
`onlyChanged`	boolean	In monitor mode, skip unchanged items in the dataset.
`stateStoreName`	string	Named key-value store for monitor state.

Pricing

Pay per event: Domain checked $0.002, robots/llms.txt changed (monitor) $0.01. Plus Apify's small per-run start fee. Apify's free plan includes monthly credit to try it.

How to run

Open the Store page and click Try for free.
Fill in the input (or paste the example JSON in the JSON tab).
Click Start, then export results as CSV, Excel or JSON, or call it via the Apify API.
Optional: add a Schedule and a webhook/Slack/email integration for ongoing monitoring.

Open robots.txt & llms.txt Monitor on Apify →