Tools / Website to Markdown Crawler for AI, RAG & LLMs

Website to Markdown Crawler for AI, RAG & LLMs

Turn any website into clean Markdown for ChatGPT, RAG pipelines, vector databases and AI agents. Input: startUrls. Output (JSON/CSV): one row per item with url, title, wordCount, language, description, markdown.

Run it on Apify Store →

What it's for

Feed company docs or a website into a RAG chatbot or vector DB
Build LLM training/eval corpora from sites you have rights to
Clean-text copies of competitor pages for analysis

Example input

{
 "startUrls": [
  "https://therealtour360.com",
  "https://docs.apify.com/platform/actors",
  "not a url ::"
 ],
 "crawl": false
}

Input reference

Field	Type	Notes
`startUrls` required	array	Single pages, or site homepages when crawl is on.
`crawl`	boolean	Follow the sitemap and internal links from each start URL.
`maxPagesPerSite`	integer	Only used when crawling.
`useSitemap`	boolean	Seed the crawl from the site's sitemap.
`mainContentOnly`	boolean	Strip navigation, headers, footers and sidebars (Readability).
`includeLinks`	boolean	Keep hyperlinks in the Markdown.
`respectRobots`	boolean	Skip pages disallowed by robots.txt.
`maxConcurrency`	integer	Max concurrent sites

Pricing

Pay per event: Page converted to Markdown $0.001. Plus Apify's small per-run start fee. Apify's free plan includes monthly credit to try it.

How to run

Open the Store page and click Try for free.
Fill in the input (or paste the example JSON in the JSON tab).
Click Start, then export results as CSV, Excel or JSON, or call it via the Apify API.
Optional: add a Schedule and a webhook/Slack/email integration for ongoing monitoring.

Open Website to Markdown Crawler for AI, RAG & LLMs on Apify →