Tools / Website to RAG Chunks: LLM-Ready Text for Vector Databases
Website to RAG Chunks: LLM-Ready Text for Vector Databases
Crawl a website or docs site (sitemap + internal links) and get clean, heading-aware Markdown chunks ready for embeddings. Input: startUrls. Output (JSON/CSV): one row per item with url, title, headingPath, chunkIndex, totalChunks, tokenEstimate, text, chunkId.
What it's for
- Build a RAG chatbot over your docs or help center
- Load a website into a vector database without writing a chunker
- Refresh embeddings when pages change (stable chunk IDs + content hashes)
Example input
{
"startUrls": [
"https://docs.apify.com/academy",
"https://therealtour360.com",
"not a url ::"
],
"maxPagesPerSite": 6,
"chunkSize": 400
}Example output
{
"chunkId": "6ccb6c366c25a517e75b",
"url": "https://docs.apify.com/academy",
"title": "Apify Academy | Academy | Apify Documentation",
"language": "en",
"headingPath": [],
"chunkIndex": 0,
"totalChunks": 1,
"tokenEstimate": 300,
"charCount": 1200,
"contentHash": "a9e5b1509ae7255d",
"crawledAt": "2026-10-12T04:44:03.470Z"
}Input reference
| Field | Type | Notes |
|---|---|---|
startUrls required | array | Site homepages, docs roots or single pages. |
crawl | boolean | Follow the sitemap and internal links. |
maxPagesPerSite | integer | Max pages per site |
useSitemap | boolean | Use sitemap.xml |
chunkSize | integer | ~4 characters per token. |
chunkOverlap | integer | Chunk overlap (tokens) |
minChunkTokens | integer | Smaller pieces are merged into the previous chunk. |
mainContentOnly | boolean | Strip menus, footers and sidebars. |
includeLinks | boolean | Keep links |
respectRobots | boolean | Respect robots.txt |
maxConcurrency | integer | Max concurrent sites |
Pricing
Pay per event: Page crawled and chunked $0.0015. Plus Apify's small per-run start fee. Apify's free plan includes monthly credit to try it.
How to run
- Open the Store page and click Try for free.
- Fill in the input (or paste the example JSON in the JSON tab).
- Click Start, then export results as CSV, Excel or JSON, or call it via the Apify API.
- Optional: add a Schedule and a webhook/Slack/email integration for ongoing monitoring.