--- title: Crawl your own docs into a Bedrock agent without a vector database description: Managed Knowledge Base Web Crawler, GA with the service on 17 Jun 2026, follows up to 300 URLs per minute from 10 seeds or 3 sitemaps. A 10,000-URL help center is at least 34 minutes at that cap, and the crawler still has no per-page ACL. url: https://www.factualminds.com/blog/amazon-bedrock-managed-kb-web-crawler-2026/ datePublished: 2026-09-23T00:00:00.000Z dateModified: 2026-09-23T00:00:00.000Z author: palaniappan-p category: Generative AI tags: amazon-bedrock, knowledge-bases, rag, web-crawler --- # Crawl your own docs into a Bedrock agent without a vector database > Managed Knowledge Base Web Crawler, GA with the service on 17 Jun 2026, follows up to 300 URLs per minute from 10 seeds or 3 sitemaps. A 10,000-URL help center is at least 34 minutes at that cap, and the crawler still has no per-page ACL. On **17 Jun 2026**, Amazon Bedrock Managed Knowledge Base became generally available with six connectors: S3, SharePoint, Confluence, Google Drive, OneDrive, and **Web Crawler**. The crawler is the one you pick when the corpus is HTML you are allowed to index and you do not want to stand up a vector database. It is the wrong connector when two employees must not see the same pages. This is the operator guide for that crawler as documented on **23 Sep 2026**. The [Q2 announcements roundup](/blog/aws-major-announcements-q2-2026-enterprise-roundup/) names the connector in one line. The [classic RAG pipeline post](/blog/how-to-build-rag-pipeline-amazon-bedrock-knowledge-bases/) is the older customer-managed setup. Do not copy that post's hourly vector-store figure onto Managed Knowledge Base. This page does not restate Managed Knowledge Base retrieval pricing. Check the Bedrock pricing page before you forecast the bill. > **Reproduce this** — Copy [`webcrawler-managed-connector.json`](/examples/architecture-blog-2026/managed-kb-web-crawler/webcrawler-managed-connector.json), [`robots.txt`](/examples/architecture-blog-2026/managed-kb-web-crawler/robots.txt), and [`crawl-scope-worksheet.csv`](/examples/architecture-blog-2026/managed-kb-web-crawler/crawl-scope-worksheet.csv). Fill the blank row in the worksheet before you create the data source. The JSON is a public-site example, depth **2**, **50** URLs per minute, `PATH_SPECIFIC`, with `/private/` excluded. --- ## What you actually get The crawler starts at seed URLs (maximum **10**) or sitemap URLs (maximum **3**), follows child links, and respects `robots.txt` under RFC 9309. The first sync is a full crawl. Later syncs are incremental: added, updated, and deleted content. It dedupes URLs and retries failed requests. The user-agent is `amazon-bedrock-knowledgebase-on-behalf-of-HASH`. AWS's News Blog (updated **19 Jun 2026**) says the Web Crawler connector keeps HTML structure, including tables and embedded images. That is the reason to crawl a docs site instead of flattening it to plain text yourself. | Control | Range | Default when AWS states one | | --- | --- | --- | | Crawl depth | 0–10 | **2**. Depth 0 indexes only the URLs you listed. | | Links followed per URL | 1–1000 | **100** | | URLs per minute | 1–300 | Not stated. Set it. | | Scope | `PATH_SPECIFIC`, `DOMAINS_ONLY`, `SUB_DOMAINS` | Same host and same initial path as the seed | | Attachment size | Numeric megabytes, as a string | **500** | | Auth | `NO_AUTH`, `BASIC_AUTH`, `FORM`, `SAML` | Required field. Public sites use `NO_AUTH`. | **Planning number, not a benchmark we ran.** The rate cap is **300 URLs per minute**. A **10,000-URL** help center takes at least **34 minutes** of crawl time if the crawler stays on that cap the whole sync (10,000 / 300). Lower the rate when the origin returns HTTP 429. Raise `implicitWaitInSeconds` when the HTML arrives after the page is "ready." Regions at the **17 Jun 2026** GA: us-east-1, us-west-2, eu-west-1, eu-central-1, eu-west-2, ap-southeast-2, ap-northeast-1, and us-gov-west-1. Confirm the list in AWS Capabilities by Region before you promise a knowledge agent in another Region. **Opinion:** Start with a sitemap, depth **2** or **0**, scope `PATH_SPECIFIC` or `DOMAINS_ONLY`, and an exclusion regex for anything you would not paste into a support agent. `SUB_DOMAINS` is how a docs seed quietly indexes a marketing blog, a status site, and a staging host on the same registrable domain. --- ## How it differs from the other crawlers **Classic Bedrock web crawler.** Still preview. AWS says only Amazon OpenSearch Serverless is available with that data source, and the page tells you to prefer Managed Knowledge Base. The API shape is different: classic `type: WEB` plus `webConfiguration`, versus managed `MANAGED_KNOWLEDGE_BASE_CONNECTOR` with `connectorParameters.type: WEB`. The classic user-agent is `bedrockbot-UUID` (then generic `bedrockbot`). A `robots.txt` written for `bedrockbot` does not name the managed crawler. **SharePoint, OneDrive, and S3 on the same managed knowledge base.** AWS names these as the connectors that support document-level ACLs. The web crawler does not. Sign-in is not an ACL. **Amazon Kendra's crawler.** Wrong default for new customers after **30 Jul 2026**, when Kendra enters maintenance for new customers. Map that date in the [Quick Suite history post](/blog/amazon-quick-suite-history-q-business-kendra-migration-2026/). **Your own crawler, or the S3 connector.** Wins when navigation is click handlers, when you already have the files, or when you need a permission model the web crawler cannot express. [LangChain on Bedrock](/blog/langchain-aws-bedrock-2026/) makes the same call: do not rebuild a sidecar index when a Bedrock connector already fits, and do not skip ACLs. | You have | Use | | --- | --- | | Docs or a help center you own, no per-user ACL | Managed Web Crawler, sitemap preferred | | SharePoint or OneDrive with different readers | That connector, not the crawler | | HTML already exported, or a JS app with no real links | S3 connector | | You must choose the vector store | Classic knowledge base. Accept the preview crawler only if you still need it. AWS's recommendation is Managed. | --- ## Sign-in gets the bot in. It does not filter retrieval | Method | What it does | When | | --- | --- | --- | | `NO_AUTH` | No credentials | Public docs | | `BASIC_AUTH` | HTTP Basic from Secrets Manager | The browser username dialog | | `FORM` | Posts the HTML login form. You supply the login URL and XPath for username, password, and submit | Ordinary form login | | `SAML` | Same shape, against the IdP login form | SAML SSO that is still a form | Credentials live in Secrets Manager. `CreateDataSource` is asynchronous: status moves from `CREATING` to `AVAILABLE`. HTTP 401, 403, a login redirect loop, or a session timeout means the secret is wrong or the XPath missed the control. Validate each XPath in browser devtools before you blame the knowledge base. AWS CLI v2, Agents for Amazon Bedrock build-time endpoint. Replace the knowledge base id. The JSON is the sample in the artifact, not your production scope. ```bash aws bedrock-agent create-data-source \ --name "WebCrawler-connector" \ --knowledge-base-id "your-knowledge-base-id" \ --data-source-configuration file://webcrawler-managed-connector.json ``` --- ## The robots.txt contradiction Two sentences in the same guide disagree in practice: 1. If `robots.txt` is **not found**, the crawler **disallows**. 2. If it **cannot fetch** `robots.txt` because of blocking, a parse error, or a timeout, it treats the file as missing and **crawls**. A WAF rule that drops the bot is not a deny. A missing file is a deny. Publish an explicit file for `amazon-bedrock-knowledgebase-on-behalf-of-HASH`, allow the paths you want indexed, and disallow the rest. Page-level `noindex` and `nofollow` meta tags are honored, but the crawler has to fetch the page to read them. A `robots.txt` disallow prevents that refetch. Do not disallow a URL you still want the crawler to notice a new `noindex` on. > **What broke (AWS-documented)** — Symptom from the Web Crawler troubleshooting table: the sync succeeds and only the seed URL is indexed. Cause: navigation is wired to click, scroll, or dynamic menus instead of ordinary anchor tags. The crawler renders JavaScript and does not simulate those interactions. Detection: the data source is `AVAILABLE` and the indexed URL count is the seed list. Recovery: add a sitemap of every URL, add more seeds, or move the export to the S3 connector. --- ## What to Do This Week 1. Fill [`crawl-scope-worksheet.csv`](/examples/architecture-blog-2026/managed-kb-web-crawler/crawl-scope-worksheet.csv). If the ACL cell is yes, stop and use SharePoint, OneDrive, or S3. 2. Publish `robots.txt` with the managed user-agent before the first sync. Confirm the hash from a request log. The sample file uses the placeholder `HASH`. 3. Prefer a sitemap. Set depth to **0** until the sitemap is the source of truth. 4. Exclude `/private/`, staging hosts, and search result URLs. Cap the rate under what the origin can serve. Watch for HTTP 429. 5. Run one sync in a GA Region and count indexed URLs against the sitemap. A successful sync with a tiny URL count is the JavaScript-navigation failure, not a green bill of health. If the knowledge base is for a customer-support or catalog agent, pair the connector with the [RAG pattern](/patterns/generative-ai-rag-on-bedrock/) and keep money-moving actions behind a human. The crawler does not grant the agent permission to act. --- ## What This Post Doesn't Cover Managed Knowledge Base retrieval price, embedding and rerank choices, and agentic-retrieval quality. Chunking and Guardrails. Crawl of sites you do not operate. The classic OpenSearch Serverless minimum. We did not run a crawl of a customer site for this post. The 34-minute figure is arithmetic on the published **300** URL/minute cap. ## FAQ ### When should I not use the Managed Knowledge Base Web Crawler? Skip it when each reader must see a different slice of the corpus. The crawler has no document-level ACL. Basic, form, and SAML sign-in only get the crawler past the login page. Anyone who can query the knowledge base can retrieve every indexed page. Use the SharePoint, OneDrive, or S3 connector when you need ACL filtering. ### What goes wrong if the site navigation is built in JavaScript? AWS documents a sync that succeeds and indexes only the seed URL. The crawler renders JavaScript and does not click, scroll, or open menus. Links that exist only as event handlers are invisible. Give it a sitemap that lists every URL, extra seed URLs, or export the pages to S3 and use the S3 connector. ### How is this different from the classic Bedrock web crawler? The classic connector is still a preview data source and, per AWS, only Amazon OpenSearch Serverless is available with it. AWS tells you to prefer Managed Knowledge Base. The managed connector uses type MANAGED_KNOWLEDGE_BASE_CONNECTOR and connectorParameters.type WEB. The classic connector uses type WEB plus webConfiguration, and its user-agent is bedrockbot-UUID, not amazon-bedrock-knowledgebase-on-behalf-of-HASH. ### What happens if robots.txt is missing? The crawler defaults to disallow when robots.txt is not found. The opposite happens when robots.txt exists but the crawler cannot fetch it because of blocking, a parse error, or a timeout. AWS says it then behaves as if robots.txt does not exist and crawls the site. Publish a real robots.txt. Do not rely on a network block as a deny. ### Can I crawl a site I do not own? No. AWS requires you to crawl only your own pages or pages you are authorized to crawl, and to follow the Amazon Acceptable Use Policy. A public URL is not authorization. ### Does Kendra's web crawler replace this? No. Amazon Kendra enters maintenance for new customers after 30 Jul 2026. New knowledge-agent work belongs on Bedrock Managed Knowledge Base, with the connector that matches your ACL needs. --- *Source: https://www.factualminds.com/blog/amazon-bedrock-managed-kb-web-crawler-2026/*