Crawl your own docs into a Bedrock agent without a vector database

On 17 Jun 2026, Amazon Bedrock Managed Knowledge Base became generally available with six connectors: S3, SharePoint, Confluence, Google Drive, OneDrive, and Web Crawler. The crawler is the one you pick when the corpus is HTML you are allowed to index and you do not want to stand up a vector database. It is the wrong connector when two employees must not see the same pages.

This is the operator guide for that crawler as documented on 23 Sep 2026. The Q2 announcements roundup names the connector in one line. The classic RAG pipeline post is the older customer-managed setup. Do not copy that post’s hourly vector-store figure onto Managed Knowledge Base. This page does not restate Managed Knowledge Base retrieval pricing. Check the Bedrock pricing page before you forecast the bill.

Reproduce this — Copy webcrawler-managed-connector.json, robots.txt, and crawl-scope-worksheet.csv. Fill the blank row in the worksheet before you create the data source. The JSON is a public-site example, depth 2, 50 URLs per minute, PATH_SPECIFIC, with /private/ excluded.

What you actually get

The crawler starts at seed URLs (maximum 10) or sitemap URLs (maximum 3), follows child links, and respects robots.txt under RFC 9309. The first sync is a full crawl. Later syncs are incremental: added, updated, and deleted content. It dedupes URLs and retries failed requests. The user-agent is amazon-bedrock-knowledgebase-on-behalf-of-HASH.

AWS’s News Blog (updated 19 Jun 2026) says the Web Crawler connector keeps HTML structure, including tables and embedded images. That is the reason to crawl a docs site instead of flattening it to plain text yourself.

Planning number, not a benchmark we ran. The rate cap is 300 URLs per minute. A 10,000-URL help center takes at least 34 minutes of crawl time if the crawler stays on that cap the whole sync (10,000 / 300). Lower the rate when the origin returns HTTP 429. Raise implicitWaitInSeconds when the HTML arrives after the page is “ready.”

Regions at the 17 Jun 2026 GA: us-east-1, us-west-2, eu-west-1, eu-central-1, eu-west-2, ap-southeast-2, ap-northeast-1, and us-gov-west-1. Confirm the list in AWS Capabilities by Region before you promise a knowledge agent in another Region.

Opinion: Start with a sitemap, depth 2 or 0, scope PATH_SPECIFIC or DOMAINS_ONLY, and an exclusion regex for anything you would not paste into a support agent. SUB_DOMAINS is how a docs seed quietly indexes a marketing blog, a status site, and a staging host on the same registrable domain.

How it differs from the other crawlers

Classic Bedrock web crawler. Still preview. AWS says only Amazon OpenSearch Serverless is available with that data source, and the page tells you to prefer Managed Knowledge Base. The API shape is different: classic type: WEB plus webConfiguration, versus managed MANAGED_KNOWLEDGE_BASE_CONNECTOR with connectorParameters.type: WEB. The classic user-agent is bedrockbot-UUID (then generic bedrockbot). A robots.txt written for bedrockbot does not name the managed crawler.

SharePoint, OneDrive, and S3 on the same managed knowledge base. AWS names these as the connectors that support document-level ACLs. The web crawler does not. Sign-in is not an ACL.

Amazon Kendra’s crawler. Wrong default for new customers after 30 Jul 2026, when Kendra enters maintenance for new customers. Map that date in the Quick Suite history post.

Your own crawler, or the S3 connector. Wins when navigation is click handlers, when you already have the files, or when you need a permission model the web crawler cannot express. LangChain on Bedrock makes the same call: do not rebuild a sidecar index when a Bedrock connector already fits, and do not skip ACLs.

Sign-in gets the bot in. It does not filter retrieval

Credentials live in Secrets Manager. CreateDataSource is asynchronous: status moves from CREATING to AVAILABLE.

HTTP 401, 403, a login redirect loop, or a session timeout means the secret is wrong or the XPath missed the control. Validate each XPath in browser devtools before you blame the knowledge base.

AWS CLI v2, Agents for Amazon Bedrock build-time endpoint. Replace the knowledge base id. The JSON is the sample in the artifact, not your production scope.

aws bedrock-agent create-data-source \
  --name "WebCrawler-connector" \
  --knowledge-base-id "your-knowledge-base-id" \
  --data-source-configuration file://webcrawler-managed-connector.json

The robots.txt contradiction

Two sentences in the same guide disagree in practice:

1. If robots.txt is not found, the crawler disallows.
2. If it cannot fetch robots.txt because of blocking, a parse error, or a timeout, it treats the file as missing and crawls.

A WAF rule that drops the bot is not a deny. A missing file is a deny. Publish an explicit file for amazon-bedrock-knowledgebase-on-behalf-of-HASH, allow the paths you want indexed, and disallow the rest.

Page-level noindex and nofollow meta tags are honored, but the crawler has to fetch the page to read them. A robots.txt disallow prevents that refetch. Do not disallow a URL you still want the crawler to notice a new noindex on.

What broke (AWS-documented) — Symptom from the Web Crawler troubleshooting table: the sync succeeds and only the seed URL is indexed. Cause: navigation is wired to click, scroll, or dynamic menus instead of ordinary anchor tags. The crawler renders JavaScript and does not simulate those interactions. Detection: the data source is AVAILABLE and the indexed URL count is the seed list. Recovery: add a sitemap of every URL, add more seeds, or move the export to the S3 connector.

What to Do This Week

1. Fill crawl-scope-worksheet.csv. If the ACL cell is yes, stop and use SharePoint, OneDrive, or S3.
2. Publish robots.txt with the managed user-agent before the first sync. Confirm the hash from a request log. The sample file uses the placeholder HASH.
3. Prefer a sitemap. Set depth to 0 until the sitemap is the source of truth.
4. Exclude /private/, staging hosts, and search result URLs. Cap the rate under what the origin can serve. Watch for HTTP 429.
5. Run one sync in a GA Region and count indexed URLs against the sitemap. A successful sync with a tiny URL count is the JavaScript-navigation failure, not a green bill of health.

If the knowledge base is for a customer-support or catalog agent, pair the connector with the RAG pattern and keep money-moving actions behind a human. The crawler does not grant the agent permission to act.

What This Post Doesn’t Cover

Managed Knowledge Base retrieval price, embedding and rerank choices, and agentic-retrieval quality. Chunking and Guardrails. Crawl of sites you do not operate. The classic OpenSearch Serverless minimum. We did not run a crawl of a customer site for this post. The 34-minute figure is arithmetic on the published 300 URL/minute cap.