Getting Started with Web Ingestion in Docket

Last updated: August 10, 2026

Web Ingestion brings website content into Docket so agents can answer with current, approved web-based knowledge.

When to Use Web Ingestion

Use Web Ingestion when the source content lives on a website and should be available as a Docket knowledge source. Common use cases include:

  • Ingesting a product documentation site.

  • Adding a focused set of onboarding pages.

  • Using a site map to control which pages are included.

  • Refreshing web content after a website update.

For Google Drive, OneDrive, SharePoint, or similar file links, use the dedicated integration instead of Web Ingestion.

Choose an Ingestion Method

Docket supports three web ingestion patterns.

Entire Domain

Use an entire domain when you want Docket to discover and ingest content from a full website.

This can be useful for large content libraries, but it can also include pages you did not intend to use. Review the discovered URLs and consider using site maps when you need tighter control.

Site Maps

Use site maps when you want a more controlled ingestion scope.

When available, Docket can detect site maps for a domain. You can then select all site maps or choose specific ones to ingest. This helps keep agent knowledge focused on the most relevant pages.

Individual URLs

Use individual URLs when you want to start with a small set of pages.

This is useful for onboarding, testing, or quickly adding a few approved pages without crawling an entire site.

Add Web Content

  1. Log in to app.docketai.com.

  2. Open Knowledge Sources.

  3. Open the Web URLs tab.

  4. Click Add Source.

  5. Choose the ingestion method that matches your source: domain, site map, or individual URLs.

  6. Submit the source and wait for ingestion to start.

After submission, use Ingestion Jobs to monitor progress.

image.png

Monitor Ingestion

Ingestion visibility helps you confirm whether the source is ready for agents. Review:

  • Discovered URLs

  • Completed pages

  • Pages that need attention

  • Error categories

  • Job start time

  • Next scheduled sync

  • Crawl duration

  • URL-level statuses

If pages fail, review the error category before adding the source to an agent.

image.png

Understanding Ingestion Jobs and States

Ingestion Jobs is the place to check whether a source is still processing, ready to use, or needs attention. Use it after adding a domain, site map, individual URL, file, or synced source.

Treat the current Done status as the launch gate. If a job is not Done, wait for processing or resolve the status before assigning the source to a live agent.

Re-Sync Web Content

Ingested web content can re-sync automatically. The current setup defaults to 30 days, with 14-day and 7-day choices for pages that change more frequently.

If a website changes and you need the update sooner, use the record menu to trigger a manual re-sync.

Use Web Content in an Agent

After the source is ingested:

  1. Open Marketing Agent Configuration.

  2. Select the agent.

  3. Open the Knowledge tab.

  4. In Knowledge Base, click Sources.

  5. Select the ingested web source if it is appropriate for that agent.

Verify Web Ingestion

  1. Confirm the source row has the expected Type: Domain or Page.

  2. Confirm Status is Done and review the page count for unexpected crawl breadth.

  3. Open the intended agent, select the source, and Preview a question whose answer appears on an ingested page.

  4. Preview an out-of-scope question to confirm the agent does not treat the entire domain as relevant to every request.

  5. After a material website change, re-sync and repeat the expected-answer test.

The ingestion is ready when the job is Done, the page count matches the intended scope, and Preview uses the expected web content.

Best Practices

  • Prefer site maps when you need precise control over included pages.

  • Start with individual URLs when testing a new source.

  • Review discovered URLs before relying on a broad domain crawl.

  • Check ingestion errors before assigning the source to an agent.

  • Re-sync after major website changes.

  • Avoid adding internal or sensitive pages unless the agent is explicitly intended to use them.

Troubleshooting

Issue

What to check

Too many pages were ingested

Use a site map or individual URLs instead of a broad domain.

Important pages are missing

Check discovered URLs, site map selection, and ingestion errors.

Source is not ready

Open Ingestion Jobs and review the current status.

Agent does not use the web content

Confirm the source is selected in the agent's Knowledge tab.

Website changed but answers are outdated

Trigger a manual re-sync or confirm the automatic sync schedule.

Related Articles