DocsUser GuideAdding Web URLs Adding Web URLs
In addition to file uploads, you can add web pages to your knowledge base by pasting a URL. Opentrace will crawl the page and extract its content.
How It Works
- Open the Knowledge Base tab in your project
- Click the Add URL button
- Paste in a valid URL (e.g.,
https://example.com/blog-post) - The backend uses ScrapingBee to render and crawl the page
- The extracted HTML content is then processed through the same ingestion pipeline as file uploads
What Gets Extracted
ScrapingBee renders the page in a headless browser and returns the full HTML, including:
- Page text content (headings, paragraphs, lists)
- Table data
- Inline images (if accessible)
JavaScript-rendered content is supported since ScrapingBee uses a full browser engine.
Only publicly accessible URLs are supported. Pages behind login walls or paywalls cannot be crawled.
Use Cases
- Blog posts — add company blog articles for the AI to reference
- Documentation pages — ingest external API docs or wiki pages
- News articles — add articles for research and analysis
- Product pages — ingest competitor or your own product information
After adding a URL, the processing stages are identical to file uploads. You can track progress in real-time from the documents list.
