From 554c76eea221045bc3cf21b8bf7f32c766f8fffc Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sat, 15 Aug 2026 17:18:53 +0200 Subject: [PATCH 01/15] feat: drop browserless --- .../guides/backward-compatibility.mdx | 82 +----- .../ruby-gem/guides/custom-http-requests.mdx | 1 - .../guides/handling-dynamic-content.mdx | 83 +++--- .../docs/ruby-gem/reference/cli-reference.mdx | 80 ++---- .../docs/ruby-gem/reference/strategy.mdx | 243 +++++------------- .../docs/troubleshooting/troubleshooting.mdx | 30 +-- .../reference/env-variables.mdx | 3 +- src/data/docker.ts | 1 - 8 files changed, 149 insertions(+), 374 deletions(-) diff --git a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx index 9357cf48..252f1c99 100644 --- a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx +++ b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx @@ -7,6 +7,13 @@ import { Code } from "@astrojs/starlight/components"; This page outlines recent breaking changes, purged legacy aliases, and migration steps for older feed configurations. +## 0.26.0 + +Upgrade notes for configs and integrators targeting gem **0.26.0**: + +1. **Browserless strategy removal** — The `browserless` strategy, `puppeteer-ruby` dependency, interaction budgets, and `request.browserless.*` options have been completely removed. +2. **Botasaurus as browser transport** — Browser-based scraping and anti-bot bypassing now run through `botasaurus` (`BOTASAURUS_SCRAPER_URL`). Use `strategy: auto` (which tries `faraday` -> `botasaurus`) or pin `strategy: botasaurus` directly. + ## 0.25.0 Upgrade notes for configs and integrators targeting gem **0.25.0**: @@ -16,78 +23,13 @@ Upgrade notes for configs and integrators targeting gem **0.25.0**: 3. **Channel metadata in `auto`** — `auto_source` automatically extracts channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page. 4. **Actionable `NoFeedItemsExtracted` error** — `auto` mode fails loud with diagnostic surface classifications (blocked surface, app-shell, unsupported extraction surface) when zero items are found. -## 0.24.0 - -Upgrade notes for configs and integrators targeting gem **0.24.0**: - -1. **`auto` strategy chain** — Default `strategy: auto` is `faraday` → `botasaurus` only. Pin `strategy: browserless` (or `--strategy browserless`) when you need Browserless preload/interaction; Browserless is not a fallback tier. -2. **RSS enclosures** — Configure media with the `enclosure` selector. RSS does not promote `image` into ``; images stay on the description / JSON Feed `image`. -3. **Ruby dual-format / telemetry** — Prefer `Html2rss.feed_result` for one scrape that must render RSS and JSON Feed (or be Marshal-cached). Read `result.status.to_h` for scrape telemetry (`selected_strategy`, `attempt_count`, `strategy_attempts` under `auto`). - -## Removed Legacy Selector Aliases - -In previous versions, `html2rss` accepted legacy selector names with a deprecation warning. These shims have been removed: - -| Removed Name | Required Name | Description | -| :----------- | :------------- | :-------------------- | -| `link` | `url` | Item destination URL | -| `pubDate` | `published_at` | Item publication date | -| `updated` | `published_at` | Item publication date | - -### Migration Example - -Update any occurrences of `link`, `pubDate`, or `updated` in your `selectors` block to `url` or `published_at`: - - - -## Removed Channel Attributes - -`strategy` and `headers` were previously tolerated inside the `channel` block. These legacy shims have been removed. Both keys must now be placed at the top level of your feed configuration: - - - ## Migration Checklist When upgrading to modern `html2rss` releases: -1. **Rename URL selectors**: Ensure article URL selectors use `url` rather than `link`. -2. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`. -3. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file. -4. **Pin Browserless when needed**: If you relied on `auto` falling through to Browserless, set `strategy: browserless` (or `--strategy browserless`). +1. **Migrate Browserless to Botasaurus**: Update any feeds using `strategy: browserless` or `request.browserless` to `strategy: botasaurus` or `strategy: auto` with `BOTASAURUS_SCRAPER_URL`. +2. **Rename URL selectors**: Ensure article URL selectors use `url` rather than `link`. +3. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`. +4. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file. 5. **Use `enclosure` for RSS media**: Do not rely on `image` becoming an RSS ``; select podcast/media URLs with `enclosure`. -6. **Prefer `feed_result` for dual-format / cache**: Integrators that need RSS + JSON Feed from one scrape (or Marshal caching) should use `Html2rss.feed_result` and `status.to_h`. -7. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema. +6. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema. diff --git a/src/content/docs/ruby-gem/guides/custom-http-requests.mdx b/src/content/docs/ruby-gem/guides/custom-http-requests.mdx index 3c385a00..7a935b19 100644 --- a/src/content/docs/ruby-gem/guides/custom-http-requests.mdx +++ b/src/content/docs/ruby-gem/guides/custom-http-requests.mdx @@ -73,7 +73,6 @@ Request budgets are configured under `request`, not as top-level keys: - `request.max_redirects` limits redirect hops - `request.max_requests` limits the total request budget for the feed build -- `request.browserless.*` is reserved for Browserless-only behavior such as preload actions - `request.botasaurus.*` is reserved for Botasaurus-only behavior such as navigation mode and retries ## Common Use Cases diff --git a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx index fbe522f4..cb6224ab 100644 --- a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx +++ b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx @@ -1,6 +1,6 @@ --- title: Handling Dynamic Content -description: "Learn how to handle JavaScript-heavy websites and dynamic content with html2rss using browser-based extraction strategies." +description: "Learn how to handle JavaScript-heavy websites and dynamic content with html2rss using Botasaurus browser-based extraction strategies." --- import { Code } from "@astrojs/starlight/components"; @@ -9,26 +9,25 @@ Some websites load their content dynamically using JavaScript. Static fetch path ## Solution -Default `strategy: auto` already tries `faraday` then `botasaurus` (when `BOTASAURUS_SCRAPER_URL` is configured). That covers many JS-rendered listing pages without pinning a strategy. +Default `strategy: auto` automatically tries `faraday` first, then falls back to `botasaurus` when `BOTASAURUS_SCRAPER_URL` is configured. This handles many JS-rendered listing pages without needing custom configuration. -Pin [`browserless`](/ruby-gem/reference/strategy/#browserless) when you need headless Chrome with preload (wait, click, scroll) or other Browserless-only controls — Browserless is not part of the `auto` chain. - -Keep the strategy at the top level and put request-specific options under `request`: +When a site requires browser rendering or anti-bot bypass by default, you can explicitly set `strategy: botasaurus` and configure request controls under `request.botasaurus`: -## When to Use Browser-Based Extraction +## When to Use Botasaurus Extraction -A browser-based extraction strategy is necessary when: +Botasaurus extraction is ideal when: -- **Content loads after page load** - JavaScript fetches data from APIs +- **Content loads after page load** - JavaScript fetches data dynamically - **Single Page Applications (SPAs)** - React, Vue, Angular apps -- **Infinite scroll** - Content loads as you scroll -- **Dynamic forms** - Content changes based on user interaction +- **Anti-bot interstitials** - Cloudflare, Google search referrers, or challenge pages +- **Scroll-triggered content** - Infinite scroll lists (`scroll_to_bottom: true`) -## Preload Actions +## Botasaurus Request Controls -For dynamic sites, rendering once is often not enough. Use `request.browserless.preload` to wait, click, or scroll before the -HTML snapshot is taken. +Configure browser actions under `request.botasaurus`: -### Wait Before Capturing Dynamic Content +### Wait for Elements -### Click "Load More" Buttons +### Scroll for Infinite Content -### Scroll Infinite Lists +### Anti-Bot Bypass Navigation Modes -These preload steps can be combined in a single config when a site needs several interactions before all items appear. - ## Performance Considerations -Browser-based extraction is slower than default static HTTP fetching because it: +Browser-based extraction uses more resources than static HTTP fetching because it: -- Launches a headless Chrome browser -- Renders the full page with JavaScript -- Takes more memory and CPU resources +- Renders pages in an automated Chrome instance +- Executes JavaScript and handles DOM events +- Manages browser pools and network emulation -**Use static HTTP fetching for static content** and switch to browser-based extraction when needed. See the [Strategy Reference](/ruby-gem/reference/strategy/) for concrete transports, defaults, and environment requirements. +Use static HTTP fetching (`faraday`) for static content, and lean on `auto` or explicit `botasaurus` strategies when browser rendering is required. See the [Strategy Reference](/ruby-gem/reference/strategy/) for details. ## Related Topics - **[Strategy Reference](/ruby-gem/reference/strategy/)** - Complete strategy documentation - **[Troubleshooting](/troubleshooting/troubleshooting/)** - Common issues with dynamic content -- **[Advanced Features](/ruby-gem/guides/advanced-features/)** - Performance optimization tips diff --git a/src/content/docs/ruby-gem/reference/cli-reference.mdx b/src/content/docs/ruby-gem/reference/cli-reference.mdx index 0089ce33..2bcb0538 100644 --- a/src/content/docs/ruby-gem/reference/cli-reference.mdx +++ b/src/content/docs/ruby-gem/reference/cli-reference.mdx @@ -21,13 +21,12 @@ Automatically discovers items from a page and prints the generated RSS or JSONFe `botasaurus`. Pin `browserless` explicitly when you need headless Chrome (preload/interaction). +- `--strategy`: Optional request strategy (`auto`, `faraday`, `botasaurus`, `local_file`). Defaults to `auto`, which tries `faraday` -> `botasaurus`. - `--format`: Output format for the auto-sourced feed (`rss` or `jsonfeed`). Defaults to `rss`. - `--limit`: Maximum number of articles to extract during discovery (defaults to `20`). - `--items_selector`: Optional CSS selector hint for item extraction. @@ -66,9 +65,9 @@ When possible, pass a direct listing/update URL instead of a top-level homepage When no extractable items are found, `auto` classifies likely causes instead of only returning a generic message: - `blocked surface likely (anti-bot or interstitial)`: - - try a more specific public listing URL + - try a more specific public listing URL or configure `BOTASAURUS_SCRAPER_URL` - `app-shell surface detected`: - - switch to a direct listing/update URL + - switch to a direct listing/update URL or configure `BOTASAURUS_SCRAPER_URL` - `unsupported extraction surface for auto mode`: - switch to listing/changelog/category URLs - use explicit selectors in a feed config @@ -79,47 +78,13 @@ If all fallback tiers run but still extract zero items, html2rss raises: - `No RSS feed items extracted after auto fallback ...` -If failures continue after URL/surface fixes, ensure `BOTASAURUS_SCRAPER_URL` is set so the `auto` Botasaurus tier can run, or pin `--strategy browserless` when you need Browserless preload/interaction. +If failures continue after URL/surface fixes, ensure `BOTASAURUS_SCRAPER_URL` is set so the `auto` Botasaurus tier can run. Start by changing the input URL to a direct listing/update page, then move to explicit selectors if needed. -#### Browserless Setup And Diagnostics (CLI) - -`browserless` is an explicit override for CLI usage. - - - -If you see `Browserless connection failed`, check: - -- `BROWSERLESS_IO_WEBSOCKET_URL` points to a reachable Browserless endpoint -- `BROWSERLESS_IO_API_TOKEN` matches the Browserless `TOKEN` -- the Browserless service is running and reachable from your shell environment - -For custom Browserless endpoints, `BROWSERLESS_IO_API_TOKEN` is required. - #### Botasaurus Environment Requirement (CLI) -`botasaurus` is an explicit override for CLI usage and requires `BOTASAURUS_SCRAPER_URL`: +`botasaurus` is an explicit strategy option for CLI usage and requires `BOTASAURUS_SCRAPER_URL`: `botasaurus`. - **`faraday`**: Makes a direct HTTP request. It is fast but does not execute JavaScript. - **`botasaurus`**: Delegates fetching to a Botasaurus scrape API. Included in the `auto` chain; requires `BOTASAURUS_SCRAPER_URL` when that tier runs (or when you pin `strategy: botasaurus`). -- **`browserless`**: Renders the website in a headless Chrome browser. **Explicit only** — set `strategy: browserless` or `--strategy browserless` (not part of `auto`). - **`local_file`**: Reads HTML content directly from a local file on disk without making network requests. `strategy` is a top-level config key. Request-specific controls live under `request`. -`auto` falls back to the next strategy when the current attempt errors or extracts zero items. Pin a concrete strategy when you need a specific transport (for example Browserless preload/interaction, or a forced Botasaurus-only run). +`auto` falls back to `botasaurus` when `faraday` errors or extracts zero items. Pin a concrete strategy when you need a specific transport (for example, forcing `botasaurus` directly). ## `auto` (default) @@ -23,8 +22,6 @@ The default strategy chain is: `faraday` -> `botasaurus` -`browserless` is not in this chain. Pin it when you need headless Chrome with preload (wait/click/scroll) or other Browserless-only controls. - Auto fallback shares one request budget across all strategy attempts. For pagination-heavy or dynamic pages, increase `request.max_requests` (or `--max-requests`) when retries exhaust the budget. Under `auto`, `Html2rss.feed_result(...).status` exposes scrape telemetry: `selected_strategy`, `attempt_count`, and `strategy_attempts` (see [Managing Feed Configs](/ruby-gem/guides/managing-feed-configs/#ruby-api-feedresult)). Auto fallback decisions are also visible at `LOG_LEVEL=info` (hidden at the default `LOG_LEVEL=warn`). @@ -33,66 +30,93 @@ Under `auto`, `Html2rss.feed_result(...).status` exposes scrape telemetry: `sele `faraday` is the standard static HTTP transport. When no custom `User-Agent` header is supplied in `headers`, `html2rss` sends `html2rss/` by default. -## `local_file` +## `botasaurus` -`local_file` parses content directly from a local file on disk. It is designed for offline testing, local fixtures, and CLI `--input` workflows. +`botasaurus` delegates page fetching to a Botasaurus scrape API endpoint. It runs as the second tier of `auto`, or when you pin `strategy: botasaurus`. -Because no remote network connection is established, `local_file` skips remote network guards and SSRF preflight checks. +Requirements: + +- `BOTASAURUS_SCRAPER_URL` set to your Botasaurus scrape API base URL (for example `http://localhost:4010`) +- pin `strategy: botasaurus` when you want to skip Faraday and force this transport directly + +html2rss enforces local request policy preflight and timeout budget. When a total request timeout remains, Botasaurus `max_retries` and `wait_timeout_seconds` are clamped so upstream work fits the remaining budget. ### Configuration -## `browserless` +Supported `request.botasaurus` options: -To use the `browserless` strategy, you need a running instance of [Browserless.io](https://www.browserless.io/). +- `navigation_mode` (`auto`, `get`, `google_get`, `google_get_bypass`; default `auto`) +- `max_retries` (`0..3`; default `1`) +- `wait_for_selector` (string) +- `wait_timeout_seconds` (integer) +- `block_images` (boolean) +- `block_images_and_css` (boolean) +- `wait_for_complete_page_load` (boolean) +- `headless` (boolean, default `false`) +- `proxy` (string) +- `user_agent` (string) +- `window_size` (two-item integer array, for example `[1920, 1080]`) +- `lang` (string, for example `en-US`) -### Docker +### Request Payload Shape -You can run a local Browserless.io instance using Docker: +Example scrape-API payload shape: + +### Command-Line Usage + + -### Configuration +## `local_file` -Set the `strategy` at the top level of your feed configuration and put request controls under `request`: +`local_file` parses content directly from a local file on disk. It is designed for offline testing, local fixtures, and CLI `--input` workflows. + +Because no remote network connection is established, `local_file` skips remote network guards and SSRF preflight checks. + +### Configuration -### Request Structure +## Request Structure Use this split consistently: -- `strategy`: selects `auto`, `faraday`, `browserless`, `botasaurus`, or `local_file` +- `strategy`: selects `auto`, `faraday`, `botasaurus`, or `local_file` - `headers`: top-level headers shared by remote strategies (defaults to `html2rss/` User-Agent) - `request.max_redirects`: redirect limit for the request session (default: `5`) - `request.max_requests`: total request budget for the whole feed build (default: `10`) - `request.total_timeout_seconds`: maximum total wall-clock budget for the entire feed build - `request.local_file_path`: file path for `local_file` strategy -- `request.browserless.*`: Browserless-only options - `request.botasaurus.*`: Botasaurus-only options Example: -### Browserless Preload - -Browserless can interact with the page before html2rss captures the final HTML. Configure preload steps under -`request.browserless.preload`. - - - -- `wait_after_ms`: inserts a fixed wait before or after preload steps -- `click_selectors`: clicks matching elements until they disappear or `max_clicks` is reached -- `scroll_down`: scrolls until the page height stops growing or `iterations` is reached - -If preload triggers a real navigation or redirect, html2rss keeps the final document metadata. Relative links and follow-up -pagination therefore resolve against the page that was actually rendered after preload completed. - -### Command-Line Usage - -You can also specify the strategy on the command line: - - - -### Browserless Troubleshooting - -If Browserless cannot connect, html2rss surfaces a `Browserless connection failed (...)` error with endpoint/token hints. - -Check these first: - -- `BROWSERLESS_IO_WEBSOCKET_URL` is reachable from where html2rss runs -- `BROWSERLESS_IO_API_TOKEN` matches your Browserless `TOKEN` -- your Browserless service is running and accepting connections - -For custom Browserless websocket endpoints, `BROWSERLESS_IO_API_TOKEN` is mandatory. The local default endpoint (`ws://127.0.0.1:3000`) can use the default local token `6R0W53R135510`. - -## `botasaurus` - -`botasaurus` delegates page fetching to a Botasaurus scrape API endpoint. It runs as the second tier of `auto`, or when you pin `strategy: botasaurus`. - -Requirements: - -- `BOTASAURUS_SCRAPER_URL` set to your Botasaurus scrape API base URL (for example `http://localhost:4010`) -- pin `strategy: botasaurus` only when you want to skip Faraday and force this transport - -html2rss still enforces local request policy preflight and timeout budget. When a total request timeout remains, Botasaurus `max_retries` and `wait_timeout_seconds` are clamped so upstream work fits the remaining budget. Botasaurus handles browser navigation/rendering internals, so some policy details are delegated to upstream execution. - -### Configuration - - - -Supported `request.botasaurus` options: - -- `navigation_mode` (`auto`, `get`, `google_get`, `google_get_bypass`; default `auto`) -- `max_retries` (`0..3`; default `1`) -- `wait_for_selector` (string) -- `wait_timeout_seconds` (integer) -- `block_images` (boolean) -- `block_images_and_css` (boolean) -- `wait_for_complete_page_load` (boolean) -- `headless` (boolean, default `false`) -- `proxy` (string) -- `user_agent` (string) -- `window_size` (two-item integer array, for example `[1920, 1080]`) -- `lang` (string, for example `en-US`) - -### Request payload shape - -Example scrape-API payload shape: - - - -### Command-Line Usage - - - --- For detailed documentation on the Ruby API, see the [official YARD documentation](https://www.rubydoc.info/gems/html2rss). diff --git a/src/content/docs/troubleshooting/troubleshooting.mdx b/src/content/docs/troubleshooting/troubleshooting.mdx index 03b2034b..e2757f20 100644 --- a/src/content/docs/troubleshooting/troubleshooting.mdx +++ b/src/content/docs/troubleshooting/troubleshooting.mdx @@ -58,42 +58,35 @@ Known anti-bot interstitial patterns (for example Cloudflare challenge pages) ar When all auto fallback tiers complete but still extract zero items, html2rss raises `No RSS feed items extracted after auto fallback ...`. -If failures continue after URL/surface fixes, retry with an explicit browser-based override (`--strategy browserless`), or `--strategy botasaurus` when `BOTASAURUS_SCRAPER_URL` is configured. +If failures continue after URL/surface fixes, configure `BOTASAURUS_SCRAPER_URL` or use `--strategy botasaurus` for browser rendering. -### Browserless Connection / Setup Failures +### Botasaurus Setup & Diagnostics -If you receive `Browserless connection failed (...)`: +If you receive a Botasaurus configuration error or connection failure: -1. Confirm Browserless is running and reachable from the machine running `html2rss`. -2. Confirm `BROWSERLESS_IO_WEBSOCKET_URL` points at that running service. -3. Confirm `BROWSERLESS_IO_API_TOKEN` matches the Browserless `TOKEN`. +1. Confirm the Botasaurus scrape API service is running and reachable from the machine running `html2rss`. +2. Confirm `BOTASAURUS_SCRAPER_URL` points at your Botasaurus instance (for example `http://localhost:4010`). -Example local startup: +Example local startup via Docker: - + Then run with: -For custom websocket endpoints, `BROWSERLESS_IO_API_TOKEN` is required. - ### Configuration Errors Common configuration-related errors: - **`UnsupportedResponseContentType`:** The website returned content that html2rss can't parse (not HTML or JSON). -- **`UnsupportedStrategy`:** The specified strategy is not available. Use `auto`, `faraday`, `browserless`, or `botasaurus`. +- **`UnsupportedStrategy`:** The specified strategy is not available. Use `auto`, `faraday`, `botasaurus`, or `local_file`. - **`BOTASAURUS_SCRAPER_URL is required for strategy=botasaurus.`:** Set `BOTASAURUS_SCRAPER_URL` to your Botasaurus scrape API base URL when using `--strategy botasaurus`. - **`BOTASAURUS_SCRAPER_URL is invalid`:** Fix the URL format and retry. - **`Configuration must include at least 'selectors' or 'auto_source'`:** You need to specify either manual selectors or enable auto-source. @@ -105,7 +98,7 @@ If parts of your items (e.g., title, link) are missing, check the following: - **Selector:** Ensure the selector for the missing part is correct and relative to the `items.selector`. - **Extractor:** Verify that you are using the correct `extractor` (e.g., `text`, `href`, `attribute`). -- **Dynamic Content:** `faraday` does not render JavaScript. If content loads dynamically, run with `--strategy browserless` (with Browserless available) or `--strategy botasaurus` (with `BOTASAURUS_SCRAPER_URL` configured) so the page can be rendered before extraction. +- **Dynamic Content:** `faraday` does not render JavaScript. If content loads dynamically, configure `BOTASAURUS_SCRAPER_URL` or run with `--strategy botasaurus` so the page can be rendered before extraction. ### Date/Time Parsing Errors @@ -159,7 +152,6 @@ If you are getting a "command not found" error, try the following: - Try a more specific listing, newsroom, changelog, or updates URL before changing infrastructure - If your first-run stack includes Botasaurus, ensure the `botasaurus` service is running -- Add Browserless later only when harder sites prove they need it - Check the feed configuration in `feeds.yml` for typos or invalid selectors - Look for parsing errors in the logs: diff --git a/src/content/docs/web-application/reference/env-variables.mdx b/src/content/docs/web-application/reference/env-variables.mdx index eb01a0aa..56db4c43 100644 --- a/src/content/docs/web-application/reference/env-variables.mdx +++ b/src/content/docs/web-application/reference/env-variables.mdx @@ -13,8 +13,7 @@ description: "Configuration reference for html2rss-web environment variables." | `BUILD_TAG` | release metadata used in logs; published Docker images set this to the release version | | `GIT_SHA` | deployed commit metadata used in logs; published Docker images set this to the released commit | | `SENTRY_DSN` | optional; enables Sentry errors/logs when set | -| `BROWSERLESS_IO_WEBSOCKET_URL` | optional; Browserless websocket endpoint for harder sites when you add Browserless later | -| `BROWSERLESS_IO_API_TOKEN` | optional unless you explicitly add Browserless to your stack or point at a custom Browserless endpoint | +| `BOTASAURUS_SCRAPER_URL` | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass | | `AUTO_SOURCE_ENABLED` | `true` by default in development/test, `false` otherwise | | `ASYNC_FEED_REFRESH_ENABLED` | optional boolean; default `false` | | `ASYNC_FEED_REFRESH_STALE_FACTOR` | optional integer `>= 1`; default `3` | diff --git a/src/data/docker.ts b/src/data/docker.ts index 014f2d69..75f87c14 100644 --- a/src/data/docker.ts +++ b/src/data/docker.ts @@ -1,7 +1,6 @@ export const dockerHubRepository = 'html2rss/web'; export const dockerHubUrl = `https://hub.docker.com/r/${dockerHubRepository}`; export const webImage = `${dockerHubRepository}:1`; -export const browserlessImage = 'ghcr.io/browserless/chromium'; export const caddyImage = 'caddy:2-alpine'; export const watchtowerImage = 'containrrr/watchtower'; export const botasaurusImage = 'html2rss/botasaurus-scrape-api:latest'; From 21e113c7e1c47539aa5cb4c5ef179a18b24e1bbf Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 16 Aug 2026 01:50:34 +0200 Subject: [PATCH 02/15] docs(adapters): document new botasaurus options and header/cookie forwarding Add execution_mode, organic_get, scroll, scroll_to_bottom, block_trackers, headers, and cookies to the botasaurus options table in strategy reference. Update the custom HTTP requests guide to mention header/cookie forwarding. These options were added in commits 9ca09a0 and a41ab3f of the core repo. --- src/content/docs/ruby-gem/guides/custom-http-requests.mdx | 2 +- src/content/docs/ruby-gem/reference/strategy.mdx | 8 +++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/src/content/docs/ruby-gem/guides/custom-http-requests.mdx b/src/content/docs/ruby-gem/guides/custom-http-requests.mdx index 7a935b19..e9302d8b 100644 --- a/src/content/docs/ruby-gem/guides/custom-http-requests.mdx +++ b/src/content/docs/ruby-gem/guides/custom-http-requests.mdx @@ -73,7 +73,7 @@ Request budgets are configured under `request`, not as top-level keys: - `request.max_redirects` limits redirect hops - `request.max_requests` limits the total request budget for the feed build -- `request.botasaurus.*` is reserved for Botasaurus-only behavior such as navigation mode and retries +- `request.botasaurus.*` is reserved for Botasaurus-only behavior such as navigation mode, retries, header/cookie forwarding, and scroll options ## Common Use Cases diff --git a/src/content/docs/ruby-gem/reference/strategy.mdx b/src/content/docs/ruby-gem/reference/strategy.mdx index d728d39e..6cee536b 100644 --- a/src/content/docs/ruby-gem/reference/strategy.mdx +++ b/src/content/docs/ruby-gem/reference/strategy.mdx @@ -60,18 +60,24 @@ html2rss enforces local request policy preflight and timeout budget. When a tota Supported `request.botasaurus` options: -- `navigation_mode` (`auto`, `get`, `google_get`, `google_get_bypass`; default `auto`) +- `execution_mode` (`auto`, `request`, `browser`; default `auto`) +- `navigation_mode` (`auto`, `get`, `google_get`, `google_get_bypass`, `organic_get`; default `auto`) - `max_retries` (`0..3`; default `1`) - `wait_for_selector` (string) - `wait_timeout_seconds` (integer) +- `scroll` (boolean) +- `scroll_to_bottom` (boolean) - `block_images` (boolean) - `block_images_and_css` (boolean) +- `block_trackers` (boolean) - `wait_for_complete_page_load` (boolean) - `headless` (boolean, default `false`) - `proxy` (string) - `user_agent` (string) - `window_size` (two-item integer array, for example `[1920, 1080]`) - `lang` (string, for example `en-US`) +- `headers` (hash, forwarded to the Botasaurus scrape API alongside config-level `headers`) +- `cookies` (hash) ### Request Payload Shape From 41313890c6c1fd3850709c6bfa98bf9ba71396e4 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 16 Aug 2026 12:26:57 +0200 Subject: [PATCH 03/15] docs(adapters): document capture and mcp in cli and mcp server references --- src/content/docs/ruby-gem/index.mdx | 4 +- .../docs/ruby-gem/reference/cli-reference.mdx | 47 ++++++ .../docs/ruby-gem/reference/mcp-server.mdx | 139 ++++++++++++++++++ 3 files changed, 188 insertions(+), 2 deletions(-) create mode 100644 src/content/docs/ruby-gem/reference/mcp-server.mdx diff --git a/src/content/docs/ruby-gem/index.mdx b/src/content/docs/ruby-gem/index.mdx index 4fe10115..40a2f2a6 100644 --- a/src/content/docs/ruby-gem/index.mdx +++ b/src/content/docs/ruby-gem/index.mdx @@ -23,5 +23,5 @@ If you are getting started with `html2rss`, we recommend starting with the [**fi ## Documentation Sections - **[Tutorials](/ruby-gem/tutorials/your-first-feed/)**: Step-by-step guides to help you get started with `html2rss`. -- **[Guides](/ruby-gem/guides/)**: Practical examples and solutions for common tasks. -- **[Reference](/ruby-gem/reference/)**: Detailed information on configuration options. +- **[Guides](/ruby-gem/guides/)**: Practical examples and solutions for common tasks, including [AI agent workflows](/ruby-gem/guides/ai-agent-workflows/). +- **[Reference](/ruby-gem/reference/)**: Detailed information on configuration options, CLI commands, and the [MCP server](/ruby-gem/reference/mcp-server/). diff --git a/src/content/docs/ruby-gem/reference/cli-reference.mdx b/src/content/docs/ruby-gem/reference/cli-reference.mdx index 2bcb0538..d4e8355d 100644 --- a/src/content/docs/ruby-gem/reference/cli-reference.mdx +++ b/src/content/docs/ruby-gem/reference/cli-reference.mdx @@ -129,6 +129,53 @@ Available options: The CLI keeps `strategy` as a top-level override and writes runtime request limits into the generated config under `request`. +### Capture + +Analyzes a URL using auto-source structural analysis and prints a reusable YAML feed configuration with derived CSS selectors to stdout. + + + +Command: `html2rss capture [URL]` + +Available options: + +- `--strategy`: Optional request strategy (`auto`, `faraday`, `botasaurus`, `local_file`). Defaults to `auto`. +- `--items_selector`: Optional CSS selector hint for item extraction. +- `--limit`: Maximum number of articles to keep (defaults to `25`). +- `--max-redirects`: Maximum redirects to follow per request. +- `--max-requests`: Maximum requests to allow for this feed build. +- `--input`: Local HTML file path to read input from without making network requests. + +### MCP + +Starts the Model Context Protocol (MCP) server for integration with AI clients and agent workflows (Cursor, Claude Desktop, Copilot). + + + +Command: `html2rss mcp` + +Available options: + +- `--transport`: MCP transport protocol (`stdio` or `http`). Defaults to `stdio`. +- `--port`: Port for HTTP transport (default: `8080`). Bound strictly to `127.0.0.1` (loopback only). + +HTTP transport requires the `rack`, `rackup`, and `webrick` gems. See the [MCP Server Reference](/ruby-gem/reference/mcp-server/) and [AI Agent Workflows Guide](/ruby-gem/guides/ai-agent-workflows/) for full configuration and tool schemas. + ### Schema Prints or writes the exported JSON Schema for the current gem version. diff --git a/src/content/docs/ruby-gem/reference/mcp-server.mdx b/src/content/docs/ruby-gem/reference/mcp-server.mdx new file mode 100644 index 00000000..43c64a1c --- /dev/null +++ b/src/content/docs/ruby-gem/reference/mcp-server.mdx @@ -0,0 +1,139 @@ +--- +title: MCP Server Reference +description: Complete protocol reference for the html2rss Model Context Protocol (MCP) server, tools, resources, prompts, and transports. +--- + +import { Code } from "@astrojs/starlight/components"; + +`html2rss` includes a built-in [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server that exposes gem capabilities directly to AI agents, clients, and IDEs (such as Cursor, Claude Desktop, and GitHub Copilot). + +## Overview + +The MCP server allows AI agents to inspect pages, capture reusable feed configurations, validate configs, and generate feeds without executing ad-hoc shell scripts or parsing CLI output. + + + +## Transports & Security + +The MCP server supports two transport mechanisms: + +| Transport | Command | Default Port | Description | +| :---------- | :------------------------------------------ | :----------- | :------------------------------------------------------------------------------ | +| **`stdio`** | `html2rss mcp --transport stdio` | N/A | Standard input/output communication. Recommended for Cursor and Claude Desktop. | +| **`http`** | `html2rss mcp --transport http --port 8080` | `8080` | Streamable HTTP transport bound strictly to `127.0.0.1` (loopback only). | + +### Security & Dependencies for HTTP + +- **Loopback binding:** The HTTP transport only binds to `127.0.0.1`. Never expose it to public interfaces without authentication and reverse proxy origin controls. +- **Dependencies:** The HTTP transport requires the `rack`, `rackup`, and `webrick` gems. If they are not present, starting the server raises a descriptive `LoadError`. + +## Strategy Resolution in MCP + +Inside the MCP server, passing `strategy: "auto"` (or omitting strategy) resolves directly to `faraday` without running the multitenant Botasaurus fallback chain. + +If a scraped target is JavaScript-rendered, Cloudflare-protected, or returns an empty article list under Faraday: + +1. Use `inspect_url` to confirm page structure and scraper eligibility. +2. Retry `scrape_url` or `capture_config` with `strategy: "botasaurus"`. +3. Ensure the Botasaurus service is running and `BOTASAURUS_SCRAPER_URL` is set in the environment (e.g. `http://127.0.0.1:4010`). + +## Tools Reference + +The server registers 5 primary tools for AI agent orchestration. + +### `scrape_url` + +One-shot article extraction as JSON Feed items. Use when you need articles immediately without authoring or saving a YAML feed config. + +- **Parameters:** + - `url` _(string, required)_: Source page URL to scrape. + - `strategy` _(string, optional)_: Request strategy (`auto`, `faraday`, `botasaurus`). Default: `auto`. + - `limit` _(integer, optional)_: Maximum articles to keep. Default: `25`. + - `items_selector` _(string, optional)_: CSS selector hint for items. +- **Return value:** JSON string containing an array of article objects (`title`, `url`, `description`, `published_at`, `author`, `image`, `categories`). +- **Metadata (`_meta`):** Includes `total`, `strategy`, and `channel_title`. +- **Error handling:** Unhandled exceptions return `isError: true` with error message text. + +### `inspect_url` + +Diagnostic page analysis. Inspects HTTP response headers, content type, scraper eligibility, Semantic Structural Tree (SST) node count, and discovered article segments. + +- **Parameters:** + - `url` _(string, required)_: Source page URL. + - `strategy` _(string, optional)_: Request strategy (`auto`, `faraday`, `botasaurus`). Default: `auto`. +- **Return value:** Pretty-printed JSON object with diagnostic details: + - `content_type`: Detected MIME type. + - `html_response`: Boolean indicating whether the payload was parsed as HTML. + - `scraper_eligibility`: Eligible scrapers (e.g. `wordpress_api`, `schema`, `microdata`, `semantic_html`). + - `sst_stats`: Node count and degradation status. + - `sst.segment_stats`: Number of segments discovered, strategies matched, and sample CSS tag paths. + +### `capture_config` + +Analyzes a target URL and derives a reusable `html2rss` feed configuration hash with calculated CSS selectors. + +- **Parameters:** + - `url` _(string, required)_: Source page URL. + - `strategy` _(string, optional)_: Request strategy (`auto`, `faraday`, `botasaurus`). Default: `auto`. + - `items_selector` _(string, optional)_: Optional CSS selector hint for items. +- **Return value:** Pretty-printed JSON configuration hash containing `:channel` and `:selectors`. +- **Metadata (`_meta`):** Includes `articles_count`, `channel_title`, `has_selectors` (boolean), and `strategy`. + +### `validate_config` + +Validates a feed configuration hash against the official `html2rss` JSON schema and runtime constraints. + +- **Parameters:** + - `config` _(object, required)_: Feed configuration object with `channel` and `selectors`. +- **Return value:** Text response `Config is valid.` on success. +- **Error handling:** Returns `isError: true` with a formatted error dictionary if schema validation fails. + +### `apply_config` + +Executes a validated feed configuration against a page and returns valid RSS 2.0 XML. + +- **Parameters:** + - `url` _(string, required)_: Source page URL (populates `channel.url` if omitted from config). + - `config` _(object, required)_: Feed configuration object. +- **Return value:** RSS 2.0 XML string. + +## Resources Reference + +The MCP server exposes reference data under the `html2rss://` URI scheme: + +| URI | MIME Type | Description | +| :-------------------------- | :----------------- | :----------------------------------------------------------------------------------------------------------------------- | +| **`html2rss://schema`** | `application/json` | The complete JSON Schema for `html2rss` feed configurations, including selector rules, extractors, and request controls. | +| **`html2rss://extractors`** | `application/json` | Alphabetical list of all registered extractor names (`attribute`, `html`, `href`, `text`, `static`, etc.). | +| **`html2rss://strategies`** | `application/json` | List of all registered request strategies (`auto`, `faraday`, `botasaurus`, `local_file`). | + +## Prompts Reference + +The server publishes guided prompt workflows for AI assistants: + +### `scrape-webpage` + +Guided one-shot extraction prompt that instructs the agent to run `scrape_url`, inspect the response if empty or JS-gated, and retry with `botasaurus` when needed. + +- **Arguments:** + - `url` _(string, required)_: Target URL to scrape. + +### `capture-feed-config` + +Four-step workflow prompt for generating durable feed configurations: + +1. Call `capture_config` to derive initial selectors and check `articles_count`. +2. If selectors are weak, call `inspect_url` and retry with `botasaurus`. +3. Call `validate_config` to ensure schema conformance. +4. Call `apply_config` to verify the resulting RSS feed XML. + +- **Arguments:** + - `url` _(string, required)_: Target URL to analyze. From cc1f3b2e2e2214ebf117eddffd9fd0216e506eac Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 16 Aug 2026 12:27:03 +0200 Subject: [PATCH 04/15] docs(use-cases): add guide for ai agent workflows with mcp --- .../ruby-gem/guides/ai-agent-workflows.mdx | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx diff --git a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx new file mode 100644 index 00000000..df090b5b --- /dev/null +++ b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx @@ -0,0 +1,103 @@ +--- +title: AI Agent Workflows with MCP +description: Guide to integrating html2rss with AI agents, Claude Desktop, and Cursor using the Model Context Protocol (MCP). +--- + +import { Code } from "@astrojs/starlight/components"; + +`html2rss` includes a native [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server. This allows AI assistants in tools like Cursor, Claude Desktop, and GitHub Copilot to scrape pages, capture feed configurations, and generate RSS feeds autonomously. + +## Why Use MCP? + +Without MCP, agents must invoke shell commands, manage output buffers, and parse unstructured text. With MCP: + +- The agent automatically discovers available tools, arguments, and return types. +- Responses are structured (JSON Feed objects, configuration schemas, and RSS XML). +- Agents can inspect, capture, validate, and verify feeds iteratively in a closed feedback loop. + +## Client Setup + +### 1. Version Manager Shims (mise, asdf, rbenv, chruby) + +Because version managers manage Ruby runtimes and gem paths through environment shims, configure your MCP client to invoke the version manager executable rather than a bare `html2rss` command. + + + +### 2. Cursor Configuration + +Add `html2rss` to your Cursor MCP settings (`~/.cursor/mcp.json` or `.cursor/mcp.json`): + + + +_(If using `asdf`, replace `"command": "mise"` with `"asdf"` and `"args": ["exec", "html2rss", "mcp"]`.)_ + +### 3. Claude Desktop Configuration + +Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS or `%APPDATA%\\Claude\\claude_desktop_config.json` on Windows): + + + +## Autonomous Workflow Patterns + +### Pattern A: One-Shot Content Scraping + +When an agent needs articles immediately without saving a feed configuration: + +1. The agent calls `scrape_url` with the target URL. +2. `html2rss` runs auto-source extraction (Schema.org, JSON state, semantic HTML) and returns a JSON Feed items array. +3. If the page is protected or rendered with JavaScript, the agent calls `inspect_url` to diagnose the structure, then retries `scrape_url` with `strategy: "botasaurus"`. + +### Pattern B: Iterative Feed Config Authoring + +When an agent is tasked with creating a durable YAML feed configuration: + +1. **Inspect:** The agent calls `inspect_url` to check content type, SST node counts, and eligible scrapers. +2. **Capture:** The agent runs `capture_config` to derive CSS selectors for items, title, link, and description. +3. **Refine:** The agent reviews the derived selectors or asks the human user for domain-specific adjustments. +4. **Validate:** The agent passes the configuration to `validate_config` to verify schema conformance. +5. **Apply:** The agent tests the final configuration with `apply_config` to produce and inspect live RSS XML. + +## JavaScript-Rendered Sites (Botasaurus) + +For dynamic JavaScript single-page applications or sites protected by anti-bot measures, launch the Botasaurus scrape service: + + + +Ensure `BOTASAURUS_SCRAPER_URL` is accessible (typically `http://127.0.0.1:4010`) in the environment where the MCP server runs. Agents can then pass `strategy: "botasaurus"` to `scrape_url`, `inspect_url`, and `capture_config`. From a76f9d70795b408134a26d6f592b09ce16136857 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 16 Aug 2026 12:50:03 +0200 Subject: [PATCH 05/15] docs(reference): reinforce schema resource links and json error details in mcp docs --- src/content/docs/ruby-gem/reference/mcp-server.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/content/docs/ruby-gem/reference/mcp-server.mdx b/src/content/docs/ruby-gem/reference/mcp-server.mdx index 43c64a1c..0da546c3 100644 --- a/src/content/docs/ruby-gem/reference/mcp-server.mdx +++ b/src/content/docs/ruby-gem/reference/mcp-server.mdx @@ -78,7 +78,7 @@ Diagnostic page analysis. Inspects HTTP response headers, content type, scraper ### `capture_config` -Analyzes a target URL and derives a reusable `html2rss` feed configuration hash with calculated CSS selectors. +Analyzes a target URL and derives a reusable `html2rss` feed configuration hash with calculated CSS selectors. Full schema options live in resource `html2rss://schema`. - **Parameters:** - `url` _(string, required)_: Source page URL. @@ -89,12 +89,12 @@ Analyzes a target URL and derives a reusable `html2rss` feed configuration hash ### `validate_config` -Validates a feed configuration hash against the official `html2rss` JSON schema and runtime constraints. +Validates a feed configuration hash against the official `html2rss` JSON schema (`html2rss://schema`) and runtime constraints. - **Parameters:** - `config` _(object, required)_: Feed configuration object with `channel` and `selectors`. - **Return value:** Text response `Config is valid.` on success. -- **Error handling:** Returns `isError: true` with a formatted error dictionary if schema validation fails. +- **Error handling:** Returns `isError: true` with a serialized JSON error dictionary if schema validation fails. ### `apply_config` From e9be3e2fa394e19116e270b7ba030d00477581ba Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 16 Aug 2026 16:17:23 +0200 Subject: [PATCH 06/15] docs(use-cases): add capture guide and XHR dynamic-content path Document html2rss capture as a draft-config workflow and note that Botasaurus browser-tier XHR JSON feeds AutoSource xhr_articles. --- .../guides/capturing-feed-configs.mdx | 118 ++++++++++++++++++ .../guides/handling-dynamic-content.mdx | 4 + src/content/docs/ruby-gem/index.mdx | 2 +- 3 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx diff --git a/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx new file mode 100644 index 00000000..0701eeff --- /dev/null +++ b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx @@ -0,0 +1,118 @@ +--- +title: Capturing Feed Configs +description: Derive a reusable YAML feed config from a URL with html2rss capture, then validate and refine selectors. +--- + +import { Aside, Code } from "@astrojs/starlight/components"; + +`html2rss capture` analyzes a page through the auto-source pipeline and prints a reusable feed config with derived CSS selectors. Use it when you want a first draft faster than hand-writing selectors from scratch. + + + +## When to Capture + +Use capture when: + +- you are writing a new custom feed and want a starting YAML +- auto-source finds articles, but you want durable selectors for review +- an AI agent or MCP client should produce a config before `validate` / `apply` + +Prefer [automatic feed generation](/web-application/guides/use-automatic-feed-generation/) when you only need items now and do not need a saved config. + +## CLI + +Print YAML to stdout: + + my-feed.yml +`} + lang="bash" +/> + +Common options: + +- `--strategy` — `auto`, `faraday`, `botasaurus`, or `local_file` (default `auto`) +- `--items_selector` — CSS selector hint for item cards +- `--limit` — maximum articles kept while deriving selectors +- `--max-redirects` / `--max-requests` — request budget overrides +- `--input` — local HTML file (sets `local_file` strategy) + +See the [CLI reference](/ruby-gem/reference/cli-reference/#capture) for the full flag list. + +## Ruby API + + + +## How It Works + +1. **Request** — fetches the page with the chosen strategy +2. **Discover** — runs AutoSource to extract articles +3. **Analyze** — normalizes the page into an SST document and maps segment positions back to articles +4. **Derive** — builds CSS selectors from SST tag paths for items, title, link, and description +5. **Assemble** — returns a config hash ready for YAML or `Html2rss.feed` + +Capture segment discovery currently uses the list Segmenter strategy only (not AutoSource cluster/semantic heuristics). When the draft is weak, pass `--items_selector` or refine selectors by hand. + +## What Capture Derives + +Capture focuses on: + +- `channel.url` (and related channel defaults) +- `selectors.items` +- `selectors.title` +- `selectors.link` (derived href selector) +- `selectors.description` when a distinct description root exists + +Rename `selectors.link` to `selectors.url` before shipping. Item URLs must use the canonical `url` selector key for RSS `` output. + +It does not invent author, `published_at`, categories, or enclosure selectors. Add those manually when the page exposes them reliably. + +Description is omitted when it would resolve to the invalid CSS selector `.` (item root equals description root). + +## Recommended Follow-Up + +1. Rename `selectors.link` → `selectors.url` if present +2. Validate: `html2rss validate my-feed.yml` +3. Render: `html2rss feed my-feed.yml` +4. Tighten selectors, strategy, or `request.botasaurus` options if needed +5. For Feed Directory contributions, add `directory.topics` (see [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config)) + +## Related + +- [CLI reference — Capture](/ruby-gem/reference/cli-reference/#capture) +- [AI agent workflows (MCP `capture_config`)](/ruby-gem/guides/ai-agent-workflows/) +- [Creating custom feeds](/creating-custom-feeds/) +- [Auto Source](/ruby-gem/reference/auto-source/) diff --git a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx index cb6224ab..dd206831 100644 --- a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx +++ b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx @@ -88,6 +88,10 @@ Configure browser actions under `request.botasaurus`: lang="yaml" /> +### JSON Loaded Over XHR + +When Botasaurus uses the browser tier, captured JSON XHR/fetch bodies feed AutoSource `xhr_articles` automatically (enabled by default). Prefer `strategy: botasaurus` (or `auto` with `BOTASAURUS_SCRAPER_URL`) for SPA listing pages that hydrate article lists over the network rather than embedding them in HTML. See [Auto Source](/ruby-gem/reference/auto-source/) and [Strategy](/ruby-gem/reference/strategy/#botasaurus). + ## Performance Considerations Browser-based extraction uses more resources than static HTTP fetching because it: diff --git a/src/content/docs/ruby-gem/index.mdx b/src/content/docs/ruby-gem/index.mdx index 40a2f2a6..91ce54fb 100644 --- a/src/content/docs/ruby-gem/index.mdx +++ b/src/content/docs/ruby-gem/index.mdx @@ -23,5 +23,5 @@ If you are getting started with `html2rss`, we recommend starting with the [**fi ## Documentation Sections - **[Tutorials](/ruby-gem/tutorials/your-first-feed/)**: Step-by-step guides to help you get started with `html2rss`. -- **[Guides](/ruby-gem/guides/)**: Practical examples and solutions for common tasks, including [AI agent workflows](/ruby-gem/guides/ai-agent-workflows/). +- **[Guides](/ruby-gem/guides/)**: Practical examples and solutions for common tasks, including [capturing feed configs](/ruby-gem/guides/capturing-feed-configs/) and [AI agent workflows](/ruby-gem/guides/ai-agent-workflows/). - **[Reference](/ruby-gem/reference/)**: Detailed information on configuration options, CLI commands, and the [MCP server](/ruby-gem/reference/mcp-server/). From 87dba698360af638a7641925ba06ebabb8afd1bf Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 16 Aug 2026 16:17:23 +0200 Subject: [PATCH 07/15] docs(domain): document xhr_articles auto_source scraper Describe always-on Botasaurus XHR captures as a first-class AutoSource strategy alongside json_state, with enable toggle and limitations. --- .../docs/ruby-gem/reference/auto-source.mdx | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/src/content/docs/ruby-gem/reference/auto-source.mdx b/src/content/docs/ruby-gem/reference/auto-source.mdx index 605804a3..ce9c9bc6 100644 --- a/src/content/docs/ruby-gem/reference/auto-source.mdx +++ b/src/content/docs/ruby-gem/reference/auto-source.mdx @@ -28,18 +28,20 @@ To enable it, add `auto_source: {}` to your configuration: 4. **`schema`:** Parses `