feat: gem v0.26.0 updates (drop browserless, capture, mcp, botasaurus improvements) - #1232
Merged
Conversation
gildesmarais
force-pushed
the
feat/drop-browserless
branch
from
August 15, 2026 15:22
2afb822 to
267266d
Compare
gildesmarais
changed the title
…warding Add execution_mode, organic_get, scroll, scroll_to_bottom, block_trackers, headers, and cookies to the botasaurus options table in strategy reference. Update the custom HTTP requests guide to mention header/cookie forwarding. These options were added in commits 9ca09a0 and a41ab3f of the core repo.
Document html2rss capture as a draft-config workflow and note that Botasaurus browser-tier XHR JSON feeds AutoSource xhr_articles.
Describe always-on Botasaurus XHR captures as a first-class AutoSource strategy alongside json_state, with enable toggle and limitations.
Document the controlled topics vocabulary, capture-first authoring workflow, and topic/language filters on the Feed Directory page.
Explain xhr_responses forwarding into captured_responses and add DataDome blocked-surface recovery steps in troubleshooting.
gildesmarais
force-pushed
the
feat/drop-browserless
branch
from
August 16, 2026 14:18
ce1515d to
d31f69c
Compare
gildesmarais
marked this pull request as ready for review
August 16, 2026 15:34
gildesmarais
changed the title
gildesmarais
commented
Aug 16, 2026
gildesmarais
commented
Aug 16, 2026
gildesmarais
commented
Aug 16, 2026
Co-authored-by: Gil Desmarais <gildesmarais@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request updates documentation and configuration to reflect the migration from the deprecated Browserless strategy to Botasaurus for browser-based scraping, improves guidance for creating and sharing custom feed configs, and introduces new documentation for advanced workflows and config capture. It also updates dependencies and simplifies Docker Compose usage by removing Watchtower snippets.
Migration to Botasaurus and Strategy Updates:
browserlessstrategy tobotasaurusfor browser-based scraping, including configuration examples, migration instructions, and option keys. This clarifies that Botasaurus is now the standard for JavaScript-rendered and anti-bot sites, and removes Browserless-specific documentation. [1] [2] [3] [4] [5] [6]Feed Config Authoring and Contribution Guidance:
html2rss capture, and clarified when and how to escalate to Botasaurus.lib/html2rss/configs/domain.com/name.ymland must includedirectory.topicsmetadata from a controlled vocabulary. Provided example metadata and topic list. [1] [2]New and Expanded Documentation:
capture) with CLI and Ruby API usage, best practices, and follow-up steps.Dependency and Docker Compose Updates:
html2rssgem dependency from a Git reference to a versioned release (~> 0.26) in theGemfile, ensuring stability and compatibility.Feed Directory and Navigation Improvements: