Both cases look identical to a working setup unless you're watching request timings. From #1762.
- With
retry_on_blocked=False, _raise_for_session_blocked_status_code is never called, so record_domain_delay never runs and every 429 is ignored.
- Behind a
RequestManagerTandem — e.g. request_list.to_tandem(throttler) — the isinstance(self._request_manager, ThrottlingRequestManager) checks in BasicCrawler all miss. 429 backoff and robots.txt crawl-delay are both dropped, and the "you are not using ThrottlingRequestManager" warning fires at you while you are. The TS port works around this by forwarding record_domain_delay/set_crawl_delay through the tandem.
#1762 added a fair amount of warning machinery for the other misconfigurations; these two slip past all of it.
Both cases look identical to a working setup unless you're watching request timings. From #1762.
retry_on_blocked=False,_raise_for_session_blocked_status_codeis never called, sorecord_domain_delaynever runs and every 429 is ignored.RequestManagerTandem— e.g.request_list.to_tandem(throttler)— theisinstance(self._request_manager, ThrottlingRequestManager)checks inBasicCrawlerall miss. 429 backoff and robots.txt crawl-delay are both dropped, and the "you are not usingThrottlingRequestManager" warning fires at you while you are. The TS port works around this by forwardingrecord_domain_delay/set_crawl_delaythrough the tandem.#1762 added a fair amount of warning machinery for the other misconfigurations; these two slip past all of it.