A request dispatching system that routes requests to registered handlers based on their labels.
The Router allows you to define and register request handlers for specific labels. When a request is received, the router invokes the corresponding request_handler based on the request's label. If no matching handler is found, the default handler is used.
from crawlee.crawlers import HttpCrawler, HttpCrawlingContext
from crawlee.router import Router
router = Router[HttpCrawlingContext]()
# Handler for requests without a matching label handler
@router.default_handler
async def default_handler(context: HttpCrawlingContext) -> None:
context.log.info(f'Request without label {context.request.url} ...')
# Handler for category requests
@router.handler(label='category')
async def category_handler(context: HttpCrawlingContext) -> None:
context.log.info(f'Category request {context.request.url} ...')
# Handler for product requests
@router.handler(label='product')
async def product_handler(context: HttpCrawlingContext) -> None:
context.log.info(f'Product {context.request.url} ...')
async def main() -> None:
crawler = HttpCrawler(request_handler=router)
await crawler.run()
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4