Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://crawlee.dev/python/api/class/AbstractHttpCrawler below:

AbstractHttpCrawler | API | Crawlee for Python · Fast, reliable Python web crawlers.

• __init__(*, configuration, event_manager, storage_client, request_manager, session_pool, proxy_configuration, http_client, request_handler, max_request_retries, max_requests_per_crawl, max_session_rotations, max_crawl_depth, use_session_pool, retry_on_blocked, additional_http_error_status_codes, ignore_http_error_status_codes, concurrency_settings, request_handler_timeout, statistics, abort_on_error, keep_alive, configure_logging, statistics_log_format, respect_robots_txt_file, status_message_logging_interval, status_message_callback, _context_pipeline, _additional_context_managers, _logger): None

• Overrides BasicCrawler.__init__

  Parameters
  • optionalkeyword-onlyconfiguration: Configuration | None = None
  • optionalkeyword-onlyevent_manager: EventManager | None = None
  • optionalkeyword-onlystorage_client: StorageClient | None = None
  • optionalkeyword-onlyrequest_manager: RequestManager | None = None
  • optionalkeyword-onlysession_pool: SessionPool | None = None
  • optionalkeyword-onlyproxy_configuration: ProxyConfiguration | None = None
  • optionalkeyword-onlyhttp_client: HttpClient | None = None
  • optionalkeyword-onlyrequest_handler: Callable[[TCrawlingContext], Awaitable[None]] | None = None
  • optionalkeyword-onlymax_request_retries: int = 3
  • optionalkeyword-onlymax_requests_per_crawl: int | None = None
  • optionalkeyword-onlymax_session_rotations: int = 10
  • optionalkeyword-onlymax_crawl_depth: int | None = None
  • optionalkeyword-onlyuse_session_pool: bool = True
  • optionalkeyword-onlyretry_on_blocked: bool = True
  • optionalkeyword-onlyadditional_http_error_status_codes: Iterable[int] | None = None
  • optionalkeyword-onlyignore_http_error_status_codes: Iterable[int] | None = None
  • optionalkeyword-onlyconcurrency_settings: ConcurrencySettings | None = None
  • optionalkeyword-onlyrequest_handler_timeout: timedelta = timedelta(minutes=1)
  • optionalkeyword-onlystatistics: Statistics[TStatisticsState] | None = None
  • optionalkeyword-onlyabort_on_error: bool = False
  • optionalkeyword-onlykeep_alive: bool = False
  • optionalkeyword-onlyconfigure_logging: bool = True
  • optionalkeyword-onlystatistics_log_format: Literal[table, inline] = 'table'
  • optionalkeyword-onlyrespect_robots_txt_file: bool = False
  • optionalkeyword-onlystatus_message_logging_interval: timedelta = timedelta(seconds=10)
  • optionalkeyword-onlystatus_message_callback: Callable[[StatisticsState, StatisticsState | None, str], Awaitable[str | None]] | None = None
  • optionalkeyword-only_context_pipeline: ContextPipeline[TCrawlingContext] | None = None
  • optionalkeyword-only_additional_context_managers: Sequence[AbstractAsyncContextManager] | None = None
  • optionalkeyword-only_logger: logging.Logger | None = None
  Returns None

• async add_requests(requests, *, forefront, batch_size, wait_time_between_batches, wait_for_all_requests_to_be_added, wait_for_all_requests_to_be_added_timeout): None

• Inherited from BasicCrawler.add_requests

  Parameters
  • requests: Sequence[str | Request]
  • optionalkeyword-onlyforefront: bool = False
  • optionalkeyword-onlybatch_size: int = 1000
  • optionalkeyword-onlywait_time_between_batches: timedelta = timedelta(0)
  • optionalkeyword-onlywait_for_all_requests_to_be_added: bool = False
  • optionalkeyword-onlywait_for_all_requests_to_be_added_timeout: timedelta | None = None
  Returns None

• error_handler(handler): ErrorHandler[TCrawlingContext]

• Inherited from BasicCrawler.error_handler

  Parameters
  • handler: ErrorHandler[TCrawlingContext | BasicCrawlingContext]
  Returns ErrorHandler[TCrawlingContext]

• async export_data(path, dataset_id, dataset_name): None

• Inherited from BasicCrawler.export_data

  Parameters
  • path: str | Path
  • optionaldataset_id: str | None = None
  • optionaldataset_name: str | None = None
  Returns None

• failed_request_handler(handler): FailedRequestHandler[TCrawlingContext]

• Inherited from BasicCrawler.failed_request_handler

  Parameters
  • handler: FailedRequestHandler[TCrawlingContext | BasicCrawlingContext]
  Returns FailedRequestHandler[TCrawlingContext]

• async get_data(dataset_id, dataset_name, kwargs): DatasetItemsListPage

• Inherited from BasicCrawler.get_data

  Parameters
  • optionaldataset_id: str | None = None
  • optionaldataset_name: str | None = None
  • kwargs: Unpack[GetDataKwargs]
  Returns DatasetItemsListPage

• async get_dataset(*, id, name): Dataset

• Inherited from BasicCrawler.get_dataset

  Parameters
  • optionalkeyword-onlyid: str | None = None
  • optionalkeyword-onlyname: str | None = None
  Returns Dataset

• async get_key_value_store(*, id, name): KeyValueStore

• Inherited from BasicCrawler.get_key_value_store

  Parameters
  • optionalkeyword-onlyid: str | None = None
  • optionalkeyword-onlyname: str | None = None
  Returns KeyValueStore

• async get_request_manager(): RequestManager

• on_skipped_request(callback): SkippedRequestCallback

• pre_navigation_hook(hook): None

• Parameters
  • hook: Callable[[BasicCrawlingContext], Awaitable[None]]
  Returns None

• async run(requests, *, purge_request_queue): FinalStatistics

• Inherited from BasicCrawler.run

  Parameters
  • optionalrequests: Sequence[str | Request] | None = None
  • optionalkeyword-onlypurge_request_queue: bool = True
  Returns FinalStatistics

• Inherited from BasicCrawler.stop

  Parameters
  • optionalreason: str = 'Stop was called externally.'
  Returns None

router: Router[TCrawlingContext]

statistics: Statistics[TStatisticsState]

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4