Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
Version: 0.7.15 Depends: R (≥ 3.0.0) Imports: stringr (≥ 1.0.0), httr (≥ 1.0.0), spiderbar (≥ 0.2.0), future.apply (≥ 1.0.0), magrittr, utils Suggests: knitr, rmarkdown, dplyr, testthat, covr, curl Published: 2024-08-29 DOI: 10.32614/CRAN.package.robotstxt Author: Pedro Baltazar [aut, cre], Peter Meissner [aut], Kun Ren [aut, cph] (Author and copyright holder of list_merge.R.), Oliver Keys [ctb] (original release code review), Rich Fitz John [ctb] (original release code review) Maintainer: Pedro Baltazar <pedrobtz at gmail.com> BugReports: https://github.com/ropensci/robotstxt/issues License: MIT + file LICENSE URL: https://docs.ropensci.org/robotstxt/, https://github.com/ropensci/robotstxt NeedsCompilation: no Materials: NEWS In views: WebTechnologies CRAN checks: robotstxt results Documentation: Downloads: Reverse dependencies: Linking:Please use the canonical form https://CRAN.R-project.org/package=robotstxt to link to this page.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4