The 'Robots Exclusion Protocol' <https://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files.
Version: 0.2.5 Depends: R (≥ 3.2.0) Imports: Rcpp LinkingTo: Rcpp Suggests: covr, robotstxt, tinytest Published: 2023-02-11 DOI: 10.32614/CRAN.package.spiderbar Author: Bob Rudis (bob@rud.is) [aut, cre], SEOmoz, Inc [aut] Maintainer: Bob Rudis <bob at rud.is> BugReports: https://github.com/hrbrmstr/spiderbar/issues License: MIT + file LICENSE URL: https://github.com/hrbrmstr/spiderbar NeedsCompilation: yes In views: WebTechnologies CRAN checks: spiderbar results Documentation: Reference manual: spiderbar.pdf Downloads: Package source: spiderbar_0.2.5.tar.gz Windows binaries: r-devel: spiderbar_0.2.5.zip, r-release: spiderbar_0.2.5.zip, r-oldrel: spiderbar_0.2.5.zip macOS binaries: r-release (arm64): spiderbar_0.2.5.tgz, r-oldrel (arm64): spiderbar_0.2.5.tgz, r-release (x86_64): spiderbar_0.2.5.tgz, r-oldrel (x86_64): spiderbar_0.2.5.tgz Old sources: spiderbar archive Reverse dependencies: Reverse imports: robotstxt Linking:Please use the canonical form https://CRAN.R-project.org/package=spiderbar to link to this page.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4