On 2006-10-03 20:10:14 +0200, A.M. Kuchling wrote: > I've added a robots.txt to keep crawlers out of /dev/. Isn't there a lot of useful, search-engine worthy stuff in /dev? I search for peps with google, and I suppose the 'explanation' section, as well as the developer faq and subversion instructions, are good pages that deserve to be in the google index. Should /dev really be Disallow:'ed entirely in robots.txt? kind regards, Gerrit Holl.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4