Gerrit Holl wrote: > On 2006-10-03 20:10:14 +0200, A.M. Kuchling wrote: >> I've added a robots.txt to keep crawlers out of /dev/. > > Isn't there a lot of useful, search-engine worthy stuff in /dev? > I search for peps with google, and I suppose the 'explanation' section, > as well as the developer faq and subversion instructions, are good pages > that deserve to be in the google index. Should /dev really be > Disallow:'ed entirely in robots.txt? I think that refers to docs.python.org/dev. Georg
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4