If there is anyone here with an interest in web-spiders, it would be nice if someone else could take a look at http://bugs.python.org/issue21469 which addresses the risk of false positives with the robots.txt parser. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20140511/b234a720/attachment.html>
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4