RetroSearch Browse

Sat Feb 28 11:37:09 CET 2009 · https://mail.python.org/pipermail/python-dev/2009-February/086547.html

Georg Brandl <g.brandl <at> gmx.net> writes:
> 
> Guido van Rossum schrieb:
> > I think the better syntax would be to add site:mail.python.org to the
> > query, but you're right, that doesn't seem to find recent messages.
> > Maybe the absence of a robots.txt file on mail.python.org could be a
> > partial explanation?
> 
> Doesn't the absence of a robots.txt mean "you may index everything"?

It does.
However, pages such as:
    http://mail.python.org/pipermail/python-dev/
(and, it seems, all other pipermail-generated archive pages)
have the following HTML tag in them:
    <META NAME="robots" CONTENT="noindex,follow">
which explicitly instructs Web spiders *not* to index contents nor follow links.

Regards

Antoine.

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://mail.python.org/pipermail/python-dev/2009-February/086547.html below:

[Python-Dev] Googlebot and the mail.python.org python-dev archive