Spiders: May 2004 Archives

I have seen plenty of robots spider my site but by far the most polite as far as time goes has got to be the miragorobot. It waited a full minute between each page request which is almost unheard of.

The spiders have now collected over 500,000 pages. Considering I have been taking it easy it was surprisingly fast. I now need another 500,000 to get my goal of 1 Million for testing.