I work with a dozen or so clients at any given time, and in the last three (or thereabouts) weeks I’ve noticed that some sites on small servers with limited capacity are being “eaten alive” by spidery searchbots. And not just the usual suspects—Google, Yahoo, MSN—but by specialized searchbots that exhibit a kind of behavior I haven’t seen very much before. It used to be that web site owners prayed for the searchbots to come by, and searchbots by and large were sparing in their examination of pages, not hitting a site very hard at all, but building an overall image of the pages on the site over a long time. [1. Illustration: “Spider & Crossbones” pirate flag]
But times are changing rapidly! Even a site with very little human traffic may be suddenly and catastrophically overwhelmed by searchbot traffic. [Read more…]