Sky's Blog

Looking back at some of the talk in the halls of the US Congress for about the last year, particularly about clarifying (or solidifying) Executive branch (they’re saying the President) authority to shut down some Internet capabilities in the case of an emergency¹, such as a cyber-attack, makes me think that humans are fond of just shutting their eyes when faced with danger. (continue reading…)

When the web was new, the goal was to get as many “eyeballs” as possible looking at your site content—to aggregate readership with your site being the aggregation point. This pretty much followed the old rules of advertising and promotion—you needed people to see your advertising in order to succeed financially¹. The phrases “visit us often” or “bookmark this site” or “come back frequently” were the conventional wisdom, and web surfers used bookmarks  to remember what sites they wanted to go back to and read later. But they mostly never did except for the big news or entertainment portals.

RSS feeds and news readers began to change that. (Thanks Dave².) I use NetNewsWire’s standalone software on my Mac, and online services like Google Reader let you integrate feeds into your iGoogle home page. You can also sync your Google Reader settings across multiple programs and devices. But in the last couple of months, the scene is greatly changing is subtle ways I think people haven’t spotted yet… (continue reading…)

I work with a dozen or so clients at any given time, and in the last three (or thereabouts) weeks I’ve noticed that some sites on small servers with limited capacity are being “eaten alive” by spidery searchbots. And not just the usual suspects—Google, Yahoo, MSN—but by specialized searchbots that exhibit a kind of behavior I haven’t seen very much before. It used to be that web site owners prayed for the searchbots to come by, and searchbots by and large were sparing in their examination of pages, not hitting a site very hard at all, but building an overall image of the pages on the site over a long time. ¹

But times are changing rapidly! Even a site with very little human traffic may be suddenly and catastrophically overwhelmed by searchbot traffic. (continue reading…)