I wrote about this – a few months ago in “Are hungry searchbots eating your site alive?” – but the saga continues! I need a rescue mission, so please will someone send in the SWAT team?
[Geek warning—this post is really for geeks only]
Here’s the short version:
- If you tweet your blog posts, there are hundreds of bots reading the twitter feed and waiting for your post;
- These bots immediately descend on your web server (following a tweet) and spider all over the place;
- If your blog is WordPress-powered or requires significant CPU or database resources to generate a page, this can slow your server at exactly the time when you most need the capacity for human visitors;
- The majority of these swarming bots do not properly identify themselves to your server; and
- The majority of them are coming from AWS now.
- It’s time to firewall unidentified bots hosted at AWS out of our blogs!
Much as my best days are those on which I “learn a lot,” I find that some of my best days are also “totally scattered and almost devoid of billable hours.” In the last three days I’ve probably addressed ten problems for ten different people, and although I’ve billed out a good number of hours to a couple of clients, the majority of the others aren’t getting a bill at all. How do you feel when you’re in this kind of situation—is this extreme attention deficit disorder, or is there something useful to be learned from this kind of behavior? 
Digital nomads, you can finally and really be the system administrator for your cloud (and other) servers from your iPad. Since December, each time I’ve left town, I have intentionally left my MacBook Pro at home in favor of my iPad. I found that just having a few specific apps allowed me to fully administer my cloud servers from the pad. Please note that a bluetooth (or other) keyboard is required for some of these apps to function fully. But generally I can do everything I need to when I’m on the road. 
My “toughest” days are those I spend solving some completely intractible technical problem or other — days when I feel like I’m beating my head against a wall and the problem just will not yield to either intelligent analysis or brute force. The problem just persists, and I keep trying different approaches, and each one leads me a little way down the path but doesn’t solve the problem.