Here’s my history of hosting on the Internet. Before 1994 I built software for everything from supercomputers to home computers (I’ll tell you more about DesignWare some day).

• 1994- Virtual hosts on Best.com; 1999- Virtual hosts on Verio (bought Best.com)
• 2003-2005: New X-Serves on Maccius (Mac-only host in San Jose)
• 2009: Slicehost, and then Rackspace cheap virtual hosts- “slices” in the cloud
• 2012: WPEngine — specializing in WordPress, with caveats

Why this particular migration and what would I recommend that you do today?1994-2003: Virtual hosts were the way to go when the Web was first ramping up. They’re easy to start up (click a few buttons) and you can use FTP to upload, and most of the provide root access so you can load any software you want to. But they were relatively expensive, given that they are on shared computers and you’re at the mercy of the other users who’re on that same computer – if someone runs away with the CPU you end up with a very slow site. (And I’ve been guilty of messing up the other users too…)^[1]

2003-2008: Later on, I tired of paying $95 a month for a slice of a larger computer and bought my own Apple servers. These machines really rock. Lots of power, and dedicated to me alone. No need to worry about other users getting in my way. For $125 a month I get a space in a rack and I can run my server at 100% CPU if I want to. Cranks out sites really quickly.

2009: Virtual servers have come down in price. I looked around early in the year and chose Slicehost, which has a simple $25/month server with 256MB of “RAM” and 10GB of RAID storage, and they give me 100GB of data transfer per slice per month. I wouldn’t have gone back to virtual hosting except for the low price. Slicehost was acquired by Rackspace, and made a part of their cloud late this year, and I also had a few virtual servers on Rackspace. So they’re all consolidated under Rackspace now.

2012: WordPress specialists, such as WPEngine, have come along, providing service for those who use WP but don’t care to do the security and system upgrading.

Only for the brave: The pros and cons of slicing up a big server: Well, 256MB of RAM is pretty small, and that’s what Slicehost allowed you to start with. Rackspace starts at 512MB now. You gotta install a lean operating system (Ubuntu 8.04 was my choice — but now Ubuntu 12.04 LTS), and you gotta really tune it up so it functions well in tight places. The nicest thing about the old Rackspace slices is the 4 powerful CPUs, which give you lots of headroom if you have to crank out a page. If you buy a new Rackspace VPS now (December, 2012) you’ll get a single CPU allocated, with 512MB of RAM as the minimum.

Yup, if you like getting your hands really dirty, then a slice might be for you. Ubuntu has gotten pretty easy to install – and on Rackspace you just click a button and a few minutes later they send you a hostname and IP address and it’s running. And adding new features is about as simple as running “apt-get install foo” and then you do a few configuration changes. Well, almost that easy. Every once in a while I run into something that I think is going to take an hour and it takes two days. But I can configure a new host from scratch in about 25 minutes (Apache, PHP, MySQL) and bring up new web sites in about 20 minutes each.

WordPress specialists: I’ve got to say something about WPEngine here because it’s a love/hate relationship. First, WPE provides a great service because they give you a functioning WP engine in one click, for as little as USD $25 a month. They provide a one-click staging version of your site where you can test new plugins and do your own PHP coding without upsetting your live site. They host under your own domain name or theirs — you pick. They “curate” common plugins, taking care of any problems and updating them so you’re always up to date. The one caveat is that you can’t use certain banned plugins (mostly those that are database intensive), and they sometimes push upgrades without warning you, which might break your non-curated plugins! I recommend them if you have plain vanilla WordPress needs, but my jury is still out in terms of whether more sophisticated sites should use them or not.[2]

[1] A Virtual Server or Virtual Host or Virtual Private Server [VPS] is a piece of software that subdivides a big server into smaller “servers” that from a user’s viewpoint look like independent machines, but in real life are just logical (thus “virtual”) subdivisions of the larger computer.

[2] I’ve brought up several sites that needed large bursts of speed and high reliability. WPE is great for the bursts, which would have bogged down a VPS. However, we’ve been surprised several times when our sites just stopped working due to unexpected WPE upgrades. They do not tell you in advance when they’re going to change some plugin out in the middle of the night, so you have to be on call 24/7 watching for such things. You just don’t have the same degree of control here in terms of staging your upgrades and improvements and then rolling them on particular days because they might do one suddenly that you didn’t expect.

The post Hosting- Rackspace on steroids and on to WPEngine (for some) appeared first on Sky's Blog.

[Geek warning—this post is really for geeks only]

Here’s the short version:

• If you tweet your blog posts, there are hundreds of bots reading the twitter feed and waiting for your post;
• These bots immediately descend on your web server (following a tweet) and spider all over the place;
• If your blog is WordPress-powered or requires significant CPU or database resources to generate a page, this can slow your server at exactly the time when you most need the capacity for human visitors;
• The majority of these swarming bots do not properly identify themselves to your server; and
• The majority of them are coming from AWS now.
• It’s time to firewall unidentified bots hosted at AWS out of our blogs!

Here are the details:

One of my clients insists on tweeting every post that goes up on her web site, and that means dozens of posts a day because the site syndicates posts (in other words, it “repeats” posts from the blogs of her contributors) from about 100 different sources. It’s a great idea to tweet your posts, because in theory it generates lots of human traffic. But, in this case within seconds of any tweet there are hundreds of Twitter-following searchbots swarming all over the site checking every possible page[1. These are bots on servers of companies that presumably “live” off reading tweets and then directing traffic to the tweeted web sites.]. They do (HTTP) GETs of the pages, and they don’t bother looking at photos or support files (which is nice of them, actually, because it lowers the server load) — they just hit the pages — and sometimes a single bot will hit a page 4 or 5 times right in a row. Why would a bot need to GET a page 4 times in a row? It only needs it once, doesn’t it?

The problem is that on a WordPress-powered site, each GET causes the full generation of all of the dynamic content on that page (if it’s not cached yet). This can take a considerable amount of CPU and database time. It can take “more than clock time” to generate a page…meaning that it can take several seconds of CPU time to generate a page. Bots that hit pages, like the home page, that are seen frequently will get a cached version of that page, which is “easy on the server” and requires very few resources to generate, but bots that crawl all over the site cause the regeneration of uncached pages, and that can put a significant load on a small server! For example, the bot hitting a page four times in a second can cause the need for 4 x 3 = 12 seconds of CPU time if it requires 3 CPU seconds to generate the page (it’s uncached, remember?). In one second they can generate the need for 12 seconds of CPU time. (I know I’m repeating myself.) It’s this unrestricted (and way too fast!) crawling that causes a server to fall to its knees and surrender — and stop serving pages to human visitors. Unfortunately, the crawlers hit the site immediately after a tweet goes out, which is when you want the server to be at maximum readiness to handle human traffic!

The majority of these swarming bots are now coming from IP addresses assigned to Amazon AWS. And although bots usually identify themselves to a site when they GET a page[2. …by including a string called the User-Agent in their request], saying who or what company is doing the search, and usually a URL so you can find out more about the company or the searchbot, the majority of these new bots I’m seeing do not identify themselves. Well-behaved bots like Google and Bing, Yahoo and MSN, all identify themselves and they only GET pages on a slower schedule[3. …that I can define in the robots.txt file]. It’s the outlaws that are causing the problems.

What I am seeing these days is dozens of AWS-based bots spidering the site, making many requests (each) per second and no identification at all!

I had hoped I could come to a compromise solution that would keep these guys from clobbering the client site but still let them in, but I can’t find one. So the firewalls went up today to refuse service to most AWS-based searchers. Sorry guys, but bad behavior on the part of many has caused grief for the few human visitors, and we have to pay attention to our real readers, who are the humans, after all.

The post Swarming Searchbots from Amazon AWS appeared first on Sky's Blog.

Oh, and I subscribe to Scientific American digital (monthly) and read it as a PDF on the screen rather than get all that paper that just piles up before I can get to it.

As in quantum-tunneling^[1] effects, you can get me past the initial resistance to an ebook if:

• The price of the ebook is 60% or less than the price of the physical book; or if
• I don’t want the physical book hanging around anyway after I’ve read it; or if
• It’s available in PDF so I can read it anywhere (though I do purchase prioprietary DRM formats frequently); or if
• It’s $9.99 even if I think I could find a paperback for slightly less somewhere else.

It is just so much easier to take an ebook with me and read it on my screen (or iPhone in the case of the Kindle^[2] and Fictionwise readers)!

[2] There’s a Kindle book reader iPhone app that allows you to buy and download Kindle books from Amazon to read them on your iPhone. No reason this wouldn’t also work on the iPad, since they say 140,000 apps already run on it. (I wonder who took the time to test that assumption…) Fictionwise.com also has a reader available in the app store.

The post The $9.99 ebook appeared first on Sky's Blog.

Geek alert! This article is for Slicehost geeks only.

Talk about playing on the edge, I almost fell off this time. I have eight slices (virtual servers, that is) on Slicehost (which is now owned by Rackspace) and believe me they are tiny!^[1] But they can serve low-volume to medium-volume web sites pretty well. For instance, Shaping Youth (several thousand visitors a day) and GirlsHorseClub are on these little servers and they serve up pretty rapidly.^[2] Nice thing about Slicehost is that any host can be scaled up from the tiny size to 16x that size, and because it’s cloud computing, you can also bring up multiple instances of any server and have them share load. This allows me to implement the traditional three-tiered architecture (web/application/database) quite well without having to worry about whether my servers will be able to handle the load because I can scale ’em up whenever I need to.

But, the catch is that when you start with the tiniest slice (256MB RAM) you really have to fine-tune your operating system and applications in order to get good performance out of them.

This has led me to some fun discoveries about 1) how to make Apache work in a really tight space; 2) how to add additional web serving using NGINX (engine-X); 3) how to run MySQL in less space; and finally 4) how to use Java and Tomcat wedged into a non-existent memory slice.

Today’s lesson for me was installing Tomcat 5.5 on Slicehost. I started by reading How to Install Tomcat on Ubuntu in the mkyong blog. Not bad at all. The essence is to first find out what version of Tomcat is available on your slice:

sudo apt–cache search tomcat

Then install that version and the admin app for it:

apt–get ––fix–missing install tomcat5.5
apt–get ––fix–missing install tomcat5.5-webapps
apt–get ––fix–missing install tomcat5.5-admin

Then I like to have /usr/local/tomcat defined (like on all my other servers):

ln –s /usr/share/tomcat5.5/server/ /usr/local/tomcat

That was about all it took. It serves by default on port 8180 – and the admin app was there and ready to go. My next step was to port a Java app that I have been running on bigger servers over to this little server. That actually worked quite well, requiring about 6 hours to port, recompile (to eliminate a few warnings) and test.

[2] These two slices actually had to be boosted to 512MB recently in order to handle anticipated high traffic. But I can reduce their size after the stress-out period passes.

The post Slicehost- Tomcat installation on a tiny virtual server appeared first on Sky's Blog.