Well am I ever surprised! I would have thought that inserting a robots.txt file that tells googlebot to “go away” would cause it to “not index the site.”

User-agent: *
Disallow: /

Instead, I discovered that the googlebot may still spot the site and then put up a message saying that the site exists but is not indexed. i.e. the Googlebot still publicizes the existence of the site. It makes Google look like the good guys and us look like the bad guys for putting up a robots.txt. Yay for Google liberating all online information! Boo for us trying to keep our site un-indexed until we’re ready to make it public. [Read more…]