Google Archives - Sky's Blog

Even Robots.txt won’t keep the googlebot away

6 November, 2012 by sky Leave a Comment

Well am I ever surprised! I would have thought that inserting a robots.txt file that tells googlebot to “go away” would cause it to “not index the site.”

User-agent: *
Disallow: /

Instead, I discovered that the googlebot may still spot the site and then put up a message saying that the site exists but is not indexed. i.e. the Googlebot still publicizes the existence of the site. It makes Google look like the good guys and us look like the bad guys for putting up a robots.txt. Yay for Google liberating all online information! Boo for us trying to keep our site un-indexed until we’re ready to make it public. [Read more…]

Google and YouTube censorship

25 September, 2010 by sky 1 Comment

As a part of their effort to provide greater transparency about the use and blocking of their services, Google provides some interesting information which is available in “real time” online. Their Transparency Report: Traffic shows the relative traffic to their various services by country.

Just for example, if you look at traffic to YouTube from Iran, you’ll see this chart… showing high traffic until June, 2009, when YouTube was blocked in Iran. The scale runs from 0 to 100 and is “normalized” rather than showing absolute bandwidth that’s being used in each country. So it quite nicely illustrates various cases of heavy-handed content blocking.

To see how censorship effectively blocks YouTube in other countries, try looking at Bangladesh, China, Libya, and maybe some others you can discover in the data.

Google and government requests for data or removal

24 September, 2010 by sky Leave a Comment

Google receives requests, including court orders, from many countries, for data or for removal of information on the various services it provides (search, YouTube, etc.).

They’re experimenting with putting up a page that shows the number of requests they’ve received, and (partially) the action they took, for the most recent six months.

You can view their map and click the pushpins to see country-specific data. For China, it says

Chinese officials consider censorship demands to be state secrets, so we cannot disclose that information at this time.

Isn’t it interesting that rather than saying “removal requests” Google used the word “censorship” in this case?

To read more about legitimate (legal) requests and requests that do not have the force of law behind them and may simply be trying to intimidate a web site owner, visit the Chilling Effects Clearinghouse.

The Open Net Initiative seeks to identify and document Internet filtering and surveillance.

Google and censorship in China – the tale continues

8 September, 2009 by sky Leave a Comment

With Kai-Fu Lee leaving Google.cn [see Wall Street Journal article which first announced the departure] the story of how Google has gone from a company that provides access to “everything” on the web, to one that provides access that is modified by special interests—albeit national interests, such as those of China—is once again in the spotlight.

This June 16, 2009 c|net article — Google’s censorship struggles continue in China describes many of the ups and downs. I’ve written a few articles about this relationship, including Censorship as Punishment and Search engines’ blocking in China “improves.” [See also the Related articles section below]

Google is reported to have around 15% of the online search market in China, compared to 75% for Baidu.com. Both companies comply with Chinese government site blocking (censorship) regulations. [Network World article cites the China Internet Network Information Center (CNNIC) as the data source – they publish periodic reports on Internet use in China]

CNNIC’s reports are quite interesting, and for a statistician they’re a lot of fun to puzzle through. For instance, of 338 million Internet users, the “accounts or passwords of 110 million were stolen.” And among all users, 320 million (94%) are on “broadband” – but note that 46% of users access the Internet from “mobile” connections…which seems to me like they would in no way be considered broadband. Also, 28% of mobile Internet users said they would access the Internet by 3G mobile phone – what are the rest using? Maybe wi-fi is considered mobile Internet? Anyone have the answer?

Previous articles I’ve written on Google and censorship in China:

Censorship as Punishment

Use OpenID on your WordPress blog!

12 June, 2009 by sky Leave a Comment

I like OpenID^[1] although I think it’s more complex than most people can handle — and that’s a big hurdle. OpenID gives visitors to your blog or web site a chance to log in (create an account on your site) using their login information from a participating OpenID web site (like gmail). In other words, they don’t have to create a separate account at your blog – they just reuse their Yahoo (or gmail or other^[2]) account. In theory this should make it easier to remember account names and passwords because you just use one account to log in at many sites.Ever since OpenID was announced (2005) I’ve loved the idea. There are OpenID providers, and then there are other sites that allow users to utilize OpenID for the creation of accounts.

[Read more…]