Has anybody seen http://www.whitehouse.gov/robots.txt? It’s unbelievable! If you don’t know what a robots.txt file is, it’s a list of pages on a web site that a search engine (i.e. Google, Yahoo, MSN) should avoid, basically telling it “if you see this file, please don’t add it in to your database.” It’s often used for stuff like hiding address books or other such data from search engines. However, if you look at the one at whitehouse.gov, you’ll see exclusion list has an incredible amount of exclusions. A quick read through shows that it excludes “911”, “iraq”, “deptofhomeland”, “europe”, “medicare”, etc, etc. Nearly every contraversial topic of the Bush administration. Is this example of limiting exposure through lack of a Google visiblity? It feels oh, so very Karl Rove.
Of course, this has had much coverage in the blogosphere: http://www.sethf.com/infothought/blog/archives/000455.html, http://www.bway.net/~keith/whrobots/whresp.html, http://www.deadparrots.net/archives/politics/0310sinister_doings_at_whitehousegov.html, and on http://www.2600.com/news/view/article/1803.