Submitted by : Simon Michael at: 2009-03-01T10:49:49-08:00 (8 years ago)
Name :
Category : Severity : Status :
Optional subject :  
Optional comment :

(new) robots.txt too restrictive --Simon Michael, Sun, 01 Mar 2009 10:49:51 -0800 reply

robots.txt was permitting only google and internet archive for a long time - a clumsy solution to load problems caused by rude bots. This probably makes us harder for non-google users to find (I just tried - yahoo search finds us, microsoft search doesn't), and blocks useful tools like browsershot.org. I've opened it up to all, excluding a few expensive urls, we'll see what happens.

... --simon, Wed, 28 Oct 2009 11:25:47 -0700 reply

Category: => user-browsing Status: => closed