Submitted by : simon at: 2003-10-26T21:31:38+00:00 (17 years ago)
Name :
Category : Severity : Status :
Optional subject :  
Optional comment :

It would be neat if someone wrote some robots.txt lines to make spiders stay off the reparent, subscribe, edit, etc links. I think my perms are set right, but as I watch the access log tail by, I always wonder if some spider is going to unwittingly delete all my pages.

2002/09/16 02:58 GMT (via web):
Previous discussion

this site's current robots.txt

I disallowed all except google today to cut down excessive robot activity. One new problem area is the tracker, which produces a lot (infinite ?) of urls.

See also #269 rename/delete/reparent methods are too open to anonymous abuse.


not supported by robots.txt --SimonMichael, 2003/08/02 23:18 GMT reply
Status: open => closed