Edit detail for #699 too many urls for robots to crawl revision 1 of 1

1
Editor: simon
Time: 2004/05/29 09:45:25 GMT+0
Note: allowed contents to scroll again, cost is reasonable

changed:
-
zwiki.org is unusually slow and conflict-prone again. The reason is we are being crawled by one or more robots, and the skin now includes more expensive links for them to trigger. We need to look at the general issue of robot-robustness again.

Those hidden links I added for access keys are a problem :(, especially things like /print (rendering many pages) and /clearCache (causing large pages to be re-rendered next time). For now I am removing at least these two urls from the zwiki.org skin. 

Standard links like contents, recent changes, index may be expensive as well. Right now there's a different contents url for each page, maybe we need to get by with just one.

From SimonMichael Thu Feb 5 13:45:45 -0800 2004
From: SimonMichael
Date: Thu, 05 Feb 2004 13:45:45 -0800
Subject: property change
Message-ID: <[email protected]>

Title: 'IssueNo0699 current skin is too open to robots, makes sites slow' => 'IssueNo0699 current skin is too open to robots' 


From SimonMichael Thu Feb 5 13:55:52 -0800 2004
From: SimonMichael
Date: Thu, 05 Feb 2004 13:55:52 -0800
Subject: semi-serious
Message-ID: <[email protected]>

This is not serious in most cases, but it's serious enough for a large wiki which allows robots and has the print dtml method installed (zwiki.org).

From SimonMichael Fri Feb 6 01:05:54 -0800 2004
From: SimonMichael
Date: Fri, 06 Feb 2004 01:05:54 -0800
Subject: improved for 0.27.1
Message-ID: <[email protected]>

The clearCache and print links are disabled, and the contents link now defaults to non-scrolling (so just one contents url per wiki).

From SimonMichael Fri Feb 6 01:08:14 -0800 2004
From: SimonMichael
Date: Fri, 06 Feb 2004 01:08:14 -0800
Subject: property change
Message-ID: <[email protected]>

Status: open => closed 


From simon Sat Apr 17 19:30:05 -0700 2004
From: simon
Date: Sat, 17 Apr 2004 19:30:05 -0700
Subject: the fix is irritating for users, though
Message-ID: <[email protected]>

Title: 'IssueNo0699 current skin is too open to robots' => 'IssueNo0699 too many urls for robots to crawl' 
Severity: serious => normal 
Status: closed => open 


From simon Sat May 29 09:45:25 -0700 2004
From: simon
Date: Sat, 29 May 2004 09:45:25 -0700
Subject: allowed contents to scroll again, cost is reasonable
Message-ID: <[email protected]>

Status: open => closed 


Submitted by : SimonMichael at: 2004-02-03T11:01:01+00:00 (17 years ago)
Name :
Category : Severity : Status :
Optional subject :  
Optional comment :

zwiki.org is unusually slow and conflict-prone again. The reason is we are being crawled by one or more robots, and the skin now includes more expensive links for them to trigger. We need to look at the general issue of robot-robustness again.

Those hidden links I added for access keys are a problem :(, especially things like /print (rendering many pages) and /clearCache (causing large pages to be re-rendered next time). For now I am removing at least these two urls from the zwiki.org skin.

Standard links like contents, recent changes, index may be expensive as well. Right now there's a different contents url for each page, maybe we need to get by with just one.


comments:

property change --SimonMichael, Thu, 05 Feb 2004 13:45:45 -0800 reply
Title: IssueNo0699 current skin is too open to robots, makes sites slow => IssueNo0699 current skin is too open to robots

semi-serious --SimonMichael, Thu, 05 Feb 2004 13:55:52 -0800 reply
This is not serious in most cases, but it's serious enough for a large wiki which allows robots and has the print dtml method installed (zwiki.org).

improved for 0.27.1 --SimonMichael, Fri, 06 Feb 2004 01:05:54 -0800 reply
The clearCache and print links are disabled, and the contents link now defaults to non-scrolling (so just one contents url per wiki).

property change --SimonMichael, Fri, 06 Feb 2004 01:08:14 -0800 reply
Status: open => closed

the fix is irritating for users, though --simon, Sat, 17 Apr 2004 19:30:05 -0700 reply
Title: IssueNo0699 current skin is too open to robots => IssueNo0699 too many urls for robots to crawl Severity: serious => normal Status: closed => open

allowed contents to scroll again, cost is reasonable --simon, Sat, 29 May 2004 09:45:25 -0700 reply
Status: open => closed