Submitted by : simon at: 2004-05-22T12:21:05+00:00 (13 years ago)
Name :
Category : Severity : Status :
Optional subject :  
Optional comment :

Check out SlowFreeformLinks.

property change --simon, Tue, 01 Feb 2005 10:42:21 -0800 reply

Name: '#818 freeform links causing slowdowns again ?' => '#818 pages with many freeform links render slowly'

property change --Wed, 28 Sep 2005 02:32:05 -0700 reply

Category: user-browsing => user-editing

it affects browsing users --simon, Wed, 28 Sep 2005 18:36:27 -0700 reply

Category: user-editing => user-browsing

**(property change) ** --Mon, 16 Jan 2006 17:50:23 -0800 reply

Name: '#818 pages with many freeform links render slowly' => '#818 Frank Johnson'

**(property change) ** --simon, Mon, 16 Jan 2006 18:45:30 -0800 reply

Name: '#818 Frank Johnson' => '#818 pages with many freeform links render slowly'

My Observations / Recommendation --Warren H. Holt, Tue, 21 Mar 2006 08:20:55 -0800 reply

So far, this is what I have deduced.<br> In all pages types, you have a render function that calls<br> <pre> &nbsp; &nbsp;&nbsp; ZWikiPage.renderMarkedLinksIn &nbsp;&nbsp;&nbsp;&nbsp; ZWikiPage.renderLink &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ZWikiPage.pageWithFuzzyName &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ZWikiPage.pageIds &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ZWikiPage.pages &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ...code... &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; for p in self.pageObjects(): ZWikiPage.pageObjects &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; results.append(self.metadataFor(p)) utils.metadataFor &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ...code... <br></pre>

I made a ZWiki page using a sub-set of the free formed links on your
test page. I then put a logger into ZWikiPage and used it to trace the time / route for the links on this page. The above psudo-code is the route to a for loop which is a major time consumer (about 140 msec on my 2.4 GHz Dell). My ZWiki is not running with a catalog but I do not believe this matters much.
So, first off, I consider this code to be link centric. Every free formed link
on a page ALWAYS returns ALL the page objects then appends the brains for every object. EVERY link! And for ZWiki's without catalogs, a brain instance is created (in utils.metadataFor) for EVERY page object for EVERY link! Further, the pageObjects function always returns ALL the objects in the ZWiki. Again, this happens for EVERY free formed link on a page.
So, here is my suggestion. Change the renderMarkedLinksIn function to be
page centric code. By this I mean renderMarkedLinksIn would gather all the page objects (really all the ZWiki objects) and append the brains then simply pass this to the lower functions. This way the objects and brains would only be gotten for each page rather than for every link. I am not that intimate with all your code but I believe this would work, though it would require at least some re-coding.
P.S. - I also wonder just how happy garbage collection is with all those brain
instances being created for every link but then not used! I have to believe if you can reduce this it would only help!

My Observations / Recommendation --Simon Michael, Tue, 21 Mar 2006 12:11:03 -0800 reply

Warren, thanks very much for clarifying this! A few notes:

pageWithFuzzyName just needs a list of page ids. pageIds() was intended to be cheap, almost a no-op. I made it call pages() back when I was trying to make wiki links span an entire plone site (and reduce zodb loads through more use of catalog). pages() was intended to be fairly quick as well; and for large sites, it was intended to have a catalog to help it.

I'd be interested to know what you see in these two scenarios:

1. when you uncomment the old code in pageIds(), so that it doesn't call pages()

  1. when you add a catalog (PAGE/setupCatalog)

and also how many pages you have and whether your zope is getting them all from zodb cache or having to load a bunch of them every time.

My Observations / Recommendation --Simon Michael, Tue, 21 Mar 2006 12:13:33 -0800 reply

I meant to add: caching expensive stuff up in the page render method seems like a good idea, where needed. And you may be right about that garbage collection.

My Observations / Recommendation --Warren H. Holt, Tue, 21 Mar 2006 12:47:31 -0800 reply

<br>I have 59 pages in my ZWiki

Using the commented out code in pageIds() results in about 3 or so renders every 15 msec and all the free form links on the page (17) rendering in just under a second.

I will try the catalog tomorrow (hopefully - depends on work load)

Also, please advise - how do I tell if pages are in zodb cache or loading them every time?

My Observations / Recommendation --Simon Michael, Tue, 21 Mar 2006 13:03:15 -0800 reply

> Using the commented out code in pageIds() results in about 3 or so renders every 15 msec Ie you can render a page with many freeform links in about 5ms ? That would be good, but I think I'm misunderstanding.

Really, I'd expect pretty much anything to be very fast in a 60 page wiki. (Except for saving a huge stx/rst page).

(property change) fixed! --simon, Tue, 21 Mar 2006 13:29:21 -0800 reply

Status: open => closed

I've reverted to the old pageIds() on, and SlowFreeformLinks rendering time went from ~20s to ~1s. I knew it used to be quick.

This is a safe and I think reasonable change, until someone wants to try trans-folder wiki linking again. pages() should still be looked at to see how it can be made more efficient, it's used all the time.

Closing this regression issue, which I reported on 2004/05/22. You rock!!

My Observations / Recommendation --Warren H. Holt, Wed, 22 Mar 2006 03:17:33 -0800 reply

Clarification! The renders I was talking about were link renders, not page renders.

... --simon, Wed, 03 May 2006 13:52:57 -0700 reply

Name: '#818 abagail' => '#818 pages with many freeform links render slowly' Category: => user-browsing Severity: funded => serious Status: open => closed