Zwiki/Zope can chew up quite a lot of memory as wikis grow large, depending on configuration (number of pages, threads, traffic, catalog..). Not a lot, but more than the 50-100M available in your average zope hosting account. And exceeding the available memory makes zope crash.

So it would be a good thing for SiteReliability if Zwiki/Zope required less memory, and also for general scalability and performance. (A related issue is how to prevent zope crashes). This page aims to understand these issues and document solutions. Please correct any misunderstandings.

Here's a reading of this server's current process size: (see WikiStats)

memusage.py is an external method for printing zope memory usage.

The situation (old - how things were with zwiki 0.21)

Zope uses lots of memory when serving large wikis, because Zwiki tends to load up all pages into the zodb cache. (On this site the average cached page costs about 20K.) There's a separate cache for each zope thread.

Ideally it would load only the pages being accessed into cache. Current Zwiki (0.21) is quite good in this respect, but I've found the following "expensive" operations which load up all pages:

Also, the following are "semi-expensive":

A user requesting these expensive ops occasionally may not be a big deal; if the whole wiki is cached, no problem, otherwise those requests will be slow. I surmise they are triggered more frequently when we're crawled by google though. And when this happens in multiple threads - either concurrently or not, I'm unsure - the overall memory footprint becomes large. And unless the target cache size is large enough to allow each thread to keep all pages fully loaded, all this cache loading causes big slowdowns.

We can do better by relying more on the catalog, when present. We should optimize some or all of these to use catalog as backlinks and RecentChanges? do. Then Zwiki would rarely or never load all pages at once, and be more scalable.

TresSeaver wrote about this long ago:

Currently, ZWiki handles backlinks, AllPages, and RecentChanges? using "brute force" iteration over the ZWikiPage(s) in the current folder.

This technique makes each invocation of SomeWikiPage/backlinks, AllPages, and RecentChanges? not only linear in the total number of pages, but also defeats ZoPe's caching mechanism (each invocation activates every page).

Using a ZCatalog would make backlinks linear in only the number of matches. AllPages and RecentChanges? would still be linear in the total number of pages, but would run entirely against the data cached in the catalog. Much more cache-friendly.

That sounds good. The only doubt I have is this issue described on the lists where concurrent writes to a shared resource, eg a zcatalog, are not handled well. Am I right about this ?

I have seen the discussion, but haven't yet experienced the problem. I have a client who is just beginning to test a ZClass?/ZCatalog-based solution with multiple users, so I should know more this week.

For Zwiki 0.22, the expensive operations above now use the catalog if possible, reducing the memory footprint of a large wiki and making mail-in, contents, search and renaming faster.

As far as I know only naive DTML code will load up all pages in a single transaction now.. but there may be some other situations I've missed.

Calculating peak memory usage

Note that the target cache size that you configure is only a guideline, which applies across but not during transactions. It seems that each zope thread will transiently grab as much memory as it needs to load all pages, plus any other objects needed, when doing one of these ops, for the duration of the transaction. If it can't, the request fails with a memory error or disconnected error, or zope restarts or hangs!

How do we know when we become exposed to these problems ? If I understand correctly, the maximum size of the main process of a zope server running zwiki will be:

 Mmax ~= Z + T * (P * S + O)

 ie Zope base memory usage + number of threads * ((number of pages * average page size) + overhead)

On this server, with it's 100M per-process memory quota, this implies:

 100M = 27M + 2 * (Pmax * 20K + 5M)

(5M overhead is just a guess) so in theory the maximum number of pages we could reliably run here (assuming no leakage) would be:

 ((100000 - 27000) / 2 - 5000) / 20 = 1575

But page sizes may be significant too. WikiStats.


subtopics:


comments:

Ouch --BillSeitz, Fri, 08 Aug 2003 23:32:26 +0000 reply
I have roughly 39k objects in my db. I think I have roughly 2k pages in my /wiki/ folder.

How can I most easily calculate the avg page size (in terms of cache cost)?

It seems to me that even if I use a catalog as much as possible, if I don't include the full body in the catalog, then a single Search will dash my plans, and drag everything into the cache anyway. Is that a fair interpretation?

is Zope that dumb? --BillSeitz, Sat, 09 Aug 2003 00:04:36 +0000 reply
To grab object IDs? it ends up loading all entire objects into cache? (e.g. for standard_error_msg, or in renaming)

make showing ages of links pages an option? --BillSeitz, Sat, 09 Aug 2003 00:06:40 +0000 reply
It's cool but not a feature I actually use much.

I could see treating it as an option set by either (a) the admin for the folder, or (b) a UserOptions? (defaulting to "no", or maybe the admin sets the default).

serving "reliably" --BillSeitz, Sat, 09 Aug 2003 00:11:37 +0000 reply
My site's been up 22 days at this point. So from that standpoint there's not much of a problem. But I'm often distressed at how slow it is (in a highly variable way).

Ugh, and I just thought about how my BillSeitz:FrontPage sucks up boatloads of linked pages to load, to grab those last-edited bits... sob...

serving "reliably" --SimonMichael, Sat, 09 Aug 2003 00:41:21 +0000 reply
That's great.. except, er, well wanting to count your pages I viewed your contents (ok) then did a search for blank (not ok - site now appears hung). If it were zwiki.org I'd say it's deadlocked trying to allocate additional memory beyond 100M - restart required.

Your variable speed sounds like cache being loaded. What's your target cache size ? Are you running the default 4 threads ?

objectIds() does not load pages. objectValues() doesn't either - but accessing attributes on the pages will load them.

Anyhow, looks like you have about 2800 and take the crown for largest zwiki right now. Welcome to a world of pain.. (just kidding :).

serving "reliably" --SimonMichael, Sat, 09 Aug 2003 00:49:42 +0000 reply
PS I estimated average page size by checking zope's base memory usage with top or ps (after startup - probably ~27Mb), checking it again after viewing wiki contents, and dividing the difference by the number of pages (assuming no activity from other clients).

Good to have these other large wikis to compare notes with.

serving "reliably" --BillSeitz, Sat, 09 Aug 2003 03:09:54 +0000 reply
Oy, I'm glad I checked email again, so I could restart.

I'm ratcheted down to 2 threads.

Since flushing cache and packing the zodb, I know have 24K objects in the db. Targeted cache size is 3300, actual total number is now 4083. If I have 2800 pages should I increase the target to 5600? How much does the target matter?

If objectIds() doesn't load pages, then why does a bad URL trigger loading the pages? Because of using title_or_id?

Might it be time to start some optimizing, in terms of tweaking feature sets to support largish wikis? (Note: JerryMichalski?'s "TheBrain" db has over 32k nodes in it. Now most of them basically have no content other than a label, a URL, and links to other nodes. But...)

Also, I wonder whether imeme would consider increasing memory space for some people for a very nominal fee? Have you ever talked to them about such things?

serving reliably --Simon Michael, Sat, 09 Aug 2003 20:15:31 +0000 reply

Since flushing cache and packing the zodb, I know have 24K objects in the db. Targeted cache size is 3300, actual total number is now 4083. If I have 2800 pages should I increase the target to 5600? How much does the target matter?

Total objects in the zodb doesn't matter. Target cache size affects performance, but let's concentrate on the "not crashing" issue for now. As I understand it the only thing that matters for this is:

If not, then sooner or later requests are going to be met with a ClientDisconnected? error, a MemoryError? plus zope restart (always ?), or an apparent zope hang with no restart (bad). NB I haven't noticed a hang on zwiki.org since installing AutoLance?, but it may just be luck.

On imeme, with 2800 pages you'd have to go to one thread to be safe. With two threads I estimate you could safely run about 1500 pages. Needs to be verified of course.

If objectIds() doesn't load pages, then why does a bad URL trigger loading the pages? Because of using title_or_id?

standard_error_message does a search

Might it be time to start some optimizing, in terms of tweaking feature sets to support largish wikis? (Note: JerryMichalski?'s "TheBrain" db has over 32k nodes in it. Now most of them basically

Yup, that's what I propose above (and patches are welcome). As next milestone I'd like to be able to serve 10K pages in an imeme account without breaking a sweat.

Also, I wonder whether imeme would consider increasing memory space for some people for a very nominal fee? Have you ever talked to them about such things?

Not directly. They've set up some expansion deals, but they're not very attractive.

serving reliably --SimonMichael, Sat, 09 Aug 2003 20:44:58 +0000 reply

standard_error_message does a search

Or does it ? It seems to just call pageWithFuzzyName, which I thought was cheap. Well, needs a looking-at.

data point. --SimonMichael, Thu, 14 Aug 2003 19:02:06 +0000 reply
Item: http://handhelds.org/z/wiki/HandheldsWiki . From browsing around a bit, this is the fastest zope site I've seen, zwiki or otherwise. I inquired further.

~1200 pages, daily edits. Old version of zwiki, 0.9.8. 8 threads. No catalog. Hardware: dual 1Ghz pentium with 1G ram.

Zope process size is about 100M, with 30M resident. Requesting the /map (should load all pages) 8 times concurrently had no significant effect on this! Also, no noticeable slowdown while browsing during this. Oh - and his target cache size is just 400. And Data.fs is 80Mb.

This is not at all what we've been seeing on zwiki.org. What could explain this ?

data point. --SimonMichael, Wed, 20 Aug 2003 00:49:43 +0000 reply
Aha. 0.9.8 didn't do prerendering or prelinking, so pages took up quite a bit less space. Modern Zwiki stores this cached information as attributes on every page:

    _prerendered = ''   # cached partially rendered text
    _prelinked = []     # cached links & rendered text regions
    _links = []         # cached unique links

A good insight.

how expensive is showing the page's parentage? --BillSeitz, Tue, 09 Sep 2003 06:59:02 -0700 reply
Because for each generation of the hierarchy it has to go grab the parents...

If you give an engineer a feature... --DeanG, Thu, 30 Oct 2003 12:28:19 -0800 reply
A "development" or "administration" header would be quite handy, including:

Totally wrong calculation -- JensN? -- Tue, 31 Aug 2004 12:09:23 -0700 reply
Multiplying the memory with every thread is not right according to http://zope.org/Members/tseaver/Projects/HighlyAvailableZope/BleedingLikeAStuckPig Thus, Mmax ~= Z + T (P S + O) is wrong but rather Mmax ~= Z + (P * S + O) should be right.

fresh data --Simon Michael, Sun, 21 Nov 2004 14:28:59 -0800 reply
14-day zodb pack: 850 -> 200Mb.

I have been looking at memory and hit logs for the last few days. NB in these discussions I will use Mb to mean 1000Kb, generally.

Restarts are often triggered by jumps of 30Mb or more in memory usage, due to zwiki.org/SearchPage?; because of a bug preventing use of catalog, a search loads all pages into zodb cache. (30Mb for 2000 page objects means an average of 15K each; remember pages hold prerendered data as well as raw source text.)

This made it clear that restarts were resulting just from normal traffic (base memory usage + 3 threads x (30Mb for all pages in cache + overhead for catalog etc.) exceeds the 150Mb memory limit I had most recently set). As a quick fix I have reduced the number of threads from 3 to 2.

Now after warming up zwiki.org and zopewiki.org, the two big wikis on this server (by doing full recentchanges and a non-catalog search in both threads), I can get memory usage up to 140Mb where it stays. I have increased the restart limit to 180Mb. To do:

Totally wrong calculation -- JensN? --simon, Sun, 21 Nov 2004 14:43:16 -0800 reply
That document is talking about the linux process sizes reported by top. It would be wrong to add all those up. Here, I am adding up numbers for the zope threads, each of which keeps its own zodb cache in ram, and which all together add up to the number shown in top.

fresh data --Bob McElrath?, Mon, 22 Nov 2004 01:37:32 -0800 reply
Simon Michael [simon@joyful.com]? wrote:

14-day zodb pack: 850 -> 200Mb.

I have been looking at memory and hit logs for the last few days. NB in these discussions I will use Mb to mean 1000Kb, generally.

Restarts are often triggered by jumps of 30Mb or more in memory usage, due to zwiki.org/SearchPage?; because of a bug preventing use of catalog, a search loads all pages into zodb cache. (30Mb for 2000 page objects means an average of 15K each; remember pages hold prerendered data as well as raw source text.)

So my new stx will store pre-rendered pages as a list of substrings, wikilinks and non-wikilinks.

One could seriously reduce pre-rendered data by storing the offsets and lengths of wikilinks.

Would this be interesting from a memory-usage perspective? The page text would have to be loaded anyway to do the rendering...

fresh data --simon, Tue, 23 Nov 2004 09:11:23 -0800 reply
I increased the limit to 200Mb to give it more room. It's regularly growing to that size; see the graph below. Here are things that seem to be responsible for some of these jumps:

Next, I'll set a lower zodb cache target size, so that it will flush out some old objects in between requests. After watching that for a bit I'll fix SearchPage?. Also I'll get LeakFinder? and ForensicLogger? working.

fresh data --simon, Thu, 25 Nov 2004 10:38:50 -0800 reply
Memory usage for the last two days. I have a makefile which runs the above actions to try and prime the cache, so you'll sometimes see an abrupt spike after startup; usually it's more gradual. At some point I changed the target cache size from unlimited to 5000 objects; this seemed to hold it at just under 200Mb (coincidentally). Last night I reduced this to 4000 objects, and you can see the lower steady state. In both cases a SearchPage? call eventually causes a spike beyond that. It's hard to be sure whether this is a memory leak, or just normal expansion of some cache or other (there's a zeo client cache which as far as I can tell is separate from the zodb object cache but I'm not clear).