Submitted by : simon at: 2003-10-26T21:31:37+00:00 (14 years ago)
Name :
Category : Severity : Status :
Optional subject :  
Optional comment :

Summary: Python on FreeBSD? tends to crash a lot when running zope, zwiki, plone etc. You can fix it by rebuilding your python with a larger thread stack size. 1Mb (1<<20) is recommended.

Patching will be done by a installing a recent Python 2.1.3 ports version. See PatchPython213 for details.

Alternate quick workaround: if you get this when saving a particular page, break it up into smaller pieces. Packing your zodb and restarting zope may also help a little.

This can be caused by different things; use the zope event log to see what error is printed as zope crashes (EventLog). This page deals with the usual cause, a known python/python-freebsd bug (python stack overflow). See also #162 browsing diffs & some other things cause zope to crash with Aieee! error code 11 (SIGSEGV).

See also ZopeIssue:560, PythonBug?:?, [[#395 zwiki's regular expressions may fail with large pages/sites].


2002/10/04 10:15 GMT (via web):
I'm encountering this problem consistently - following links such as UserOptions, Recent Changes etc. in the ZWiki footer.

My environment is:

Zope version: (Zope 2.5.1 (source release, python 2.1, linux2), python 2.1.3, freebsd4) 
Python version: 2.1.3 (#1, Aug 19 2002, 14:43:12) [GCC 2.95.3 20010315 (release) [FreeBSD]] 
ZWiki-0.11.0

The STUPID_LOG_FILE entries at the point of crash are:

2002-10-04T11:24:25 INFO(0) ZServer PCGI Server started at Fri Oct  4 12:24:25 2002
        Unix socket: /usr/local/www/Zope/var/pcgi.soc
------
2002-10-04T11:25:19 ERROR(200) zdaemon zdaemon: Fri Oct  4 12:25:19 2002: Aiieee! 35894 exited with error code: 11
------
2002-10-04T11:25:19 INFO(0) zdaemon zdaemon: Fri Oct  4 12:25:19 2002: Houston, we have forked
------
2002-10-04T11:25:19 INFO(0) zdaemon zdaemon: Fri Oct  4 12:25:19 2002: Hi, I just forked off a kid: 35899
------
2002-10-04T11:25:19 INFO(0) zdaemon zdaemon: Fri Oct  4 12:25:19 2002: Houston, we have forked

Any Clues?

JimW?


2002/10/10 15:34 GMT (via web):
I really haven't looked into this, but it might be a problem with your FreeBSD? Python. See also: http://lists.zope.org/pipermail/zope/2002-October/124478.html and http://www.zopezen.org/Members/zopista/News_Item.2002-10-09.4129

-- PieterB


2002/10/10 15:36 GMT (via web):
Do you use Python from the ports-collection or Python from source?

It might be a good idea to give a create a hyperlink between http://collector.zope.org/Zope/560 and this page. I cannot create a comment on the Zope collector, for some stupid reason.

-- PieterB


Bob Finch, 2002/10/18 05:20 GMT (via web):
I had the same problem as JimW; clicking on UserOptions would crash zope every time. I was using the FreeBSD python-2.1.3 package.

The following patch to Python/thread_pthread.h seems to fix the problem. It's based on the patch PieterB referenced above, but properly handles the case where PTHREAD_SYSTEM_SCHED_SUPPORTED is undefined:

  *** /usr/ports/distfiles/python/Python-2.1.3/Python/thread_pthread.h  Thu Jan 10 03:12:20 2002
  --- work/Python-2.1.3/Python/thread_pthread.h Thu Oct 17 21:59:48 2002
  ***************
  *** 128,141 ****
    {
        pthread_t th;
        int success;
  - #ifdef PTHREAD_SYSTEM_SCHED_SUPPORTED
        pthread_attr_t attrs;
  ! #endif
            dprintf(("PyThread_start_new_thread called\n"));
        if (!initialized)
                PyThread_init_thread();
  - #ifdef PTHREAD_SYSTEM_SCHED_SUPPORTED
        pthread_attr_init(&attrs);
        pthread_attr_setscope(&attrs, PTHREAD_SCOPE_SYSTEM);
    #endif

  --- 128,141 ----
    {
        pthread_t th;
        int success;
        pthread_attr_t attrs;
  ! 
            dprintf(("PyThread_start_new_thread called\n"));
        if (!initialized)
                PyThread_init_thread();
        pthread_attr_init(&attrs);
  +     pthread_attr_setstacksize (&attrs, (1<<17));
  + #ifdef PTHREAD_SYSTEM_SCHED_SUPPORTED
        pthread_attr_setscope(&attrs, PTHREAD_SCOPE_SYSTEM);
    #endif

  ***************
  *** 153,163 ****
                                 func,
                                 arg
    #elif defined(PY_PTHREAD_STD)
  - #ifdef PTHREAD_SYSTEM_SCHED_SUPPORTED
                                 &attrs,
  - #else
  -                              (pthread_attr_t*)NULL,
  - #endif
                                 (void* (*)(void *))func,
                                 (void *)arg
    #endif
  --- 153,159 ----

To install it, I did the following:

  1. cd /usr/ports/lang/python21
  2. make extract
  3. Apply the patch to work/Python-2.1.3/Python/thread_pthread.h
  4. make
  5. make install FORCE_PKG_REGISTER=1

-- Bob


SimonMichael, 2002/10/18 21:18 GMT (via web):
Thanks for this Bob. I'd love to try it here but haven't had time to learn enough freebsd. I get no such directory at step 1, any clues would be appreciated.


Bob Finch, 2002/10/18 22:19 GMT (via web):
Simon, sounds like you don't have the FreeBSD? ports collection installed. You can find instructions on how to install the FreeBSD? ports collection in the FreeBSD? Handbook, Chapter 4.5 Using the Ports Collection:

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-using.html

You might also want to check that the python21 port you're building is Python 2.1.3. To verify this, look at the PORTVERSION line in /usr/ports/lang/python21. If it isn't 2.1.3, you should update your ports collection via cvsup. Instructions are in the Handbook chapter I referenced above.

-- Bob


Bob Finch, 2002/10/18 23:13 GMT (via web):
It might also be that you've got an older copy of the ports collection. If /usr/ports/lang/python15 exists, that's probably the case. Follow the instructions in the handbook to update your ports collection.


2002/11/01 23:03 GMT (via web):
For those of you who don't like diff (just like me), try to use:

http://gewis.nl/~pieterb/zwiki/thread_pthread.h.txt (patched file for Python 2.1.3 on FreeBSD?)

-- PieterB


2002/11/01 23:36 GMT (via web):
For those who have problems downloading the ports can also install: http://gewis.nl/~pieterb/zwiki/usr-ports-lang-python21.tgz (14Kb) in /usr/ports/langs/python21

Then use BobFinch's recipe to get Python 2.1.3 patched.

I just e-mailed the maintainer of the FreeBSD? port. I hope it will be installed as default, because that would make Zope/Zwiki life much easier.

-- PieterB


SimonMichael, 2002/11/02 15:09 GMT (via web):
Added "on freebsd" to this bug's title.


SimonMichael, 2002/11/02 17:59 GMT (via web):
Pieter extracted the above patch & instructions to PatchPython213. This is now installed on this server, but I'm still seeing SIGSEGV and others: SIGBUS, SIGILL etc when I save a long discussion page. I've tried increasing the stack size as far as 1<<20 (1Mb), didn't help.

I think I'm applying the patch correctly - I'm doing:

 # cd /usr/ports/lang/python21
 # make clean
 # make extract
 # cp sm-thread_pthread.h work/Python-2.1.3/Python/thread_pthread.h
 # make
 # make install FORCE_PKG_REGISTER=1
 # (restart zope) 


PieterB, 2002/11/02 19:17 GMT (via web):
Simon: please post (or send me) the sm-thread_pthread.h

Here it is:

sm-thread_pthread.h


DeanGoodmanson, 2002/11/15 06:43 GMT (via web):
After editing a large page twice, I got a proxy error and then (on next attempt) : KeyError?: 154599424 (No trace in source)

Is this related? Sorry if this isn't the general crashes issue I intended to add it to...


JustinClift, 2002/11/17 14:05 GMT (via web):
If it's any help, the Zwiki 0.12.0 + Zope 2.6.0 + PatchPython213 setup here (http://techdocs.postgresql.org/guides) hasn't crashed at all over the last fortnight.

Not sure if that's just because we're still implementing things and it may not have been tested well enough.

Regards and best wishes,

Justin Clift

2002/12/11 10:42 GMT (via web):
Progress!

I stayed up past my bedtime and made progress with this. The build we're running now should be less prone to these crashes.

Recap: I searched the python and freebsd bug databases and didn't find the issue. I learned a lot about the zope startup scripts on this server, and got a backtrace of a segfault with gdb: it was deep in a recursion in the re code, which made sense. I found a particular diff that always triggered the crash, when viewed on a live zope server (calling it in the debugger didn't crash). Using debug log statements I narrowed it down to the first regexp in diff, which is apparently expensive. Double-checked my python patch, made sure zope was running the rebuilt python, and got it working. Experimentation showed that the default 64k and 128, 256, 512k stack sizes failed but 1M worked. I don't know why I didn't see it work before, maybe it's borderline. So far no crashes since. I don't know if this is a python or an re bug, or what exactly the correct behaviour should be - warn of insufficient stack space for regexp, ideally. I wonder what the stack size is on linux etc. and if they can also experience this. We should watch for remaining crashes and also review/optimize all the zwiki regexps (and maybe reduce this server's stack size again). I wonder how to measure how expensive each regexp is. --SM

DeanGoodmanson, 2002/12/11 15:01 GMT (via web):
Can you point out the problematic page/diff so we can try it on other platforms? I'm particularly interested to see if OS X is plagued with this problem. Would an export retain the diff info?

Simon Michael, 2002/12/11 18:46 GMT (via mail):
Re: [on FreeBSD?, python may crash repeatably when viewing or saving certain wiki pages]

zwiki@zwiki.org (DeanGoodmanson) writes:
> Can you point out the problematic page/diff so we can try it on other
> platforms? I'm particularly interested to see if OS X is plagued with
> this problem. Would an export retain the diff info?

Don't know about that, plus it's not purely data dependent, other factors (number of pages/zodb size/... ?) affect it. I used to be able to reproduce it very quickly by stepping back through the diffs of different pages, so try that. Also try posting a lot of mixed text to a page, eg a couple of months of GeneralDiscussion would trigger it quickly. Also FWIW I was getting it from posting the exact source (html & body tags included) of apache's server-status output.

It should be possible to boil these down into some very small regexp tests which you could run at the command line to get some idea of your vulnerability. These might exist in the python bug tracker already.

Ah, a little more insight on [[#395 zwiki's regular expressions may fail with large pages/sites] zwiki's regular expressions may fail with large pages/sites].

More on crash testimiony from OS X point of view

Last few days I've experienced similar crashes on Zwiki 0.11.0rc1, SPVI Zope 2.5.1 on OS X. Database: 85M, in need of packing to usual ~25M.

  1. Create a page, add a huge chunk of Word Generated HTML html & body tags incl. Later added without and no crash.
  2. Browsing through the diff of a 48K page.

Simon Michael, 2003/01/19 23:24 GMT (via mail):
test mailin to long page names

Just a little test.. if this does not to go to #226 on freebsd, python/zope may crash repeatably when browsing diffs or saving certain pages, we still have #394 can't mail in to long freeform page names ? to deal with.

closing --SimonMichael, 2003/02/24 09:59 GMT
This has been well and truly solved.

python binary --2003/06/13 13:54 GMT
BillSeitz is having trouble with this over on GeneralDiscussion. I am uploading the freebsd python binary I built for this site (stack size 1<<20), which may work for other imeme or freebsd users.

python.bigstack

From: BillSeitz Date: 2003/06/16 14:28 GMT Subject: how use python.bigstack?

Do I copy this to /bin/ and then rename it to python? (after killall)

From: BillSeitz Date: 2003/06/16 20:02 GMT Subject: actually, the .bigstack file doesn't seem to exist

or, at least, I end up with an empty file.


comments:

property change --simon, Wed, 18 Feb 2004 13:35:04 -0800 reply
Title: IssueNo0226 on FreeBSD, python may crash repeatably when viewing or saving certain wiki pages => IssueNo0226 on freebsd, python/zope may crash repeatably when browsing diffs or saving certain pages

a fix for diff browsing --simon, Wed, 18 Feb 2004 14:00:21 -0800 reply
This ancient code was using horrible regexps, they are now gone. This means diff browsing should no longer trigger this bug. Saving large pages will still do it, but that is more rare. This should make running zwiki on freebsd less annoying.

a fix for diff browsing -- Mon, 24 May 2004 04:02:33 -0700 reply
It seems I have come across something similar in Zope 2.7.0 / Python 2.3.3 (from source) FreeBSD? 5.2.1 and mxODBCZopeDA : doing a test database query leads to a segmentation fault and core dump. The p_thread code no longer looks quite the same for the patch. Ideas?

Regards Charlie Clark charlie at egenix.com