Submitted by : wlang at: 2007-04-10T08:11:52+00:00 (10 years ago)
Name :
Category : Severity : Status :
Optional subject :  
Optional comment :

After upgrading from zwiki 0.53 to 0.59 (Zope-2.7.7) i get an UnicodeDecodeError at almost every page i try:

Site Error

An error was encountered while publishing this resource.

UnicodeDecodeError

Traceback (innermost last):

  * Module ZPublisher.Publish, line 180, in publish_module_standard
  * Module ZPublisher.Publish, line 131, in publish
  * Module Zope.App.startup, line 204, in zpublisher_exception_hook
  * Module ZPublisher.Publish, line 101, in publish
  * Module ZPublisher.mapply, line 88, in mapply
  * Module ZPublisher.Publish, line 39, in call_object
  * Module Products.ZWiki.ZWikiPage, line 253, in __call__
  * Module Products.ZWiki.ZWikiPage, line 266, in render
  * Module Products.ZWiki.pagetypes.rst, line 60, in render
  * Module Products.ZWiki.Views, line 697, in addSkinTo
  * Module Shared.DC.Scripts.Bindings, line 306, in __call__
  * Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec
  * Module Products.PageTemplates.ZopePageTemplate, line 222, in _exec
  * Module Products.PageTemplates.PageTemplate, line 96, in pt_render
    <ZopePageTemplate at /wikis/wikipage used for /wikis/BachDocs/DvuBesprechung20070329>
  * Module TAL.TALInterpreter, line 189, in __call__
  * Module TAL.TALInterpreter, line 233, in interpret
  * Module TAL.TALInterpreter, line 670, in do_useMacro
  * Module TAL.TALInterpreter, line 233, in interpret
  * Module TAL.TALInterpreter, line 408, in do_optTag_tal
  * Module TAL.TALInterpreter, line 393, in do_optTag
  * Module TAL.TALInterpreter, line 388, in no_tag
  * Module TAL.TALInterpreter, line 233, in interpret
  * Module TAL.TALInterpreter, line 693, in do_defineSlot
  * Module TAL.TALInterpreter, line 233, in interpret
  * Module TAL.TALInterpreter, line 408, in do_optTag_tal
  * Module TAL.TALInterpreter, line 393, in do_optTag
  * Module TAL.TALInterpreter, line 388, in no_tag
  * Module TAL.TALInterpreter, line 233, in interpret
  * Module TAL.TALInterpreter, line 670, in do_useMacro
  * Module TAL.TALInterpreter, line 233, in interpret
  * Module TAL.TALInterpreter, line 408, in do_optTag_tal
  * Module TAL.TALInterpreter, line 393, in do_optTag
  * Module TAL.TALInterpreter, line 388, in no_tag
  * Module TAL.TALInterpreter, line 233, in interpret
  * Module TAL.TALInterpreter, line 636, in do_condition
  * Module TAL.TALInterpreter, line 233, in interpret
  * Module TAL.TALInterpreter, line 262, in do_startEndTag
  * Module TAL.TALInterpreter, line 290, in do_startTag
  * Module TAL.TALInterpreter, line 356, in attrAction_tal
  * Module Products.PageTemplates.TALES, line 227, in evaluateText
  * Module Products.PageTemplates.TALES, line 221, in evaluate
    URL: head
    Line 47, Column 4
    Expression: standard:'here/summary'
    Names:

    {'container': <Folder instance at be61b260>,
     'context': <ZWikiPage 'DvuBesprechung20070329' at 0xbef27ca0>,
     'default': <Products.PageTemplates.TALES.Default instance at 0xbef8cf0c>,
     'here': <ZWikiPage 'DvuBesprechung20070329' at 0xbef27ca0>,
     'loop': <SafeMapping instance at be5e4150>,
     'modules': <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter instance at 0xbef923cc>,
     'nothing': None,
     'options': {'args': (<ZWikiPage 'DvuBesprechung20070329' at 0xbef27ca0>,
                          <HTTPRequest, URL=https://bach.wu-wien.ac.at/11080/wikis/BachDocs/DvuBesprechung20070329>),
                 'body': '<div class="document">\n<p>Am 29.3.2007 fand eine DVU-Besprechung und ein Erfahrungsaustausch mit den Univerist\xe4ten,...</div>\n'},
     'repeat': <SafeMapping instance at be5e4150>,
     'request': <HTTPRequest, URL=https://bach.wu-wien.ac.at/11080/wikis/BachDocs/DvuBesprechung20070329>,
     'root': <Application instance at be8002c0>,
     'template': <ZopePageTemplate at /wikis/wikipage used for /wikis/BachDocs/DvuBesprechung20070329>,
     'traverse_subpath': [],
     'user': Anonymous User}

  * Module Products.PageTemplates.Expressions, line 201, in __call__
  * Module Products.PageTemplates.Expressions, line 196, in _eval
  * Module Products.PageTemplates.Expressions, line 103, in render
  * Module Products.ZWiki.Utils, line 143, in summary

UnicodeDecodeError

Reason: Patch #1303 (Illegal utf-8 handling in Utils.py) assumes that the wiki string is coded in ascii or utf-8, which may or may not be true. The wiki string could also be latin-1 (as in our case).

The bug is easy to see in zope debug mode:

zope@bach-s17:~/InstanceHomeDevel$ ./bin/zopectl debug

>>> app.wikis.BachDocs.SamplePage
<ZWikiPage 'SamplePage' at 0xbeeddcd0>

>>> app.wikis.BachDocs.SamplePage.raw
'Universit\xe4t'

>>> app.wikis.BachDocs.SamplePage.summary()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/local/zope/Zope-2.7.2devel/lib/python/Products/ZWiki/Utils.py", line 143, in summary
    t = t.decode(charset)  # go to unicode.
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 9-10: unexpected end of data

Unfortunately i do not know, what's the right fix for this issue. Is there a way to determine the encoding of the wiki page?

I worked around it with following patch:

zope@bach-s17:.../Products/ZWiki$ diff -u Utils.py.ori  Utils.py
--- Utils.py.ori        2007-04-02 20:54:20.000000000 +0200
+++ Utils.py 2007-04-06 23:22:30.000000000 +0200
@@ -129,7 +129,7 @@
         strings = filter(lambda x:type(x)==type(''), strings)
         return len(join(strings,''))

-    def summary(self,size=200,paragraphs=1,charset="utf-8"):
+    def summary(self,size=200,paragraphs=1,charset="latin-1"):
         """
         Give a short plaintext summary of this page's content.

But that works obviously only for latin-1 sites and of course cannot be the real fix!

utf-8 is hardcoded into modern zwikis --betabug, Tue, 10 Apr 2007 09:22:02 +0000 reply

See ReleaseNotes: "If you use a character encoding other than utf-8: zwiki has been hard-coding utf-8 since 0.31 - see IssueNo0855?." -- that was way back in 0.32. #855 0.31 hard-codes utf-8, ignores cmf/plone's default_charset

I expect you will run into more trouble with your iso-8859-1 wiki in the long run. I suggest converting the contents of the wiki to utf-8, I believe there have been instructions to do this somewhere on zwiki.org.

for conversion see... --betabug, Tue, 10 Apr 2007 09:56:08 +0000 reply

#944 Need an automatic method to convert from old encoding (iso-8859-1) to UTF-8

... --EmmaLaurijssens, Mon, 16 Apr 2007 10:53:59 +0000 reply

Category: general => admin-installing Severity: serious => normal Status: open => closed

See the script at #944. Also, don't forget to set the rest-* encoding settings in your zope.conf.