Edit detail for #608 saving international characters in ReStructuredText mode gives unicode error revision 1 of 1

1
Editor: betabug
Time: 2007/02/21 08:20:35 GMT+0
Note:

changed:
-
 From RestructuredText:

RuedigerBlach wrote:
 > Problem: ``edit`` with German "Umlaute" ==> ``save`` ==> Unicode Error
 >
 > Is this a problem of !ZWiki or Zope or Python? Any solution?

Suggestion for Fix:

 Try putting a file called "sitecustomize.py" into $ZOPEDIR/bin/lib.
  It should contain:
   import sys
   sys.setdefaultencoding("iso8859-1")
 that should help with the problem. At least it worked with my Windows installation.

  ThomasSprinzing



[email protected] wrote:
<br>><i> This is a problem with your Python setup in the first place. You need</i>
<br>><i> to convince Unicode to use a defaultencoding which is not ASCII. see</i>
<br>><i> /etc/python2.1/site.py , line 252, above # Enable to support locale</i>
<br>><i> aware default string encodings. The other problem is Zope related or,</i>
<br>><i> well, maybe one could say that the reStructuredText integration could</i>
<br>><i> be a little bit tighter. At least I didn't have such problems with</i>
<br>><i> StructuredText. To get your umlauts for Zope, you need to install a</i>
<br>><i> proper locale on your Linux box, and then make sure that Zope is</i>
<br>><i> started with an option like this: "-L de_DE" Edit /etc/zope/zopectlrc,</i>
<br>><i> locate the comment telling you how to add options to ZOPEOPTS and add</i>
<br>><i> this: ZOPEOPTS="${ZOPEOPTS} -L de_DE"</i>

From SimonMichael Thu Feb 26 20:02:06 -0800 2004
From: SimonMichael
Date: Thu, 26 Feb 2004 20:02:06 -0800
Subject: property change
Message-ID: <[email protected]>

Category: zwiki: general => general 


From unknown Thu Mar 4 09:58:35 -0800 2004
From: 
Date: Thu, 04 Mar 2004 09:58:35 -0800
Subject: Fiddling with default string is not the way to do it
Message-ID: <[email protected]>

I believe the consensus on both Zope 2 and Zope 3 mailing lists is that Python applications should not depend on sys.setdefaultencoding.  Instead the proper encoding needs to be figgured out and used explicitly.

I believe that ZWiki can get away with using UTF-8 everywhere.  As a rule web browsers submit forms in the same encoding they received the original web page in, and ZWiki sets UTF-8 as the charset on all pages.

From unknown Thu Mar 4 10:36:09 -0800 2004
From: 
Date: Thu, 04 Mar 2004 10:36:09 -0800
Subject: Using UTF-8 for RestructuredText
Message-ID: <[email protected]>

Here's a patch.  It is tested.

<pre>
--- PageTypes.py.orig   Thu Mar  4 20:26:24 2004
+++ PageTypes.py        Thu Mar  4 20:26:44 2004
@@ -294,7 +294,8 @@
<!-- empty lines break things -->
     def renderRstIn(self,t):
         if reStructuredText:
-            return reStructuredText.HTML(t,report_level=0) # doesn't work:(
+            return reStructuredText.HTML(t,report_level=0, # doesn't work:(
+                        input_encoding="UTF-8",output_encoding="UTF-8")
         else:
             return "&lt;pre>Error: could not import reStructuredText&lt;/pre>\n"+t
</pre>


From unknown Sun Jul 4 06:20:43 -0700 2004
From: 
Date: Sun, 04 Jul 2004 06:20:43 -0700
Subject: Using UTF-8 for RestructuredText
Message-ID: <[email protected]>
In-Reply-To: <[email protected]>

I encountered the same problem, and it took me a little research until I found your patch, which solved the problem. The patch seems to be available already since March. Could we incorporate it into the !ZWiki-Distribution? I guess it would make sense to do that, wouldn't it? Benedikt

From simon Sun Jul 4 08:44:53 -0700 2004
From: simon
Date: Sun, 04 Jul 2004 08:44:53 -0700
Subject: Using UTF-8 for RestructuredText
Message-ID: <[email protected]>
In-Reply-To: <[email protected]>

We do hard-code utf-8 in a number of other places, but we probably shouldn't. Wouldn't setting the RST encoding in zope.conf be equivalent and better ? Please let us know if that works.

From unknown Mon Jul 19 08:03:34 -0700 2004
From: 
Date: Mon, 19 Jul 2004 08:03:34 -0700
Subject: Using UTF-8 for RestructuredText
Message-ID: <[email protected]>
In-Reply-To: <[email protected]>

Yes, it works. I agree that it is a better way than hardcoding. Benedikt

From unknown Sun Jan 30 08:51:59 -0800 2005
From: 
Date: Sun, 30 Jan 2005 08:51:59 -0800
Subject: property change
Message-ID: <[email protected]>

Category: user-pagetypes => user-editing-rst 


From stefanrank Sat Apr 2 01:34:39 -0800 2005
From: stefan rank
Date: Sat, 02 Apr 2005 01:34:39 -0800
Subject: is it ok if I close this? (it's a zope.conf configuration issue)
Message-ID: <[email protected]>

Status: open => closed 


From betabug Wed Feb 21 08:20:35 +0000 2007
From: betabug
Date: Wed, 21 Feb 2007 08:20:35 +0000
Subject: 
Message-ID: <[email protected]>

Name: '#608 chancey' => '#608 saving international characters in ReStructuredText mode gives unicode error' 
Category: general-docs => user-editing-rst 
Severity: wishlist => normal 
Status: open => closed 


Submitted by : simon at: 2003-10-26T21:32:36+00:00 (17 years ago)
Name :
Category : Severity : Status :
Optional subject :  
Optional comment :

From RestructuredText:
RuedigerBlach wrote:
> Problem: edit with German "Umlaute" ==> save ==> Unicode Error > > Is this a problem of ZWiki or Zope or Python? Any solution?

Suggestion for Fix:

Try putting a file called "sitecustomize.py" into $ZOPEDIR/bin/lib.
It should contain:
import sys sys.setdefaultencoding("iso8859-1")

that should help with the problem. At least it worked with my Windows installation.

ThomasSprinzing

zwiki-wiki@zwiki.org wrote: <br>><i> This is a problem with your Python setup in the first place. You need</i> <br>><i> to convince Unicode to use a defaultencoding which is not ASCII. see</i> <br>><i> /etc/python2.1/site.py , line 252, above # Enable to support locale</i> <br>><i> aware default string encodings. The other problem is Zope related or,</i> <br>><i> well, maybe one could say that the reStructuredText integration could</i> <br>><i> be a little bit tighter. At least I didn't have such problems with</i> <br>><i> StructuredText. To get your umlauts for Zope, you need to install a</i> <br>><i> proper locale on your Linux box, and then make sure that Zope is</i> <br>><i> started with an option like this: "-L de_DE" Edit /etc/zope/zopectlrc,</i> <br>><i> locate the comment telling you how to add options to ZOPEOPTS and add</i> <br>><i> this: ZOPEOPTS="${ZOPEOPTS} -L de_DE"</i>

property change --SimonMichael, Thu, 26 Feb 2004 20:02:06 -0800 reply

Category: zwiki: general => general

Fiddling with default string is not the way to do it --Thu, 04 Mar 2004 09:58:35 -0800 reply

I believe the consensus on both Zope 2 and Zope 3 mailing lists is that Python applications should not depend on sys.setdefaultencoding. Instead the proper encoding needs to be figgured out and used explicitly.

I believe that ZWiki can get away with using UTF-8 everywhere. As a rule web browsers submit forms in the same encoding they received the original web page in, and ZWiki sets UTF-8 as the charset on all pages.

Using UTF-8 for RestructuredText --Thu, 04 Mar 2004 10:36:09 -0800 reply

Here's a patch. It is tested.

<pre> --- PageTypes?.py.orig Thu Mar 4 20:26:24 2004 +++ PageTypes?.py Thu Mar 4 20:26:44 2004 @@ -294,7 +294,8 @@ <!-- empty lines break things -->

def renderRstIn(self,t):
if reStructuredText:
  • return reStructuredText.HTML(t,report_level=0) # doesn't work:(
  • return reStructuredText.HTML(t,report_level=0, # doesn't work:(
  • input_encoding="UTF-8",output_encoding="UTF-8")
else:
return "&lt;pre>Error: could not import reStructuredText&lt;/pre>n"+t

</pre>

Using UTF-8 for RestructuredText --Sun, 04 Jul 2004 06:20:43 -0700 reply

I encountered the same problem, and it took me a little research until I found your patch, which solved the problem. The patch seems to be available already since March. Could we incorporate it into the ZWiki-Distribution? I guess it would make sense to do that, wouldn't it? Benedikt

Using UTF-8 for RestructuredText --simon, Sun, 04 Jul 2004 08:44:53 -0700 reply

We do hard-code utf-8 in a number of other places, but we probably shouldn't. Wouldn't setting the RST encoding in zope.conf be equivalent and better ? Please let us know if that works.

Using UTF-8 for RestructuredText --Mon, 19 Jul 2004 08:03:34 -0700 reply

Yes, it works. I agree that it is a better way than hardcoding. Benedikt

property change --Sun, 30 Jan 2005 08:51:59 -0800 reply

Category: user-pagetypes => user-editing-rst

is it ok if I close this? (it's a zope.conf configuration issue) --stefan rank, Sat, 02 Apr 2005 01:34:39 -0800 reply

Status: open => closed

... --betabug, Wed, 21 Feb 2007 08:20:35 +0000 reply

Name: '#608 chancey' => '#608 saving international characters in ReStructuredText? mode gives unicode error' Category: general-docs => user-editing-rst Severity: wishlist => normal Status: open => closed