For discussion of WikiMail and related issues.


comments:

mailin access -- Sun, 11 Jan 2004 14:36:10 -0800 reply
I've been attempting to integrate ZWiki and Mailman as described in ZwikiAndMailman and ZwikiAndMailingLists. I have an email alias that's forwarding messages via curl as described. My problem is, they only show up on GeneralDiscussion if I allow anonymous users View and ZWiki:Add Comments to Pages permission. I've tried adding the messages' "From" address to the subscriber_list and the mail_from property. I've tried setting posting_policy (and then mailin_policy) to open, and when that didn't work, I've set it to closed and added the "From" address to a mail_accept_nonmembers property. Nothing seems to work.

How can I get the messages redirected by curl to post to GeneralDiscussion while barring anonymous users from Viewing and Adding comments?

I'm using Zwiki 0.22.0 and Zope 2.6.3 on freezope.org.
Thanks.

mailin access --Simon Michael, Mon, 12 Jan 2004 01:52:33 -0800 reply
The curl process must have permission to add comments. You can use it's -u option, or better -n which allows you to keep the name and password more securely in your ~/.netrc file. Give it a login with Access contents information, View and Zwiki: Add comments permissions, probably.

mailout stopped working --Michael Sanborn, Fri, 16 Jan 2004 10:05:51 -0800 reply
Well, I had everything working perfectly for a few days there. But the night before last, the wiki stopped mailing out changes to subscribers. I don't know what I would have changed to cause that. Since I'm on freezope, I don't think I have access to the file system to check logs, etc. Any hints as to what I could do to debug the problem?

oh... --Michael Sanborn, Sun, 18 Jan 2004 16:14:57 -0800 reply
It turns out it was a problem on the server end, having nothing to do with the wiki. Neverrr minnnd...

using procmail for robust wikimail --simon, Wed, 21 Jan 2004 14:26:39 -0800 reply
There are two known issues that prevent Zwiki-only mail handling being reliable - 1. incoming mail will be dropped if the zope server is down and 2. outgoing mail can be duplicated due to ZODB conflict resolution retries. Until now, to give a wiki robust mail handling I've always added a mailman list. But this is complicated, especially if you don't need all the mailman features. I'd like us to have a standard easy-to-set-up procmail configuration which can solve problems 1 and 2 in a simpler way.

Pieter worked on this at WikiMailNG but I can't find the solution he posted - has anyone seen it ?

Re: using procmail for robust wikimail --PieterB, Wed, 21 Jan 2004 15:29:22 -0800 reply
I had a harddisk crash during last Christmas. Data is saved in a backup, but I didn't have time to put it back online. I'll have a look. If anybody has it, please let me know. Otherwise I'll try to dig it up.

Re: using procmail for robust wikimail --simon, Wed, 21 Jan 2004 18:15:30 -0800 reply
I was pretty sure you had posted it, but I don't see it among the FileUploads? or on any of the obvious wiki pages. Darn.

procmailrc (alpha) --PieterB, Thu, 22 Jan 2004 00:54:28 -0800 reply
I found a wikimail-dir with procmailrc, and zipped it at: http://gewis.nl/~pieterb/zwiki/wikimail.zip I also put back my zwiki-dir, so that the picture at WikiMailNG works again. I don't have time to do more wiki-hacking at this moment, but I'll hope you can use my prototypes. I don't know if these were the latest versions, but I don't think other versions are saved (on my system).

procmailrc (alpha) --Simon Michael, Thu, 22 Jan 2004 06:08:56 -0800 reply
Great, thanks Pieter.

strange digest header --DeanGoodmanson, Wed, 17 Mar 2004 22:32:38 -0800 reply
I hope everybody isn't seeing my name, but are they seeing theres? :

 zwiki Digest 18 Mar 2004 04:59:24 -0000 Issue 6
Topics (messages 86 through 109):
[GeneralDiscussion] BlogMethod calling
        86 by: DeanGoodmanson
[MoinMoinMarkupMode] Changes to ZWikiPage.py
        87 by: DeanGoodmanson
[StructuredTextSucks] (new) Headers in StructuredText
        88 by: DeanGoodmanson
        89 by: DeanGoodmanson

Mailout on seperate thread --DeanGoodmanson, Wed, 12 May 2004 08:49:19 -0700 reply
Was this implemented in a recent version? Is there a workaround/alternative to MailHost?

In the event of a busy SMTP server, the lag in updating a page while waiting for the mail to be sent out, can get quite slow.

Mailout on seperate thread --simon, Wed, 16 Jun 2004 20:31:24 -0700 reply
SecureMailHost? has async support, but I don't know how to enable it.

There is an older solution that zope.org used to use, whose name I can't remember; that one uses an external process as a helper.

wiki mail change --Simon Michael, Sat, 10 Jul 2004 19:56:18 -0700 reply
Resending, this time cc'ing the wiki as intended:

As proposed in recent GeneralDiscussion (thoughts on discussion ), mailed-in messages with no bracketed page name in the subject are now discarded by default. This is the start of a new phase in WikiMail: list traffic goes to the wiki only if you make it do that.

The thinking is that a wiki is an expensive place to keep reams of old discussion, most of it never used again, but chewing up memory, slowing down renames etc. Better to leave most of that in mail list archives and gmane, and keep the wiki for more permanent and high-value content.

This may seem obvious, but the experiment was worthwhile. The simplicity of the mail = wiki setup (at least conceptually) may still have it's place.

de-emphasize Discussion pages ? --simon, Sat, 10 Jul 2004 21:54:18 -0700 reply
Does this mean Discussion pages could/should be de-emphasized or go away ?

Maybe. DevDiscussion? could probably go first, replaced by the list. GeneralDiscussion could be kept with an explanation that it's for drop-in, new user discussion and may not be preserved.. join the list for full traffic. Older Discussion archives could be archived elsewhere or deleted. This would make renaming pages a bit quicker.

de-emphasize Discussion pages ? --simon, Sat, 10 Jul 2004 21:56:50 -0700 reply
Then again, look at this page. It's focussed and makes it very easy to read the past several years' discussion on this topic. So I don't know. Time will tell.

fixed wiki -> list --simon, Sat, 10 Jul 2004 22:15:04 -0700 reply
The last few comments didn't show up due to a ezmlm/zwiki disagreement.

permalinks --Simon Michael, Sat, 10 Jul 2004 22:19:21 -0700 reply
I'd like a solution for this problem: comment urls change when discussion is moved to an archive page.

automatic archiving --Simon Michael, Sat, 10 Jul 2004 22:27:50 -0700 reply
And somewhat related: it would be nice if monthly or yearly archiving of discussion pages could be automated.

Yes. Now that I'm talking about getting rid of them, I want to beef them up.

Impersonating Users on Plone -- Sun, 01 Aug 2004 20:55:29 -0700 reply
I made a few changes to mailin.py and Mail.py so that the username is used in the comment rather than the name in the From field. Rather than going through the whole portal_membership and checking for a match for the From address I made allSubscriptionsFor and isSubscriber return the username in the subscriber_list that matched the E-mail address given. This is because Plone adds the username to the subscriber_list instead of the E-mail address. This meant that I also had to make the changes suggested by #810 "subscribers only" mailin to a plone site ZWiki fails to get Plone to recognise the usernames in the subscriber_list. I then set the value returned to mailin.py by allSubscriptionsFor to be self.FromUserName.

I decided to do this because users could impersonate others simply by changing the name in their From Field. I'm not sure if this issue has been solved already or if there are some inherit problems in my solution that I haven't realised (seems to work fine) but if you want to know more, perhaps the actual code I used just reply.

Impersonating Users on Plone --Simon Michael, Sat, 07 Aug 2004 14:21:16 -0700 reply

I made a few changes to mailin.py and Mail.py so that the username is used in the comment rather than the name in the From field. Rather

Thanks for working on this. Yes please, I'd like to see the patch, here or by mail or darcs send. But note the mail code changed for 0.33.

I'm not sure what username you're using during mailin. The authenticated username, if any, is just the login used by curl, and will always be the same, won't it. But I probably don't understand your fix yet.

... --Simon Michael, Mon, 09 Aug 2004 23:13:58 -0700 reply
Moving a bit from the "monitor all changes" use case towards "reduce noise-to-signal ratio": creating an empty page or posting an empty comment no longer sends a mail-out.

Impersonating Users on Plone -- Fri, 13 Aug 2004 01:18:36 -0700 reply
OK, here are the changes, I appologise if this isn't the appropriate place to post them, maybe they should go in an Issue? If so feel free to move them. These changes apply to ZWiki 0.33.0, they make mailin work for Plone and they make mailin use the username in the comment instead of the From Field in the E-mail.

I changed three functions in Mail.py, instead of removing the old ones though I just copy and pasted them and added "New" on the end, I did this because I wasn't sure whether these functions were being used elsewhere. It could be that you can just modify the old functions:

    def isSubscriberNew(self,email,parent=0):
        """
        Is this email address or member id subscribed to this page ?

        With parent flag, check the parent folder's subscriber list
        instead.  Note "email" may be either an email address
        (case-insensitive) or a CMF member id.  We'll accept either, and
        find subscriptions using either.
        """
        subscriber = email
        if subscriber:
            email = self.emailAddressFrom(subscriber)
            if email: email = string.lower(email)
            usernames = self.usernamesFrom(subscriber)
            for sub in self.subscriberList(parent):
                if not sub: continue
                if ((email and (self.emailAddressFrom(sub) == email)) or
                    (usernames and (sub in usernames))):
                    return sub
                    # sub is the username stored in the subscriber_list
        return ""

    # created a new function that returns username instead of 1 or 0
    def isWikiSubscriberNew(self,email):
        """whole-wiki version of isSubscriber"""
        return self.isSubscriberNew(email,parent=1)

    def allSubscriptionsForNew(self, email):
        """
        Return the ids of all pages to which a subscriber is subscribed
        ('whole_wiki' indicates a wiki subscription).

        XXX catalog case duplicates isSubscriber code
        """
        subscriber = email
        subscriptions = []
        # subscriber may be an email address or a member id, and
        # they may be subscribed as either
        email = self.emailAddressFrom(subscriber)
        if email: email = string.lower(email)
        usernames = self.usernamesFrom(subscriber)

        if not (email or usernames):
            return ""
        if self.isWikiSubscriberNew(subscriber):
            subscriptions.append('whole_wiki')
        # optimization: try to use catalog for memory efficiency..
        # XXX can we do better - index subscriber_list and search
        # it directly ?
        if self.hasCatalogIndexesMetadata(
            (['meta_type','path'], ['subscriber_list'])):
            pages = self.pages()
            for page in pages:
                for sub in page.subscriber_list:
                    if not sub: continue
                    if ((email and (self.emailAddressFrom(sub) == email)) or
                        (usernames and (sub in usernames))):
                        return sub
                        # sub is the username stored in the subscriber_list
        else:
            # poor caching
            for id, page in self.folder().objectItems(spec=PAGE_METATYPE):
                sub = page.isSubscriberNew(subscriber)
                if (sub != ""):
                    return sub
                    # sub is the username returned by the new isSubscriber function
        return ""

The only real difference in these functions is that they allow allSubscriptionsFor to return the username, sub that they found in the subscriber_list or "" instead of 0 or 1. This only works in CMF/Plone because in Zope the E-mail address is added to the subscriber_list, not the username.

What this means is that mailin.py now has access to the Plone username without going through the whole portal_membership. So here is the diff to 'mailin.py':

 diff mailin.py mailin_new.py 
 364a365,366
 >         fromUsername = self.workingpage.allSubscriptionsForNew(self.FromEmail)
 >         senderUsername = self.workingpage.allSubscriptionsForNew(self.senderEmail)
 368c370,371
 <             self.senderEmail in accept or
 ---
 >             self.senderEmail in accept):
 >           return
 370,371c373,376
 <             self.workingpage.allSubscriptionsFor(self.FromEmail) or
 <             self.workingpage.allSubscriptionsFor(self.senderEmail)):
 ---
 >         elif (fromUsername != ""):
 >           self.FromUserName = fromUsername; 
 >       elif (senderUsername != ""):
 >           self.FromUserName = senderUsername;

All this does is assign the value returned by allSubscriptionsForNew to self.FromUserName?.

This fix seems to work but I'm not guaranteeing anything :)

MailoutPolicyLogs -- Wed, 15 Sep 2004 08:18:05 -0700 reply
The following page will discuss an idea I mentioned on ZwikiDevDiscussion? .

Need this page to a. Define the feature, b. Work out nuances (such as the StickyChangeNote? in editing)

Changine URL in mail-outs --davidm, Wed, 07 Dec 2005 22:31:33 -0800 reply
Hi, Where can I change the page URL attach to mailouts - or switch it off as it's currently an internal network URL? david..

Changine URL in mail-outs --Simon Michael, Thu, 08 Dec 2005 07:05:14 -0800 reply
Hi.. make a mail_signature property on the folder and leave it blank.