PieterB's Next Generation WikiMail Proposal (v0.1)

(discuss at WikiMailDiscussion)

I've been thinking of a new architecture for the WikiMail. Things I would like to improve:

Here are my current thoughts in a picture.

WikiMailNG v0.1 architecture

Use Case: sending e-mail to a wiki

  1. the users sends an e-mail to zwiki at zwiki.org or replies to a zwiki message
  2. the email is handled by the zwiki.org mail transfer agent (MTA) and will be forwarded to a user wikimail on the local machine
  3. the user wikimail has procmail installed in it's .forward.
  4. the procmail filters out duplicates of messages (based on message-id's). This prevents mailloops and will also decrease the ammount of spam, because spam tends to use similar message-id's
  5. the message is scanned for spam by using http://spamassassin.org/
  6. then the mail is either sent to Zope/Zwiki or to mailman ("switch" component in the picture). This is done based on a secret header which will be added by Zwiki. If the header is present the mail will be forwarded to MailMan?. Otherwise it will be saved in a local mailqueue on the filesystem and zwiki will be notified that a new message has arrived. I think it's better to save the email on the filesystem and let zwiki pull it from there. In that way:
    • messages can be processed in first-in first-out order (serial)
    • the message doesn't need to be encoded for HTTP posting
    • it will make the load of the http-upload much lower
    • ... <<input wanted on this issue!!!>>
  7. the mail is forwarded to Zope/Zwiki using a http-interface: curl or an improved version of [ZwikiFrontend]?.
  8. i would like to introduce something like a "Zwiki Proces engine" that is responsible for handling all the actions on wiki documents (e.g. add comments trough e-mail, add comment through the web, edit pages, rename pages, etc.). The message will be handled by something like a "HandleIncomingMail?" method, that will make sure the incoming message is processed sucessfully. The incoming mail handler will also trigger the outgoing mailhandler for sending the mail out (via mailman).
  9. the outgoing mail handler will add a secret header to the outgoing message (e.g. "X-Sent-to-mailman: Yeah!") and will sent it to zwiki@zwiki.org over SMTP (using Mailhost or maildrophost).
  10. the MTA will then forward the message to the user's wikimail's procmail.
  11. even the outgoing message can be checked for spam if necessary
  12. the "switch" detects the secret header, will remove the secret header and will forward the message to the mailinglist manager mailman (e.g. "|/usr/local/mailman/mail/wrapper post zwiki").
  13. the mailinglist manager will sent the message to the wiki subscribers.


status update --simon, Thu, 22 Jan 2004 06:12:32 -0800 reply
WikiMailDiscussion is the place to discuss this now.

I wrote :

There are two known issues that prevent Zwiki-only mail handling being reliable - 1. incoming mail will be dropped if the zope server is down and 2. outgoing mail can be duplicated due to ZODB conflict resolution retries. Until now, to give a wiki robust mail handling I've always added a mailman list. But this is complicated, especially if you don't need all the mailman features. I'd like us to have a standard easy-to-set-up procmail configuration which can solve problems 1 and 2 in a simpler way.

Pieter worked on this at WikiMailNG but I can't find the solution he posted - has anyone seen it ?

Pieter found an old copy and I am tweaking it - see http://zwiki.org/zwikidir/procmail .