Forum OpenACS Q&A: E-mail responses to forum/bboard postings?

I'm starting a new OpenACS project, and my client is interested in having the ability for people to respond to forum postings via e-mail.  That is, anyone who receives e-mail notifications from forums should be able to use their e-mail client to contribute to the forum.

My first question is if this has been implemented in either the (new) forums package or the (old) bboard package.  I poked around, and didn't see any obvious indications, and wanted to double-check this issue here before telling the client that this would require custom development.

In addition, thinking about this problem has raised a bunch of questions in my head.  For example, how do we pass e-mail messages to the right forum?  (qmail would make this relatively simple, and I use qmail, but is that now the de facto required MTA?)    More importantly, how do we guarantee security and/or ensure that people are who they claim to be and/or handle private or moderated forums?  This is not a simple problem to solve, and I wouldn't be surprised if any implementation that has been incorporated to date falls short on some or all of this.

Collapse
Posted by Bart Teeuwisse on
Reuven,

the (new) forums package can accept e-mail responses and post them to the forum. I haven't activated this feature myself and can't comment on the robustness of the implementation. However, I'm inclined to agreee with your security concerns.

Don't forget the need to extract only the meaningfull text from the e-mail response. This is particularly an issue for batched notifications which contain more than one post to which the receiver can respond.

See forum-reply-procs.tcl for the current qmail based response handling.

/Bart

Hi Reuven,
well, this is a real neat feature that a lot of people would like to see, and probably not just for forums but as a general service.

I remember that in ACES (ACS 4.x) they tried to do something like that for forums, but not sure right now, maybe Caroline know about it. AFAIK there is no implementation of this yet for oacs.

Basicly, just come to my mind, you'll need a parser for handling diffent email formats (there's one good Oscar said) for all the incoming emails, and embeded ID for obvious reasons, including checking "email from" for security issues. There is no de facto MTA for oacs, or is one? I also use qmail.

Hi Bart!
you have quick fingers!
where is that implemtation? can you point me to the procs...
Collapse
Posted by Oscar Bonilla on
I've tought a little about this and there are a couple of ways
to do it. Basically they all involve the following:

1. A way to pipe the email to a script that connects to the
database and inserts it in the appropriate forum. This does
not have to be qmail (I also use qmail ;), it could be
sendmail with procmail or whatever. The important thing would
be to be able to pipe the mail to a script.

2. A way to tell to which forum the user is posting and if
this is a reply to a post. The reply part is easy since we
could set some mail headers in the original message, for
instance X-Forum-Id, and X-Reply-To-Message-Id or whatever.
If it's a new posting the forum name would have to be
embedded in the destination address and would have to be
looked up. You would have to trust the return email address
since there is no easy way (except maybe PGP) to
authenticate the user via email.

3. A way to get the content. This is really, really hard.
Having worked with email processing systems before I can
assure you there is NO standard on how email clients encode
their responses. It could be plain text, html, MIME with
tons of different charsets, Multipart MIME (think recursive
encodings) which Outlook likes a lot. Assuming you're able
to decode the email, you'd still have to find the relevant
part (the new content) since most email clients quote
original text.

I haven't seen any mail parsers for Tcl, but I've used
the Perl MIME parsers. One solution would be to write a
Perl script to do what I've described. It would
not be elegant, but it would do the job (isn't that perl's
philosophy?).

I've glanced at the forum-reply-procs.tcl file but
IMHO it doesn't look complicated enough to do the job
right ;)

Regards.

Oscar,
I would say for:

1. setting an specific standard account, lets say, oacs-emails,
so then you just make your proc to go the directory and schedule it to parse them all! The solution should support any kind of MTA.

2. could have
email-id, so you get a mapping table where the answer belong to

user-id (or a crypted form of it), and you can verify if the users emails from where you are getting the message is the same to the registered email from that user-id. I don't see another clear way of checking it right now.

3. its better to use already proven stuff, specially for email parsing that changes time to time

Collapse
Posted by Caroline Meeks on
Hi Rauven,

Sloan has had email response capability in production since last September. It works. Its not without some problems, but its working for Sloan and I bet most of what you need has probably been done.

Areas where you can contribute (aka "known problems")

  • Html email in the reply does not work.
  • Attachments do not work.
  • Can't reply from a batched message (do we really want to implement that? Might be very confusing to the user)
Security: I think that the email must have the reply address the same as their login address.

There is also a useful thread in the .LRN testing forum on the problem of vacation replies. Not sure if the solutions have been implemented in the latest release yet.

https://openacs.org/forums/message-view?message_id=82536

Collapse
Posted by Tilmann Singer on
It's working fine for us on a small intranet site as well. I logged some issues with it in the bugtracker under the notifications component - the most annoying I can think of right now is the fact that mails with a wrong From address get silently discarded instead of generating a bounce.
Collapse
Posted by Reuven Lerner on
Thanks for all of the feedback!

Ironically, it seems that the client was primarily asking if it was possible to get notifications via e-mail, not submit responses via e-mail.  But the information that you folks provided here was very useful.  If we decide to go ahead with forum submissions via e-mail, I'll be sure to coordinate my work with others, so that everyone can benefit from the result.

This is a (very) old post but, is it possible to answer a forum (for example developer forum here) via e-mail (not via web form).

I suppose that it needs configuration in email server, but is it work (Openacs Forum or as a general service)

Collapse
Posted by Gustaf Neumann on
Up to my knowledge, There is nothing ready to use for forums, but it is not unlikely, that several have implemented this based on the available infrastructure. The acs-mail-lite (part of acs-core) has some functionality to process incoming mails (e.g. mail bounces) and provides means to extend mail processing via callbacks and a ReplyPrefix. Arbitrary application packages can provide callback to handle the parsed mails. If you are interested, you can check in the notification package or in the bug-tracker for examples of such callbacks

A short online description is here:
https://openacs.org/api-doc/proc-view?proc=acs_mail_lite%3a%3aload_mails

Collapse
Posted by Dave Bauer on
Cesareo,

There is a feature for accepting email replies for forums.

The notifications package has a feature to handle incoming notifications and one route is via email.

The basic idea is to setup an incoming email and deposit the valid replies into a Maildir formatted directory with one file for each incoming message.

I have listed some quite old instructions for this that might be of value, but will most likely require some interpretation and updating:

This tutorial guides you in setting up "reply by email" for the forums package. The forums package will email an article to users who are subscribed to notifications. The "reply by email" feature allows recipients of the notification to reply to the forum post by email. The system is configured to receive the email and post the reply to the appropriate forum article.

1. Have the mails delivered in a Maildir formatted directory which the site has read and write access to.

Assuming user web is running the server, the email sender prefix is "notification" and the domain is "domain.com", here are a couple of ways to do this:

a. Mail server in the same machine
- Install a mail server which supports Maildir (e.g. Postfix) in the server.
- Add an MX entry for domain.com pointing to the server on your DNS.
- Configure the mail to deliver all mailto:notification-*@domain.com to user web.
- Create a .forward file in web's home directory containing the location of the Maildir directory

b. Mail server in a different machine (RECOMMENDED)
- Install fetchmail and procmail
- In web's home directory, create a .fetchmailrc file and put:

poll <your.mail.server> protocol pop3 user "<email.account.username>" password "<password>", is web here and wants mda "</location/of/procmail>"
- get demime from http://scifi.squawk.com/demime.stable, save it as demime.pl and give it execute permissions

chmod 755 demime.pl
- Then create a .procmailrc file and put:

:0 w
\* ^subject:.*Out of Office AutoReply
/dev/null

:0 B f b w i
\* ^>
| /usr/bin/sed -e '/^ *>/d'

:0 f w
| /home/<project_name>/demime.pl -

:0 w
</location/of/Maildir/>
NOTE : remove the '\' in '\*'

The first three lines filter out Outlook auto-replies.
The second set of three lines remove quoted replies.
While the 3rd set of lines calls demime.pl to clean the html of attachments and other html that may have been produced by richtext formatting in MS Outlook.

2. Setup the OpenACS notifications package

EmailDomain: domain.com
EmailQmailQueue: /location/of/Maildir
EmailQmailQueueScanP: 1
EmailReplyAddressPrefix: notification
NOTE : Line breaks seems to be getting ignored. To prevent this in forums, locate forums/tcl/. Remove the ad_convert_html_to_text proc around \$reply(content).

3. Restart OpenACS

Hi

Amazing! 😊

One of my "openacs mantras" is "if you have thought about it somebody has done it" (sorry about my spanglish)

I'll try to set it up in some days and I'll keep my experience here (now I'm upgrading my system).

Thanks!

Hi

I'm trying to set this up (email responses to forum). So far I could do it only by email. I mean, mail is not sourced to forum thread.

So far it was very easy. All I have to do was to do: https://openacs.org/xowiki/incoming_email

But, I think each package (i.e forum) need to parse those email, right? Any work on this?

Collapse
Posted by Dave Bauer on
Forums package already supports this.

If acs-mail-lite is scanning the emails it should be picked up by the forums package.

My settings now:

* Mail Service .. Parameters
BounceDomain www.myserver.com
BounceMailDir /home/openacs/Maildir
BounceDomain www.myserver.com
BounceScanQueue 90 (every 1.5 minute)

* Notification .. Parameters (I'd missed this)
EmailDomain www.myserver.com
EmailQmailQueue /home/openacs/Maildir
EmailQmailQueueScanP 1
EmailReplyAddressPrefix notification

I had to add EmailDomain to fix email domain so I could get an alias in postfix to manage all @www.myserver.com addressses (mailto:notification*@www.myserver.com and mailto:bounces*@www.myserver.com)

BUT It seems that acs-mail-lite doesn't scan openacs/Maildir

Any idea to solve that?

I had a problem (an alias for openacs user). I've removed that alias and it seems to work.

My problem now is some "garbage" in content. I'd like to "post automatically" only the main content (and not all reply).