Forum OpenACS Development: new bboard package

Okay, OpenForce is working on a new bboard package for dotLRN. We haven't begun the coding yet for this part. We have an aggressive schedule, but we'd like to share our immediate thoughts for rearchitecturing the system.

Functionality

• bboard forums either Q&A or threaded (and certainly a simple architecture for other presentation styles)
• categorization of postings (optional system-wide, and optional per forum)
• ability to restrict posting of new questions
• ability to moderate
• ability to manage existing threads (mostly for deapproving/ deleting threads).
• emailing a post to a friend (nice new feature of v4)
• editing your posts... although this should be optional per forum, as this is a hotly contested feature since it changes the historical trail of a thread.
• incoming email submissions to bboard... another hotly contested feature, but certainly let's implement it as an optional thing.

Under the Hood

• forums are objects scoped to packages
• bboard postings are *not* objects, they're *not* CR items, they're *not* versioned. Thus, they're not permissioned individually, nor should they be. Only forums are permissioned.
• use of sortkeys for message hierarchy

1. Trying to get permissions on a large result set is a problem. I'm not sure this is completly a permission problem since putting plsql in any query returning a large (>100 rows) result set can be a problem. If a function takes 10 ms and you run it 100 times it adds up. I need to mess with the all_object_party_privilage_map and see how it works.

2. Updating the context_id on an object takes several seconds right now. I'm testing a fix for this right now.

I've found the object system to be invaluable on my project. It's great to be able to pass around a number and figure out what kind of object it is, what it's attributes are and what's related to it. The only problem is when you what to know who has access to it.

From what I've seen if there were no permission system the object model would have little impact on performance. With permission I started having problems around 250,000 objects. To be fair I've read the permssion system design goals and I think it's a spec problem more than an implementaion problem. The goal seems to have been return permission on a single object quickly and return a result set with permission checking at least as fast and the result set without permission checking. This means select * from acs_objects where permssion = 'read' will take a with if you have more than a few hundred objects.

I think two things need to be done.
1. Make the current permsission model as fast as possible.
2. Make the permission system optional on each object.

The first just requires fixing stuff as it shows up. The second I'm planing to implement by adding a check_perm flag to acs_objects and maintaining a table with object_id, context_id with a trigger. Then change the permsission views to use that table. Since I only need to permssion about 10% of my objects that's a big win. It would also help the bboard if you don't want permission checks on all posts.

Trying to get permissions on a large result set is a problem. I'm not sure this is completly a permission problem since putting plsql in any query returning a large (>100 rows) result set can be a problem. If a function takes 10 ms and you run it 100 times it adds up. I need to mess with the all_object_party_privilage_map and see how it works.

So far, every permissions performance problem that has raised its ugly head has been solvable (knock wood).

In other words a dimensional slider that slices index page
different ways.

Walter

Also, somebody once told me that each post is assigned an
anchor tag in the 4.x bboard. It would be useful to have this
feature and to have it *accessible to the users.* Sometimes
(particularly in a long thread) you want to send somebody directly
to a specific post on the thread. The UI might be tricky on this, so
you may want to start by just including the anchor tags and then
thinking through how to make them acessible later.

- not sure I read this correctly in the comment above by Jun, but each forum is *not* a package. A forum is scoped to a package. I was basically refering to the fact that current bboard v4 is not completely correctly package-scoped (half-scoped stuff is about as annoying as can be), which makes multi-instantiation of bboard difficult.

- categorization: I see, in theory, the use of a central categorization scheme. Often, however, the categories only make sense inside a particular forum. For example, inside the OpenACS 4.x forum, we might have the categories "design", "backwards compatibility", "sample code", "documentation". These categories are pretty specific to OpenACS 4.x and wouldn't apply to say, photography if we also had a discussion forum about that. Michael - and others - maybe you've already thought about / solved this problem?

- anchor tags: great idea. And not that hard to implement at all.

- CR and versioning: there's one good point here - if we allow editing of posts, we *must* allow versioning, and actually, we should technically allow users to view old versions of posts (otherwise the stream of discussion might make no sense and make someone look like a fool for responding to an issue that is no longer there). How do people feel about this editing issue? It seems to bring up far more problems than it solves....

categorization: I see, in theory, the use of a central categorization scheme. Often, however, the categories only make sense inside a particular forum. For example, inside the OpenACS 4.x forum, we might have the categories "design", "backwards compatibility", "sample code", "documentation". These categories are pretty specific to OpenACS 4.x and wouldn't apply to say, photography if we also had a discussion forum about that. Michael - and others - maybe you've already thought about / solved this problem?

It's absolutely true that categories can need to be local rather than global sometimes. So you'd need some sort of scoping mechanism in your categories system and really, you'd need to be able to scope not only specific categories but entire taxonomies to do it right. It seems to me that this conversation has come up before under the guise of keywords, but I couldn't fully follow the technical ins and outs so I may be mistaken.

FWIW, though, even the examples you gave above may be better scoped globally--not across bboard instances, but across packages. It would be nice, for example, to have a portlet that shows all bboard posts, file-storage docs, and wimpypoint presentations relevant to "backwards compatibility" (particularly if those items in the portlet could be filtered or ranked by their aggregate user rating).

CR and versioning: there's one good point here - if we allow editing of posts, we *must* allow versioning, and actually, we should technically allow users to view old versions of posts (otherwise the stream of discussion might make no sense and make someone look like a fool for responding to an issue that is no longer there). How do people feel about this editing issue? It seems to bring up far more problems than it solves....

I'd want editing. Despite the post confirmation screen after clicking the "submit" button, people still pull the trigger to soon sometimes. You can say "too bad, you had your chance" but I would hesitate to take that position unless the performance penalty you'd pay for it is pretty significant.

categories: so, getting to this global namespace requires building a taxonomy for your site, which is a big deal. I want to find a solution that doesn't impose too much on the user. Maybe a generic categorization scheme (as discussed in a different thread) that supports local or global taxonomies. That said, I obviously want to keep things simple for now, with an eye towards future enhancements.

I completely agree that setting categories in a single bboard should not require the creation of a site-wide taxonomy. However, it should allow integration with a site-wide taxonomy. From my limited understanding of service contracts, I'm thinking that you'd want to create a contract that enables a package (or package instance) to use a simple (i.e., instance-scoped) or complex (i.e, site-wide) category system. For the immediate purposes, you might want to just implement the local version for bboard as long as the plumbing is there to enable other hackers to do broader scoping when that itch is ready to be scratched.

- editing: the issue is not so much performance as it is usability: I don't know of many other bboard systems that allow editing. Not because it's hard, but because it completely breaks the flow of conversation. If you write a message on May 10th, and I respond on May 11th, and then you edit your message on May 12th, then whether you change the posting date to May 12th or leave it to May 10th, chances are my response makes no sense. However, if you simply allow someone to post a new correction on May 12th, then it all makes sense. That's the real issue, IMO.

I don't see usability as a huge issue as long as you allow readers to view previous versions, much the way you can view previous versions of a doc in file-storage. Alternatively (and somewhat more simply), you could only allow editing as long as the post being edited is the most recent one. You could make these admin configurable.

I admit that there's no perfect solution to the editing problem, but I'm not convinced that not providing editing capabilities is the best compromise.

Another argument against allowing someone to edit bboard
postings is the fact that there are email alerts on the postings.
Will you send out corrected postings? Or would you make the
corrected posting "stealthy"?

But admin editing needn't drive versioning in the way user editing would.  If we disallow user editing, which seems to be the growing consensus, then we don't really need versioning.

If we had the more lightweight CR fantasized above I'd argue for sticking posts in the CR anyway, since the current Tcl API does simplify handling large objects and in Oracle unfortunately posts must be CLOBs.  But straight content one knows will always be text is easy enough to stuff into CLOBs directly so making non-versionable posts be non-CR objects is fine.  I'll just consider it an editorial comment on the current state of the CR ...

And, how about automatically closing open tags in posts?

At Greenpeace we've discussed hooking tidy and OACS together, but not for form verification (good idea, though).  We were actually thinking about running tidy against every page we serve in developer support mode.  I think having tidy available as an AOLserver module would be fantastic if ... it ... is ... threadsafe!

As far as categorization is concerned I thought you should know that I helped develop quite a feature rich categorization service (any category tree could be scoped to whole site, an object type, or any acs object i.e. a bboard forum) about a year ago at aD in Munich. Even though you may not be able to use the source code directly (it's based on Oracle and aD may not want to release the source code - not my call) you might be able to derive some good ideas from it. You can find the documentation here. I think the requirements document is most useful to read.

Let me know if there are questions!

Some points about the above comments:

- editing of messages without versioning? I'm strongly opposed to that. I can't see that ever being a good application. Total flexibility is not always good if you allow features that start to confuse users. If discussions start to get confused by unversioned message editing, the package will not be used as much as it could, because it won't be as useful in fostering useful, long-term-archived exchanges.

- Making each forum its own package. This assumes that little or no aggregation will be done across forums. That you won't ever ask "hmmm, what other crazy rants has Ben posted on OpenACS Evangelization now that i see how nutty he is being on OpenACS Design." Note that, in dotLRN, we are attempting to do cross-package aggregation. It's taken a *long* time to get anything right, and I'm not sure we've finalized that architecture just yet. I certainly don't think we should dabble in that for a core package like bboard. I'm still strongly in favor of multiple forums per package. If there are other arguments, let me know, but killing aggregation is, IMO, not an option.

My proposal for the first version, following the principle of simple design first, iteration later, is as follows:

- forums are objects, scoped to a particular package

- messages are objects, directly associated to a forum (as opposed to indirectly mapped to forums through a third table), not CR items.

- no categorization system yet. I think we need a lot more thought to create a general categorization mechanism. I don't want to create a bogus system to do a bad job of this right now. We can add this later, especially if we design a good, modular categorization scheme.

- two different views: threaded and flat

- revival of features from 3.x: can users create new threads? moderation.

- admins can edit messages in a permanent manner, no versioning. admins can delete whole threads or individual messages.

- ability to post in HTML/text, with toggling after the fact, as that is a display issue and not a content versioning issue.

Note that I wouldn't mind some help on the HTML tag closing stuff! Yes, we really should have a good way to do this once and for all.

http://dotlrn.openforce.net/doc/forums.adp.

This pretty much summarizes the stuff we talked about in this thread, and formalizes things a tad more. I've started writing this code and will import it in the CVS tree shortly. I'm focusing on functionality and page flow for now, not so much individual page UI.

This document is not done, I will be updating it as things move along, and I look forward to more comments and suggestions!