Forum OpenACS Development: OpenACS.org Priorities

Posted by Talli Somekh on 06/28/03 11:37 PM

Hi guys and gals,

I think that there are some issues with OpenACS.org that we need to prioritize on the same level as getting bugs fixed and releases out. It's a real bummer to position the OACS as a scaleable, high-performance toolkit when the website is so slow.

This is not to say that those who are spending their valuable time getting that work out are doing something wrong. But I feel we're doing a disservice not addressing this stuff on the same level.

The two critical issues are:

Forums scalability This is really horrid. Personally, I'm finding that the amount of time to get a post confirmed is a discouragement to responding to posts.
Download area We need both kernel, official and contrib package APM area.

Again, I understand people are overwhelmed getting these things out, but I really think that these should be critical showstoppers for getting at least v5.1 out.

talli

2: Re: OpenACS.org Priorities (response to 1)

Posted by Randy O'Meara on 06/28/03 11:54 PM

I agree with the forums posting nap time...

Talli, do you have rights to add the download areas? If so, why not just do it? If it can be made better later, it will.

BTW, it's great to see more forum activity on the weekend. I haven't seen this much Saturday activity in some time.

Randy

(i'm going to take a nap now, while I await posting completion)

3: Re: OpenACS.org Priorities (response to 1)

Posted by Talli Somekh on 06/29/03 12:00 AM

Because I'm not a programmer.

talli

4: Re: OpenACS.org Priorities (response to 1)

Posted by Jarkko Laine on 06/29/03 06:35 PM

Talli,

I agree on your every point. One more thing I (and at least a few others) would like to see in the download area is the availability to wget the tarballs. The current form solution makes it impossible. (Assuming you want the packet to your server) It's a real nuisance to launch lynx and try to find your way through openacs.org when you could as well copy the location from Mozilla to your ssh window and wget it.

OK, this was just one (though IMHO quite typical) scenario. Making this work would probably need some tweaking on file-storage package but I think it'd be worth it.

5: Re: OpenACS.org Priorities (response to 1)

Posted by Jonathan Ellis on 06/30/03 03:59 AM

viewing a forum thread list is none too speedy, either. I'm guessing the main slowdown is computing the "new" tags for each thread.

if instead we simply sorted the threads by most recent reply, the thread index page could be cached globally. you could provide another page sorted by thread creation date if you liked, but from experience users seem to prefer the first by a wide margin.

6: Re: OpenACS.org Priorities (response to 1)

Posted by Dave Bauer on 06/30/03 04:17 AM

Ok everyone. As noone else seemed willing to actually take any action of these items, I took some of my weekend to implement direct download from the download package. If you click on a file it will now download directly. This took about 10 mins to implement. Besides this I really want a more sourceforge like download page so that all the files will be shown on one page. I will also implement a feature similar to sourceforge for those who still which to ask questions before offering downloads. It will return a page with a link to download the file with a META refresh tag to redirect to start the download.

Anyway, as you are all aware, the source code the site runs on is freely available. The site would improve much more quickly if more than one or two people helped to maintain and improve it. Besides improving this site, and fixes or improvements can also be added to the toolkit.

I don't mind discussing ways to improve openacs.org in the forums, but let's also keep track of what needs to be fixed in the bugtracker http://openacs.org/bugtracker/openacs.org setup specifically for the web site.

Maybe someone will look in there and possibly even submit a patch. I am sure we can find someone to review these and apply then as needed.

Thanks.

7: Re: OpenACS.org Priorities (response to 1)

Posted by Jonathan Ellis on 06/30/03 04:26 AM

As big as I am on "if you don't like it, fix it yourself," it does suck to spend time on a patch to see it disappear into the Great Big Patch Graveyard In The Sky.

8: Re: OpenACS.org Priorities (response to 1)

Posted by Dave Bauer on 06/30/03 04:45 AM

Jonathan,

We have two ways to get code into openacs and openacs.org. Either get commit privileges on CVS or submit a patch. The process does work. If there are outstanding patches in the openacs.org bugtracker I will handle them this week. I admit I still have some outstanding patches for the survey package. (I peronsally wouldn't mind a weekly reminder of these things.)

Anyway, if you have a problem with a certain package email the maintainer. Yes, I know, how do you find out? We all know that perhaps everything isn't ideal with openacs, but if we work together it can improve.

I was going to add myself as maintainer of the openacs.org bugtracker, but it uses the all 3000 users is a dropdown box interface. I'll do it in psql tomorrow. Beyond that I will look at all the open bugs this week and see what I can do. If anyone wants to volunteer to help out let me know.

9: Re: OpenACS.org Priorities (response to 1)

Posted by Mark Aufflick on 06/30/03 05:09 AM

Maybe we need a few people identified as "openacs.org feature enhancers" - like Don proved and as we all know, small feature improvements are often trivial to add.

The problem as I see it is that there is no official decision about who can make decisions like providing a direct download link. Maybe someone decided once that every download had to be logged to a community member, and adding a direct download link would ruin those stats.

If there was a smallish team who could together make such decisions and act on them I think it wouldn't be a lot of work (I would be happy to be on such a team).

The forums scalability is of course another matter. Do we have any information about where the bottleneck is? I guess the way forward is to replicate openacs.org onto someone's dev box and start testing with the real data set like collaboraid did with the bug tracker recently.

Has anyone done this, or does anyone have rough ideas about where the scalability problem is? I know Lars has suggested we remove the CVS commit forum, but it is not the biggest forum by a long shot, so I don't see that it is the main problem.

10: Re: OpenACS.org Priorities (response to 1)

Posted by Robert Locke on 06/30/03 05:47 AM

Hi,

Yes, a week or two ago I replicated the site on a dev server and identified a couple of bottlenecks. One is mentioned on this thread:

http://openacs.org/forums/message-view?message_id=108290

It has to do with a scheduled proc in the notifications package which runs every minute and is a *huge* resource hog.

Creating new objects is also slow (fyi, creating a new post creates 3 objects I think). Specifically, there is a query in the "acs_objects_insert_tr" trigger which seems to take a while:

select max(tree_leaf_key_to_int(tree_sortkey))
from acs_objects
where context_id is null;

Don says he has ideas on how to improve this. However, the bigger gain would be fixing the notifications query I mentioned earlier. There may be other issues, I don't know.

I apologize to the group that I haven't made the time to work on these issues. I will try to do so this week.

11: Re: OpenACS.org Priorities (response to 1)

Posted by Jonathan Ellis on 06/30/03 05:50 AM

The problem as I see it is that there is no official decision about who can make decisions like providing a direct download link.

This is my primary concern with the current process.

12: Re: Notifications package (response to 1)

Posted by Mark Aufflick on 06/30/03 06:14 AM

simple thought i'm sure has been thought of before:

Would it help to simply reduce the scheduled proc to only run every 5 minutes - or is it the actual bulk of notifications rather than just the querying that is expensive?

If it would help, i think we can all wait a few minutes to be emailed new posts.

13: Re: Notifications package (response to 12)

Posted by Robert Locke on 06/30/03 06:28 AM

Yes, I think that might help. It would be best if someone could optimize the query though. And, I wouldn't be surprised if the actual sending out of notifications to everyone was a bit of a hog (acs_mail_lite::sweeper is run every 60 seconds), though I have that disabled on my test system since I don't want to send mail to everyone.

14: Re: Notifications package (response to 13)

Posted by Malte Sussdorff on 06/30/03 09:31 AM

Luis Mosteiro wrote a patch for acs-mail-lite (both mat's and the cvs version) that immediatly sends out mails and only stores them in the database if the mailer returned an error. This should reduce the number of read/writes to the database while mailing.

15: Re: OpenACS.org Priorities (response to 1)

Posted by Mark Aufflick on 06/30/03 09:58 AM

ok - it seems there are a number of issues and lots of good and potential solutions to a problem that is *annoying* a lot of people, so let's do something about it!

I suggest the following:

1/ I will write up a page listing the known issues contributing to the performance problems in openacs.org forums and suggested avenues for solution

2/ we can discuss that page via comments, and it can be opened via etp for the people working on the problem

3/ the different problems can be allocated to different people to fix as takes their fancy

4/ when we agree on solutions we can test them on Robert's copy of openacs.org (Is this possible Robert? otherwise I can set up a test server, but it seems a waste to do it all again)

5/ Don (or someone) can vet our solutions and decide which should go into the cvs tree and which might be openacs.org specific fixes (and of course which are too ugly for public consumption at all...)

16: Re: OpenACS.org Priorities (response to 15)

Posted by Robert Locke on 06/30/03 10:57 AM

> when we agree on solutions we can test them on Robert's
> copy of openacs.org (Is this possible Robert?

But ofcourse! The machine's at home and I'm behind a semi-unreliable DSL connection, but I'd even be willing to give SSH access as needed.

17: Re: OpenACS.org Priorities (response to 1)

Posted by Dave Bauer on 06/30/03 01:03 PM

Mark,

This sounds like a great application of bugtracker. We should list the problems at http://openacs.org/bugtracker/openacs.org/

The items can be assigned, and commented on. Patches can be submited, or the person could be given commit to the openacs.org tree.

18: Re: OpenACS.org Priorities (response to 1)

Posted by Don Baccus on 06/30/03 06:03 PM

Malte - the change to acs-mail-lite is great, there's no reason to db the message if it goes out immediately and doesn't bounce. That's cool ... has anyone looked into putting it in yet?

That's not going to help notifications, though, not much, as it queues things up first then mails 'em later even if the notification is "immediate". Well, it will help by getting rid of one insert per message, actually, that is a good thing.

I'm going to start a new thread on forums scalability under PG ... stay tuned.

19: Re: OpenACS.org Priorities (response to 1)

Posted by Mark Aufflick on 07/01/03 02:24 AM

Dave - good point. I will working on a free form doco though which I think is better for discusison (wiki style) and then the tasks can be put into a ticket-tracker instance of the new "everything including the kitchen sink (except for the in-sinkerator)" bug-tracker.

Especially since a number of the tasks won't be bugs per se - just things that should be done.

Maybe i'm wrong actually...