Forum OpenACS Q&A: Assessment - urgent

http://strauss.gast.it.uc3m.es/

There you have the three brances oacs-5-1, oacs-5-2 and HEAD, already with the assessment package mounted in the commuities. The code is checked out from the repository and reinstaled each day, so I think that's what you are looking for.

jopez

http://dotlrn5.uni-mannheim.de:10000/likert.jpg

Please note:
- question is places left of the choices
- choices can either have their own header or a section header
- choices can have min or max header or individual headers

And if it takes 1 hour to understand I guess this is a good argument for documentation. Is programming involved or simply knowing where all the all the bells and whistles are?

- Not sure I understand, but if you want to have multiple lecturer answer and grade the same assessment then I *think* you could easily put the assessment into multiple communities. Otherwise it is "just" a question of permissions.

- List of current submitters is there. Option to write to those who did not answer.
- You get a CSV file with all the answers to an assesment which will show you the number a particular answer was given. Is this what you want ?!?
- You can notify all users who
-- have answered the assessment
-- have not answered the assessment
-- are participating in the assessment

- The assessment code is called "Password" in assessment. Only if you have the password you can access the assessment.

- Reciept for the student? This should be an after assessment action which you can define that generates a PDF for the student (or whatever you want. This is your custom code then).

What would be needed is rather a default user who requires no login but must provide a unique transaction number that can only be used once to submit an assessment.

So the quest is whether assessment accepts multiple submits from a single account or from a user without an account and instead checks against a list of valid transaction numbers assigned to it.

We have now about 1.5M cr-items and 2M cr-revisions. If you have less than 0.5M cr-items, you should be in the safe side.
The functionality is very robust, but the UI for creating an assessment is overly complex.

In general, UI for creating assessments & scalability needs to be enhanced.

http://dotlrn5.uni-mannheim.de:10000/likert.jpg

Somehow I don't like the idea that we need to copy an assessment each time a course is registered. Thus maybe there should be a single assessment with a first section to determine the relevant course and to check whether the user has a valid code. But now comes the big question, how can we include at the end the customizable portion of the assessment depending of the course that has been selected?
What is surely not wanted is that there is one large assessment with many optional sections. Rather each professors should be able to define a section which is automatically included in the assessment at its end.

Any idea what is already there and what needs to be coded?

I talked to Eduardo and the dotlrn-catalog package is now installed and mounted automatically =)

I'm not sure about what courses are you talking about. There is a community (test) which is created automatically each time the code is updated and the database reinstalled. Are you talking about that?

About the likert question, I don't have too much experience with that, but I have just seen the "Final Test" assessment and it has a likert question type, I guess it was you :-p. So I think it works.

Regards

I found some bugs though:
- Currently you can select from any assessment defined on the system
- Once you assign an assessment it is assumed under /assessment/assessment?assessment_id=foo so I guess we need to resolve the URL depending on the Id passed
- The assessment link is labeled as "enroll" but assessment could be really several things: an enrollment form, a test, an evaluation at the end...

Therefore I would recommend to allow a course to be associated with several assessments and let the professor decided what it means. And going that way why not allow a professor to associate any acs object to a course? If someone uses ETP or XoWiki in a course you could provide a link to the website and likewise to a forum, to the profesors homepage, to a blog ....

What do you think?

Who is actually the developer of dotlrn-catalog? I like it very much.

Greetings,
Nima

We have been quietly using (and improving) it within Partners Healthcare for a pilot of .LRN (that will be expanded considerably in 2006 -- Partners has 30K+ employees). So far about 500 people have signed up for courses using the pilot. We have also integrated the system with our internal Active Directory using real time sync (we hope to improve integration of AD as well this quarter).

Take a look at the limited public UI here: http://hub.partners.org/catalog/

Here is a general project discription:
http://lcs.mgh.harvard.edu/projects/hub.html

I am happy to give you an admin account on our dev or staging machines Nima so you can try it out.

We are using a version that integrates ecommerece and uses the dotlrn course catalog API.

Eventually we hope to clean up the two packages (dotlrn-catalog and dotlrn-ecommerece), but at this point they are best taken together (e.g. I am pretty sure there is code in dotlrn-ecommerce that should probably be in dotlrn-catalog -- we put the two puppies together and ended up with a mixed breed).

It uses assessment for data collection (note: we are also dedicated to cleaning up assessment).

The original work on dotlrn-catalog was done for Heidelberg Medical School by Viaro, but it was not put into place so it could be fine tuned and tested before I started my fellowship at MGH (in addition Heidelberg was not yet in a position to actually use the code). Heidelberg Medical School will be using the same code starting in March and I will be working with them on improvments in February as well.

Also note, that the Museum of Science in Boston is using a variant of this code (for courses that imho are much more interesting than the statistics courses we have listed at MGH right now). See: https://openacs.org/forums/message-view?message_id=324851

Eventually (contingent on further internal funding) we will use this as the main course interface (Hamilton just created a parameter that allows replacement of the standard course add/drop interface) and start migrating groups in /dotlrn to OpenACS subsites and the new portals in the first half of 2006 (what I hope will become .LRN 3.0 "codename Whiteboard" - Don gets credit for that one).

Carl

P.S. We are also adding anonymous evaluation using assessment. Work started on that yesterday.

P.P.S. Feedback is greatly appreciated, both are under active development.

http://xarg.net/tools/cvs/change-set-details?key=22801

Below is info I found in the Postgres Doc. Maybe Roel or Eduardo (the author in this case) could chime in the interest of incremental improvements? I am wondering if it had to do with a user double clicking or if it is more serious. I was actually doing some live usability testing when this came up on Monday and I think the person I was watching might have double clicked between sections (although she was not sure and I was not watching her hand).

Carl

"Use of explicit locking can cause deadlocks, wherein two (or more) transactions each hold locks that the other wants. For example, if transaction 1 acquires an exclusive lock on table A and then tries to acquire an exclusive lock on table B, while transaction 2 has already exclusive-locked table B and now wants an exclusive lock on table A, then neither one can proceed. PostgreSQL automatically detects deadlock situations and resolves them by aborting one of the transactions involved, allowing the other(s) to complete. (Exactly which transaction will be aborted is difficult to predict and should not be relied on.)

The best defense against deadlocks is generally to avoid them by being certain that all applications using a database acquire locks on multiple objects in a consistent order. One should also ensure that the first lock acquired on an object in a transaction is the highest mode that will be needed for that object. If it is not feasible to verify this in advance, then deadlocks may be handled on-the-fly by retrying transactions that are aborted due to deadlock.

So long as no deadlock situation is detected, a transaction seeking either a table-level or row-level lock will wait indefinitely for conflicting locks to be released. This means it is a bad idea for applications to hold transactions open for long periods of time (e.g., while waiting for user input)."

I have also thought about double clicking as the cause of the problem, but i am not so sure now. Prior to our last big evaluation, i have added the following to the submit button in
packages/assessment/www/assessment-section-submit.adp:

input type=button value="assessment.Submit" onclick="this.form.submit(); this.disabled=true; this.value='Verarbeitung läuft...'; return true;"

The javascript code disables the submit button and the user cannot easily submit the form twice (maybe with the return button on the keyboard). Despite this change, we had the deadlocks appearing :(

Imagine you are serving an assessment with 20 questions, to 20 persons at the same time, and the creation of an new cr_item takes about 100 ms, that will end up in big delays to serve the request (if you increase the deadlock timeout) or you'll end having the deadlock. In this example, takes 2000 ms to create all the items of a given assessment, by 20 persons is about 40 seconds in total to create all the items requested at the same time, eventually you'll have deadlocks, and they will propagate among all the applications that might be using (creating / updating) cr_items / revisions at the same time.

Finally, intensive concurrent use of the CR is NOT scaling so far.

Retry to process something is just good if you DO NOT even care having your users waiting long periods of time for the page to be served, something that you don't want specially when you aim to serve pages in less than 1 sec.

Requests, are actions that will be executed after someone reviews it.
Play with it =)

If it is, I'd be very interetsed to know what about the CR causes this.

Roel is going to research more on our installation as well. I look forward to more information on what is causing this behavior.

I think if we can isolate the code, we can come up with a design that will work.

Despite the fact that i know how a deadlock can appear in general, i am not sure why it happens that often in the assessments package. When 2 users are submitting the same assessment in the same moment, where the questions are in the same order, why would there be a deadlock?

Now, for something different. You are aware that you can turn of randomization and then the pages are served very fast as we do intelligent caching ?

ACS-Authentication.

I commited documentation to oacs-5-2 before release (which I just dicovered is not the version labeled 5.2 on openacs.org/doc)

Here it is:

http://cvs.openacs.org/cvs/*checkout*/openacs-4/packages/acs-authentication/www/doc/ext-auth-ldap-install.html?rev=1.4.6.1

Please update and improve if possible Nima.