Forum OpenACS Development: Wiki modifications for mediawiki format

Over the last 2 weeks, I made some modifications to the existing wiki package to support a mediawiki-like syntax. This is literally a day late and probably a few dollars short in comparison to Gustaf's XoWiki package, mentioned in this post. I haven't played extensively with XoWiki or browsed through the code in detail, but the demo looks amazing. Thanks Gustaf! That said, I think that some of the changes I made to the wiki package might be worth merging/integrating into XoWiki.

I setup a live demo of the modified wiki package. A brief overview of the support for mediawiki markup is outlined on that page.

I really like the way Gustaf implemented the ability to do adp includes, and also the markup syntax for using divs with the >>name<< delimiters. It also seems as though XoWiki has the potential to deal with "type_links" or "namespaces" (the mediawiki term), but doesn't do anything specific with the link_type information at the moment.

One way to use the link_type information is for rendering a certain type of content, such as an image. This is how the mediawiki [[Image:name|options]] namespace works. You type in something that tells the system to replace it with an image, <name>, using the appropriate display <options>. Alternatively, this information could be used to execute an action or create a relationship, such as adding an article to a Category (i.e. [[Category:OpenACS]] would add the page to the OpenACS category). As far as I can tell, mediawiki uses the Category namespace to categorize articles. They also use language namespaces to render content in foreign languages.

As an example, I implemented a "Photo" namespace/link_type for the wiki package that will display a specified photo in the photo-album. The major advantage of this approach is that users can include images safely without using IMG tag. However, this approach is limited to images stored in the system (inside an instance of the photo-album in this case).

You can read more about how this works on the demo page mentioned above. Ultimately, this code ends up calling template::adp_include to generate HTML for displaying the photo, but I'm not clear as to whether there is a better approach. Specifically, I wasn't sure if cr_items store information about how to display themselves.

I'm still fairly new to OpenACS, although I've been silently lurking on the forums (on and off) for awhile, and trying to understand how to build apps with the api. I'm a scientist by training, and I play with OpenACS in my free time, which there never seems to be enough of these days. Based on my limited exposure, it seems like it would be better to extend XoWiki, rather than continue to use the wiki package. If any of the above makes sense, and seem like good ideas, I'm volunteering to help implement them in XoWiki. Either way, I'd appreciate thoughts and feedback from the community. Thanks.

Collapse
Posted by Gustaf Neumann on
Thank you for the nice words about xowiki. Conceptually includes {{...}} are a more powerful construct than links [[...]], since every include can return an HTML link markup.

We could easily use {{link WIKIPAGE -label "Some Label"}} as long-form for [[WIKIPAGE|Some Label]]. Similarly, we could use {{start sidebar}} instead of >>sidebar<<. so, one construct could be sufficient.

So, i am not sure, whether links with an "image:" prefix is currently the best way to go. The image could come from different sources, from the filestore, icons, or photo-album, it can be linked or included. so, unless we have a canonical way to refer to images/files/news articles/... it seems better to me to use the include syntax and provide a small adp-file if needed, such as e.g.

{{adp photo joe.jpg album album1}}

Still, the nicer way to handle images is via an image selector, that has some of this logic built in. We have developed an image selector plugin for xinha for our elearning application learn@wu, which could be extended this way. i will give it a closer look soon. This has the advantage that it is more straight-forward for a end-user, it is more wysiwyg, and there is no need to handle all HTML image options via the include syntax. The disadvantage is a bigger dependence on the richtext package.

Similarly, for categories, i would like to have a adp-snippet that handles this instead of extending the syntax of [[...]]. Categories can be used for many things loke for example for structuring a book with chapters, where pages are wiki-pages and the menu-structure comes from categories. We use in learn@wu our own category system to build a "concept space" containing text-pages and exercises etc for every lecture unit. It would be interesting to experiment with this by using the oacs categories as sketched...

i am not sure what you have implemented in detail, but certainly, all contributions to xowiki are welcome.

Collapse
Posted by Brian Chapados on
I realize that includes {{...}} are indeed more flexible/powerful for integrating and rendering different types of content. Clearly, [[page|label]] could be short for {{link page -label "label"}}. As soon as I saw the demo page, I knew you had nailed the problem.

What I'm having a hard time wrapping my head around is whether the {{...}} syntax provides an easy way to relate the included item to the wiki page. There will certainly be cases where you don't need that functionality, but if you want to render a different type of cr_item, such as an image, it would be nice to have a way to setup a cr_relationship between the page and the image. I noticed that you're using an xowiki_references table to store page references. In the wiki package, Dave B. created cr_item relationships to do this. I'm not sure which way is better, but xowiki seems to be more robust in terms of handling page references right now.

Anyway, my point is that the link [[...]] construct implies a relationship between the page and the linked item, thus using the [[image:myimage.jpg]] might be a convenient way to achieve this. Your code for handling references already supports a link_type attribute. Using [[linktype:item_name|options]] can be implemented so that it ultimately resolves to {{adp /adp/stub/for/linktype item_name -options options}} to generate HTML for the item. The difference would be that the [[...]] construct creates a relationship. Alternatively, I suppose you could just add a -link or -relate option to the {{adp ...}} syntax, which would do the same thing.

Also, I have mixed feelings about WYSIWYG editors. I understand that people like them, however, for composing anything other than simple documents, I find them annoying. I especially hate them for any application where you need to insert several graphics into a document. I use latex when possible to write papers/protocols/technical documents, ect.. So, I like the idea of the xowiki supporting some type of _optional_ text markup system. I'm guessing that there might be others (though maybe a small percentage) that feel the same way. It occurred to me that much of the formatting for mediawiki (or any wiki really) could be handled by a proc such as:

ad_html_text_convert -from text/wiki

We could just add support for a text/wiki type. It would render everything except links [[...]], includes {{...}} and div &gt;&gt;...&lt;&lt; markups. This might be the best way to add support for using wiki markup in xowiki.

Collapse
Posted by Gustaf Neumann on
it sounds like a nice idea to provide some include shortcuts that generate relations of certain types automatically. One could use e.g. {{image album/name}} to include the image and to create automatically a link of the form (page, 'image', image_id). later it is possible to query, what images are used in which pages.... Just now, i made a small adp file for including wiki pages in wiki pages. Here one, could think about a 'wiki-inclusion' link. What's needed is a better way to communicate between different inclusion layers.

It should be certainly possible to create a different type of widget page with a different edit-form (without the rich text editor) that uses as well a differnt rendering function. The current infrastructure should already be able to support this..

Collapse
Posted by Brian Chapados on
The original wikit and mediawiki parser are fairly generic in that you can pass the name of an external procedure for parsing wiki links/references ([[..]]). I'm in the process of cleaning up this code, and then hopefully releasing it as a separate procedure that can be called from any application that would normally accept HTML. As DaveB pointed out in IRC, this would let other applications deal with that information perhaps through a callback. It shouldn't take too long to do this and then hook it up to XoWiki.

Being able to include wiki pages in other wiki pages is a very useful feature. For the way you have it setup now, is the included wiki page just rendered, or are link relationships for the current page updated to encompass links from pages that reference the included page? Hmm... Hope that last bit made sense. Also of note is this warning that I ran across a few weeks ago while browsing the source for the mediawiki parser:

/**
* Variable substitution O(N^2) attack
*
* Without countermeasures, it would be possible to attack the parser by saving
* a page filled with a large number of inclusions of large pages. The size of
* the generated page would be proportional to the square of the input size.
* Hence, we limit the number of inclusions of any given page, thus bringing any
* attack back to O(N).
*/

Collapse
Posted by David Kuczek on
Hey Brian,

this looks like a nice and quick solution for markup and display. Do you know if there is a script for OpenACS that wiki-like displays differences between two revisions of a wiki page?

Collapse
Posted by Brian Chapados on
I'm not aware of any scripts for openacs that displays diffs for wiki text. It's on my todo list, and I think others would also like to have this feature in the wiki/xowiki packages. You might want to look at the software that we use for displaying diffs for source code in the cvs browser. If it's possible to reuse that code it would be a good starting point. Alternatively, you can check the source code for mediawiki to see how they implement displaying page diffs.

Also, if you're interested in using mediawiki format with xowiki or other apps, I have some code you might be interested in. I've separated the parsing and rendering functions that handle mediawiki text so that they can be used in xowiki (or anywhere). I'm still debugging a few things. I should have some code on the web site within a week or so for others to download/review.

Collapse
Posted by Vinod Kurup on

I've been doing some minor hacking on the wiki package, namely adding the ability to see diffs between versions. I looked around for a diff procedure coded in TCL, but couldn't find one. I did find some procs in tcllib which compute the longest common subsequence. From those, you can get a list of the changes between 2 pieces of text. So, I created this proc which takes 2 pieces of text and then returns a HTML fragment with the changes outlined with the CSS tags 'diff-added' or 'diff-deleted'.

You can see what this looks like at my dev server (if it's up). Click on 'history', then pick 2 versions to diff. Feel free to poke around.

I haven't had a chance to do much testing of it yet, so I wouldn't venture to say that it's complete. But it's a start. Here's the proc:

ad_proc -public wiki::diff {
    -old
    -new
} {
    Perform a UNIX diff on 'old' and 'new', and return a HTML fragment of the changes.

    Requires struct::list (from tcllib)

    @author Vinod Kurup vinod@kurup.com
    @creation-date 2005-10-18

    @param old original text
    @param new new text
    @return HTML fragment of differences between 'old' and 'new'
} {
    package require struct::list

    set old [split $old " "]
    set new [split $new " "]

    # tcllib procs to get a list of differences between 2 lists
    # see: http://tcllib.sourceforge.net/doc/struct_list.html
    set len1 [llength $old]
    set len2 [llength $new]
    set result [::struct::list longestCommonSubsequence $old $new]
    set result [::struct::list lcsInvert $result $len1 $len2]

    # each chunk is either 'deleted', 'added', or 'changed'
    set i 0
    foreach chunk $result {
        set action [lindex $chunk 0]
        set old_index1 [lindex [lindex $chunk 1] 0]
        set old_index2 [lindex [lindex $chunk 1] 1]
        set new_index1 [lindex [lindex $chunk 2] 0]
        set new_index2 [lindex [lindex $chunk 2] 1]
        
        while {$i < $old_index1} {
            lappend output [lindex $old $i]
            incr i
        }

        if { $action eq "changed" } {
            lappend output <d>
            foreach item [lrange $old $old_index1 $old_index2] {
                lappend output $item
            }
            lappend output </d>
            lappend output <a>
            foreach item [lrange $new $new_index1 $new_index2] {
                lappend output $item
            }
            lappend output </a>
            incr i [expr $old_index2 - $old_index1 + 1]
        } elseif { $action eq "deleted" } {
            lappend output <d>
            foreach item [lrange $old $old_index1 $old_index2] {
                lappend output $item
            }
            lappend output </d>
            incr i [expr $old_index2 - $old_index1 + 1]
        } elseif { $action eq "added" } {
            while {$i < $old_index2} {
                lappend output [lindex $old $i]
                incr i
            }
            lappend output <a>
            foreach item [lrange $new $new_index1 $new_index2] {
                lappend output $item
            }
            lappend output </a>
        }
    }
    
    # add any remaining words at the end.
    while {$i < $len1} {
        lappend output [lindex $old $i]
        incr i
    }

    set output [join $output " "]
    set output [string map {"<d>" {<span class="diff-deleted">}
        "</d>" </span>
        "<a>" {<span class="diff-added">}
        "</a>" </span>} $output]

    return "$output"
}
Collapse
11: diff in Tcl (response to 8)
Posted by Andrew Piskorski on
Vinod, nice, I like the colored diff output.

See also the Wiki page diff in Tcl and Wiki History Diff. That includes the same longestCommonSubsequence code that was later included in Tcllib, plus other misc. info.

Collapse
Posted by David Kuczek on
Hey Brian,

I've separated the parsing and rendering functions that handle mediawiki text so that they can be used in xowiki (or anywhere)

... that sounds very good. This is exactly what I would need! 😊

Hey Vinod,

I saw a revision displayer at wiki.tcl.tk ... they might have some code in tcl, but I didn't really check into it:
http://mini.net/tclrevs/590-78

Collapse
Posted by Malte Sussdorff on
Vinod, your code looks nice and should be part of acs-tcl. It would allow us to enhance ETP and other CRM systems as well with a diff view.

Would you put it in there? Or do you think it is not appropriate.

Collapse
Posted by Nick Carroll on
Vinod,

The diff proc looks great! It definitely should go in acs-tcl, as I would like to use it in non-wiki apps too.

Thanks.

Collapse
Posted by Vinod Kurup on
Thanks for the feedback everyone.

Andrew and David, I did look at those pages, but I couldn't find any pure tcl code that does diffs. The 'diffs in TCL' page is outdated. I think all of that code is now in tcllib and that's the code that my proc uses. AFAIK, the Tcl'ers Wiki runs tkdiff to do the diffs and I didn't want to have to install Tk. The link to Pascal Scheffer's code 404's, but I think he uses the CVS diff function.

Malte and Nick, I'd be happy to place this in acs-tcl. Maybe as util::diff? I also don't have CVS commit access anymore, so either someone could restore that (username was vinodk) or I'd be happy to mail the patch to someone who could commit it.