Here is an update on the work being done to integrate OpenFTS with OpenACS:

OpenFTS is a PostgreSQL-based search engine that makes use of the GiST interface available in PostgreSQL. It provides online full text indexing of data and relevance ranking for database searching. It can be used to find documents containing terms with the same linguistic root as the specified word and it can also be used for indexing/searching of multi-lingual and non-text documents. Currently, OpenFTS is implemented as a collection of PERL-scripts.

We are working with Oleg Bartunov and Teodor Sigaev (XWare) to help them open source their search tools under the GPL license. Dan, is currently porting the PERL-scripts into TCL and he is going to move some of the functionality into an aolserver module.

OpenFTS uses PostgreSQL as a database backend where documents are stored as arrays of integers. The index access structure for the array of integers is constructed as an RD-Tree which is implemented using the GiST interface that is available in PostgreSQL. The RD-Tree is a variant of the R-Tree, a popular access method for spatial data. RD stands for "Russian Doll", which describes the transitive containment relation that is fundamental to the tree structure. The RD-Tree data structure implementation provides three predicates between sets: superset, subset, and overlap.

For indexing, a parser is used that reads the document and converts it into a stream of lexemes. Then, morphology or stemming is applied in order to get the base form and finally, an algorithm calculates an ID for each of the lexemes. The resulting array of integers is stored into the database.

When a search query is received, the parser converts it into a stream of lexemes and morphology or stemming is applied to get the base form. Then, each lexeme is assigned an integer ID and finally, SQL queries are generated and executed.

A prominent feature of OpenFTS is the ability to rank documents according to proximity between the words of the search query -- this is accomplished by maintaining coordinate information of the lexemes of each document. For example, if the query is "full text search", documents containing the phrase "full text search" will be ranked higher than documents where words "full", "text", "search" occur in different places.

Information about GiST support in PostgreSQL can be found here (http://www.sai.msu.su/~megera/postgres/gist/).

Additionally, OpenFTS maintains 10 (this is configurable) indexing tables where it stores the frequency and the position of each of the lexemes in a document. The frequency of each lexeme is calculated as:

         occurences of the lexeme in the document
freq = ----------------------------------------------
            number of all lexemes in the document

OpenFTS supports stopwords and stemming. The default method for stemming uses Porter's algorithm.

> There's no reason why static content couldn't be mapped via the
> content repository, too, now that it knows about stuff stored in
> the filesystem ...

Sounds good to me

OK. I have a fairly sizeable project for a non-profit to bid on, for which (from initial reading of the RFP) OpenACS 4.x using CMS and acs-workflow will be a good basis for - if I can get good searching to work too.

Question #1: Any idea when search will become usable (beta level OK) using OpenFTS? Would adding some (TCL? Perl? other?) developer hours help speed that up, or is the issue mostly licencing-related at this stage?

Question #2: How easy/hard/trivial/works already (!) is it for searches to only return results that a user actually has permission to view? And how horribly does doing that extra check on each result hurt in terms of perceived search performance?

Question #3: Are we really going to have a Beta1 of OpenACS 4.x on 01 September 2001? And do we expect it will have OpenFTS searching included in it? What can I usefully do to make that more likely?

I don't want to have to recommend ACS 4.2 and Intermedia, if I can realistically get a working system using OpenACS 4.x and OpenFTS!

Thanks,

The text can come from anywhere, but given that the guts require Postgres one can safely presume it will be used universally by those using the PG version of OpenACS and not at all by those using another RDBMS.

Hi,

I'm about to announce a release of OpenFTS search engine.

Short blurb:

  OpenFTS (Open Source Full Text Search engine)
  is an advanced PostgreSQL-based search engine
  that provides online indexing of data and relevance
  ranking for database searching. Close integration
  with database allows use of metadata to restrict
  search results.

This is actually what we use at fts.postgresql.org and
several other sites. Perl version is available for download,
TCL version (with AOL server support) will be available soon -
actually it just needs to wrap a release.

The OpenFTS project web site - http://openfts.sourceforge.net/

OpenFTS team:
  Oleg Bartunov       - Project Manager
  Teodor Sigaev       - Principal Developer
  Daniel Wickstrom    - Developer of TCL Version
  Neophytos Demetriou - Documentation Writer

Copyright:

  OpenFTS is Copyright 2000-2001 XWare and
  licensed under the GNU General Public License,
  version 2 (June 1991). This means you can use it
  and modify it in any way you want. If you choose to
  redistribute OpenFTS, you must do so under the
  terms of the GNU license.

        Regards,
                Oleg