Here is an update on the work being done to integrate OpenFTS with OpenACS:
OpenFTS is a PostgreSQL-based search engine that makes use of the GiST interface available in PostgreSQL. It provides online full text indexing of data and relevance ranking for database searching. It can be used to find documents containing terms with the same linguistic root as the specified word and it can also be used for indexing/searching of multi-lingual and non-text documents. Currently, OpenFTS is implemented as a collection of PERL-scripts.
We are working with Oleg Bartunov and Teodor Sigaev (XWare) to help them open source their search tools under the GPL license. Dan, is currently porting the PERL-scripts into TCL and he is going to move some of the functionality into an aolserver module.
OpenFTS uses PostgreSQL as a database backend where documents are stored as arrays of integers. The index access structure for the array of integers is constructed as an RD-Tree which is implemented using the GiST interface that is available in PostgreSQL. The RD-Tree is a variant of the R-Tree, a popular access method for spatial data. RD stands for "Russian Doll", which describes the transitive containment relation that is fundamental to the tree structure. The RD-Tree data structure implementation provides three predicates between sets: superset, subset, and overlap.
For indexing, a parser is used that reads the document and converts it into a stream of lexemes. Then, morphology or stemming is applied in order to get the base form and finally, an algorithm calculates an ID for each of the lexemes. The resulting array of integers is stored into the database.
When a search query is received, the parser converts it into a stream of lexemes and morphology or stemming is applied to get the base form. Then, each lexeme is assigned an integer ID and finally, SQL queries are generated and executed.
A prominent feature of OpenFTS is the ability to rank documents according to proximity between the words of the search query -- this is accomplished by maintaining coordinate information of the lexemes of each document. For example, if the query is "full text search", documents containing the phrase "full text search" will be ranked higher than documents where words "full", "text", "search" occur in different places.
Information about GiST support in PostgreSQL can be found here (http://www.sai.msu.su/~megera/postgres/gist/).