MERESCO combines components written in various programming languages. It uses Python to tie these components together. It integrates Java using JTool.
It began with Lucene
Lucene is a well-known Java library for full-text search. MERESCO used PyLucene which compiled Lucene to native machine code. PyLucene was unstable and did not cover all of Lucene. In 2008 it changed strategy and the performance dropped significantly. We decided to try a completely different approach and that turned out to work very well.
What did we try?
We quickly discovered that compiling Lucene with GCJ was easy and that it resulted in robust, fast and reliable programs. Then we created a Python extension called JTool which mirrors the complete Java API in Python.
How to use?
Here is how you use it in Python:
$ python
>>> import jtool
>>> jtool.load('liblucene-core.so') # compiled lucene-core.jar
>>> from org.apache.lucene.index import IndexReader
>>> reader = IndexReader.open("/indexdir")
This is how all of Lucene is accessed in MERESCO. It runs fast, reliable and with low memory footprint. The code base of JTool is only 1500 lines, there is no code generation and it is completely generic. So the next question is:
Will JTool work for other Java libraries?
In February 2010 we started looking for a more scalable Triple Store for MERESCO. Our choice was OWLIM…. written in Java. While Lucene is quite a large library, OWLIM is even larger. The latter depends on 22 other Java projects, including the Sesame RDF Framework.
Compilation of OWLIM took a bit more effort as we needed to gather all needed jar files and make sure some factories did not get duplicated in the final library. Then we tried to load this library in Python using JTool:
>>> import jtool
>>> jtool.load("libowlim-core.so")
>>> from org.openrdf.repository.sail import SailRepository
>>> from org.openrdf.query import QueryLanguage
>>> ...
This enabled us to insert RDF and execute SPARQL queries on the triple store. Yes it works!
Future of JTool
JTool can not yet call methods with NULL-parameter or Java 5 varargs. It also does not support callbacks in Python yet. We have solutions for these omissions which we will implement this year. Meanwhile, it is easy enough to create a Java wrapper and use this via JTool. So JTool allows us to quickly integrate any Java libraries in MERESCO.
Avaliability
Sources for JTool up to version 4 are available JTool Sources.
JTool version 5 and up are available in binary form JTool Binaries.