reduced the creation step from 5 minutes or so to 40 seconds (this on
If you are filtering and just get a factor of 13 I said it your system that is slow. If I remember correctly you are just requestíng the certain information so somebody else is doing the work!
It's not a server application, so the client is (potentially) doing the indexing if he want to update the catalog. Its the indexing that takes 40 s.
The index is at about 33.5 mb, and should compress into < 10mb. Probably enough to be included into the application.
Hardcoding data of that size into the program is not feasible. Though most newer computers can load it into memory quite quickly. gives you a factor of 100 if everything is in memory that is why perl is so fast.
Including everything in memory would more than double my program heap, and don't forget that this is a java application, so it would never be released before it ending (or at least a sub process ending). Besides, as lucene uses files, i think i can't use the in memory index to search the rdf (using LuceneSail, that uses Sesame and Lucene)