The Lehigh University Benchmark (LUBM) is probably the most popular benchmarking framework for semantic repositories. It uses a relatively simple ontology describing a university organization structure with synthetically generated datasets. The data generated for each university include a number of departments and related individuals together with relevant descriptions and existing relations between them. The largest 'standard' dataset, LUBM(50), includes 50 universities, described in 1000 OWL files with a total size of 583MB and almost 7 million explicit statements.

The LUBM benchmark consists of sample ontologies (and data), Java benchmark programs, and 'adapter' interfaces, making it easy to adapt to different repositories. The OWLIM 'adapter' for LUBM, which is provided together with the distribution of each OWLIM version, enables users to reproduce the results in their own environment. The framework also contains a data generator, so that one can easily generate a dataset with the desired size.

Last but not least, LUBM comes with 14 queries, the correct evaluation of which requires inference with respect to both the ontology and the data. The semantics of the ontology and the datasets is fully covered by the OWL Horst language fragment. The complexity of the inference process is relatively low because the generated datasets are loosely interconnected.

Loading and Inference

Note: The results presented below have been achieved by generating data for 90,000 universities, which is well above any of the standard LUBM data set sizes.

Highlights:

• BigOWLIM can perform reasoning against 12 billion explicit statements - it loaded a 90,000 LUBM university dataset (run b31lu90kH) at an average speed of 11,543 explicit statements per second;
• BigOWLIM can manage 20 billion statements on a single server worth less than $10,000 - this is the total number of statements inserted in the indices after materialization of LUBM(90k) in run b31lu90kH; the speed of storage and indexing is 18 KSt/second;
• BigOWLIM can manage 1 billion statements on a desktop machine worth $2000 (runs b31lu8kH and b31lu8kN); it takes it less than 5 hours to load the dataset (66 KSt/sec.); loading with inference and materialization takes 14 hours (21 KSt/sec.)
• The "full cycle" run of LUBM(8000), including loading, inference, and query evaluation, takes 15.2 hours (run b31lu8kH).

Query Evaluation

The table presents query evaluation time and number of results per query for different runs

Query evaluation statistics for LUBM(8000) on a desktop machine (run b31lu8kH):

• Average query time: 199 sec. Total query time: 46 min.
• Average QTPR (query-time-per-result): 27 milliseconds.
• Total number of results: 149,185,222. Average number of results per query: 10,656,087.
• Total time for the 'full cycle' run, including loading, inference, and query evaluation: 15.2 hours.

Query evaluation statistics for the run on LUBM(25000):

• Average query time: 349 sec. Total query time: 81 min.
• Average QTPR (query-time-per-result): 36 milliseconds.
• Total number of results: 465,849,803. Average number of results per query: 33,274,986.