OWLIM Benchmarking: BSBM

The Berlin SPARQL Benchmark evaluates the performance of query engines in an e-commerce use case: searching products and navigating through related information. Randomized 'query mixes (each consisting of 25 queries) are evaluated continuously through a client application that communicates with the repository through a SPARQL end-point. The benchmark can be executed using datasets of various sizes as well as different numbers of clients each issuing simultaneous query requests. Recent evaluation results for some of the most popular engines are published here.

Although created for benchmarking SPARQL engines, the design of BSBM favors relational databases and other raw-store-based implementations, because it deals with a single fixed data schema and a uniformly dense data representation. Generally speaking, RDF databases are designed to deal efficiently with diverse data, integrated from multiple data sources, encoded against different data schema, resulting in sparse data tables in relational databases.

BSBM also allows for benchmarking relational engines, as there is a SQL-based version of the dataset and queries. However, one should note that the semantics of some of the queries in the SQL version is simpler than those of their SPARQL equivalents, i.e. the SQL versions are less powerful and return different results. The evaluation results published in here show that relational databases are really more suitable for such loads - even though the results are not truly comparable, it is nonetheless significant that relational engines are one order of magnitude faster.

Unlike LUBM, BSBM does not require any inference - an engine can answer the queries correctly without any interpretation of the semantics of the data. Still, the benchmark is useful for evaluating the level of SPARQL support by the engines and their efficiency in handling of multiple client loads.

The table below provides query evaluation results for a growing number of simultaneous clients (1, 4, 8, 16) against the 25M and 100M BSBM dataset versions (containing 25 million and 100 million explicit statements respectively). The values in the table give the total number of query mixes per hour (QMpH) executed by all connected clients.

Performance highlights:

• BigOWLIM scales up well given multiple cores. In the 25M run, 4 clients can execute nearly three times as many queries compared to the single client case, which is close to the theoretical gains, given the quad-core CPU of the desktop machine used. In the 100M run, 4 clients process 2.5 times as many query mixes.
• The performance of OWLIM matches the best results of native triple stores published at the BSBM site.