$Revision: 1.34 $Copyright © 2010 W3C® ( MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, and document use rules apply.
This document describes the testing process used by the SPARQL Working Group.
Working Document.
Organization
Manifest Vocabularies
Manifest Structure
File Names
Syntax Tests
Query Evaluation Tests
CSV Result Format Tests
Entailment Evaluation Tests
Update Evaluation Tests
Federated Query Tests
Service Description Tests
Test annotations
How to run the Test Cases
This document is a work in progress. As the SPARQL working group finalizes work on the SPARQL 1.1 standards, the test suite and its description below will be updated to reflect the standards. However, the current tests and descriptions may be incomplete with respect to the current state of the standards.
The SPARQL Working Group uses a test-driven process. The test area is a collection of the current test cases of the working group extending and updating the testcases of the Data Access Working Group.
Tests are divided into collections (corresponding to directories) for manageability. Each collection of tests has a manifest file within its directory (usually named manifest.ttl, but sometimes manifest.n3). There is also a number of overall manifests containing entries pointing to the individual test collection manifests:
@@@ Fix URLs!
The test cases are organised in two directories
The tests as-is shall constitute a test suite that the group will use to generate an implementation report for the SPARQL1.1 Query and SPARQL1.1 Update languages.
@@@ What about other tests, will we use the same structure for e.g. http-update, service-description tests, will we (a) have separate manifest(s) for entailment?
The SPARQL1.1 test manifest files define five vocabularies to express tests and results:
All examples below use these prefix bindings (specified in turtle):
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix mf: <http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#> . @prefix dawgt: <http://www.w3.org/2001/sw/DataAccess/tests/test-dawg#> . @prefix qt: <http://www.w3.org/2001/sw/DataAccess/tests/test-query#> . @prefix ut: <http://www.w3.org/2009/sparql/tests/test-update#> . @prefix sd: <http://www.w3.org/ns/sparql-service-description#> . @prefix rs: <http://www.w3.org/2001/sw/DataAccess/tests/result-set#> .
A manifest is a list (RDF Collection) of tests. Every test has a name (mf:name); many tests also have a comment (rdfs:comment) explaining the purpose of the test. The dawgt:approval predicate relates a test to its official Working Group status (e.g. dawgt:Approved). Tests are grouped (via their rdf:type) as:
Typically, in the test case suite, we use the following suffixes to indicate different file types:
@@@ Fix/check final URL for http://www.w3.org/TR/sparql11-results-json/!
@@@ Will we update the SPARQL Query Results XML Format spec for SPARQL1.1?
Each syntax test has an mf:action, the object of which is a resource identifying a (possible) query string. An example definition of a syntax test is:
<#syntax-basic-01> mf:name "syntax-basic-01.rq" ;
rdf:type mf:PositiveSyntaxTest ;
mf:action <syntax-basic-01.rq> ;
dawgt:approvedBy <http://lists.w3.org/Archives/Public/public-rdf-dawg/2007JanMar/0047> ;
dawgt:approval dawgt:Approved .
A SPARQL implementation passes a mf:PositiveSyntaxTest if it parses the query string without error. A SPARQL implementation passes a mf:NegativeSyntaxTest if it raises an error while attempting to parse the query string.
Each query evaluation test has an mf:action and an mf:result. The object of mf:action is a resource with properties taken from the query evaluation test vocabulary. At a minimum, a test's action includes a qt:data relation and a qt:query relation. The qt:data predicate points to a URI that can be dereferenced to yield the default graph for the test. The qt:query prediate points to a URI that can be dereferenced to yield the query string for the test. Query evaluation tests may also use the qt:graphData predicate to indicate the named graph components of the test's RDF dataset.
Query evaluation tests also contain an mf:result which points to a URI that can be dereferenced to yield the expected results of the test query. These results are expressed in one of several possible ways:
A SPARQL implementation passes a query evaluation test if the graph
produced by evaluating the query against the RDF dataset (and encoding in the
DAWG result set vocabulary, if necessary) is
equivalent [RDF-CONCEPTS]
to the graph named in the result (after encoding in the DAWG result set
vocabulary, if necessary). Note that, solution order only is considered relevant, if the result is expressed in the test suite in the DAWG result set vocabulary, with explicit rs:index triples; otherwise solution order is considered irrelevant for passing. Equivalence can be tested by
checking that the graphs are isomorphic and have identical IRI and
literal nodes. Note that testing whether two result sets are isomorphic is simpler than full graph isomorphism. Iterating over rows in one set, finding a match with the other set, removing this pair, then making sure all rows are accounted for, achieves the same effect.
Query evaluation tests that involve the REDUCED keyword have slightly different passing criteria. These tests are indicated in the manifest files with the mf:resultCardinality predicate with an object of mf:LaxCardinality. To pass such a test, the result set produced by a SPARQL implementation must contain each solution in the expected result set at least once and no more than the number of times that the solution occurs in the expected result set. (That is, the expected result set contains the solutions with cardinalities as they would be if the query did not contain REDUCED; to pass the test, an implementation must produce the correct results with cardinalities between one and the cardinlity in the expected result set.)
An example definition of a query evaluation test is:
<#dawg-regex-002> a mf:QueryEvaluationTest ;
mf:name "regex-query-002" ;
dawgt:approval dawgt:Approved ;
dawgt:approvedBy <http://lists.w3.org/Archives/Public/public-rdf-dawg/2007AprJun/0029.html> ;
rdfs:comment
"Case insensitive unanchored match test" ;
mf:action
[ qt:query <regex-query-002.rq> ;
qt:data <regex-data-01.n3> ] ;
mf:result <regex-result-002.n3> .
CSV Result Format tests are meant to test a SPARQL implementation's ability to serialize query results in the SPARQL 1.1 Query Results CSV Format. This is a lossy format, however, and so cannot be tested in the same manner as the other result formats. Care should be taken to ensure that results produced in the CSV format are compared properly to the expected result values. In all other respects, CSV tests should be treated as query evaluation tests.
An example result set in the CSV Format is:
s,p,o http://example.org/s2,http://example.org/p2,2.2
Due to the lossy nature of the CSV format, the single expected result could match any of the following actual results:
| s | p | o |
|---|---|---|
| <http://example.org/s2> | <http://example.org/p2> | "2.2" |
| <http://example.org/s2> | <http://example.org/p2> | "2.2"^^xsd:string |
| <http://example.org/s2> | <http://example.org/p2> | "2.2"^^xsd:decimal |
Entailment Evaluation tests are special query evaluation tests that additionally (slightly ab-)use the sd:entailmentRegime property from the SPARQL 1.1 Service Description vocabulary to further specify the object of the mf:action property, indicating the expected entailment regime for graphs in the dataset.
A SPARQL implementation passes a query evaluation test if its answers over any graphs in the dataset using the sd:entailmentRegime property comply with the criteria formalised in the SPARQL 1.1 Entailment Regimes document. Apart from that passing tests is as defined for query evaluation tests.
An example definition of an entailment evaluation test is:
:rdf01 rdf:type mf:QueryEvaluationTest ;
mf:name "RDF inference test" ;
dawgt:approval dawgt:NotClassified ;
mf:action
[ qt:query <rdf01.rq> ;
qt:data <rdf01.ttl> ;
sd:entailmentRegime <http://www.w3.org/ns/entailment/RDF> ] ;
mf:result <rdf01.srx>
.
Note that, strictly speaking, this use of the sd:entailmentRegime property - by its specified domain - makes the object of the mf:action attribute a member of the sd:NamedGraph class; the sd:NamedGraph class though has no meaning within the context of test cases, so this domain specification can be savely ignored.
Each update evaluation test has an mf:action and an mf:result. The object of mf:action for an Update evaluation test case is a resource with properties taken from the update evaluation test vocabularies. The latter is used among others to describe the graph store's state before and after the execution of an update. At a minimum, a test's action includes a ut:request relation.
The optional ut:data and ut:graphData relations within the mf:action of an update test case point the status of the Graph Store prior to the update execution described in terms of at most one ut:data property denoting the unnamed graph and optional ut:graphData properties denoting named graphs. The object of the ut:data property is a URI reference to an RDF graph, whereas the objects of the ut:graphData property indicate the named graph components of the Graph Store. Named graphs are described either - analogous to the qt:graphData property from the query test vocabulary - by explicit URI reference (in which case the graph name is supposed to correspong to the respective URI reference), or the object of the ut:graphData property may be a resource further described in terms of an ut:graph and an rdfs:label property. Here, the ut:graph property is a URI reference to an RDF graph, whereas the rdfs:label property with plain literal value indicates the graph's name under which it is accessible in the graph store. The ability to assign a "name" different from the URI reference name explicitly to a named graph in a graph store is needed to denote different graphs by the same name to describe the status of a named graph prior and after execution of an update.
In the case of absence of both ut:data and ut:graphData properties within the mf:action, the graph store is supposed to be empty (i.e., with an empty default graph and no named graphs) prior to execution of the update.
The ut:request predicate points to a URI that can be dereferenced to yield the update query string for the test.
Update evaluation tests also contain an mf:result. The object of mf:result is a resource described in terms of the ut:data and ut:graphData properties. The optional ut:data and ut:graphData properties within a an update evaluation test result denote the state of the graphstore after execution of the query analogous to the ut:data and ut:graphData properties occurring in the mf:action of an update evaluation test. In the case of absence of both ut:data and ut:graphData properties within the mf:result, the graph store is supposed to be empty after execution of the update.
A SPARQL implementation passes a update evaluation test if the graphs in the graph store are equivalent [RDF-CONCEPTS] to the graphs denoted in the mf:action (and mf:result) property, respectively) prior to the update execution (after update execution, respectively). Equivalence can be tested as described above for query evaluation tests.
An example definition of an update evaluation test is:
:insert-data-spo1 a mf:UpdateEvaluationTest ;
mf:name "Simple insert data 1" ;
rdfs:comment "This is a simple insert of a single triple to the unnamed graph of an empty graph store" ;
dawgt:approval dawgt:NotClassified ;
mf:action [
ut:request <insert-data-spo1.ru> ;
ut:data <empty.ttl>
] ;
mf:result [
ut:data <spo.ttl>
] .
In SPARQL 1.1 Federated Query, tests cases contain new vocabulary not used before. These tests check whether the queries with that operator are correct or not. Queries using the SERVICE keyword for accessing remote SPARQL endpoints need a way to describe the data comming from these endpoints which was not previously defined.
The predicate qt:serviceData helps to describe the data comming from these remote SPARQL endpoints in a query. This predicate can also contain the predicate qt:endpoint which contains the URL of the remote SPARQL endpoint.
An example definition of a Federated Query test is:
:service1 rdf:type mf:QueryEvaluationTest ;
mf:name "SERVICE test 1" ;
dawgt:approval dawgt:NotClassified ;
mf:feature sd:BasicFederatedQuery ;
mf:action [
qt:query <service01.rq> ;
qt:data <data01.ttl> ;
qt:serviceData [
qt:endpoint <http://example.org/sparql> ;
qt:data <data01endpoint.ttl>
]
] ;
mf:result <service01.srx> .
A testing service for SPARQL 1.1 Protocol implementations supporting SPARQL 1.1 Service Descriptions has been set up at http://www.w3.org/2009/sparql/sdvalidator. The service performs a number of tests to verify that a submitted endpoint returns RDF that conforms to the service description vocabulary specification. Using content negotiation to request RDF (supporting both Turtle and RDF/XML), the service can be used to generate an EARL implementation report.
@@@ This section might need reconsideration to reflect extensions (like new library functions) that have become standard in SPARQL1.1 @@@
A number of tests in the open-world directory illustrate features of SPARQL by depending on how a SPARQL query processor can extend the set of core types and operations as defined by the operator table [http://www.w3.org/TR/rdf-sparql-query/#OperatorMapping].
These tests are marked by property mf:requires and an object value from one
of the URIs described below.
mf:XsdDateOperationsmf:StringSimpleLiteralCmpxsd:string with the same
lexicial form. This is covered by rules "xsd 1a" and "xsd 1b" from RDF
Semantics [http://www.w3.org/TR/rdf-mt/#DtypeRules].mf:KnownTypesDefault2Neqmf:LangTagAwarenessThis annotation indicates a feature of SPARQL that implementers might note:
mf:IllFormedLiteral@@@ This section shall contain some hints on how to actually run the test suite and generate implementation reports for implementers.