$Revision: 1.20 $Copyright © 2004 W3C® ( MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, and document use rules apply.
This document describes the testing process used by the RDF Data Access Working Group.
Working Document.
The RDF Data Access Working Group (DAWG) uses a test-driven process. The test area is a collection of the current test cases of the working group.
Tests are divided into collections (corresponding to directories) for manageability. Each collection of tests has a manifest file within its directory (usually named manifest.ttl, but sometimes manifest.n3). There is also a pair of overall manifests containing entries pointing to the individual test collection manifests: One is a manifest of syntax-only tests (positive and negative tests); the other is a manifest of query-evaluation tests.
The Working Group recently completed reorganizing the test suite. The output of this reorganization process can be found in the data-r2 directory. The tests here are restructured copies of tests from the previous test suite along with new tests aimed at covering as much of the SPARQL Query Language as possible. The purpose of this restructuring is to enhance usability, clear away obsolete tests and provide an up-to-date, consistent, and easy-to-use suite of test cases that SPARQL query language implementors can use to evaluate and report on their implementation.
The Working Group decided on 21 Aug 2007 that the tests as-is constitute a test suite that the group will use to generate an implementation report for the SPARQL Query Language for RDF.
The DAWG's test manifest files define three vocabularies to express tests:
All examples below use these prefix bindings (specified in turtle):
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix mf: <http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#> . @prefix dawgt: <http://www.w3.org/2001/sw/DataAccess/tests/test-dawg#> . @prefix qt: <http://www.w3.org/2001/sw/DataAccess/tests/test-query#> .
A manifest is a list (RDF Collection) of tests. Every test has a name (mf:name); many tests also have a comment (rdfs:comment) explaining the purpose of the test. The dawgt:approval predicate relates a test to its official Working Group status (e.g. dawgt:Approved). Tests are grouped (via their rdf:type) as syntax tests (an rdf:type of either mf:PositiveSyntaxTest or mf:NegativeSyntaxTest) or query-evaluation test (an rdf:type of mf:QueryEvaluationTest).
Each syntax test has an mf:action, the object of which is a resource identifying a (possible) query string. An example definition of a syntax test is:
<#syntax-basic-01> mf:name "syntax-basic-01.rq" ;
rdf:type mf:PositiveSyntaxTest ;
mf:action <syntax-basic-01.rq> ;
dawgt:approvedBy <http://lists.w3.org/Archives/Public/public-rdf-dawg/2007JanMar/0047> ;
dawgt:approval dawgt:Approved .
A SPARQL implementation passes a mf:PositiveSyntaxTest if it parses the query string without error. A SPARQL implementation passes a mf:NegativeSyntaxTest if it raises an error while attempting to parse the query string.
Each query-evaluation test has an mf:action and an mf:result. The object of mf:action is a resource with properties taken from the query-evaluation test vocabulary. At a minimum, a test's action includes a qt:data relation and a qt:query relation. The qt:data predicate points to a URI that can be dereferenced to yield the default graph for the test. The qt:query prediate points to a URI that can be dereferenced to yield the query string for the test. Query-evaluation tests may also use the qt:graphData predicate to indicate the named graph components of the test's RDF dataset.
Query evaluation tests also contain an mf:result which points to a URI that can be dereferenced to yield the expected results of the test query. These results are expressed in one of several possible ways:
A SPARQL implementation passes a query-evaluation test if the graph produced by evaluating the query against the RDF dataset (and encoding in the DAWG result set vocabulary, if necessary) is href="http://www.w3.org/TR/rdf-concepts/#section-graph-equality">equivalent [RDF-CONCEPTS] to the graph named in the result (after encoding in the DAWG result set vocabulary, if necessary). Equivalence can be tested by checking that the graphs are isomorphic and have identical IRI and literal nodes. Note that testing whether two result sets are isomorphic is simpler than full graph isomorphism. Iterating over rows in one set, finding a match with the other set, removing this pair, then making sure all rows are accounted for, achieves the same effect.
Query evaluation tests that involve the REDUCED keyword have slightly different passing criteria. These tests are indicated in the manifest files with the mf:resultCardinality predicate with an object of mf:LaxCardinality. To pass such a test, the result set produced by a SPARQL implementation must contain each solution in the expected result set at least once and no more than the number of times that the solution occurs in the expected result set. (That is, the expected result set contains the solutions with cardinalities as they would be if the query did not contain REDUCED; to pass the test, an implementation must produce the correct results with cardinalities between one and the cardainlity in the expected result set.)
An example definition of a query-evaluation test is:
<#dawg-regex-002> a mf:QueryEvaluationTest ;
mf:name "regex-query-002" ;
dawgt:approval dawgt:Approved ;
dawgt:approvedBy <http://lists.w3.org/Archives/Public/public-rdf-dawg/2007AprJun/0029.html> ;
rdfs:comment
"Case insensitive unanchored match test" ;
mf:action
[ qt:query <regex-query-002.rq> ;
qt:data <regex-data-01.n3> ] ;
mf:result <regex-result-002.n3> .
A number of tests in the open-world directory illustrate features of SPARQL by depending on how a SPARQL query processor can extend the set of core types and operations as defined by the operator table [http://www.w3.org/TR/rdf-sparql-query/#OperatorMapping].
These tests are marked by property mf:requires and an object value from one
of the URIs described below.
mf:XsdDateOperationsmf:StringSimpleLiteralCmpmf:KnownTypesDefault2Neqmf:LangTagAwarenessThis annotation indicates a feature of SPARQL that implementers might note:
mf:IllFormedLiteral