2009/11/25

Automatic XQuery unit tests for OSB projects

Within the OSB 10.3 you use XQuery code to perform data transformations. These transformations can be very complex. I am in favor of TDD and i want to use automatic tests for XQueries. You can use the OSB Console to test XQueries but this is not very practical with complex Queries and takes a lot of time.You can also use XQuery and JDeveloper to run XQueries and this inspired me to write a simple XQUnit module to test xqueries.
This blog shows you how to set up automatic tests with the use of JUnit, XMLUnit and Oracle XQuery engine.

The contraints for me were:
1) The XQuery code should be as-is, so no changes with scripts or whatever. The XQueries are used within the test framework as used within the OSB.
2) The XQuery engine of the WLS must be used. I could use some other XQuery engine, but this would not test the real WLS engine and still errors could occur runtime.
3) The test framework can be used within Java unit tests. This way it can also be used within Maven for instance.


XQuery transformations
XQueries are used within the OSB to perform data transformations. These XQuery modules use external variables that are binded to OSB $variables.

An example of such an XQuery (test.xq) is:
declare namespace wn = "http://www.example.com/werknemer/v1";
declare namespace xf = "http://tempuri.org/test/";
declare namespace np = "http://www.example.com/natuurlijkpersoon/v1";

declare function xf:Medewerker2Werknemer($medewerker as element())
    as element(wn:Werknemer) 
{
  
    
       {concat(data($medewerker/np:Voorvoegsels), " ",  
        data($medewerker/np:Achternaam))}
    
  
};

declare variable $medewerker external;
xf:Medewerker2Werknemer($medewerker)

As mentioned already i used some ideas from Biemond's blog to run an xquery. I added some constructions from XMLUnit and developed the following XQUnit code.
The libraries needed are:
* JUnit.jar (3.7 or higher is good)
* Xmlunit-1.3.jar (Downloaded from XMLUnit site)
* XML Parser v1
* XML Query


package unittest;


import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.Reader;
import java.io.StringWriter;
import javax.xml.namespace.QName;

import javax.xml.transform.stream.StreamSource;

import oracle.xml.parser.v2.DOMParser;
import oracle.xml.parser.v2.XMLDocument;
import oracle.xml.parser.v2.XMLNode;
import oracle.xml.parser.v2.XMLParseException;
import oracle.xml.xqxp.datamodel.XMLSequence;
import oracle.xquery.PreparedXQuery;
import oracle.xquery.XQueryContext;
import oracle.xquery.exec.Utils;

import org.custommonkey.xmlunit.DetailedDiff;
import org.custommonkey.xmlunit.Diff;

import org.custommonkey.xmlunit.NodeInputStream;
import org.custommonkey.xmlunit.jaxp13.Validator;
import org.custommonkey.xmlunit.XMLAssert;

import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

import org.xml.sax.SAXException;


public class XQUnit {

   public XQUnit() {

   }
   
    public static boolean checkSchema(Document doc, String element, String schemaLocation) {
        NodeList list = doc.getElementsByTagName(element);
        Node bodyNode = list.item(0);
        Validator bodyValidator = new Validator();
        File orgschema = new File(schemaLocation);
        bodyValidator.addSchemaSource(new StreamSource(orgschema));
        NodeInputStream bodyStream = new NodeInputStream(bodyNode);
        StreamSource src = new StreamSource(bodyStream);
        boolean valid = bodyValidator.isInstanceValid(src);
        
        return valid;
    }
    
    public static boolean checkSchema(String file, String element, String schema) {
        boolean result = false;
        try {
            DOMParser prs = new DOMParser();
            InputStream is = Utils.getStream(file);
            prs.parse(is);
            XMLDocument doc = prs.getDocument();
            result = checkSchema(doc, element, schema);
        } catch (SAXException e) {
            // TODO
        } catch (IOException e) {
            // TODO
        }
        
        return result;
    }

   public String runXQuery(String xq, String testdoc, String extvar) {
        return runQuery(xq, testdoc, extvar);
    }
   
   public void testXQuery(String xq, String testdoc, String var, String expected) {
     boolean testresult = false;
     try {
       // Run the query
       String resultxml = runQuery(xq, testdoc, var);
                     
       // Create expected result XMLDocument
        int ch;
        StringBuffer strContent = new StringBuffer("");
        FileInputStream fin = null;
         fin = new FileInputStream(expected);
         while( (ch = fin.read()) != -1)
           strContent.append((char)ch);
        fin.close();

       // Check the difference      
       Diff dif = new Diff(strContent.toString(), resultxml);
       XMLAssert.assertXMLEqual(new DetailedDiff(dif), true);

       } catch (FileNotFoundException e) {
        e.printStackTrace();
       } catch (IOException e) {
         e.printStackTrace();
       } catch (SAXException e) {
         e.printStackTrace();
       }   
   }

   private String runQuery(String xq, String testdoc, String extvar)  {
     String resultxml = null;
     try {
       // Create XQuery
       XQueryContext ctx = new XQueryContext();
       Reader strm = new FileReader(xq);
       PreparedXQuery xquery = ctx.prepareXQuery(strm);
       
       // Create test XMLDocument
       DOMParser prs = new DOMParser();
       InputStream is = Utils.getStream(testdoc);
       prs.parse(is);
       XMLDocument doc = prs.getDocument();
       
       // Run XQuery
       xquery.setNode( new QName(extvar), (XMLNode)doc.getDocumentElement());
       XMLSequence seq = xquery.executeQuery();
       
       // Convert to String
       seq.next();
       StringWriter wr = new StringWriter();
       seq.getCurrentItem().getNode().print(wr);
       resultxml = wr.toString();
     } catch (FileNotFoundException e) {
        e.printStackTrace();
        resultxml = null;
     } catch (SAXException e) {
        resultxml = null;
     } catch (IOException e) {
        resultxml = null;
     }
     System.out.println("runXQuery result: " + resultxml);
     return resultxml;

   }

   public static void main(String[] args) {
       XQUnit testXQuery = new XQUnit();
       testXQuery.testXQuery("test.xq", "TestXML.xml",
        "medewerker", "TargetWerknemer1.xml");
   }
}
The following example XML file (Medewerker1.xml) is used as a test example:
 
        Kimmenade
        van de


The expected result of the test xquery is (TargetWerknemer1.xml):

   van de Kimmenade



The test code to test is then (AllTests.java):
package unittest;

import junit.framework.TestSuite;
import junit.textui.TestRunner;

public class AllTests {
    
    public static void main(String[] args) {
        AllTests tests = new AllTests();
        TestRunner.run(tests.suite());
    }
    
    public static TestSuite suite() {
        System.out.println("Start Werknemer2WerkgeverTest");
        
        TestSuite suite = new TestSuite("All");
        suite.addTestSuite(unittest.AchternaamTest.class);
        
        return suite;
    }
}
The next is the actual test case for Achternaam:
package unittest;

import org.custommonkey.xmlunit.XMLTestCase;

public class AchternaamTest extends XMLTestCase {

    protected void setUp() {}
   
    public void testTransformation1() throws Exception {
        XQUnit xq = new XQUnit();
        xq.testXQuery("test.xq", "Medewerker1.xml",
         "medewerker", "TargetWerknemer1.xml");
    }
}

Only test part of the result

In case you want to test only some parts of the result XQuery you can use some handy XMLUnit constructions. The following example shows it.
package unittest;

import java.util.HashMap;

import org.custommonkey.xmlunit.SimpleNamespaceContext;
import org.custommonkey.xmlunit.XMLTestCase;
import org.custommonkey.xmlunit.XMLUnit;

public class VoorkeurnaamTest extends XMLTestCase {

    private SimpleNamespaceContext ctx;

    protected void setUp() {
        HashMap m = new HashMap();
        m.put("wn", "http://www.example.com/werknemer/v1");
        ctx = new SimpleNamespaceContext(m);
        XMLUnit.setXpathNamespaceContext(ctx);
        
    }

    public void testAchternaam() throws Exception {
        XQUnit xq = new XQUnit();
        
        String res = xq.runXQuery("test.xq", "Medewerker1.xml",
         "medewerker");
        assertXpathEvaluatesTo("van de Kimmenade", "/wn:Werknemer/wn:Achternaam", res);
    }
}

Conclusions
* The XQUnit framework is not complete, but can be extended to support multiple external variables.
* The XQUnit framework also supports the check of XMLSchema, but this is not shown in an example.
* At the moment the JDeveloper XQuery engine is used (Somebody knows which WLS jar to include to use the real WLS XQuery engine?) 


Feel free to comment on this blog item!

17 comments:

  1. Thanks for sharing the example. I am just wondering do you have any suggestions for doing file validation using OSB for incomplete files, bad schema, corrupt files etc... Any suggestions are welcome.

    ReplyDelete
  2. @Khad: Do you mean within testing or at runtime on the OSB? Furthermore what are the type of files?

    See also here for an example of custom file to XML format: http://tinyurl.com/yjn89c2
    You can check the XML against an XSD afterwards.

    ReplyDelete
  3. I would like to do some validation for XML and JPEG files which will be my input files to OSB. Before I do any further processing on these files, I would like to check for whether files are incomplete, empty files, bad schema, corrupt files etc... If you come across with these scenario's please let me know. I am not sure whether we can do anything like this in OSB or do we just have to write some java code and call it as a jar file in OSB. Any ideas.... Thanks

    ReplyDelete
  4. XML files can be checked by the Validate action on the OSB (VETO Pattern = Validate Enhance Transform Operate).
    This is of course the syntactic checking. Semantic checking must be done within Java (unfortunately XML Schematron is not supported yet, as i know).
    JPEG also has to be done by Java callout, i'm afraid.

    ReplyDelete
  5. Hi Roger, I cactually cannot see where you are using the WEBLOGIC Xquery engine ??? You are importing oracle.* xquery classes, or ?

    -K

    ReplyDelete
  6. Hello K,
    You are right, I used the JDeveloper libraries as I state in the Conclusion.
    Both implement the XQuery standard, so this should not be a problem (in theory). In practice I did not ran into differences yet.

    Thank you for your comment.

    Regards
    Roger

    ReplyDelete
  7. Hi Roger,

    Where do i download the xquery.jar from?

    Regards,
    Kiran

    ReplyDelete
  8. Hello Kiran,

    You can find the XQuery and XML Pareser v2 jars from the JDeveloper installation directory.

    Regards
    Roger

    ReplyDelete
  9. Thanks Roger,

    I am getting below runtime exception:
    I am trying to test my own xq, Also got the same error with your example xq & input, output files.

    Any idea on how to solve this.

    Tx,
    Kiran






    Exception in thread "Main Thread" oracle.xquery.XQException: XPTY0004: It is a type error if, during the static analysis phase, an expression is found to have a static type that is not appropriate for the context in which the expression occurs, or during the dynamic evaluation phase, the dynamic type of a value does not match a required type as specified by the matching rules in 2.5.4 SequenceType Matching.
    at oracle.xquery.exec.BindExpr.checkType(BindExpr.java:161)
    at oracle.xquery.exec.BindExpr.setNode(BindExpr.java:118)
    at oracle.xquery.comp.SAXComp.setNode(SAXComp.java:4778)
    at oracle.xquery.PreparedXQuery.setNode(PreparedXQuery.java:315)
    at be.telenet.soa.testing.unittests.XQUnit.runQuery(XQUnit.java:122)
    at be.telenet.soa.testing.unittests.XQUnit.testXQuery(XQUnit.java:82)
    at be.telenet.soa.testing.unittests.XQUnit.main(XQUnit.java:146)

    ReplyDelete
  10. With your example XQuery, I got below error:

    Exception in thread "Main Thread" oracle.xquery.XQException: XPTY0004: It is a type error if, during the static analysis phase, an expression is found to have a static type that is not appropriate for the context in which the expression occurs, or during the dynamic evaluation phase, the dynamic type of a value does not match a required type as specified by the matching rules in 2.5.4 SequenceType Matching.
    Detail: Encounter a type error during the static type checking phase
    at oracle.xquery.exec.StaticTypingVisitor.raiseStaticTypeError(StaticTypingVisitor.java:1888)
    at oracle.xquery.exec.StaticTypingVisitor.handleError(StaticTypingVisitor.java:1903)
    at oracle.xquery.exec.StaticTypingVisitor.handleError(StaticTypingVisitor.java:1897)
    at oracle.xquery.exec.StaticTypingVisitor.normalizeParameter(StaticTypingVisitor.java:348)
    at oracle.xquery.exec.StaticTypingVisitor.visitFunctionDefn(StaticTypingVisitor.java:2615)
    at oracle.xquery.func.FunctionDefn.staticTypeChecking(FunctionDefn.java:154)
    at oracle.xquery.exec.QueryState.staticTypingFnDecls(QueryState.java:1022)
    at oracle.xquery.PreparedXQuery.staticTyping(PreparedXQuery.java:511)
    at oracle.xquery.PreparedXQuery.prepare(PreparedXQuery.java:434)
    at oracle.xquery.PreparedXQuery.executeQuery(PreparedXQuery.java:544)
    at oracle.xquery.PreparedXQuery.executeQuery(PreparedXQuery.java:562)
    at be.telenet.soa.testing.unittests.XQUnit.runQuery(XQUnit.java:125)
    at be.telenet.soa.testing.unittests.XQUnit.testXQuery(XQUnit.java:82)
    at be.telenet.soa.testing.unittests.XQUnit.main(XQUnit.java:147)

    ReplyDelete
  11. Hello Kiran,

    Could you zip your JDeveloper project and I will have a look at it! (send to rvdkimmenade@gmail.com)

    Regards
    Roger

    ReplyDelete
  12. Hi,

    I am keep getting following error at runtime. Is this an issue of jar file version?

    My xquery version is : XDK 11.1.0

    Exception in thread "main" java.lang.NoSuchMethodError: oracle.xquery.comp.CompState.setModulesTable(Ljava/util/Hashtable;)V
    at oracle.xquery.PreparedXQuery.(PreparedXQuery.java:157)
    at oracle.xquery.PreparedXQuery.(PreparedXQuery.java:147)
    at oracle.xquery.PreparedXQuery.(PreparedXQuery.java:211)
    at oracle.xquery.XQueryContext.prepareXQuery(XQueryContext.java:464)

    Code Line :
    PreparedXQuery xq = ctx.prepareXQuery(queryString);

    Thanx,
    Chirag

    ReplyDelete
  13. @Chirag: Yes it is a jar issue. I used the JDeveloper libraries of version 10.x.
    You should try them.

    Let me know if this solves your problem.

    ReplyDelete
  14. Hi Roger,
    Can you please share the project used in this example.

    Thanks,
    Kris

    ReplyDelete
  15. @Kris

    Hello, unfortunately i do not have this code anymore, my system was crashed where i kept it.
    What is you problem, maybe i can help.

    Thx
    Roger

    ReplyDelete
  16. Hi Roger,
    I am getting the below error
    oracle.xquery.XQException: XPTY0004: It is a type error if, during the static analysis phase, an expression is found to have a static type that is not appropriate for the context in which the expression occurs, or during the dynamic evaluation phase, the dynamic type of a value does not match a required type as specified by the matching rules in 2.5.4 SequenceType Matching.

    I have emailed you my JDeveloper project to rvdkimmenade@gmail.com. can you please tell me how to proceed further.

    Thanks,
    Kris

    ReplyDelete