2009/11/25

Automatic XQuery unit tests for OSB projects

Within the OSB 10.3 you use XQuery code to perform data transformations. These transformations can be very complex. I am in favor of TDD and i want to use automatic tests for XQueries. You can use the OSB Console to test XQueries but this is not very practical with complex Queries and takes a lot of time.You can also use XQuery and JDeveloper to run XQueries and this inspired me to write a simple XQUnit module to test xqueries.
This blog shows you how to set up automatic tests with the use of JUnit, XMLUnit and Oracle XQuery engine.

The contraints for me were:
1) The XQuery code should be as-is, so no changes with scripts or whatever. The XQueries are used within the test framework as used within the OSB.
2) The XQuery engine of the WLS must be used. I could use some other XQuery engine, but this would not test the real WLS engine and still errors could occur runtime.
3) The test framework can be used within Java unit tests. This way it can also be used within Maven for instance.


XQuery transformations
XQueries are used within the OSB to perform data transformations. These XQuery modules use external variables that are binded to OSB $variables.

An example of such an XQuery (test.xq) is:
declare namespace wn = "http://www.example.com/werknemer/v1";
declare namespace xf = "http://tempuri.org/test/";
declare namespace np = "http://www.example.com/natuurlijkpersoon/v1";

declare function xf:Medewerker2Werknemer($medewerker as element())
    as element(wn:Werknemer) 
{
  
    
       {concat(data($medewerker/np:Voorvoegsels), " ",  
        data($medewerker/np:Achternaam))}
    
  
};

declare variable $medewerker external;
xf:Medewerker2Werknemer($medewerker)

As mentioned already i used some ideas from Biemond's blog to run an xquery. I added some constructions from XMLUnit and developed the following XQUnit code.
The libraries needed are:
* JUnit.jar (3.7 or higher is good)
* Xmlunit-1.3.jar (Downloaded from XMLUnit site)
* XML Parser v1
* XML Query


package unittest;


import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.Reader;
import java.io.StringWriter;
import javax.xml.namespace.QName;

import javax.xml.transform.stream.StreamSource;

import oracle.xml.parser.v2.DOMParser;
import oracle.xml.parser.v2.XMLDocument;
import oracle.xml.parser.v2.XMLNode;
import oracle.xml.parser.v2.XMLParseException;
import oracle.xml.xqxp.datamodel.XMLSequence;
import oracle.xquery.PreparedXQuery;
import oracle.xquery.XQueryContext;
import oracle.xquery.exec.Utils;

import org.custommonkey.xmlunit.DetailedDiff;
import org.custommonkey.xmlunit.Diff;

import org.custommonkey.xmlunit.NodeInputStream;
import org.custommonkey.xmlunit.jaxp13.Validator;
import org.custommonkey.xmlunit.XMLAssert;

import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

import org.xml.sax.SAXException;


public class XQUnit {

   public XQUnit() {

   }
   
    public static boolean checkSchema(Document doc, String element, String schemaLocation) {
        NodeList list = doc.getElementsByTagName(element);
        Node bodyNode = list.item(0);
        Validator bodyValidator = new Validator();
        File orgschema = new File(schemaLocation);
        bodyValidator.addSchemaSource(new StreamSource(orgschema));
        NodeInputStream bodyStream = new NodeInputStream(bodyNode);
        StreamSource src = new StreamSource(bodyStream);
        boolean valid = bodyValidator.isInstanceValid(src);
        
        return valid;
    }
    
    public static boolean checkSchema(String file, String element, String schema) {
        boolean result = false;
        try {
            DOMParser prs = new DOMParser();
            InputStream is = Utils.getStream(file);
            prs.parse(is);
            XMLDocument doc = prs.getDocument();
            result = checkSchema(doc, element, schema);
        } catch (SAXException e) {
            // TODO
        } catch (IOException e) {
            // TODO
        }
        
        return result;
    }

   public String runXQuery(String xq, String testdoc, String extvar) {
        return runQuery(xq, testdoc, extvar);
    }
   
   public void testXQuery(String xq, String testdoc, String var, String expected) {
     boolean testresult = false;
     try {
       // Run the query
       String resultxml = runQuery(xq, testdoc, var);
                     
       // Create expected result XMLDocument
        int ch;
        StringBuffer strContent = new StringBuffer("");
        FileInputStream fin = null;
         fin = new FileInputStream(expected);
         while( (ch = fin.read()) != -1)
           strContent.append((char)ch);
        fin.close();

       // Check the difference      
       Diff dif = new Diff(strContent.toString(), resultxml);
       XMLAssert.assertXMLEqual(new DetailedDiff(dif), true);

       } catch (FileNotFoundException e) {
        e.printStackTrace();
       } catch (IOException e) {
         e.printStackTrace();
       } catch (SAXException e) {
         e.printStackTrace();
       }   
   }

   private String runQuery(String xq, String testdoc, String extvar)  {
     String resultxml = null;
     try {
       // Create XQuery
       XQueryContext ctx = new XQueryContext();
       Reader strm = new FileReader(xq);
       PreparedXQuery xquery = ctx.prepareXQuery(strm);
       
       // Create test XMLDocument
       DOMParser prs = new DOMParser();
       InputStream is = Utils.getStream(testdoc);
       prs.parse(is);
       XMLDocument doc = prs.getDocument();
       
       // Run XQuery
       xquery.setNode( new QName(extvar), (XMLNode)doc.getDocumentElement());
       XMLSequence seq = xquery.executeQuery();
       
       // Convert to String
       seq.next();
       StringWriter wr = new StringWriter();
       seq.getCurrentItem().getNode().print(wr);
       resultxml = wr.toString();
     } catch (FileNotFoundException e) {
        e.printStackTrace();
        resultxml = null;
     } catch (SAXException e) {
        resultxml = null;
     } catch (IOException e) {
        resultxml = null;
     }
     System.out.println("runXQuery result: " + resultxml);
     return resultxml;

   }

   public static void main(String[] args) {
       XQUnit testXQuery = new XQUnit();
       testXQuery.testXQuery("test.xq", "TestXML.xml",
        "medewerker", "TargetWerknemer1.xml");
   }
}
The following example XML file (Medewerker1.xml) is used as a test example:
 
        Kimmenade
        van de


The expected result of the test xquery is (TargetWerknemer1.xml):

   van de Kimmenade



The test code to test is then (AllTests.java):
package unittest;

import junit.framework.TestSuite;
import junit.textui.TestRunner;

public class AllTests {
    
    public static void main(String[] args) {
        AllTests tests = new AllTests();
        TestRunner.run(tests.suite());
    }
    
    public static TestSuite suite() {
        System.out.println("Start Werknemer2WerkgeverTest");
        
        TestSuite suite = new TestSuite("All");
        suite.addTestSuite(unittest.AchternaamTest.class);
        
        return suite;
    }
}
The next is the actual test case for Achternaam:
package unittest;

import org.custommonkey.xmlunit.XMLTestCase;

public class AchternaamTest extends XMLTestCase {

    protected void setUp() {}
   
    public void testTransformation1() throws Exception {
        XQUnit xq = new XQUnit();
        xq.testXQuery("test.xq", "Medewerker1.xml",
         "medewerker", "TargetWerknemer1.xml");
    }
}

Only test part of the result

In case you want to test only some parts of the result XQuery you can use some handy XMLUnit constructions. The following example shows it.
package unittest;

import java.util.HashMap;

import org.custommonkey.xmlunit.SimpleNamespaceContext;
import org.custommonkey.xmlunit.XMLTestCase;
import org.custommonkey.xmlunit.XMLUnit;

public class VoorkeurnaamTest extends XMLTestCase {

    private SimpleNamespaceContext ctx;

    protected void setUp() {
        HashMap m = new HashMap();
        m.put("wn", "http://www.example.com/werknemer/v1");
        ctx = new SimpleNamespaceContext(m);
        XMLUnit.setXpathNamespaceContext(ctx);
        
    }

    public void testAchternaam() throws Exception {
        XQUnit xq = new XQUnit();
        
        String res = xq.runXQuery("test.xq", "Medewerker1.xml",
         "medewerker");
        assertXpathEvaluatesTo("van de Kimmenade", "/wn:Werknemer/wn:Achternaam", res);
    }
}

Conclusions
* The XQUnit framework is not complete, but can be extended to support multiple external variables.
* The XQUnit framework also supports the check of XMLSchema, but this is not shown in an example.
* At the moment the JDeveloper XQuery engine is used (Somebody knows which WLS jar to include to use the real WLS XQuery engine?) 


Feel free to comment on this blog item!

2009/11/16

Technical CDM model considerations

There were probably already hundreds of discussions about versioning XML schemas and WSDL documents. This Blog considers the strategy described in Thomas Erl's "Web Service Contract Design and Versioning for SOA".
How would you use these practices within a CDM? This Blog considers some options you have when developing XML Schemas out of you logical CDM.
Consider a small CDM in which the following model is used.

LegalEntity <-- NaturalPerson <-- Employee
                  <-- NonNaturalPerson

Namespaces

For these entities you have some possibilities to use namespaces:
1) A group of Entities share the same namespace
Example: http://www.blog/organisation/v1

2) Each Entity gets its own namespace
Example:
http://www.blog/organisation/legalentity/v1
http://www.blog/organisation/naturalperson/v1

The advantages of option 1 are
  • Only 1 namespace needed, namespaces are hell as we all know
  • The entities belong together so this seems logical
The disadvantages
  • The CDM entities are used within Service Contracts (WSDL) and when the version of the namespace changes all Services that use one of these Entities is changed. In fact all Service Consumers have to be adapted.
  • When schema elements are "qualified" all data transformations (XSLT and XQueries) have to be changed, because these use the same namespace within the XPath constructions.
So this option leads to a lot of rework when the major version changes.

Lets consider option1 in which each Entity gets its own namespace.
In this case the disadvantages are
  • You will get a lot of namespaces.
But the advantages are
  • When one Entity changes its namespace only those Services that use this Entity changes.
  • Not all data transformations have to be changed, only those that use this Entity.
  • Better governance on the Entities.
Separate XSD files
As also described as a SOA pattern by Thomas Erl, it is best practice to have each Entity within a separate XSD file.
  • Better reuse of Entities
  • Better governance of the Entities
So you have LegalEntity_v1.xsd and NaturalPerson_v1.xsd files.

Entity inheritance relations
As described by the logical model LegalEntity is a base of NaturalPerson.
You have the possiblity to design this as an xml:extension.

When you have an InternalEmployee and ExternalEmployee Entity how would you consider a Service in which an Employee is needed as input. You could consider to use the xml:choice construct within the message (1). Or you could define two different operations: AddInternalEmployee and AddExternalEmployee (2).

The disadvantage of option (1) is that within your Service (or on the ESB) you will have if-constructions to check with which kind of Employee you are dealing with.

This Blog described some options you have when dealing with the technical CDM.
Please leave you comments on how you are dealing with it...




2009/11/04

JPA problem: Column override on fields

Today I was struggling with WLS 10.3, Workshop 10.3, JPA and Hibernate to get an Entity persistent to a Database. I got the following error:


I used the following code (which was acutally generated by Workshop!):

@Entity()
@Table(name="werknemers")
public class Werknemer implements Serializable {
 //default serial version id, required for serializable classes.
 private static final long serialVersionUID = 1L;
 
 @Id
 private String persNummer; 
 private String emailInternet;
    public Werknemer() {
    }

 @Basic()
 @Column(name="email_internet", nullable=false, length=128)
 public String getEmailInternet() {
  return this.emailInternet;
 }
 public void setEmailInternet(String emailInternet) {
  this.emailInternet = emailInternet;
 }
}

As you can see from the error code the column name "email_internet" (as indicated by @Column) was not taken, but instead the "emailInternet" property was taken.
I could not find the problem, but the following helped. Move the JPA annotation above the private property, like this:

        @Basic()
 @Column(name="email_internet", nullable=false, length=128)
 private String emailInternet;

This solved my problem, but not sure if this is a Hibernate, JPA bug?




2009/11/02

International SOA Symposium 2009



On the 22th and 23th of october 2009 the second SOA Symposium and first Cloud symposium took place in Rotterdam. For me it was the first SOA Symposium. The main theme of this symposium was the introduction of the SOA Manifesto by some respectful people of the SOA community like Thomas Erl, Anne Thomas Manes and Grady Booch.

There were 6 SOA tracks, 2 Cloud computing tracks and a working group that worked on Candidate SOA Design patterns and the SOA Manifesto.

The first day I followed:
The Convergence and Unification of SOA, EDA & BPM, by Manas Deb, Oracle
This presentation was about to think about the business events that occur within your business and to take these events into account early. So it should be natural to combine events, services and processes to create the need for the business. Events were also used to split complex processes.
At the end of the presentation his new book was given away and I got hold of one piece (see also here for a book review).

Service Modelling: Making Sure Your Services Deliver Value, by Richard Watson, Burton Group

This talk was about focus on service design principles rather than technology. The identification and definition of sharable and reusable services is still more art than science. He believes that developers must start thinking out-of-the-box instead of in there technical world.

I agree with him to some extend, but that's what they are hired to do, maybe the role of the architect is more relevant here. I also see that architects just design point-2-point interfaces without thinking "Services".

SOA and Cloud Computing - A Match Made in Heaven? by David Chappell, Oracle


I followed this presentation to get some feeling about what Cloud computing is, and what the relationship with SOA is.
He began his talk about the weeks/months it takes to deploy new applications/services on hardware. He then introduces the Cloud in which hardware and other resources can be managed by configuration. This significantly increases the speed of deployment.
Then he also introduces the Grid in which data is made persistent and available for all services. There I get lost a little and wonder if this is the way you would want to go. I would expect this to be hided from the Service Consumer, and that this data is accessible through a Service. He foresees public- and private clouds.

Mission impossible? Applying Agile to the World of SOA and ERP, by Sander Hoogendoorn (Capgemini) and Twan van den Broerk (Ciber)
This was a great real example of how Agile can help (even) SAP projects to deliver on time and what the customer wants. This presentation was about the use of Smart Usecases and Scrum within a SAP project.


 I like the use of Scrum or other iterative and agile approaches when doing a software project. I also strongly believe in its value in integration and SOA based projects. This presentation showed its value in real example project.




 


Next Generation SOA: A Web-Centric Perspective, by Stefan Tilkov, InnoQ

Stefan Tilkov is a pioneer on exposing Services as REST services, so this talk was also on REST. After an introduction on resources, URI and representations he talked about Atom.
He concluded with some recommendations:
* Ensure your webapps are RESTful
* Expose machine readable info via HTTP Get
* Manage your metadata with RESTful HTTP
* Use WS-* if policies or legacy forces you

I personally like the REST style of interfacing, but only if this is the "best" way. I really like the fact that each resource object gets its own URI that can be used for linking. I also like the fact that a resource can have multiple representations. So a resource can be represented as a WebPage, POX (Plain Old XML), JSON or whatever. The WS-* versus the REST continues ;)

Between the sessions..
Between the sessions I had the opportunity to talk to some interesting people. I spoke among others with Brian Lokhorst who works for the DTCA ("Belastingdienst") and he also uses SOA Design Patterns to encapsulate legacy systems.
Furthermore I talked with a Belgian company (Collibra) that developed a product for managing "semantic data". This product manages the enterprise data, not only syntactically but also semantically and can generate technical models and transformations. At runtime it will keep track of semantic correctness.


There was a funny presentation by Gray Booch who could not be physically present and did his presentation via SecondLife.

Day 2

Introducing the modern ESB, by Satadru Roy from Sun and Brain Loesgen from Microsoft
The only interesting and funny part was that SUN and Microsoft did this presentation together. There was a demo of Biztalk but further nothing new.

 

Understanding SOA Governance by Harold van Aalst, Progress Software
This was a presentation about Progress Actional governance tool of Progress.
I wonder if these tool can really help you. There were only a few Dutch customers using the tool at that moment.

Real-life use cases of SOA Design patterns at the Dutch Tax and Customs Administration, by Brian Lokhorst
This talk was about the usage of two SOA design patterns used within the DTCA.

Introducing Transactions to REST, by Michael Musgrove, Red Hat
This talk was about transactions (Compensations and Atomic) the REST way. A demo was given using the
RESTeasy framework.

It was a bit ackward interface of which the client must have knowledge. It remains a complex problem to solve also within the WS-* world in which the standard is not used very much (yet)?

SOA Manifesto
But the main theme of the symposium was the introduction of the SOA Manifesto which can be seen here. I hope this will help take SOA on a higher level, let SOAlize the enterprise !!