Libraries - Third-Party Java Libraries

Introduction

This section contains information about third-party Java libraries used to build Orbeon Forms. Some of these libraries are patched by Orbeon.

See also:

Re-rooting

Several packages included with Orbeon Forms have been re-rooted (i.e. they have been moved from their standard package name to a package specific to Orbeon, under org.orbeon) because they can conflict with packages provided by applications servers or the JDK.

As of 2012-07-13, we are re-rooting the following libraries with the jarjar tool instead of doing this manually:

  • Xerces
  • MSV
In order to build these with jarjar:
  • update the source libraries under lib/src
  • update build.xml to point to the new source and destination versions
  • run the ant jarjar target
  • commit both the source and destination JAR files and remove the previous versions

Modified libraries

Saxon

As of 2012-07-13:
Purpose: XPath, XSLT, serialization support in XPL and XForms (see below)
Version used in Orbeon Forms: 9.1
Patched: yes
Re-rooted: yes
Patch reason: see commits here https://github.com/orbeon/saxon/commits/master
Source code of Orbeon version: http://github.com/orbeon/saxon

Orbeon Forms makes heavy use of Saxon:

  • It is the default XSLT transformer, for XSLT 1.0 as well as XSLT 2.0.
  • It is used as an XPath 2.0 engine by the XForms engine.
  • Serializations such as XML, HTML, and XHTML all use Saxon.

The Saxon distribution is re-rooted into package org.orbeon instead of net.sf.

When upgrading Saxon:

  • Also make sure to integrate our changes to the dom4j wrappers

Xerces

As of 2012-07-13, we use a stock Xerces 2.11 which is re-rooted by the jarjar ant target. This is possible because since this commit on 2011-03-28, we don't depend on Xerces for XInclude.

In order to produce the re-rooted version, run the ant jarjar target. The resulting JAR file must be checked in.

eXist

As of 2010-05-13:
Purpose: 1) default database for Form Runner 2) XForms persistent state store
Version used in Orbeon Forms: 1.2.6
Patched: yes
Re-rooted: no
Patch reason: see below
Source code of Orbeon version: official version patched as described below

eXist 1.4

As of 2011-01-04, eXist 1.4 is integrated into Orbeon Forms.

Here are the steps taken to patch eXist:

  1. Checkout latest stable 1.4 from eXist Subversion repository
    svn co https://exist.svn.sourceforge.net/svnroot/exist/stable/eXist-1.4.x/ eXist-1.4.x
  2. Build Orbeon Forms (so you have a build/lib/orbeon.jar)
  3. Update ORBEON_HOME and VERSION in tools/patch_exist.sh
  4. Modify line 257 (as of 2010-05-14) of MatchRegexp.java and change it to:
    if(canCache && contextSequence != null && contextSequence.isCacheable())
    NOTE: No longer needed as fix is in eXist since 2010-05-27 rev. 11828
  5. Run: rm src/org/exist/client/UserDialog.java
  6. Make sure your JAVA_HOME is set, e.g. on OS X:
    export JAVA_HOME=/Library/Java/Home
  7. Run tools/patch_exist.sh from the eXist root directory
  8. Merge conf.xml in eXist directory with our exist-conf.xml if needed
  9. [Temporary hack, 2012-05-17] Remove the javax.xml classes from the jar (newer versions are included in xqjapi.jar)
  10. Update Orbeon Forms build.xml with the new names of the jar files if needed
  11. Add/remove new/old JAR files to/from the source code repository
  12. Make sure patch file under $ORBEON_HOME/tools/ is added to the source code repository
NOTE: To build the eXist WAR file if you need it: ./build.sh dist-war

Modifications:
  • By script:
    • Any reference to org.apache.xerces changed to orbeon.apache.xerces
    • Any reference to net.sf.saxon changed to org.orbeon.saxon
    • Instead of SAXParserFactory.newInstance() use org.orbeon.oxf.xml.xerces.XercesSAXParserFactoryImpl()
    • Instead of DocumentBuilderFactory use orbeon.apache.xerces.jaxp.DocumentBuilderFactoryImpl

NOTE: Full-text indexing is configured by default in the sample database that comes with Orbeon Forms starting 2011-01-18.

Also, Alistair Miles and Jeremy Nix have published instructions to update eXist to 1.4 in Orbeon Forms.

Stable eXist on Orbeon github

Since 2011-01-20, you can also get the source of the modified eXist directly from github:
https://github.com/orbeon/eXist-1.4.x/tree/orbeon
The master branch there is meant to tracks the stable 1.4.x branch of eXist via git svn.

$ git svn info

Path: .
URL: https://exist.svn.sourceforge.net/svnroot/exist/stable/eXist-1.4.x
Repository Root: https://exist.svn.sourceforge.net/svnroot/exist
Repository UUID: 8784284e-840f-0410-bd5a-a6a6012a0d1a
Revision: 16254
Node Kind: directory
Schedule: normal
Last Changed Author: dizzzz
Last Changed Rev: 16254
Last Changed Date: 2012-04-16 08:23:48 -0700 (Mon, 16 Apr 2012)

In order to update the master branch:

$ git co master
$ git svn fetch
$ git rebase remotes/git-svn
$ git push

Rebasing to the orbeon branch:

$ git co orbeon
$ git rebase master

[Fix conflicts, build and test that everything is ok.]

$ git push

MSV

As of 2012-07-13:
Purpose: XML Schema and Relax NG validation in XPL and XForms
Version used in Orbeon Forms: CVS 20081113
Patched: yes
Re-rooted: yes [SINCE 2012-07-13]
Patch reason: see below
Source code of Orbeon version: http://github.com/orbeon/msv
Issue 1: Here is the bug. MSV has issues on OS X because DatatypeLibraryLoader in relaxngDatatype.jar is included in Java 6. On OS X, this JAR has a bug, whereby it uses a class loader which is null. It is null because the classes are loaded by the bootstrap class loader.

Fix: simply patching relaxngDatatype.jar is not enough, because DatatypeLibraryLoader is used first. Instead, we:
  • patched MSV to use a modified version of DatatypeLibraryLoader
  • also added build.xml to build MSV
Files:
NOTE 2012-07-13: The patch above might no longer be needed since MSV is re-rooted.

Issue 2: There is another issue with MSV causing a class cast exception, which requires patching com.sun.msv.grammar.DataExp.


In order to produce the re-rooted version, run the ant jarjar target. The resulting JAR file must be checked in.

Flying Saucer

As of Orbeon Forms 4.4, 2013-08-12:

Purpose: XHTML to PDF production in oxf:xhtml-to-xforms, used by Form Runner
Version used in Orbeon Forms: snapshot from 2013-01-24 with fix for NPE
Patched: no
Re-rooted: no
Source code of Orbeon version: http://github.com/orbeon/flyingsaucer
Build: mvn install  (use core and pdf JARs)

Due to #756, switched to the snapshot, which fixed it.

As of Orbeon Forms 3.8, 2010-05-13:
Purpose: XHTML to PDF production in oxf:xhtml-to-xforms, used by Form Runner
Version used in Orbeon Forms: CVS 20100512
Patched: yes
Re-rooted: no
Patch reason: thread leak, see this thread
Source code of Orbeon version: http://github.com/orbeon/flyingsaucer

Saxpath

As of 2010-05-13:
Project site: 
Original source code: 
Purpose: legacy XPath evaluator
Version used in Orbeon Forms: 
Patched: 
Re-rooted: 
Patch reason: 
Source code of Orbeon version: 

Legacy/historical

Xerces

As of 2010-05-13:
Original source code: same
Purpose: XML parsing
Version used in Orbeon Forms: 2.9
Patched: yes
Re-rooted: yes
Patch reason: 1) avoid JAR conflicts 2) XInclude notifications
Source code of Orbeon version: http://github.com/orbeon/xerces

How to re-root:

  • Move folder org/apache to orbeon/apache
  • Replace all org.apache with orbeon.apache (except org.apache.xml.resolver which comes from resolver.jar)
    • NOTE: IntelliJ only replaces in file types it knows! Some text in Xerces going to META-INF do not have extensions so be careful that those also are replaced.
  • Replace in build.xml
    • all org.apache with orbeon.apache (except org.apache.stylebook.StyleBook and org.apache.xerces.util.XJavac)
    • all org/apache with orbeon/apache
  • Enable Java debugging in build.xml with <property name="debug" value="on"/>
  • Build target "jar". This produces xercesImpl.jar

The files we produce:

  • xerces-xercesImpl-2_9_orbeon_yyyymmdd.jar
  • xerces-resolver-2_9_orbeon_yyyymmdd.jar
  • xerces-serializer-2_9_orbeon_yyyymmdd.jar
  • xerces-xml-apis-2_9_orbeon_yyyymmdd.jar

Besides re-rooting, we made some changes to Xerces 2.9. The purpose is to allow being notified of XInclude inclusions as they occur. Xerces currently does not have a mechanism to achieve this.

With Xerces 2.6.2, we extended XIncludeHandler(), copied over some code from Xerces, and somehow managed to make things work this way. However, just upgrading to 2.9 didn't allow this to work.

So with 2.9, we resolved instead to modifying orbeon.apache.xerces.xinclude.XIncludeHandler to directly support notifications. The changes are marked with "BEGIN ORBEON" and "END ORBEON".

Note that right now we use OrbeonParserConfiguration which derives from XIncludeAwareParserConfiguration. This allowed us to plug our own XIncludeHandler. However, the Xerces XIncludeHandler, from which we derive, creates instead an XIncludeParserConfiguration for sub-inclusions, and that in turn creates the stock XIncludeHandler instead of ours. This complexity is why simply forcing our own XIncludeHandler down the throat of XIncludeAwareParserConfiguration doesn't work: it works for first-level inclusions, but not for deeper levels. So for now, we just have a dummy XIncludeHandler that derives from the stock XIncludeHandler. We could probably get rid of it.

Jakarta ORO

No longer used starting 2012-06-22.

Building antlr

Antlr (http://www.antlr.org/) is a tool used by eXist to generate parsers (typically for XQuery in eXist).

There is a conflict with the standard version of antlr and that used by WebLogic, so we re-root antlr from antlr to org.orbeon.antlr. Antlr does not follow Sun's package conventions is originally under a package rooted at "antlr". This does not make it easy to re-root the package, but it is still possible, trying to rename all packages, imports, and most of the antlr occurrences in the source code.

Once the source code has been updated, the fun news is that antlr doesn't use ant. It uses its own build tool. What I did:

  • created a simple build.xml to build all of the antlr Java classes
  • ran build.xml to compile the antlr Java files
  • ran the antlr build tool (this generates errors it it can't find javac, but that is not a problem), something like:
java -Dantlr.build.root=c:/Java/antlr-2.7.4/src/org/orbeon org.orbeon.antlr.build.Tool build
  • ran build.xml again to compile the new Java files created by antlr

The result is a regular set of class files that I just zipped into a JAR file.

eXist 1.2

Latest version: release 1.2.6 (upgraded and patched 2009-07-02)

  1. Checkout eXist from Subversion (if possible using a release)
  2. Build Orbeon Forms (so you have a build/lib/orbeon.jar)
  3. Update ORBEON_HOME and VERSION in tools/patch_exist.sh
  4. Run tools/patch_exist.sh
  5. Merge conf.xml in eXist directory with our exist-conf.xml if needed
  6. Update Orbeon Forms build.xml with the new names of the jar files if needed
  7. Add/remove new/old JAR files to/from the source code repository
  8. Make sure patch file under $ORBEON_HOME/tools/ is added to the source code repository
Modifications:
  • By script:
    • Any reference to org.apache.xerces changed to orbeon.apache.xerces
    • Any reference to net.sf.saxon changed to org.orbeon.saxon
    • Instead of SAXParserFactory.newInstance() use org.orbeon.oxf.xml.xerces.XercesSAXParserFactoryImpl()
    • Instead of DocumentBuilderFactory use orbeon.apache.xerces.jaxp.DocumentBuilderFactoryImpl
  • By hand:
    • Set Last-Modified in RESTServer.java for GET
    • XmlLibraryChecker.java supports our versions of Xerces and Saxon
    • Make free-text search work in the Form Runner summary page. Line 266 of MatchRegexp.java changed to:
      if(canCache && contextSequence != null && contextSequence.isCacheable())
A patch file is created as $ORBEON_HOME/tools/.

2010-03-17: Saxon 8.8 to 9.1 upgrade

In short: lots of changes!
  • XPath function library
    • Entry has new resultIfEmpty member
    • StandardFunction.arg() has extra parameter
    • Type.STRING_TYPE -> BuiltInAtomicType.STRING, etc.
    • isAvailable() and bind() are passed StructuredQName
  • Expression.evaluateAsString() return CharSequence instead of String
  • ExpressionTool.make() has new tracing parameter
  • XMLChar.isValidNCName(), etc. -> Name10Checker.getInstance().isValidNCName()
  • Configuration by default has its own NamePool
    • pass single configuration around in XForms
    • keep single global NamePool for now, shared between XForms and XPL
  • DateTimeValue.makeDateTimeValue(), etc. instead of new DateTimeValue()
  • functions depending on context must have:
    • public int getIntrinsicDependencies() { return StaticProperty.DEPENDS_ON_CONTEXT_ITEM; }
  • IntegerValue -> Int64Value
  • StringValue -> StringLiteral in a few places
  • StringValue.equals() now always throw an exception
    • ->custom comparison, e.g. using simple codepoint comparison
  • encountered issue in unit tests' warp-server.xpl XSLT stylesheet, which exhibited the "Cannot output a namespace node for the default namespace" error
    • -> use prefixes on surrounding elements
  • use Saxon's XPathEvaluator/XPathExpression in sxpath package
  • StaticError is deprecated, use XPathException instead
  • one of the XPathContextMajor constructors is deprecated, use other one
  • Saxon high-level XPath API does not support passing a context position
    • -> added that functionality
  • Other misc API changes!

Comments