Orbeon Search

Comments? Feedback?

This wiki does not yet support public comments (a limitation of Google Sites), so we encourage you to post your comments either:

On Twitter by responding to @orbeon.

On our community mailing list: subscribe sending an email to ops-users-subscribe@ow2.org (content of subject/body doesn't matter), you'll get a response with the email to use to send your message to the community mailing list.

Recent site activity

Form Builder - Integration Notes and Persistence Layer

Contents

  1. 1 Installation
  2. 2 Form types and form data
  3. 3 Overall architecture
  4. 4 Integration points
    1. 4.1 Form Runner and Form Builder URLs
    2. 4.2 Virtual hierarchy of data
    3. 4.3 Built-in persistence
    4. 4.4 Configuring persistence through properties
    5. 4.5 Persistence API
    6. 4.6 Support for versioning
  5. 5 Data exchange format
  6. 6 Scenario

Installation

Form Builder is distributed as a Java WAR file. You install it into a Servlet container, preferably Tomcat 5.5.

Installation notes are availabe online as part of the Orbeon Forms tutorial:


We recommend, for testing, that:

  • you deploy the WAR file under the "orbeon" context
  • you deploy as an "exploded WAR" file, that is an unzipped WAR file

Once the WAR file is deployed, restart your container and access the following URL, assuming a default port of 8080 for your container:

This is the Summary page of Form Builder. The page should look like the following (most likely without any content in the summary table):


Then select "New Form". This takes you to a new empty form:
If you get to this point, the installation has succeeded.

Form types and form data

We make a distinction between:

  • Form types, for example an "address" form vs. a "claim" form
  • Form data, for example a given instance of the "address" form filled out by a particular user.

In Form Builder, a form type is identified by a two-level hierarchy of names:

  • An application name, which could be a company name such as example "orbeon", "acme", or an actual project name such as "mercury", "foobar", etc.
  • A form type name, which is local to an application name, for example "address" or "claim".

This two-level hierarchy allows for easy grouping of forms, and allows using a single instance of Form Builder to host distinct applications.

Form data is identified by a three-level hierarchy which includes:

  • the application name/form type name couple that identifies the form type
  • plus a unique form data id provided by Form Runner for each instance of form data

Overall architecture

Form Builder actually consists of two parts:

  • Form Builder proper, aimed at form authors, allows you to create, edit and manage forms types.
  • Form Runner, aimed at users of the forms, allows you to create, edit and manage form data related to published form types.

Form Builder/Runner are implemented on top of the open source Orbeon Forms platform. They are implemented entirely with the following technologies:

  • XForms for the user interface (along with XHTML and CSS)
  • XPL (XML pipelines), mainly as a glue between XForms and services such as persistence services
  • XSLT, mainly to provide higher-level components for Form Runner

Form Builder is written essentially in XForms, but it also produces, for each form type edited, an XForms file which Form Runner can load and run. In short, Form Builder is an XForms application which is able to produce other XForms applications.

Both Form Builder and Form Runner share a persistence layer, which in turn allows them to talk to databases or external services. Form Builder/Runner comes with a built-in XML database called eXist, which by provides persistence out of the box. However, you can also access an external eXist instance, or implement your entirely custom persistence layer in your own system.

The following figure illustrates the overall architecture:

Integration points

Form Builder/Form Runner integrate with other systems through two main means:

  • Plain URLs, through which you access Form Runner and Form Builder's pages
  • A configurable persistence API based on REST (that is, through HTTP)

The URLs can be accessed simply by using hyperlinks or redirects from another application.

The persistence API can be implemented either within Orbeon Forms (like for example the built-in eXist persistence layer), or within an external system.

Form Runner and Form Builder URLs

Form Runner/Form builder attempt to use friendly URLs.

The following URL patterns are followed:
  • Summary page for a given form type:
    /fr/[APPLICATION]/[FORM_TYPE]/summary
  • New empty form data:
    /fr/[APPLICATION]/[FORM_TYPE]/new
  • Edit existing form data:
    /fr/[APPLICATION]/[FORM_TYPE]/edit/[FORM_DATA]
  • Read-only HTML view:
    /fr/[APPLICATION]/[FORM_TYPE]/view/[FORM_DATA]
  • Read-only PDF view:
    /fr/[APPLICATION]/[FORM_TYPE]/pdf/[FORM_DATA]
For Form Builder itself:
  • Summary page:
    /fr/orbeon/builder/summary
  • New empty form type:
    /fr/orbeon/builder/new
  • Edit existing form type:
    /fr/orbeon/builder/edit/[FORM_TYPE_ID]
NOTE: All paths above are relative to the deployment context, e.g the actual URLs start with http://localhost:8080/orbeon/fr/...

NOTE: As of May 2009, the paths have been changed to not include a trailing slash so as to help working around a Firefox bug related to file downloads.

Virtual hierarchy of data

Form Builder/Form Runner access data under a virtual hierarchy or URLs, not unlike directories or folders in a filesystem. However this hierarchy can be physically located in different places:


  • An XML database, like eXist
  • A disk-based filesystem
  • Your own system, which you can implement on top of a database or other type of storage

Following XML database technology, we use the terms collections and resources instead of directories and files.

The hierarchy looks like this:

  • acme/
    • address/
      • form/
        • form.xhtml
        • logo.png
        • style.css
        • form.pdf
      • data/
        • 6E6FC50F4BB945235EB5B573F2C7E695
          • data.xml
          • headshot.jpg
        • C37284A414E5F266E3ECEE8C8AEDB6F0
          • data.xml
    • contract/
      • form/
        • ...
      • data/
        • ...
  • foobar/
    • ...

The hiearchy is organized as follows:

  • At the top-level there is one collection per application
  • Within an application collection, there is one collection per form type
  • Within a form type, there is one collection called "form" for the form definitions produced by Form Builder, and one collection called "data" for form data produced by Form Runner
  • Each "form" collection contains:
    • form.xhtml: the main form definition, which is an XHTML+XForms resource
    • optional attachments, such as images, CSS, and PDF files uploaded by the form author when editing the form type
  • Each "data" collection contains one collection for each form data id, identified by an automatically-generated UUID
  • Each form data collection contains:
    • data.xml: the main form data document
    • optional attachments, such as images uploaded by the user when editing the form data

Built-in persistence

Form Builder/Form Runner come with the following built-in persistence implementations:

Storage Notes More information
eXist  A full persistence layer implemented on top of the open source eXist database. You can setup this persistence layer to either use the internal eXist database, or an external one.  
Oracle A full persistence layer implemented on top of the Oracle database. Forms and data are stored using Oracle XMLType columns.  Oracle persistence layer
MySQL A full persistence layer implemented on top of MySQL. MySQL persistence layer
File system (resource) A minimal, read-only persistence layer implemented on top of Orbeon Forms' resource manager. This allows reading form definitions stored within the Orbeon Forms WAR file.  

In the future, Form Builder could include more persistence implementations, for targeting additional relational databases or CMS systems.

Configuring persistence through properties

Storage for the persistence hierarchy can be configured at multiple levels:
  • Globally
  • For each application
  • For each form type within an application
  • For each form type (the form definition and associated resources) vs. form data (filled-out form data)
This allow you for example to store certain form types on disk, while storing the associated data, as filled-out by users, in a database.

This configuration is done in an XML property file called properties.xml. This file is located under your exploded WAR file at the following location:

WEB-INF/resources/config/properties.xml

You can edit this file with any text editor. Some of the properties do not require a restart or Orbeon Forms, but in doubt you can restart the Servlet container.

Persistence configuration properties look like this in properties.xml:

    <property as="xs:anyURI"  name="oxf.fr.persistence.app.uri.*.*.*"                value="/fr/service/exist"/>
    <property as="xs:anyURI"  name="oxf.fr.persistence.app.uri.orbeon.builder.form"  value="/fr/service/resource"/>
    <property as="xs:anyURI"  name="oxf.fr.persistence.app.uri.orbeon.bookcast.form" value="/fr/service/resource"/>

All those property names start with oxf.fr.persistence.app.uri to denote that they configure the Form Runner's persistence layer per application. Follows:
  1. A string to specify the application name, like "orbeon" or "acme".
  2. A form type name, like "bookcast" or "address". Note that "builder" is a special form type name for Form Builder itself.
  3. Whether the configuration regards form data ("data"), or the form configuration files ("form").
The value of the properties are URIs, which point to the the location of the root of the virtual hierarchy.

There are two types of persistence services:
  • Local: those are handled by Orbeon Forms, and start with the prefix /fr/service/
  • Remote: those are handled by third-party software, and are absolute URLs starting with http: or https:
The properties are interpreted hierarchically: you may specify the configuration more or less specifically:
  • oxf.fr.persistence.app.uri.*.*.* -> form types and form data for all applications
  • oxf.fr.persistence.app.uri.*.*.data -> form data for all applications
  • oxf.fr.persistence.app.uri.*.*.form -> form types for all applications
  • oxf.fr.persistence.app.uri.orbeon.*.form -> form types for all forms in application "orbeon"
  • oxf.fr.persistence.app.uri.orbeon.*.data -> form data for all forms in  application "orbeon"
  • oxf.fr.persistence.app.uri.orbeon.bookcast.* -> form type and data for "orbeon bookcast"
  • oxf.fr.persistence.app.uri.orbeon.bookcast.form -> form type for "orbeon bookcast"
  • oxf.fr.persistence.app.uri.orbeon.bookcast.data -> form data for "orbeon bookcast"
More specific properties have a higher priority over less specific properties.

The default configuration properties above specify the following:
  • By default, everything is stored into eXist
  • The "orbeon" application's "builder" and "bookcast" form definitions are stored into Orbeon Forms resources
  • But their associated form data is stored into eXist
Whether storage is done into eXist is determined by the URI specified as property value:
  • /fr/service/exist points to Form Runner's internal eXist API implementation
  • /fr/service/oracle points to Form Runner's internal Oracle XML API implementation
  • /fr/service/resource points to Form Runner's internal resource manager API implementation

Persistence API

The Form Builder persistence API is based on REST. This means that Form Builder and Form Runner use HTTP to communicate with a persistence layer.

Form Builder calls URLs as follows:
  • Create, Read, update, delete form data -> HTTP GET, PUT, DELETE to:
    /crud/[APPLICATION]/[FORM_TYPE]/data/[FORM_DATA_ID]/data.xml
  • Search -> POST to:
    /search/    [APPLICATION]/[FORM_TYPE
Following the REST philosophy, HTTP methods are used to determine what operation to perform:
  • GET: read a resource
  • PUT: create or update a resource
  • DELETE: delete a resource

Support for versioning

Form Runner / Builder do not at the moment fully support versioning, but when retrieving form.xhtml, Form Runner also passes a URL parameter when possible:
  • .../form.xhtml?document=6E6FC50F4BB945235EB5B573F2C7E695
This parameter is passed when:
  • editing form data
  • viewing form data
  • producing a PDF
This parameter is NOT passed when no document information is available, e.g. when:
  • creating new form data
  • listing form data on the summary page
The value of the parameter is the id of the form data (document) being edited. This allows the persistence layer to retrieve the form type associated with that document, in case the persistence layer handles versioning itself.

In addition, the Oracle XML supports versioning to some level, in that it keeps older versions. However this is not visible at the level of the REST API.

If all you need is keep older versions in the database, it might be sufficient to use the Oracle XML persistence layer implementation, or to implement a similar system for eXist or your own persistence layer. The best will be for Form Runner and Form Builder to handle versioning natively.

Data exchange format

When using GET, PUT and DELETE to deal with resources, the body of HTTP requests just contains the resource to handle.

  • GET: request body is empty, response body contains resource
  • PUT: request body contains resource, request response is empty
  • DELETE: both request and response bodies are empty

When the resource is an XML file (e.g. form.xhtml, data.xml), the persistence layer must return an appropriate XML content-type: application/xml, or application/xhtml+xml.

When using POST to perform a search, the format is as follows:

<search xmlns="">
    <!-- Application name -->
    <app>orbeon</app>
    <!-- Form type name -->
    <form>address</form>
    <!-- Free text search query -->
    <query/>
    <!-- Structured search query -->
    <query name="first-name" path="personal-info/first-name"/>
    <query name="last-name" path="personal-info/last-name">smith</query>
    <!-- Paging -->
    <page-size>10</page-size>
    <page-number>1</page-number>
    <!-- Sorting -->
    <sort-key/>
    <!-- Language -->
    <lang/>
</search>

NOTE: In Milestone 1, sort-key is not yet supported.

The query element is the most complex element. It is used for structured search. The path attribute is the one that matters: it contains a path to the element or attribute in the XML data to retrieve.

Say your documents looks like this:

<form>
    <personal-info>
        <first-name>John</first-name>
        <last-name>Doe</last-name>
        <birth-date>1980-01-01</birth-date>
    </personal-info>
</form>

Then the path personal-info/first-name matches XML elements following the XPath notation, relative to the root element of the document. The paths sent by Form Runner are simple paths, not full XPath expressions.

If the query element is empty, it just denotes the details to return in the response. If it contains a string, then it requires performing a search.

The persistence layer must return a document of this form:

<documents total="2" page-size="10" page-number="1" query="">
    <document created="2008-03-14T12:33:15.735-07:00" last-modified="2008-03-14T12:36:51.657-07:00" name="847C6B6ADB949146C0105433A374B5CE">
        <details>
            <detail>John</detail>
            <detail>Doe</detail>
        </details>
    </document>
    <document created="2008-03-14T12:32:53.735-07:00" last-modified="2008-03-14T12:32:53.735-07:00" name="85F3E3433DEB6E1EDE9DCD6F87D24240">
        <details>
            <detail>Sofia</detail>
            <detail>Smith</detail>
        </details>
    </document>
</documents>

The order of the detail elements is determined by the order of the query elements of the request.

Scenario

This scenario describes how company Acme can go about implementing their own persistence service.

First, configure properties.xml, for example:

<property as="xs:anyURI"  name="oxf.fr.persistence.app.uri.acme.*.data" value="http://example.com/persistence"/>

What this does is tell Form Runner to dispatch all persistence API calls for applications with name "acme" to the specified URL root. Replace the URL as appropriate, but it must point to a server able to implement the persistence API's behavior. It could be implemented with Java, .NET, PHP, Ruby, Orbeon Forms itself, etc.

You must then implement a server component responding to the /persistence path on server example.com. Upon receiving a request, it must:

  • Check the HTTP method to determine the operation to perform (which CRUD operation or search operation)
  • Check the requested path to determine the location of the resource
  • In the case of PUT, read the request body and store it appropriately. This might require writing data into a CLOB or XML type column in a relational database, for example.
  • In the case of GET, read the storage and return the appropriate resource.
  • In the case of DELETE, delete the appropriate resource in storage
  • In the case of POST, perform a search. This might require generating an SQL query, for example.

Depending on the type of storage chosen, storage operations may be more or less complex.