Orbeon Search

Comments? Feedback?

This wiki does not yet support public comments (a limitation of Google Sites), so we encourage you to post your comments either:

On Twitter by responding to @orbeon.

On our community mailing list: subscribe sending an email to ops-users-subscribe@ow2.org (content of subject/body doesn't matter), you'll get a response with the email to use to send your message to the community mailing list.

Recent site activity

Form Runner/Form Builder Persistence API

Contents

  1. 1 Existing persistence layers
  2. 2 Configuration
  3. 3 Deployment
    1. 3.1 Development vs. staging vs. production
    2. 3.2 Migrating form definitions to another database
  4. 4 API
    1. 4.1 API overview
    2. 4.2 Virtual hierarchy of data
    3. 4.3 CRUD format
    4. 4.4 Deleting all data for an existing form
    5. 4.5 Search API
      1. 4.5.1 Example query
      2. 4.5.2 Query elements
    6. 4.6 Paging
    7. 4.7 Language (optional)
    8. 4.8 Query response
    9. 4.9 Deprecated query elements
    10. 4.10 Support for versioning
  5. 5 A scenario

Existing persistence layers

At runtime, forms you create with Form Builder go through a REST API for persistence. Out of the box, Orbeon Forms provides an
implementation of that REST API for a number of databases, and other implementations are provided by third parties. The table below lists the databases and CMS for which implementation of the REST API is available today. If you don't find your database of choice listed here, you can implement the REST API yourself to target that database, or Orbeon can help you with that as a part of a Development Support subscription. Should you decide to implement the REST API yourself, you'll find more about the inner workings of the REST API in the remainder of this page.


Storage Notes Availability More information
eXist  A full persistence layer implemented on top of the open source eXist database. You can setup this persistence layer to either use the internal eXist database, or an external one. Built-in  
Oracle A full persistence layer implemented on top of the Oracle database. Forms and data are stored using Oracle XMLType columns.  Built-in Oracle persistence layer
MySQL A full persistence layer implemented on top of MySQL. Built-in MySQL persistence layer
Alfresco This persistence layer isn't bundled with Orbeon Forms: it has been developed by Alexey Ermakov and is available as an add-on that you can download and configure with Orbeon Forms. Add-on Documentation
Project on GitHub
MongoDB A full persistence layer implemented on top of MongoDB. 
NOTE: This is experimental as of 2011-04-26.		 Built-in  TODO
File system (resource) A minimal, read-only persistence layer implemented on top of Orbeon Forms' resource manager. This allows reading form definitions stored within the Orbeon Forms WAR file. Built-in  

Configuration

Storage for the persistence hierarchy can be configured at multiple levels:
  • Globally
  • For each application
  • For each form type within an application
  • For each form type (the form definition and associated resources) vs. form data (filled-out form data)
This allow you for example to store certain form types on disk, while storing the associated data, as filled-out by users, in a database.

This configuration is done in an XML property file called properties-local.xml. This file must be located under your exploded WAR file at the following location:

WEB-INF/resources/config/properties-local.xml

See Configuration Properties on how to create the properties-local.xml file if it doesn't exist. You can edit this file with any text editor. Some of the properties do not require a restart or Orbeon Forms, but in doubt you can restart the Servlet container.

Persistence configuration properties look as follows in properties-local.xml with Orbeon Forms 3.8 and 3.9:

<property as="xs:anyURI"
          name="oxf.fr.persistence.app.uri.*.*.*"
          value="/fr/service/exist"/>
<property as="xs:anyURI"
          name="oxf.fr.persistence.app.uri.orbeon.builder.form"
          value="/fr/service/resource"/>
<property as="xs:anyURI"
          name="oxf.fr.persistence.app.uri.orbeon.bookcast.form"
          value="/fr/service/resource"/>

All those property names start with oxf.fr.persistence.app.uri to denote that they configure the Form Runner's persistence layer per application. Follows:
  1. A string to specify the application name, like orbeon or acme.
  2. A form type name, like bookcast or address. Note that builder is a special form type name for Form Builder itself.
  3. Whether the configuration regards form data (data), or the form configuration files (form).
The value of the properties are URIs, which point to the the location of the root of the virtual hierarchy.

There are two types of persistence services:
  • Local: those are handled by Orbeon Forms, and start with the prefix /fr/service/
  • Remote: those are handled by third-party software, and are absolute URLs starting with http: or https:
The properties are interpreted hierarchically: you may specify the configuration more or less specifically:
  • oxf.fr.persistence.app.uri.*.*.* -> form types and form data for all applications
  • oxf.fr.persistence.app.uri.*.*.data -> form data for all applications
  • oxf.fr.persistence.app.uri.*.*.form -> form types for all applications
  • oxf.fr.persistence.app.uri.orbeon.*.form -> form types for all forms in application "orbeon"
  • oxf.fr.persistence.app.uri.orbeon.*.data -> form data for all forms in  application "orbeon"
  • oxf.fr.persistence.app.uri.orbeon.bookcast.* -> form type and data for "orbeon bookcast"
  • oxf.fr.persistence.app.uri.orbeon.bookcast.form -> form type for "orbeon bookcast"
  • oxf.fr.persistence.app.uri.orbeon.bookcast.data -> form data for "orbeon bookcast"
[TODO: property names have changed after Orbeon Forms 3.9. See also this compatibility note.]

More specific properties have a higher priority over less specific properties.

The default configuration properties above specify the following:
  • By default, everything is stored into eXist
  • The "orbeon" application's "builder" and "bookcast" form definitions are stored into Orbeon Forms resources
  • But their associated form data is stored into eXist
Whether storage is done into eXist is determined by the URI specified as property value:
  • /fr/service/exist points to Form Runner's internal eXist API implementation
  • /fr/service/oracle points to Form Runner's internal Oracle XML API implementation
  • /fr/service/resource points to Form Runner's internal resource manager API implementation

Deployment

Development vs. staging vs. production

Orbeon Forms is often installed in different environments, with one or more servers dedicated to each environment. For instance:
  1. A development environment — On which form authors create and test the forms they are working on.
  2. A staging  environment — On which testing is performed before deployment.
  3. A production environment — Accessed by end users to fill out forms.
The above scenario is typical, but there is nothing in Orbeon Forms that dictates you have those 3 environments. Your setup could include more or less different environments depending on your needs.

So you get the full benefit of having different environment, you should setup the instances of Orbeon Forms in different environments to use different databases, or at least different database schemas, so you can see each environment as a silo, and never have, say, form authors accessing the development environment change any data related to the staging or production environment.

At some point, you might want to graduate data, typically forms, from one environment to the next. For instance, when form authors are done, moving forms from the development to staging. Or once testing is done, from staging to production. This move will be done by a database administrator (DBA), copying the appropriate data from one database to another database, and the exact operations to be performed by the DBA depend on the specific database you are using.

Also, you might want to setup access control differently depending on the environment. For instance, if your production system is used to just capture data, you might want to completely block the access to Form Builder in production and staging, but not in development.

Migrating form definitions to another database

If you have multiple databases for different environments, as discussed in the previous section, at some point, you might want to migrate form definitions from one database to another database, for instance from the development database to the staging database, or staging database to production database. How you do this depends on which persistence layer you're using:
  • On Oracle and MySQL, published form definitions are stored in the orbeon_form_definition table, and their attachements, if any, in orbeon_form_definition_attach. So you'll want to migrate the content of those two tables. If you wish to only migrate specific apps or forms, you can do so by exporting rows where the app and/or form columns match certain values.

  • On eXist, form definitions for an app my-app and form my-form are stored in the /orbeon/fr/your-app/your-form/form collection. If you only want to migrate a subset of the forms, it is often easier to create a backup of the whole database, and then remove from the zip the form or data you don't want to migrate before you import it on the target database.

API

API overview

The Form Runner/Form Builder persistence API is based on REST. This means that Form Builder and Form Runner use HTTP to communicate with a persistence layer implementation.

Form Builder calls URLs as follows:
  • Create, Read, update, delete (CRUD) form data → HTTP GET, PUT, DELETE to:
    /crud/[APPLICATION_NAME]/[FORM_NAME]/data/[FORM_DATA_ID]/data.xml
  • Search → HTTP POST to:
    /search/    [APPLICATION_NAME]/[FORM_NAME]
Following the REST philosophy, HTTP methods are used to determine what CRUD operation to perform:
  • GET: read a resource
  • PUT: create or update a resource
  • DELETE: delete a resource

Virtual hierarchy of data

Form Runner/Form Builder access data under a virtual hierarchy or URLs, not unlike directories or folders in a filesystem. However this hierarchy can be physically located in different places:

  • An XML database, like eXist.
  • A disk-based filesystem.
  • Your own system, which you can implement on top of a database or other type of storage.
Following XML database technology, we use the terms collections and resources instead of directories and files.

The hierarchy looks like this:
  • acme/
    • address/
      • form/
        • form.xhtml
        • logo.png
        • style.css
        • form.pdf
      • data/
        • 6E6FC50F4BB945235EB5B573F2C7E695
          • data.xml
          • headshot.jpg
        • C37284A414E5F266E3ECEE8C8AEDB6F0
          • data.xml
    • contract/
      • form/
        • ...
      • data/
        • ...
  • foobar/
    • ...
The hiearchy is organized as follows:
  • At the top-level there is one collection per application
  • Within an application collection, there is one collection per form type
  • Within a form type, there is one collection called "form" for the form definitions produced by Form Builder, and one collection called "data" for form data produced by Form Runner
  • Each "form" collection contains:
    • form.xhtml: the main form definition, which is an XHTML+XForms resource
    • optional attachments, such as images, CSS, and PDF files uploaded by the form author when editing the form type
  • Each "data" collection contains one collection for each form data id, identified by an automatically-generated UUID
  • Each form data collection contains:
    • data.xml: the main form data document
    • optional attachments, such as images uploaded by the user when editing the form data

CRUD format

When using GET, PUT and DELETE to deal with resources, the body of HTTP requests just contains the resource to handle.

  • GET: request body is empty, response body contains resource
  • PUT: request body contains resource, request response is empty
  • DELETE: both request and response bodies are empty

When the resource is an XML file (e.g. form.xhtml, data.xml), the persistence layer must return an appropriate XML content-type: application/xml, or application/xhtml+xml.

Deleting all data for an existing form

[SINCE: 2011-05-17]

To remove all instances of form data, issue a DELETE to:

/crud/[APPLICATION_NAME]/[FORM_NAME]/data/

Search API

Example query

A search query uses HTTP POST to provide an XML document containing the search criteria as well as information about the response to provide. The following shows for example a search from the demo Bookshelf form:

<search>
    <!-- Free-text search query -->
    <query/>
    <!-- Structured search query -->
    <query name="title"
           path="details/title"
           label="Title"
           type="xs:string"
           control="input"
           search-field="true"
           summary-field="true"
           match="substring">Peace</query>
    <query name="author"
           path="details/author"
           label="Author"
           type="xs:string"
           control="input"
           search-field="true"
           summary-field="true"
           match="substring"/>
    <query name="language"
           path="details/language"
           label="Language"
           type="xs:string"
           control="select1"
           search-field="true"
           summary-field="true"
           match="substring"/>
    <!-- Paging -->
    <page-size>10</page-size>
    <page-number>1</page-number>
    <!-- Language -->
    <lang>en</lang>
</search>

Query elements

The query element is the most complex element. It is used for full-text and structured search.

NOTE: As of 2011-11-22, full-text search and structured search are exclusive: either Form Runner performs one, or the other.

Full-text search

If the user is performing a full-text search, the first <query> element (which does not have attributes) contains the text of the search.

The result details to return is still determined by the subsequent <query> elements with summary-field set to true. See below for more information about the response format.

NOTE: The exact semantic of the full-text search is implementation-dependent.

Structured search
When building a structured search query, Form Runner first looks at all controls in the source form that have the class fr-summary or fr-search. These are used to build the request.

For each such control found, a <query> element is added to the request, with the following attributes and text value:
  • name attribute:
    • contains the control name as entered in Form Builder
  • path attributes:
    • contains the path to the XML element or attribute storing the data in XForms, relative to the root element
  • label attribute:
    • informative only
    • the control's label in the current language used by Form Runner
  • type attribute:
    • contains the datatype associated with the field in Form Builder
    • default: xs:string
    • other possible values include
      • optional/mandatory string: xforms:string/xs:string
      • optional/mandatory date: xforms:date/xs:date
      • optional/mandatory time: xforms:time/xs:time
      • optional/mandatory dateTime: xforms:dateTime/xs:dateTime
    • other values are possible, including XML Schema data types as well as user-defined data types
  • control attribute:
    • informative only
    • the XForms control type used in the form, including:
      • input
      • textarea
      • secret
      • select
      • select1
      • range (rarely used)
  • search-field attribute:
    • whether this is a search field
    • if true, the field might contain a search string entered by the user as text value
    • NOTE: Only the controls with class fr-search are shown to the user to enter search criteria.
  • summary-field attribute:
    • whether this is a summary field
    • if true, search must return a result detail value fo this field (assuming there are search results)
  • match attribute:
    • whether a substring or an exact match is requested
    • possible values
      • exact: an exact match is requested
      • substring: a substring match is requested
    • this is useful only for <query> elements with search-field set to true
    • as of 2011-11-22, exact is used only when querying the Form Builder summary page's app and form names, but this might be extended in the future
    • NOTE: The exact semantics for substring is not specified at the moment (the search implementation may use a "starts-with", a "contains" logic, or something else).
  • text value of the <query> element:
    • the search string entered by the user, if any
    • this is useful only for <query> elements with search-field set to true

About search paths

The path attribute can be looked at a search key. Say your documents looks like this:

<form>
    <personal-info>
        <first-name>John</first-name>
        <last-name>Doe</last-name>
        <birth-date>1980-01-01</birth-date>
    </personal-info>
</form>


Then the path personal-info/first-name matches XML elements following the XPath notation, relative to the root element of the document.

Note that for a given form definition, there is a limited number of search paths, which are fully determined by the XPath expressions present on the form definition's control bindings and model binds for controls that have the fr-summary or fr-search classes.

The paths sent by Form Runner are usually simple paths, but there are some exceptions:
  • Form Builder uses more complex paths out of the box, for example:
    xhtml:head/
      xforms:model[@id = 'fr-form-model']/
        xforms:instance[@id = 'fr-form-metadata']/
          */
            description[@xml:lang = $fb-lang]
  • Forms where user has entered XPath expressions manually to bind controls to XML data will contain those paths.
If the persistence layer is able to run XPath expressions (as eXist or other XML databases can), then the path provided by the search request can be executed.

However, XPath evaluation is not absolutely required. For example with non-XML (relational, key/value) data stores, is easier to consider the search path as an opaque search key that uniquely identifies the field to search.

Paging

The page-size and page-number elements control paging.
  • page-size attribute: how many results to return at the most
  • page-number attribute: page number to result, starting with 1

Language (optional)

The lang element contains the current language used by the user in the summary page. If the form contains language-dependent fields, this information can be used to refine the search. As of 2011-11-22, only Form Builder makes use of this capability.

For example, Form Builder allows storing a form title and description in more than one language. When searching the form title field, the expectation of the user is likely to be that, if the user interface is in English, the English title will be searched and not, for example, the Italian title.

In this case, the search path contains a special variable, $fb-lang, which allows the persistence implementation to discriminate on the language:

xhtml:head/
  xforms:model[@id = 'fr-form-model']/
    xforms:instance[@id = 'fr-form-metadata']/
      */
        description[@xml:lang = $fb-lang]

In persistence implementations that support XPath, the variable can be either scoped or replaced in the path by the value provided by the incoming <lang> element.

In persistence implementations that don't support XPath, this kind of queries are a little more tricky. The path can:
  • either be used as a plain key and ignore the <lang> element (therefore ignoring the language-sensitive search)
  • or be translated to the native query language of the persistence implementations

Query response

The persistence layer must return a document of this form:

<documents total="13" search-total="4" page-size="10" page-number="1" query="">
    <document created="2011-05-06T14:58:40.376-07:00"
              last-modified="2011-09-12T12:05:07.3-07:00"
              name="e8bfd3ba63fa12a8b59cdd5c08369a35">
        <details>
            <detail>The Terror</detail>
            <detail>Dan Simmons</detail>
            <detail>en</detail>
        </details>
    </document>
    <document created="2011-05-06T14:58:39.611-07:00"
              last-modified="2011-09-12T12:05:06.914-07:00"
              name="9531a191c77b75c417e9874427fa21f7">
        <details>
            <detail>The Little Prince</detail>
            <detail>Antoine de Saint-Exupéry</detail>
            <detail>fr</detail>
        </details>
    </document>
    <!-- ... more <document> elements ... -->
</documents>

The root element contains these attributes:
  • total attribute:
    • total number of documents in the database for this app/form
  • search-total attribute:
    • number of documents matched by the current search
  • page-size attribute:
    • echo of the query's attribute
    • this might be removed in the future
  • page-number attribute:
    • echo of the query's attribute
    • this might be removed in the future
  • query attribute:
    • echo of the full-text query text
    • this might be removed in the future
For each of the documents found, a <document> element is returned:
  • created attribute:
    • creation date in ISO format
  • last-modified attribute:
    • last modification date in ISO format
  • name attribute:
    • document identifier,
    • this must match the identifier created by Form Runner when saving the data
Each document contains one <detail> element in the order determined by the <query> elements with a summary-field set to true in the request. The text value of the <detail> element is the value of the field in the document found.

Deprecated query elements

[2011-11-22]
  • app and form elements:
    • these elements are deprecated and will be removed in the future. As of Orbeon Forms 3.9, they are present but empty
    • to obtain the app and form name being queried, extract them instead from the search URL
  • sort-key element:
    • this element was present but never used and will be removed in the future

Support for versioning

Form Runner / Builder do not at the moment fully support versioning, but when retrieving form.xhtml, Form Runner also passes a URL parameter when possible:

.../form.xhtml?document=6E6FC50F4BB945235EB5B573F2C7E695

This parameter is passed when:
  • editing form data
  • viewing form data
  • producing a PDF
This parameter is NOT passed when no document information is available, e.g. when:
  • creating new form data
  • listing form data on the summary page
The value of the parameter is the id of the form data (document) being edited. This allows the persistence layer to retrieve the form type associated with that document, in case the persistence layer handles versioning itself.

In addition, the Oracle XML supports versioning to some level, in that it keeps older versions. However this is not visible at the level of the REST API.

If all you need is keep older versions in the database, it might be sufficient to use the Oracle XML persistence layer implementation, or to implement a similar system for eXist or your own persistence layer. The best will be for Form Runner and Form Builder to handle versioning natively.

A scenario

This scenario describes how company Acme can go about implementing their own persistence service.

First, configure properties-local.xml, for example, with Orbeon Forms 3.8 or 3.9:

<property as="xs:anyURI"
          name="oxf.fr.persistence.app.uri.acme.*.data"
          value="http://example.com/my-persistence"/>

With Orbeon Forms post-3.9:

<property as="xs:string"
          name="oxf.fr.persistence.provider.acme.*.data"
          value="my-persistence"/>

<property as="xs:anyURI"
          name="oxf.fr.persistence.my-persistence.uri"
          value="http://example.com/my-persistence"/>

What this does is tell Form Runner to dispatch all persistence API calls for applications with name "acme" to the specified URL root. Replace the URL as appropriate, but it must point to a server able to implement the persistence API's behavior. It could be implemented with Java, .NET, PHP, Ruby, Orbeon Forms itself, etc.

You must then implement a server component responding to the /my-persistence path on server example.com. Upon receiving a request, it must:
  • Check the HTTP method to determine the operation to perform (which CRUD operation or search operation)
  • Check the requested path to determine the location of the resource
  • In the case of PUT, read the request body and store it appropriately. This might require writing data into a CLOB or XML type column in a relational database, for example.
  • In the case of GET, read the storage and return the appropriate resource.
  • In the case of DELETE, delete the appropriate resource in storage
  • In the case of POST, perform a search. This might require generating an SQL query, for example.
Depending on the type of storage chosen, storage operations may be more or less complex.

As an example, Orbeon Forms features an implementation for MongoDB written in Scala.