Comments? Feedback?

This wiki does not yet support public comments (a limitation of Google Sites), so we encourage you to post your comments either:

On Twitter by responding to @orbeon.

On our community mailing list: subscribe sending an email to ops-users-subscribe@ow2.org (content of subject/body doesn't matter), you'll get a response with the email to use to send your message to the community mailing list.

Recent site activity

Oracle and MySQL Persistence Layers


Overview

The setup for the Oracle and MySQL persistence layer is a 3 step process. The first two steps are database specific, so please refer to the relevant subsection below.

1Database setup
You setup the database and create a schema with a few tables. This will typically be done by a DBA.
2
Application server setup
You configure your application server to use the database.
3
Orbeon Forms setup
You configure Orbeon Forms to use the Oracle or MySQL persistence layer.

The Oracle persistence layer is an Orbeon Forms PE feature. The MySQL persistence layer does not support attachments.

Database setup

Oracle

Create a user/schema in Oracle, for instance with the following commands. (In this example "all privileges" are granted to the newly created user/schema, which is not strictly required. You might want to fine-tune permissions on your system as appropriate.) 

> sqlplus sys/password as sysdba
SQL> create user orbeon identified by password ;
SQL> grant all privileges to orbeon ;

Note:
  • If you had already created this schema and that the definition changed, or that you want to restart from scratch for some other reason, you can delete the schema with all the data it contains with:

    drop user orbeon cascade ;
  • In the above, orbeon is the name of your user or schema, which you pick; and "password" is the name of the password, which you pick as well.
Create the following tables:

create table orbeon_form_definition (
    created timestamp,
    last_modified timestamp,
    username varchar2(256),
    app varchar2(256),
    form varchar2(256),
    deleted char(1),
    xml xmltype
);

create index orbeon_form_definition_x
    on orbeon_form_definition(xml)
    indextype is ctxsys.context
    parameters ('sync (on commit)');

create index orbeon_form_definition_i1
    on orbeon_form_definition(app, form);

create table orbeon_form_definition_attach (
    created timestamp,
    last_modified timestamp,
    username varchar2(256),
    app varchar2(256),
    form varchar2(256),
    deleted char(1),
    file_name varchar2(256),
    file_content blob
);

create index orbeon_form_definition_att_i1
    on orbeon_form_definition_attach(app, form, file_name);

create table orbeon_form_data (
    created timestamp not null,
    last_modified timestamp not null,
    username varchar2(256),
    app varchar2(256) not null,
    form varchar2(256) not null,
    document_id varchar(256) not null,
    deleted char(1) not null,
    xml xmltype not null,
    constraint orbeon_form_data_pk primary key (document_id, last_modified)
);


create index orbeon_form_data_x
    on orbeon_form_data(xml)
    indextype is ctxsys.context
    parameters ('sync (on commit)');

create index orbeon_from_data_i1
    on orbeon_form_data(app, form, document_id);

create table orbeon_form_data_attach (
    created timestamp,
    last_modified timestamp,
    username varchar2(256),
    app varchar2(256),
    form varchar2(256),
    document_id varchar(256),
    deleted char(1),
    file_name varchar2(256),
    file_content blob
);

create index orbeon_from_data_attach_i1
    on orbeon_form_data_attach(app, form, document_id, file_name);


MySQL

The MySQL persistence layer relies on XML functions that have been introduced in MySQL 5.1, so you need to be using the MySQL 5.1 (which was released in November 2008) or newer.
  • Create a new user orbeon. Orbeon Forms will connect to MySQL as that user.
  • Create a new schema orbeon. This schema will contains the tables used to store your forms definitions and form data.
  • Create tables Orbeon Forms in the orbeon schema:
    • Connect to the orbeon schema with the orbeon user. Assuming the password is orbeon, run:
mysql --user=orbeon --password=orbeon orbeon
    • Run the following data definition statements to create the tables:
alter database character set utf8 collate utf8_general_ci;

create table orbeon_form_definition (
    created timestamp,
    last_modified datetime,
    username varchar(256),
    app varchar(256),
    form varchar(256),
    deleted char(1),
    xml mediumtext
);
create table orbeon_form_definition_attach (
    created timestamp,
    last_modified timestamp,
    username varchar(256),
    app varchar(256),
    form varchar(256),
    deleted char(1),
    file_name varchar(256),
    file_content longblob
);
create table orbeon_form_data (
    created timestamp,
    last_modified timestamp,
    username varchar(256),
    app varchar(256),
    form varchar(256),
    document_id varchar(256),
    deleted char(1),
    xml mediumtext
);
create table orbeon_form_data_attach (
    created timestamp,
    last_modified timestamp,
    username varchar(256),
    app varchar(256),
    form varchar(256),
    document_id varchar(256),
    deleted char(1),
    file_name varchar(256),
    file_content longblob
);


Application server setup

Oracle

General

Assuming:
  • ${HOST}: the host Oracle server is running on, for example oracle.acme.com
  • ${PORT}: the port the Oracle server is running on, for example 1521
  • ${INSTANCE}: the instance name, for example orcl
  • ${USERNAME}: the user/schema, for example orbeon
  • ${PASSWORD}: the password, for example password

Tomcat

Put the Oracle jar file that contains the JDBC driver (e.g. ojdbc14_g.jar or ojdbc5_g.jar) in the appropriate directory for your application server (on Tomcat: common/lib or simply lib, depending on the version). If you don't already have it, you can download the Oracle JDBC driver from the Oracle site.

Setup a JDBC data source for your Oracle instance. With Tomcat, this is done in server.xml, where you define a Resource pointing to the your Oracle instance. In the example below, the Oracle server is running on oracle.acme.com, the instance name is orcl, and the user/schema is orbeon with password password:

<Resource name="jdbc/oracle" auth="Container" type="javax.sql.DataSource"
      initialSize="3" maxActive="10" maxIdle="20" maxWait="30000"
      validationQuery="select * from dual"
      driverClassName="oracle.jdbc.OracleDriver"
      poolPreparedStatements="true"
      username="${USERNAME}"
      password="${PASSWORD}"
      url="jdbc:oracle:thin:@//${HOST}:${PORT}/${INSTANCE}"/>

JBoss 5.0.1 / JBoss EAP 5.0.1

Please follow the JBoss documentation first, but here are some steps that work for us in our test environment.

Place ojdbc5_g.jar into server/default/lib/.

Create an Oracle datasource as server/default/deploy/oracle-ds.xml, for example:

<?xml version="1.0" encoding="UTF-8"?>
<datasources>

    <local-tx-datasource>
        <jndi-name>OracleDS</jndi-name>
        <connection-url>jdbc:oracle:thin:@//${HOST}:${PORT}/${INSTANCE}</connection-url>
        <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
        <user-name>${USERNAME}</user-name>
        <password>${PASSWORD}</password>
        <valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name>
        <metadata>
            <type-mapping>Oracle9i</type-mapping>
        </metadata>
    </local-tx-datasource>

</datasources>

Update WEB-INF/jboss-web.xml  to:

<?xml version="1.0" encoding="UTF-8"?>
<jboss-web>
    <resource-ref>
        <res-ref-name>jdbc/oracle</res-ref-name>
        <jndi-name>java:/OracleDS</jndi-name>
    </resource-ref>
</jboss-web>


MySQL

  • Download the MySQL JDBC driver, called Connector/J, e.g. mysql-connector-java-5.1.12-bin.jar

  • Copy it in the appropriate directory for your application server (on Tomcat: common/lib or simply lib, depending on the version).

  • Setup a JDBC data source for your MySQL schema. With Tomcat, you can do this in conf/server.xml, where you define a Resource pointing to the your MySQL database and schema. In the example below, the Oracle server is running on localhost port 3306, the schema is orbeon, the username/password is orbeon/orbeon, and on the JDBC URL you're telling the MySQL driver to use Unicode and the UTF-8 encoding when talking to the database:
<Resource name="jdbc/mysql" auth="Container" type="javax.sql.DataSource"
    initialSize="3" maxActive="10" maxIdle="20" maxWait="30000"
    driverClassName="com.mysql.jdbc.Driver"
    poolPreparedStatements="true"
    username="orbeon"
    password="orbeon"
    url="jdbc:mysql://localhost:3306/orbeon?useUnicode=true&amp;characterEncoding=UTF8"/>


Orbeon Forms setup

With Orbeon Forms 3.9

In your properties-local.xml:
  1. Map an app, form, form type to the Oracle or MySQL persistence layer using the oxf.fr.persistence.app.uri.*.*.* wildcard property, For instance, the following indicates that all the form definition and form data in the acme "app" are stored in Oracle:

    <property as="xs:anyURI"  name="oxf.fr.persistence.app.uri.acme.*.*"
                              value="/fr/service/oracle"/>

  2. Set the value of the property oxf.fr.persistence.service.oracle.datasource for Oracle or oxf.fr.persistence.service.mysql.datasource for MySQL to match the name of the resource you setup in server.xml. For instance, if in server.xml the resource name is jdbc/oracle, then the property should be set to just oracle, as in:

    <property as="xs:anyURI" name="oxf.fr.persistence.service.oracle.datasource" value="oracle"/>


Dynamic datasource for MySQL

You select the datasource used by the MySQL persistence layer with the oxf.fr.persistence.service.mysql.datasource configuration property. If you have multiple MySQL datasources, you can dynamically choose which one the MySQL persistence layer uses by setting the Orbeon-Datasource HTTP header. When the header is set, it takes precedence over the value of the oxf.fr.persistence.service.mysql.datasource property. In addition to setting the header, you need to add the following property so Form Runner forwards that header to the MySQL persistence layer:

<property as="xs:string" name="oxf.http.forward-headers" value="Orbeon-Datasource"/>

The header must be present in both the HTTP requests that return a web page and the Ajax requests. This assumes that you have your own "proxy" (for instance a servlet filter) that receives requests from the browser and adds the header before passing requests along to Orbeon Forms. For security reasons, you should never return the header to the browser, as this would enable malicious users to access any datasource they want by changing the value of the header. Typically, your proxy will work as follows:

Step Users / browser Your proxy Orbeon Forms / Persistence layer 
A.1 Logs in, or accesses the system for the first time loading a web page.
 
A.2   Creates a session, and stores in the session the datasource to use based on who the user is.  
B.1 Accesses a Orbeon Forms page or sends an Ajax request.    
B.2   Based on the information stored in the session, finds what datasource to use, and adds the Orbeon-Datasource header to the request it passes along to Orbeon Forms. This header is just added to the request going to Orbeon Forms, and is not returned to the browser.  
B.3     Finds the Orbeon-Datasource header and uses the datasource specified by that header rather than the one specified by the oxf.fr.persistence.service .mysql.datasource property.

With Orbeon Forms 3.10

With a single schema

In your properties-local.xml:
  1. Map an app, form, form type to the Oracle or MySQL persistence layer using the oxf.fr.persistence.provider.*.*.* wildcard property, For instance, the following indicates that all the form definition and form data in the acme "app" are stored using the Oracle provider, use:

    <property as="xs:string" name="oxf.fr.persistence.provider.acme.*.*" value="oracle"/>

  2. Set the value of the property oxf.fr.persistence.oracle.datasource for Oracle or oxf.fr.persistence.mysql.datasource for MySQL to match the name of the resource you setup in server.xml. For instance, if in server.xml the resource name is jdbc/oracle, then the property should be set to just oracle, as in:

    <property as="xs:string" name="oxf.fr.persistence.oracle.datasource" value="oracle"/>

With multiple schemas

The single schema configuration described in the previous section uses the predefined oracle and mysql providers. To use multiple schemas you need to define you own provider names. For instance, assume that you have two apps, hr and finance, and would like both the form definition and data for those apps to be stored in two separate schemas:
  1. In your application server configuration, you setup two data sources ; let's call them hr-datasource and finance-datasource.

  2. In properties-local.xml, you use the following properties to define two providers hr and finance that you configure to use the desired persistence layer implementation (Oracle in this example) and data source:

    <!-- HR provider -->
    <property as="xs:anyURI" name="oxf.fr.persistence.hr.uri"
                             value="/fr/service/oracle"/>
    <property as="xs:string" name="oxf.fr.persistence.hr.datasource"
                             value="hr-datasource"/>

    <!-- Finance provider -->
    <property as="xs:anyURI" name="oxf.fr.persistence.finance.uri"
                             value="/fr/service/oracle"/>
    <property as="xs:string" name="oxf.fr.persistence.finance.datasource"
                             value="fiance-datasource"/>

  3. Still in properties-local.xml, you map the hr and finance app to the respective provider:

    <property as="xs:string" name="oxf.fr.persistence.provider.hr.*.*"      value="hr"/>
    <property as="xs:string" name="oxf.fr.persistence.provider.finance.*.*" value="finance"/>

Flat view or table

Orbeon Forms stores form data as XML in Oracle and MySQL, which gives it a lot of flexibility. However, it might be harder for other tools to access this XML data. For this reason, you might want to provide other tools a way to access the XML data through another "flat" table or view that has one column per form field.

Oracle

[SINCE: 2011-04-08]

Orbeon Forms can create in Oracle a form-specific view of your data, with one column for each form field. You enabled this feature by setting the following property to true. (If you define your own provider for Oracle, use that provider name in the property name, instead of oracle.)

<property as="xs:boolean" name="oxf.fr.persistence.oracle.create-flat-view" value="false"/>

When you enable this property, upon publishing a form, Oracle persistence layer creates a view specific to that form. The name of the view is based on your app and form name, and has the form: orbeon_flat_#{app}_#{form}. For instance, if your app is hr and you form is expense, then the view is named orbeon_flat_hr_expense. If upon publishing, there is already a view with that name, the Oracle persistence layer deletes it before recreating a new view.

The view always has the following 4 metadata columns, with information copied from the equivalent columns in orbeon_form_data: metadata_document_id, metadata_created, metadata_last_modified, and metadata_username. In addition to those 4 columns, you have one column per form field, and each column is named by combining the section name with the control name. You can see an example of generated column names based on section and control names in the following table:


Section name
Control name
Column name
personal-information
first-name
PERSONAL_INFOR_FIRST_NAME
 last-name
PERSONAL_INFOR_LAST_NAME
 addressPERSONAL_INFOR_ADDRESS
company
name
COMPANY_NAME
 industry
COMPANY_INDUSTRY


Oracle columns names are limited to 30 characters, so the Oracle persistence layer truncates the section name to 14 characters, the control name to 15 characters, and combines both with an underscore in between. It also converts dashes to underscores, removes any non alphanumerical character, and converts the name to uppercase (so it can be used in queries without quotes).

In the vast majority of the cases, this will result in distinct and recognizable column names. In cases where two or more columns would end up having the same name or conflict with one of the metadata column, the Oracle persistence layer adds a number prefix of the form 001_, 002_, 003_… to each column to make it unique. If this happens, you might want to change your section and/or control names to have more recognizable column names.

MySQL

Orbeon Forms doesn't (yet) provide a way to have a table or view automatically created for a form upon publishing in MySQL, as it does in Oracle. However, you can do this manually. For instance, assume you want to create a "flat" bookshelf table for the sample bookshelf form. You want that table to have 3 columns:
  • title corresponds to the title form field;
  • author corresponds to the author form field;
  • document_id corresponds to the column with the same name in orbeon_form_data.
Start by creating the bookshelf table:

create table bookshelf (
    document_id varchar(256),
    title  text,
    author text
);

Choose an appropriate type for your columns, depending on the maximum length for the fields. Then create a trigger, which will update your bookshelf table when form data is saved in orbeon_form_data:

delimiter |
create trigger bookshelf_trigger before insert on orbeon_form_data for each row begin
    if new.app = 'orbeon' and new.form = 'bookshelf' then
        delete from bookshelf where document_id = new.document_id;
        if new.deleted = 'N' then
            insert into bookshelf set document_id = new.document_id,
                title = extractValue(new.xml, '/book/details/title'),
                author = extractValue(new.xml, '/book/details/author');
        end if;
    end if;
end;
|

Since you are interested in data for Bookshelf form, which is in the app orbeon form bookshelf, the trigger only does something if new.app = 'orbeon' and new.form = 'bookshelf'. To enable auditing, the MySQL persistence layer never deletes or updates data; it only inserts new row. So your trigger only needs to be concerned about updates. On insert, you want to make sure you are not creating duplicates in your bookshelf table, hence the delete statement. When a newly inserted row has delete = 'N', this indicates that a user deleted that document, in which case you don't want to insert a row in your bookshelf table, hence the if test.

Auditing and versioning

Oracle

The Oracle persistence layer never deletes old form definitions or form data:
  • when a new form definition or form data is saved, it simply adds a new row to the table with a newer time stamp
  • when a form definitions or form data is deleted, it marks the row as deleted but does not remove the row
Note however that, as of November 2011, there is no user interface for this feature.