Advanced configuration¶

Managing the database connection pool through Apache DBCP¶

This option is the one that most users would use as it is the default option for managing the database in GeoNetwork. A typical configuration in the resources element of INSTALL_DIR/web/geonetwork/WEB-INF/config.xml uses the jeeves.resources.dbms.ApacheDBCPool class and looks something like:

<resource enabled="true">
        <name>main-db</name>
        <provider>jeeves.resources.dbms.ApacheDBCPool</provider>
        <config>
                <user>www-data</user>
                <password>www-data</password>
                <driver>org.postgis.DriverWrapper</driver>
                <url>jdbc:postgresql_postGIS://localhost:5432/geonetwork</url>
                <poolSize>10</poolSize>
                <validationQuery>SELECT 1</validationQuery>
        </config>
</resource>

The parameters that can be specified to control the Apache Database Connection Pool are described at http://commons.apache.org/dbcp/configuration.html. You can configure a subset of these parameters in your resource element. The parameters that can be specified are:

For performance reasons you should set the following parameter after GeoNetwork has created and filled the database tables it has been configured to use:

• maxOpenPreparedStatements=”300” (at least)

The following parameters are set by GeoNetwork and cannot be configured by the user:

• removeAbandoned - true
• removeAbandonedTimeout - 60 x 60 seconds = 1 hour
• logAbandoned - true
• testOnBorrow - true
• defaultReadOnly - false
• defaultAutoCommit - false
• initialSize - maxActive
• poolPreparedStatements - true, if maxOpenPreparedStatements >= 0, otherwise false

Note: Some firewalls kill idle connections to databases after say 1 hour (= 3600 secs). To keep idle connections alive by testing them with the validationQuery, set minEvictableIdleTimeMillis to something less than timeout, interval (eg. 2 mins = 120 secs = 120000 millisecs), set testWhileIdle to true and set timeBetweenEvictionRunsMillis and numTestsPerEvictionRun high enough to visit connections frequently eg 15 mins = 900 secs = 900000 millisecs and 4 connections per test. For example:

<testWhileIdle>true</testWhileIdle>
<minEvictableIdleTimeMillis>120000</minEvictableIdleTimeMillis>
<timeBetweenEvictionRunsMillis>900000</timeBetweenEvictionRunsMillis>
<numTestsPerEvictionRun>4</numTestsPerEvictionRun>

Note:

• When GeoNetwork manages the database connection pool, PostGIS database is the only database that can hold the spatial index in the database. All other database choices hold the spatial index as a shapefile. If using PostGIS, two pools of database connections are created. The first is managed and configured using parameters in this section, the second is created by GeoTools and cannot be configured. This approach is now deprecated: if you want to use the database to hold the spatial index you should use the JNDI configuration described in the next section because it uses a single, configurable database pool through GeoTools as well as the more modern NG (Next Generation) GeoTools datastore factories.
• For more on transaction isolation see http://en.wikipedia.org/wiki/Isolation_%28database_systems%29.

Database connection pool managed by the container¶

A typical configuration in the resources element of INSTALL_DIR/web/geonetwork/WEB-INF/config.xml uses the jeeves.resources.dbms.JNDIPool class and looks something like:

<resource enabled="true">
        <name>main-db</name>
        <provider>jeeves.resources.dbms.JNDIPool</provider>
        <config>
                <context>java:/comp/env</context>
                <resourceName>jdbc/geonetwork</resourceName>
                <url>jdbc:oracle:thin:@localhost:1521:XE</url>
                <provideDataStore>true</provideDataStore>
        </config>
</resource>

The configuration parameters and their meanings are as follows:

The remainder of the configuration is done in the container context. eg. for tomcat this configuration is in conf/context.xml in the resource called jdbc/geonetwork. Here is an example for the Oracle database:

<Resource name="jdbc/geonetwork"
        auth="Container"
        type="javax.sql.DataSource"
        username="system"
        password="oracle"
        factory="org.apache.commons.dbcp.BasicDataSourceFactory"
        driverClassName="oracle.jdbc.OracleDriver"
        url="jdbc:oracle:thin:@localhost:1521:XE"
        maxActive="10"
        maxIdle="10"
        removeAbandoned="true"
        removeAbandonedTimeout="3600"
        logAbandoned="true"
        testOnBorrow="true"
        defaultAutoCommit="false"
        validationQuery="SELECT 1 FROM DUAL"
        accessToUnderlyingConnectionAllowed="true"
/>

eg. for jetty, this configuration is in INSTALL_DIR/web/geonetwork/WEB-INF/jetty-env.xml. Here is an example for the Postgis database:

<Configure class="org.eclipse.jetty.webapp.WebAppContext">
  <New id="gnresources" class="org.eclipse.jetty.plus.jndi.Resource">
    <Arg></Arg>
    <Arg>jdbc/geonetwork</Arg>
    <Arg>
      <New class="org.apache.commons.dbcp.BasicDataSource">
        <Set name="driverClassName">org.postgis.DriverWrapper</Set>
        <Set name="url">jdbc:postgresql_postGIS://localhost:5432/gndb</Set>
        <Set name="username">geonetwork</Set>
        <Set name="password">geonetworkgn</Set>
        <Set name="validationQuery">SELECT 1</Set>
        <Set name="maxActive">10</Set>
        <Set name="maxIdle">10</Set>
        <Set name="removeAbandoned">true</Set>
        <Set name="removeAbandonedTimeout">3600</Set>
        <Set name="logAbandoned">true</Set>
        <Set name="testOnBorrow">true</Set>
        <Set name="defaultAutoCommit">false</Set>
        <!-- 2=READ_COMMITTED, 8=SERIALIZABLE -->
        <Set name="defaultTransactionIsolation">2</Set>
        <Set name="accessToUnderlyingConnectionAllowed">true</Set>
      </New>
      </Arg>
    <Call name="bindToENC">
      <Arg>jdbc/geonetwork</Arg>
    </Call>
      </New>
</Configure>

The parameters that can be specified to control the Apache Database Connection Pool used by the container are described at http://commons.apache.org/dbcp/configuration.html.

The following parameters must be set to ensure GeoNetwork operates correctly:

For performance reasons you should set the following parameters after GeoNetwork has created and filled the database it has been configured to use:

Notes:

• both PostGIS and Oracle will build and use a table in the database for the spatialindex if provideDataStore is set to true. Other databases could be made to do the same if a spatialindex table is created - see the definition for the spatialIndex table in INSTALL_DIR/web/geonetwork/WEB-INF/classes/setup/sql/create/create-db-postgis.sql for example.
• you should install commons-dbcp-1.3.jar and commons-pool-1.5.5.jar in the container class path (eg. common/lib for tomcat5 or jetty/lib/ext for Jetty) as the only supported DataSourceFactory in geotools is apache commons dbcp.
• the default tomcat-dbcp.jar version of apache commons dbcp for tomcat appears to work correctly for geotools and PostGIS but does not work for those databases that need to unwrap the connection in order to do spatial operations (eg. Oracle).
• Oracle ojdbc-14.jar or ojdbc5.jar or ojdbc6.jar (depending on the version of Java being used) and sdoapi.jar should also be installed in the container class path (for tomcat: common/lib or lib and for jetty: jetty/lib/ext).
• advanced: you should check the default transaction isolation level for your database driver. READ_COMMITTED appears to be a safe level of isolation to use with GeoNetwork for commonly used databases. Also note that McKoi can only support SERIALIZABLE (does anyone still use McKoi?). For more on transaction isolation see http://en.wikipedia.org/wiki/Isolation_%28database_systems%29.

Specific Database Issues¶

DB2 SQL error: SQLCODE: -805, SQLSTATE: 51002, SQLERRMC: NULLID.SYSLH203

Creating a new data directory¶

The data directory needs to be created before starting the catalogue. It must be readable and writable by the user starting the catalogue. If the data directory is an empty folder, the catalogue will initialize the directory default structure. The easiest way to create a new data directory is to copy one that comes with a standard installation - you can find this in INSTALL_DIR/web/geonetwork/WEB-INF/data.

Setting the data directory¶

The data directory variable can be set using:

• Java environment variable
• Servlet context parameter
• System environment variable

For java environment variable and servlet context parameter use:

• <webappName>.dir and if not set using:
• geonetwork.dir

For system environment variable use:

• <webappName>_dir and if not set using:
• geonetwork_dir

Java System Property¶

Depending on the servlet container used it is also possible to specify the data directory location with a Java System Property.

For Tomcat, configuration is:

CATALINA_OPTS="-Dgeonetwork.dir=/var/lib/geonetwork_data"

Run the web application in read-only mode¶

In order to run GeoNetwork with the webapp folder in read-only mode, the user needs to set two variables:

• <webappName>.dir or geonetwork.dir for the data folder.
• (optional) config overrides if configuration files need to be changed (See Configuration override).

For Tomcat, configuration could be:

CATALINA_OPTS="-Dgeonetwork.dir=/var/lib/geonetwork_data -Dgeonetwork.jeeves.configuration.overrides.file=/var/lib/geonetwork_data/config/my-config.xml"

Structure of the data directory¶

The structure of the data directory is:

data_directory/
 |--data
 |   |--metadata_data: The data related to metadata records
 |   |--resources:
 |   |     |--htmlcache
 |   |     |--images
 |   |     |   |--harvesting
 |   |     |   |--logos
 |   |     |   |--statTmp
 |   |
 |   |--metadata_subversion: The subversion repository
 |
 |--config: Extra configuration (eg. could contain overrides)
 |   |--schemaplugin-uri-catalog.xml
 |   |--codelist: The thesauri in SKOS format
 |   |--schemaPlugins: The directory used to store new metadata standards
 |
 |--index: All indexes used for search
 |   |--nonspatial: Lucene index
 |   |--spatialindex.*: ESRI Shapefile for the index (if not using spatial db)
 |
 |--removed: Folder with removed metadata.

Advanced data directory configuration¶

All sub-directories could be configured separately using java system property. For example, to put index directory in a custom location use:

• <webappName>.lucene.dir and if not set using:
• geonetwork.lucene.dir

Example:

• Add the following java properties to start-geonetwork.sh script:

  java -Xms48m -Xmx512m -Xss2M -XX:MaxPermSize=128m -Dgeonetwork.dir=/app/geonetwork_data_dir -Dgeonetwork.lucene.dir=/ssd/geonetwork_lucene_dir

• Add the following system properties to start-geonetwork.sh script:

  # Set custom data directory location using system property
  export geonetwork_dir=/app/geonetwork_data_dir
  export geonetwork_lucene_dir=/ssd/geonetwork_lucene_dir

System information¶

All catalogue configuration directory can be found using the System Information in the Administration page.

Other system properties¶

In Geonetwork there are several system properties that can be used to configure different aspects of Geonetwork. When a webcontainer is started the properties can be set. For example in Tomcat one can set either JAVA_OPTS or CATALINA_OPTS with -D<propertyname>=<value>.

• <webappname>.jeeves.configuration.overrides.file - See Configuration override
• jeeves.configuration.overrides.file - See Configuration override
• mime-mappings - mime mappings used by jeeves for generating the response content type
• http.proxyHost - The internal geonetwork Http proxy uses this for configuring how it can access the external network (Note for harvesters there is also a setting in the Settings page of the administration page)
• http.proxyPort - The internal geonetwork Http proxy uses this for configuring how it can access the external network (Note for harvesters there is also a setting in the Settings page of the administration page)
• geonetwork.sequential.execution - (true,false) Force indexing to occur in current thread rather than being queued in the ThreadPool. Good for debugging issues.

There is a usecase where multiple geonetwork instances might be ran in the same webcontainer, because of this many of the system properties listed above have <webappname>. When declaring the property this should be replaced with the webapp name the setting applies to. Typically this will be geonetwork.

Add a search field¶

Indexed fields are defined on a per schema basis on the schema folder (eg. xml/schemas/iso19139) in index-fields.xsl file. This file define for each search criteria the corresponding element in a metadata record. For example, indexing the title of an ISO19139 record:

<xsl:for-each select="gmd:identificationInfo/gmd:MD_DataIdentification/
                       gmd:citation/gmd:CI_Citation/
                       gmd:title/gco:CharacterString">
    <Field name="mytitle" string="{string(.)}" store="true" index="true"/>
</xsl:for-each>

Usually, if the field is only for searching and should not be displayed in search results the store attribute could be set to false.

Once the field added to the index, user could query using it as a search criteria in the different kind of search services. For example using:

http://localhost:8080/geonetwork/srv/en/q?mytitle=africa

If user wants this field to be tokenized, it should be added to the tokenized section of config-lucene.xml:

<tokenized>
  <Field name="mytitle"/>

If user wants this field to be returned in search results for the search service, then the field should be added to the Lucene configuration in the dumpFields section:

<dumpFields>
  <field name="mytitle" tagName="mytitle"/>

Boosting documents and fields¶

Document and field boosting allows catalogue administrator to be able to customize default Lucene scoring in order to promote certain types of records.

A common use case is when the catalogue contains lot of series for aggregating datasets. Not promoting the series could make the series “useless” even if those records contains important content. Boosting this type of document allows to promote series and guide the end-user from series to related records (through the relation navigation).

In that case, the following configuration allows boosting series and minor importance of records part of a series:

<boostDocument name="org.fao.geonet.kernel.search.function.ImportantDocument">
  <Param name="fields" type="java.lang.String" value="type,parentUuid"/>
  <Param name="values" type="java.lang.String" value="series,NOTNULL"/>
  <Param name="boosts" type="java.lang.String" value=".2F,-.3F"/>
</boostDocument>

The boost is a positive or negative float value.

This feature has to be used by expert users to alter default search behavior scoring according to catalogue content. It needs tuning and experimentation to not promote too much some records. During testing, if search results looks different while being logged or not, it could be relevant to ignore some internal fields in boost computation which may alter scoring according to current user. Example configuration:

<fieldBoosting>
  <Field name="_op0" boost="0.0F"/>
  <Field name="_op1" boost="0.0F"/>
  <Field name="_op2" boost="0.0F"/>
  <Field name="_dummy" boost="0.0F"/>
  <Field name="_isTemplate" boost="0.0F"/>
  <Field name="_owner" boost="0.0F"/>
</fieldBoosting>

Boosting search results¶

By default Lucene compute score according to search criteria and the corresponding result set and the index content. In case of search with no criteria, Lucene will return top docs in index order (because none are more relevant than others).

In order to change the score computation, a boost function could be define. Boosting query needs to be loaded in classpath. A sample boosting class is available. RecencyBoostingQuery will promote recently modified documents:

<boostQuery name="org.fao.geonet.kernel.search.function.RecencyBoostingQuery">
  <Param name="multiplier" type="double" value="2.0"/>
  <Param name="maxDaysAgo" type="int" value="365"/>
  <Param name="dayField" type="java.lang.String" value="_changeDate"/>
</boostQuery>