Java Content Repository FAQ

 

+

Search Tips  |   Advanced Search

 


Contents


JCR tables, nodes, and UUIDs

WCM content nodes are modeled through a graph of information stored across multiple tables. Deleting a node is not as simple a matter as removing one row from a table in the database. The interrelated connections between all the tables must be handled.

In many cases, the level of logic required is not encoded in any singular SQL command that can be sent to the database.

IBM does not publish the JCR database schema. The data in the JCR is separated across several tables in the database. There is no simple correlation between the IBM Lotus Web Content Management objects and database tables or rows.

In most cases there is no simple SQL statement that can be constructed to fix an application issue. The context or semantics for functions are encoded in the application and not the database itself.


JCR workspace

A workspace can be thought of as a view with a stored userid and password.

Within the JCR specification, a workspace is a container for a collection of nodes.

The content repository is divided into the following hierarchy...

All work in the repository is done by means of a workspace. When a user logs in to the repository, a "ticket" is issued, through which the user requests a workspace. It is through the workspace that all interaction with the nodes, or content, is performed. All modifications to nodes within a workspace is transient until a save operation. The workspace save operation persists all changes made to the nodes within the workspace to permanent storage, the database.


Dynamic workspaces

Changes to a regular workspace are not persisted to the backend data store until save is called on the workspace. A dynamic workspace is a workspace created from a regular workspace. It contains all the nodes that exist in the regular workspace. However, this sandbox is even more specialized because any changes made to nodes can be persisted by way of a save operation called on the workspace to the back end data store.

Changes made in the dynamic workspace are NOT made in the stable workspace. The changes made in a dynamic workspace are merged into the backing stable workspace when a merge operation is called.


Workspace user permissions

A workspace itself does not have access/permissions associated with it. All authorization decisions within the repository are made against the nodes in the workspace. Therefore, if an operation is requested within a workspace, the authorization check is made against the nodes involved in the operation. There are no workspace authorization checks.


Does each logged in user get a workspace, or is the workspace shared?

Within the repository there is only one workspace with a given name. However, every user that logs in to the repository and requests a workspace with a given name is given a distinct Java object to represent it. Thus, each user has their own personal playground copy of that workspace. Their changes are persisted only if they call save on the workspace object.

NOTE: Applications above the repository may choose to share or cache the workspace object in their own manner.


How do you back up and restore the database?

Each database provides its own tooling to accomplish a backup or data restore of a database instance. The database administrator should have a regular process of backup for the repository database, along with the other portal databases. In the event of a catastrophic failure, a restore may be the shortest path to recovery of the data in the system.

Restoring data from backup does not solve the root cause of problems affecting the appserver itself. For Cloudscape, a backup can be as simple as creating a .zip file archive of the directory structure that contains the database. For the other database platforms, use the associated vendor tools. A database administrator should be familiar with how to use them.


How many characters are allowed for a user ID in JCR?

For IBM WebSphere Portal 6.0.1.x and earlier, there is a limit of 32 characters for user ID's that are using JCR.

For IBM WebSphere Portal 6.1 and later, the user ID length can be up to 175 characters.


How do you change the database access rights after the JCR Database has been created?

For details on how to do this, refer to IBM Support Technotes...

Databases must be created using UNICODE Database and National character sets such as UTF8, AL32UTF8, or AL16UTF16.


JCR Troubleshooting


Install the latest JCR Cumulative fix

  1. To see the current recommended JCR cumulative fix, refer to the Support Technote...

  2. Select the appropriate product version and then search the page for "JCR".


Lotus WCM Support Tools Portlet

Install the Support Tools Portlet from the IBM Lotus and WebSphere Portal Business Solutions Catalog.

The Support Tools portlet is helpful in understanding the structure of the JCR database and is invaluable in troubleshooting many WCM and JCR issues.

Some features of the portlet are not enabled due to the risk of database corruption if misused. IBM Support will provide codes valid for the current week to enable these additional features, if necessary to troubleshoot an issue effectively.

It is also possible to run XPath queries that are pulled from a JCR Trace and paste them into the Support Tools Portlet.

Web Application display name from web.xml: WCMSupportTools
Context Root: /wps/PA_WCMSupportTools
Enterprise Application display name PA_WCMSupportTools


Collect a JCR trace

  1. Delete any trace*.log files in...

      6.1.x websphere/wp_profile/logs/WebSphere_Portal/
      6.0.x websphere/Portalserver/log/

  2. Log in to WebSphere Portal with the portal admin userid.

  3. Open a second tab in the same browser and copy/paste the URL from the first tab as the URL for the second tab.

  4. Navigate to the point in the first tab just before the problem is reproduced and stop at that point.

  5. In the second tab, enable the trace by selecting...

      Administration | Portal analysis | Enable tracing

  6. Add the tracestring below in the "Append these trace settings" field and click the "+" icon:

      com.ibm.icm.*=finest

  7. Reproduce the problem in first tab and remove the tracestring.


JCR performance issues

To troubleshoot JCR performance issues, start with the Support Technote...

Once the long-running queries have been identified, either work with the application to reduce the size or complexity of the query sent to JCR, or optimize the database server to reduce the time to process the query.

To optimize the database server performance, start with the WebSphere Portal Family wiki article...

One approach that can help with query optimization is to extract the query from the trace and run it from the command line. For some queries all values are populated in the trace, but for other queries the "host variables" are missing and replaced by a "?" in the trace.

When this occurs, first check the trace entries below the query to see if the values that should be substituted for the "?" are present. If not, temporarily configure the portal to output the "host variables" in the trace, using this procedure:

  1. Stop WebSphere Portal Server.

  2. Edit...

      PortalServer/jcr/lib/com/ibm/icm/icm.properties

    ...setting the following property:

      pls.debug.trackStatementCursorLeakage=+

    Note that the value above must be set to "+" and not to "true".

  3. Set trace to...

      com.ibm.icm.*=all

    ...in the WebSphere Application Server Admin console, if trace is required during WebSphere Portal startup.

    Otherwise, use the dynamic instructions in the Collect a JCR Trace section above.

  4. Restart WebSphere Portal.

    Once the query is reproduced from the command line, work with the DBA to analyze database performance. In many cases, the DBA can identify a new index that will help reduce the time to process the query.


AccessDeniedException

An example of this error is...

An AccessDeniedException error from WCM can mean one of the following:


Find locked library nodes

Use the Support Tools Portlet to locate the node(s)and their locks, if you're unsure of the current lock status.

The Display Node option of WCM Support Tools will display all the locks for the specified node and its children. Any of the nodes that are locked will be shown in the portlet as either "external" or "internal".

Use the following steps in the Support Tools Portlet to display the node and its locks:

  1. Launch Utility JSP's.

  2. Select the Display Node option.

  3. Enter the following values:

    • The node by path or UUID
    • Show Node's Children
    • Show Node's Locks

    You can get UUID for a library node by running Browse Nodes.

External locks indicate that the node is locked by another user, and the lock details shows the lock userId. So the next step in this case is to log in as that user and unlock the node(s).

Internal locks indicate that the node is locked by a draft workspace. This is the common occurrence if the content node has a pending draft. In this case the next step is to remove the content node and its drafts.

The logged-in user does not have available Portal Access Control for this object. This is a valid exception if the user does not have the correct access rights for the requested action. The administrator must grant the necessary rights through WebSphere Portal for that object and action.

Examples from trace:

Permission 8 is READ permission.

The JCR node is locked by another workspace. Here's an example trace for this issue:

This is a common occurrence if the content node is being held by another draft. In this case, delete the draft before proceeding. When the draft is deleted, JCR will remove the draft workspace.

Try using the application user interface (for example, the Web Content Management Authoring UI) to clear out draft workspaces.

If unable to unlock the node through the Authoring UI, use the Support Tools Portlet to remove the locks for that node:

Occasionally a problem will occur after a failed library delete, in which drafts are still left in the library. In this case, use the Support Tools Portlet or manageDrafts.jsp to delete the drafts and their associated workspaces. (Contact IBM Support for a copy of manageDrafts.jsp.)

The JCR node is locked by another user. It is common for another user to be working on the same node, and this is considered normal behavior. The best action for this failure is to log in as the other user and unlock the node.

AccessDeniedExceptions during import-wcm-data. Lock exceptions during import-wcm-data are caused by leftover locks from the export. The solution is to remove the locks during export, using these steps:

  1. Edit PortalServer/jcr/migration/conf/wcm60_conf.xml, replacing the "serializer" property (at around line 100) with the following:

      <!-- The class name for the serializer instance to be used -->
      <!-- property name="serializer" value="com.ibm.icm.jcr.serialization.serializer.ext.WPSDocLibNodeSerializerImpl"/ -->
      <property name="serializer" value="com.ibm.icm.jcr.serialization.serializer.ext.WPSNoLockDocLibNodeSerializerImpl"/>

  2. Export the library again to an empty directory.

  3. Retry the import based on the new export directory.


ItemNotFoundException

JCR throws an ItemNotFoundException when the requested node cannot be located, for example:

This issue can be due to two causes:

First verify that the user has access to the requested nodes in WebSphere Portal, and then contact the application's Support team regarding why the requested node has not been saved in the repository.


StaleValueException

This is a normal exception by JCR and does not indicate a JCR failure. StaleValueException is thrown when the current node being saved is different from the latest version of that node in the database.

To resolve the issue:

  1. Locate the DBAccessException in the logs, for example:

      [3/8/10 7:14:17:194 CST] 000000c4 PUpdateNodeIm E PUpdateNodeImpl persistUpdatedContents Error while calling a function updateItems of PLS data manager.
      com.ibm.icm.da.DBAccessException: [ErrorCode:10024] Error, item AB001001N10C04B700000C61D9, versionID=1 (userTable=ICMUT01074001) is stale, timestamp mismatch: 2/28/10 12:00 AM, 2/28/10 12:00 AM

  2. Compare the two timestamps in the timestamp mismatch in the exception text:

    Exactly one hour apart (for example, 2/28/10 10:00 AM, 2/28/10 11:00 AM). There is a known issue with JCR nodes that have been created or updated within the first hour after the time switch to Daylight Saving Time (DST).

    The cause of this problem is that Java compensates for DST differently than the database. During this hour of overlap, Java advances internal timestamps one hour. However, the database does not advance the timestamp during this period, resulting in an inconsistency between Java timestamps and database timestamps. The end result is that the user encounters a StaleValueException every time they try to update nodes from this hour of overlap.

NOTE: This DST issue has been addressed by ifixes PK76660 and PK82630, which are now available on all releases as part of the latest JCR Cumulative Fix.

We have also encountered a StaleValueException that is caused by problems in Oracle JDBC drivers prior to version 10.2.0.3. If you receive a StaleValueException related to DST and the above ifixes have been applied, make sure the Oracle driver is at the correct level.

Different values. This is the exception caused by normal processing. Retry the operation and make sure that no other user is updating the same content at the same time. If the problem persists, contact the application's Support team (Web Content Management, Personalization, Portal Document Manager, etc.).

The same value (for example, 2/28/10 12:00 AM, 2/28/10 12:00 AM). A problem has been observed on Oracle servers in which the Oracle database saves an invalid timestamp. This results in a StaleValueException from JCR because the Java cannot correctly read the invalid timestamp(s).

To avoid the issue, use these steps to force a resave of the data:

  1. Install the Support Tools Portlet (version 1.0.5 or later).
  2. Set the access code for update (available from IBM Support).
  3. Search for the document by IID from the DBAccessException (for example, AB001001N10C04B700000C61D9 in the example in Step 1).
  4. Edit the node and save it.
  5. If the node is Web Content Management content, update the node in Web Content Management to verify that the data syndicates correctly.


javax.transaction.RollbackException

A javax.transaction.RollbackException indicates that a previous failure has occurred and the current database transaction is being rolled back. Find and troubleshoot the previous failure.

For DB2 users, also check the db2diag.log file for the previous exception(s).


Database deadlocks

To troubleshoot a database deadlock:

  1. Ensure the database statistics are current.

  2. Verify the number of database connections and webcontainer threads. A deadlock in the JRE can occur if there are less database connections than webcontainer threads in the configured WebSphere Portal server. For more information, refer to the Support Technote...

      Unable to locate Dynamic Workspace

  3. Verify the deadlock exceptions in the trace, for example,

      "SQL0913N Unsuccessful execution caused by deadlock or timeout."

    ...or...

      COM.ibm.db2.jdbc.DB2Exception: [IBM][CLI Driver][DB2/AIX64] SQL0913N Unsuccessful execution caused by deadlock or timeout. Reason code "68"." The error description can be found in the DB2 InfoCenter. The issue is caused by either a deadlock or a lock timeout:
      # Reason Code "2": Deadlock
      # Reason Code "68": Timeout

    Also, note that "Server returned XA_RBDEADLOCK" is an example of a deadlock message from an XA Connection.

  4. Increase the DB2 lock timeout and verify the results. For more information, refer to the DB2 InfoCenter topic...

      SET CURRENT LOCK TIMEOUT statement

  5. Obtain deadlock diagnostic information:

    • Verify that the deadlock message(s) in the WebSphere Portal logs identify the tables involved.

    • with JCR trace enabled (see the JCR Trace section above).

    • snapshot of the failure. Note that the snapshot specifics vary based on database platform, so refer to the documentation for the database server (DB2, Oracle) for specific information.


InvalidWorkspaceException

JCR throws this exception when an error has occurred, rendering the current log-in session invalid. For instance:

To correct the problem, log out and then log in. Note that this exception does not require data in the repository to be deleted.

IMPORTANT: InvalidWorkspaceException is an indication that a previous problem has left the current log-in session invalid, so be sure to identify and address the original problem.




OutOfMemoryError

There are often cases in which the OutOfMemoryError is caused by the application processing too much data when using a StrongWorkspaceState, which can be identified is you see StrongWorkspaceState holding a lot of memory in the heap dump analysis.

The StrongWorkspaceState is an internal cache of all JCR nodes referenced in the current workspace. The OutOfMemoryError here indicates that more nodes are referenced than the current memory can handle; thus the application must be changed to do one of the following:

  1. Reduce the number of nodes processed per workspace.
  2. Manually clear the workspace state at an interval that can be handled by the system's memory.
  3. Use Weak workspace state.

The latest JCR Cumulative Fix for WebSphere Portal 6.0.1 and 6.1 contains a fix that helps identify the application that is improperly using the StrongWorkspaceState. It does not resolve the problem; however, if you enable trace to com.ibm.icm.jcr.StrongWorkspaceState=all and recreate the failure, then after the next OutOfMemoryError the trace will contain debug stack traces that can be used by IBM Support to identify the failure.

Otherwise, this error needs more diagnostic information to determine the cause. You'll need to collect Java cores and the heap dump of the failure, which will show the objects that are consuming the memory during the time of failure.


ReferentialIntegrityException

JCR throws a ReferentialIntegrityException when the application attempts to remove a node that contains an outstanding reference, for example:

All references to the node must be removed before removing the node itself. To resolve the issue, locate and remove the outstanding references, following these steps:

  1. Locate the IID values from the error message (for example, 'AB001001N07L28C22602B8A3B7' and 'AB001001N07L28C217029035B7')
  2. Install the Support Tools Portlet.
  3. Browse nodes, and lookup the nodes by IID.

After identifying the nodes involved with the exception, use the application (WCM Authoring) to remove the reference.


Non-atomic batch failure

To troubleshoot a non-atomic batch failure:

  1. Make sure the latest cumulative fix is applied (see above), to ensure diagnostic information for the Non-atomic batch failure is available.
  2. Enable JCR Trace (see above) and recreate the failure.
  3. Look in the logs for the database exception; the actual exception is included in the trace immediately before the Non-atomic batch failure.


Database exceptions

To troubleshoot database exceptions:

  1. Ensure that the tables are part of JCR.

    JCR tables start with ICM* and TSS*. Note that other applications (including WCM and PAC) store data in the JCR database.

  2. Get the exception and stack trace from the logs, for example:

      [4/22/09 9:33:56:592 PDT] 0000005a PGetItemsByCM E Error while calling a function retrieveItemsByCMId of PLS data manager.
      com.ibm.db2.jcc.a.SqlException: DB2 SQL error: SQLCODE: -104, SQLSTATE: 42601, SQLERRMC:

In the case of DB2, translate the SQLCODE value using the DB2 InfoCenter.

For example, the DB2 exception above translates to...

In the case of SQL Server on WebSphere Portal 6.0.1.x, verify lowercase schema issues by confirming that the following fixpack is applied...


DB2 Database hangs during WebSphere Portal upgrade.

There have been issues in which a database hang occurs while upgrading WebSphere Portal versions (e.g., upgrading to 6.0.1.4). This can be caused by the database user not having DBADM authority. Note that it is not enough to grant only SYSADM authority to the user; the user must have explicit DBADM authority.


SQLSTATE=54001 - The statement is too long or too complex

Increase the values of the DB2 statement heap (stmtheap), the application heap (applheapsz), and the cache size (pckcachesz), and then restart the database manager for these updates to take effect. If this failure occurs during a one-time operation, such as library delete, it's OK to reduce these values after delete has completed.

In addition, this error can occur if database statistics are not regularly executed. Make sure the database statistics are current. Refer to the Database performance topic in the WebSphere Portal InfoCenter for more information.


SQLCODE 805 - Missing database package.

For example:

This indicates a need to bind the db2schema.bnd file against each database by entering the following commands at a DB2 command prompt:

...where "database-name" is the name of the database to which the utilities must be bound, and "path" is the full path name of the directory where the bind files are located.


SQLCODE 302 (host variable is too large) during query.

For example:

This exception can occur if the XPath query specifies invalid properties, causing the database to throw this exception when trying to execute the invalid query. If this exception is encountered, obtain a JCR trace and verify that the XPath query does not attempt to query invalid properties.


DB2 zOS:

Failure to create database

The prefix must be unique enough to contain only the databases used for the content repository. Always use a prefix that provides unique database name conventions (for example, DPTJCRXXX, or something similar).

Note that this problem can also appear during database transfer, when JCR is attempting to write to the wrong database. This is another case in which the jcr.ZosDbPrefix is not unique.


Pause during database transfer

WebSphere Portal 6.1 has a pause option for the database transfer, allowing the administrator to customize the generated DDL before it is applied to the database. Enable the pause feature by adding "-DTransferPauseEnabled=true" to the database-transfer command.


Oracle:

Error with ordered results

There is a known defect with ordering on Oracle 9.2.0.6, in which an ordered list from JCR may not be returned in the correct order. Upgrade to Oracle 10.0.2.3 or later to resolve the problem. For more information, refer to the Support Technote...


Menu ordering is broken in WCM due to bug in Oracle Trigger failures

Example error:

This issue occurs when the triggers are created in the incorrect schema. This has been fixed with PK48092, but the problem may still occur after a database is transferred to Oracle before applying PK48092.

NOTE: PK48092 is included in the latest JCR Cumulative Fix for WebSphere Portal 6.0.1.x and is not needed for Portal 6.1.

To resolve the problem, verify:

For more details, refer to the Support Technote...


SQL Server

PK87097: Make 6.0x fault tolerant of lowercase schema

This fix allows WebSphere Portal 6.0.1.x to be tolerant of lowercase database schema and is part of the latest JCR Cumulative Fix.


Database constraint failures on ICMSTJCRNODELOCKS.

For example:

This exception can be seen in the ffdc logs. It's part of normal processing and does not indicate a failure.

Unable to extend index ICMADMIN.xxxx by nnn in tablespace yyyy ORA-06512: at line 1.

Here the DBA needs to increases the size of tablespace yyyy.


DuplicateKeyException

For example:

This is expected behavior when trying to create the same node (e.g. /.personalization) in more than one workspace (e.g. ROOTWORKSPACE and RULESWORKSPACE). Please ensure that this node (/.personalization) has been completely removed from all workspaces before continuing.


Database connection issues:

Oracle StaleConnectionException

For example:

This problem has been seen on Oracle 10.2.0.2 and is an Oracle problem. Oracle Support recommends upgrading to version 10.2.0.3.0.

This problem can occur when another process closes all database connections. It's often accompanied by NullPointerException messages within JCR, as JCR has now lost the database connection in mid-process. Instances of this problem have been resolved with PK60501 from WCM.

This error indicates that the Java Database Connectivity (JDBC) connection to the database has been closed elsewhere. Applying the latest JCR Cumulative Fix may help with the issue.


SQL30081N Failure

For example:

Connection Manager received a fatal connection error from the Resource Adaptor for resource jdbc/wpdbDS. The exception which was received is...

"SQL30081N RC 32" signifies a broken pipe between the DB2 client driver and DB2 server. In the case of one user, the issue was due to a network error and required an AIX patch.


Import Issue

Installing Portal NOW. Import goes up to 70% and then fails with...

00000063 ImportStatusC I com.ibm.icm.jcr.command.ImportStatusCommand importNode
Imported 360 files (67%)
00000063 ImportStatusC I com.ibm.icm.jcr.command.ImportStatusCommand importNode
Imported 380 files (71%)
00000063 ServletWrappe E SRVE0068E: Uncaught exception thrown in one of the service methods of the servlet: ImportNodeCommandServlet. Exception thrown : javax.servlet.ServletException: responseCode=500
at com.ibm.icm.jcr.command.AbstractCommandServlet.doPost(AbstractCommandServlet.java:147)
at com.ibm.icm.jcr.command.AbstractCommandServlet.doGet(AbstractCommandServlet.java:81)

Fix

Import latest JCR update ifix


IWKMU1062X

Caused by: com.ibm.workplace.wcm.services.repository.RepositoryException: IWKMU1062X: Message: An External Lock could not be applied to {290f16004e1014fd9df8bd55e5d01204, com.aptrix.pluto.content.Content}., Cause: com.ibm.workplace.wcm.services.repository.LockNodeFailedException: An External Lock could not be applied to {290f16004e1014fd9df8bd55e5d01204, com.aptrix.pluto.content.Content}.

Check: http://www-01.ibm.com/support/docview.wss?uid=swg21322145

PM13808 solves issues related to the "LOCKING OF SITEAREAS AND CATEGORIES MAKES THEM UNAVAILABLE VIA WEBDAV"


JCR Text Search Troubleshooting