Tune the IBM virtual machine for Java


+

Search Tips   |   Advanced Search

 

As part of configuring appservers, tune the Java runtime environment for performance.

Determine the type of JVM on which the appserver is running.

cd WAS_HOME/java/bin
java -fullversion

JVM provider information is written to SystemOut.log.

Verify the most recent supported version, and service update, is installed.

General tuning concepts...

 

Limit the number of thread dumps

In certain error conditions, multiple appserver threads might fail and the JVM requests a TDUMP for each of those threads. If a significant number of threads fail at the same time, the resulting number of TDUMPs that are taken concurrently might lead to other system problems, such as a shortage of auxiliary storage.

To specify the number of dumps the JVM produces, use the environment variable JAVA_DUMP_OPTS

For example, to configure the JVM to...

..set JAVA_DUMP_OPTS to...

JAVA_DUMP_OPTS=ONANYSIGNAL(JAVADUMP[3],SYSDUMP[1]),ONINTERRUPT(NONE)

Note that the value specified for JAVA_DUMP_OPTS does not affect the number of TDUMPS that are generated because of calls to...

com.ibm.jvm.Dump.SystemDump()

 

Optimize the startup and runtime performance

In some environments, such as a development environment, it is more important to optimize the startup performance of the appserver rather than the runtime performance. In other environments, it is more important to optimize the runtime performance. By default, IBM virtual machines for Java are optimized for runtime performance, while HotSpot-based JVMs are optimized for startup performance.

The Java Just-in-Time (JIT) compiler impacts whether startup or runtime performance is optimized. The initial optimization level that the compiler uses influences the length of time that is required to compile a class method, and the length of time that is required to start the server.

For faster startups, reduce the initial optimization level that the compiler uses.

 

Configure the heap size

Increasing the heap size supports more object creation. Because a large heap takes longer to fill, the application runs longer before a garbage collection occurs.

However, a larger heap also takes longer to compact and causes garbage collection to take longer.

The JVM uses defined thresholds to manage the storage that it is allocated. When the thresholds are reached, the garbage collector is invoked to free up unused storage.

In the majority of cases you should set the maximum JVM heap size to a value that is higher than the initial JVM heap size. This setting allows for the JVM to operate efficiently during normal, steady state periods within the confines of the initial heap.

This setting also allows the JVM to operate effectively during periods of high transaction volume because the JVM can expand the heap up to the value specified for the maximum JVM heap size. In some rare cases, where absolute optimal performance is required, we might want to specify the same value for both the initial and maximum heap size. This setting eliminates some overhead that occurs when the JVM expands or contracts the size of the JVM heap. Before changing any of the JVM heap sizes, verify that the JVM storage allocation is large enough to accommodate the new heap size.

Do not make the size of the initial heap so large that while it initially improves performance by delaying garbage collection, when garbage collection does occur, the collection process affects response time because the process has to run longer.

With new runtime provisioning functionality, theoretically we can reduce heap sizes by up to 30%.

To configure the heap sizes...

Servers | Server Types | WebSphere application servers | server_name | Server Infrastructure | Java and process management | Process definition | Java virtual machine

..and specify a new values for...

For performance analysis, the initial and maximum heap sizes should be equal.

Do benchmarking of memory on OS level (vmstat) before deployment of WAS and after creation of JVMs.

If tuning a production system where you do not know the working set size of the enterprise apps that are running on that system, an appropriate starting value for the initial heap size is 25 percent of the maximum heap size. The JVM then tries to adapt the size of the heap to the working set size of the application.

The illustration below represents three CPU profiles, each running a fixed workload with varying Java heap settings.

In the middle CPU profile, the initial and maximum heap sizes are set to 128 MB. Four garbage collections occur. The total time in garbage collection is about 15 percent of the total run.

When the heap parameters are doubled to 256 MB, as in the top CPU profile, the length of the work time increases between garbage collections and only three garbage collections occur, but the length of each garbage collection is also increased.

In the third CPU profile, the heap size is reduced to 64 MB and exhibits the opposite effect. With a smaller heap size, both the time between garbage collections and the time for each garbage collection are shorter.

For all three configurations, the total time in garbage collection is approximately 15 percent.



The optimal result for the average time between garbage collections is at least five to six times the average duration of a single garbage collection. If we do not achieve this number, the application is spending more than 15 percent of its time in garbage collection.

Run a series of tests that vary the Java heap settings. For example, run experiments with 128 MB, 192 MB, 256 MB, and 320 MB. During each experiment, monitor the total memory usage. If we expand the heap too aggressively, paging can occur.

Use the vmstat command or the Windows Performance Monitor to check for paging. If paging occurs, reduce the size of the heap or add more memory to the system.

When all the runs are finished, compare the following statistics:

If the application is not over utilizing objects and has no memory leaks, the state of steady memory utilization is reached. Garbage collection also occurs less frequently and for short duration.

If the heap free space settles at 85 percent or more, consider decreasing the maximum heap size values because the appserver and the application are under-utilizing the memory allocated for heap.

Cycle appservers after changing heap settings.

We can also use the following command-line parameters to adjust these settings. These parameters apply to all supported JVMs and are used to adjust the minimum and maximum heap size for each appserver or appserver instance.

 

Tune Java memory

  1. Check for over-utilization of objects.

    Review the counters for the JVM run time included in Tivoli Performance Viewer reports. Specify the -XrunpmiJvmtiProfiler command-line option, as well as the JVM module maximum level, to enable JVMTI counters.

    The optimal result for the average time between garbage collections is at least five to six times the average duration of a single garbage collection. If we do not achieve this number, the application is spending more than 15 percent of its time in garbage collection.

    If the information indicates a garbage collection bottleneck, there are two ways to clear the bottleneck.

    • Optimize the application to implement object caches and pools. Use a Java profiler to determine which objects to target.

    • Add memory, processors and clones. Additional memory allows each clone to maintain a reasonable heap size. Additional processors allow the clones to run in parallel.

  2. Test for memory leaks.

    Memory leaks are more damaging than memory overuse, because a memory leak ultimately leads to system instability. Over time, garbage collection occurs more frequently until the heap is exhausted and the Java code fails with a fatal out-of-memory exception.

    Memory leaks occur when an unused object has references that are never freed. Memory leaks most commonly occur in collection classes, such as Hashtable because the table always has a reference to the object, even after real references are deleted.

    High workload often causes applications to crash immediately after deployment in the production environment. These application crashes if the applications are having memory leaks because the high workload accelerates the magnification of the leakage, and a memory allocation failures occur.

    The goal of memory leak testing is to magnify numbers. Memory leaks are measured in terms of the amount of bytes or kilobytes that cannot be garbage collected. The delicate task is to differentiate these amounts between expected sizes of useful and unusable memory. This task is achieved more if the numbers are magnified, resulting in larger gaps and easier identification of inconsistencies.

    The following list provides insight on how to interpret the results of the memory leak testing:

    • Long-running test

      Memory leak problems can manifest only after a period of time, therefore, memory leaks are found during long-running tests. Short running tests might provide invalid indications of where the memory leaks are occurring. It is sometimes difficult to know when a memory leak is occurring in the Java language, especially when memory usage has seemingly increased either abruptly or monotonically in a given period of time.

      The reason it is hard to detect a memory leak is that these kinds of increases can be valid or might be the intention of the developer. We can learn how to differentiate the delayed use of objects from completely unused objects by running applications for a longer period of time. Long-running application testing gives you higher confidence for whether the delayed use of objects is actually occurring.

    • Repetitive test

      In many cases, memory leak problems occur by successive repetitions of the same test case. The goal of memory leak testing is to establish a big gap between unusable memory and used memory in terms of their relative sizes. By repeating the same scenario over and over again, the gap is multiplied in a very progressive way. This testing helps if the number of leaks caused by the execution of a test case is so minimal that it is hardly noticeable in one run.

      We can use repetitive tests at the system level or module level. The advantage with modular testing is better control. When a module is designed to keep the private module without creating external side effects such as memory usage, testing for memory leaks is easier.

      First, the memory usage before running the module is recorded. Then, a fixed set of test cases are run repeatedly. At the end of the test run, the current memory usage is recorded and checked for significant changes. Remember, garbage collection must be suggested when recording the actual memory usage by inserting System.gc() in the module where you want garbage collection to occur, or using a profiling tool, to force the event to occur.

    • Concurrency test

      Some memory leak problems can occur only when there are several threads running in the application. Unfortunately, synchronization points are very susceptible to memory leaks because of the added complication in the program logic. Careless programming can lead to kept or not-released references.

      The incident of memory leaks is often facilitated or accelerated by increased concurrency in the system. The most common way to increase concurrency is to increase the number of clients in the test driver. Consider the following points when choosing which test cases to use for memory leak testing:

      • A good test case exercises areas of the application where objects are created. Most of the time, knowledge of the application is required.

        A description of the scenario can suggest creation of data spaces, such as adding a new record, creating an HTTP session, performing a transaction and searching a record.

      • Look at areas where collections of objects are used. Typically, memory leaks are composed of objects within the same class. Also, collection classes such as Vector and Hashtable are common places where references to objects are implicitly stored by calling corresponding insertion methods. For example, the get method of a Hashtable object does not remove its reference to the retrieved object.

    Use the Tivoli Performance Viewer to help find memory leaks.

    For optimal results, repeat experiments with increasing duration, such as 1,000, 2,000, and 4,000 page requests. The Tivoli Performance Viewer graph of used memory should have a jagged shape. Each drop on the graph corresponds to a garbage collection. There is a memory leak if one of the following conditions is appears in the graph:

    • The amount of memory used immediately after each garbage collection increases significantly. When this condition occurs, the jagged pattern looks more like a staircase.

    • The jagged pattern has an irregular shape.

    • The gap between the number of objects allocated and the number of objects freed increases over time.

    Heap consumption that indicates a possible leak during periods when the appserver is consistently near 100 percent CPU utilization, but disappears when the workload becomes lighter or near-idle, is an indication of heap fragmentation. Heap fragmentation can occur when the JVM can free sufficient objects to satisfy memory allocation requests during garbage collection cycles, but the JVM does not have the time to compact small free memory areas in the heap to larger contiguous spaces.

    Another form of heap fragmentation occurs when objects that are less than 512 bytes are freed. The objects are freed, but the storage is not recovered, resulting in memory fragmentation until a heap compaction occurs.

    Heap fragmentation can be reduced by forcing compactions to occur. However, there is a performance penalty for forcing compactions. Use the Java -X command to see the list of memory options.

 

Tune garbage collection

Examining Java garbage collection gives insight to how the application is utilizing memory. Garbage collection is a Java strength. By taking the burden of memory management away from the application writer, Java applications are more robust than applications written in languages that do not provide garbage collection. This robustness applies as long as the application is not abusing objects. Garbage collection typically consumes from 5 to 20 percent of total run time of a properly functioning application.

If not managed, garbage collection is one of the biggest bottlenecks for an application.

Monitor garbage collection while a fixed workload is running, provides you with insight as to whether the application is over using objects. Garbage collection can even detect the presence of memory leaks.

Use JVM settings to configure the type and behavior of garbage collection. When the JVM cannot allocate an object from the current heap because of lack of contiguous space, the garbage collector is invoked to reclaim memory from Java objects that are no longer being used. Each JVM vendor provides unique garbage collector policies and tuning parameters.

Use the Verbose garbage collection setting in the admin console to enable garbage collection monitoring. The output from this setting includes class garbage collection statistics. The format of the generated report is not standardized between different JVMs or release levels.

To adjust the JVM garbage collection settings:

  1. In the admin console, click...

    Servers | Server Types | WebSphere application servers | server_name | Server Infrastructure | Java and process management | Process definition | Java virtual machine

  2. Enter the –X option you want to change in the Generic JVM arguments field.

  3. Click Apply.

  4. Click Save to save the changes to the master configuration.

  5. Stop and restart the appserver.

The following list describes the –X options for the different JVM garbage collectors.

The IBM virtual machine for Java garbage collector.

A complete guide to the IBM implementation of the Java garbage collector is provided in the IBM Developer Kit and Runtime Environment, Java 2 Technology Edition, V5.0 Diagnostics Guide. This document is available on the developerWorks Web site.

Use the Java -X option to view a list of memory options.

  • -Xgcpolicy

    The IBM virtual machine for Java provides four policies for garbage collection. Each policy provides unique benefits.

    • optthruput is the default policy, and provides high throughput but with longer garbage collection pause times. During a garbage collection, all application threads are stopped for mark, sweep and compaction, when compaction is needed. The optthruput policy is sufficient for most applications.

    • optavgpause is the policy that reduces garbage collection pause time by performing the mark and sweep phases of garbage collection while an application is running. This policy causes a small performance impact to overall throughput.

    • gencon, is the policy that works with the generational garbage collector. The generational scheme attempts to achieve high throughput along with reduced garbage collection pause times. To accomplish this goal, the heap is split into new and old segments. Long lived objects are promoted to the old space while short-lived objects are garbage collected quickly in the new space. The gencon policy provides significant benefits for many applications. However, it is not suited for all applications, and is typically more difficult to tune.

    • subpool is a policy that increases performance on multiprocessor systems, that commonly use more then 8 processors. This policy is only available on IBM System iSystem pand System zprocessors. The subpool policy is similar to the optthruput policy except that the heap is divided into subpools that provide improved scalability for object allocation.

    Default optthruput
    Recommended optthruput
    Usage Xgcpolicy:optthruput sets the garbage collection policy to optthruput

    Set gcpolicy to optthruput disables concurrent mark. You should get optimal throughput results when you use the optthruput policy unless we are experiencing erratic application response times, which is an indication that we might have pause time problems

    Set gcpolicy to optavgpause enables concurrent mark with its default values. This setting alleviates erratic application response times that normal garbage collection causes. However, this option might decrease overall throughput.

  • -Xnoclassgc

    By default, the JVM unloads a class from memory whenever there are no live instances of that class left. Therefore, class unloading can decrease performance.

    Use the -Xnoclassgc argument to disable class garbage collection so that the applications can reuse classes more . Turning off class garbage collection eliminates the overhead of loading and unloading the same class multiple times.

    Avoid trouble: This argument should be used with caution, if the application creates classes dynamically, or uses reflection, because for this type of application, the use of this option can lead to native memory exhaustion, and cause the JVM to throw an Out-of-Memory Exception. When this option is used, if we have to redeploy an application, you should always restart the appserver to clear the classes and static data from the pervious version of the application.gotcha

    Default Class garbage collection is enabled.
    Recommended

    Disable class garbage collection.

    Usage Xnoclassgc to disable class garbage collection.

 

Enable class sharing in a cache

The share classes option of the IBM implementation of J2RE V1.5.0 lets you share classes in a cache. Sharing classes in a cache can improve startup time and reduce memory footprint. Processes, such as appservers, node agents, and dmgrs, can use the share classes option.

The IBM implementation of J2RE V1.5.0 is currently not supported on Solaris and HP-UX

If we use this option, clear the cache when the process is not in use. To clear the cache, either call the utility...

APP_ROOT/bin/clearClassCache.bat/sh

...or stop the process and then restart the process.

To disable the share classes option for a process, specify the generic JVM argument...

-Xshareclasses:none

  1. In the admin console, click...

    Servers | Server Types | WebSphere application servers | server_name | Server Infrastructure | Java and process management | Process definition | Java virtual machine

  2. Enter -Xshareclasses:none in the Generic JVM arguments field.

  3. Click OK.

  4. Click Save to save the changes to the master configuration.

  5. Stop and restart the appserver.

Default The Share classes in a cache option are enabled.
Recommended Leave the share classes in a cache option enabled.
Usage -Xshareclasses:none disables the share classes in a cache option.

 

Enable compressed references on 64-bit environments

The compressed references option of the IBM implementation of the 64-bit Java SE Runtime Environment V6.0 lets you limit all of the memory references to 32-bit size. Typically, the 64-bit JVMs use more heap space than the 32-bit JVMs because they use 64-bit wide memory references to address memory. The heap that is addressable by the 64-bit reference is orders of magnitude larger than the 32-bit heap, but in the real world, a heap that requires all 64-bits for addressing is typically not required. Compressing the references reduces the size of the addresses and makes more efficient use of the heap. Compressing these references also improves the processor cache and bus utilization, thereby improving performance.

This is for environments such as AIX 64, Linux PPC 64, zLinux 64, and Microsoft Windows AMD64, and Linux AMD64.

The compressed references feature is not supported on:

WAS ND v7.0 automatically enables pointer compression on the supported platforms by default if the heap size (controlled by the -Xmx parameter) is set under a certain heap size (around 25 GB depending on platform), else it will default to non-compressed references. The user can override these defaults by using the command line options below.

The use of compressed references improves the performance of many applications because objects are smaller, resulting in less frequent garbage collection and improved memory cache use. Certain applications might not benefit from compressed references. Test the performance of the application with and without the option to determine if it is appropriate.

When using compressed references, the following structures are allocated in the lower area of the address space:

  1. Classes
  2. Threads
  3. Monitors

Because this is a JVM technique that is separated from WAS v7, we can only activate it by using a JVM argument on a JVM level.

In the Integrated Solutions Console navigate to...

Server | Server Types | WebSphere application servers | my_server | Server Infrastructure | Java and Process Management | Process definition | server_component (Adjunct | Control | Servant) | Java Virtual Machine

Then add the following statement to the generic JVM arguments:

-Xcompressedrefs

As always when changing some JVM settings, we have to restart the server after saving and synchronizing the modifications to activate them.

-Xcompressedrefs Enables the compressed references feature. When the JVM is launched with this command line option it would use 32-bit wide memory references to address the heap. This feature can be used up to a certain heap size (around 29GB depending on the platform), controlled by -Xmx parameter.
-Xnocompressedrefs Explicitly disables the compressed references feature. When the JVM is launches with this command line option it will use full 64-bit wide memory references to address the heap. This option can be used by the user to override the default enablement of pointer compression, if needed.

 

Tune the configuration update process for a large cell configuration

In a large cell configuration, we might have to determine whether configuration update performance or consistency checking is more important. When configuration consistency checking is turned on, a significant amount of time might be required to save a configuration change, or to deploy a several applications.

The following factors influence how much time is required:

If the amount of time required to change a configuration change is unsatisfactory, we can add the config_consistency_check custom property to the JVM settings and set the value of this property to false.

  1. In the admin console, click...

    Servers | Server Types | WebSphere application servers | server_name | Server Infrastructure | Java and process management | Process definition | Java virtual machine | Custom properties | New

  2. Enter config_consistency_check in the Name field and false in the Value field.

  3. Click OK and then Save to apply these changes to the master configuration.

  4. Restart the server.

If using the wsadmin command wsadmin -conntype none in local mode, set the config_consistency_check property to false before issuing this command.

 

Related tasks

Tuning operating systems
Enable the Java virtual machine profiler data
Tuning the application serving environment