A Just-In-Time (JIT) compiler is a platform-specific compiler that generates machine instructions for each method as needed. For more information, see the sections Using the Just-In-Time compiler and Just-In-Time compiler in the i5/OS Information Center topicIBM Developer Kit for Java.
- In the administrative console, click Servers > Application Servers > server.
- Under Server Infrastructure, click Java and Process Management > Process Definition > Java Virtual Machine.
- Select the Disable JIT option if you want to disable the JIT.
- Enter -Djava.compiler=jitc in the Generic JVM arguments field if you want to run with the full JIT compiler.
- Click Apply or OK.
- Save changes to the master configuration.
- Stop and restart the application server.
| Default: | JIT is enabled. |
| Recommended: | It is recommended that you do not disable the JIT compiler, and you enable the full JIT compiler (see step 4.d). The os400.jit.mmi.threshold can have a significant effect on performance. For more information about the JIT compiler and the os400.jit.mmi.threshold property, see the section Just-In-Time compiler in the i5/OS Information Center topic IBM Developer Kit for Java. |
In some environments, such as a development environment, it is more important to optimize the startup performance of your application server rather than the runtime performance. In other environments, it is more important to optimize the runtime performance. By default, IBM JVMs are optimized for runtime performance, while HotSpot based JVMs are optimized for startup performance.
The Java JIT compiler has a big impact on whether startup or runtime performance is optimized. The initial optimization level that the compiler uses influences the length of time it takes to compile a class method, and the length of time it takes to start the server. For faster startups, you should reduce the initial optimization level that the compiler uses. However if you reduce the initial optimization level, the runtime performance of your applications might be degraded because the class methods are now compiled at a lower optimization level.
- -Xquickstart
This setting influences how the IBM JVM uses a lower optimization level for class method compiles. A lower optimization level provides for faster server startups, but lowers runtime performance. If this parameter is not specified, the IBM JVM defaults to starting with a high initial optimization level for compiles, which results in faster runtime performance, but slower server starts.
Default: High initial compiler optimization level Recommended: High initial compiler optimization level Usage: -Xquickstart provides faster server startup.
The heap size settings control garbage collection in the i5/OS JVM. The initial heap size is a threshold that triggers new garbage collection cycles. For example, if the initial heap size is 10 MB, a new collection cycle is triggered as soon as the JVM detects that since the last collection cycle, 10 MB are allocated.
Smaller heap sizes result in more frequent garbage collections than larger heap sizes. If the maximum heap size is reached, the garbage collector stops operating asynchronously, and user threads are forced to wait for collection cycles to complete. This situation has a significant negative impact on performance. A maximum heap size of 0 (*NOMAX) assures that garbage collection always operates asynchonously.
The maximum heap size can affect application performance. The maximum heap size specifies the maximum amount of object space the garbage collected heap can consume. If the maximum heap size is too small, performance might degrade significantly, or the application might receive out of memory errors when the maximum heap size is reached.
Because of the complexity of determining a correct value for the maximum heap size, a value of 0 (meaning there is no size limit) is recommended unless an absolute limit on the object space for the garbage collected heap size is required.
If you want to determine the proper value for the maximum heap size, run multiple tests, because the appropriate value is different for each configuration or workload combination. If you want to prevent a run-away JVM, set the maximum heap size larger than you expect the heap to grow, but not so large that it affects the performance of the rest of the machine. For one of the tests you should:
- Run your application server under a heavy load with a maximum heap value of 0
- Use the DMPJVM command or iDoctor to determine the maximum size of the garbage collected heap for the JVM.
- Multiply the size of the garbage collection heap by 1.25. The result is a reasonable estimate for maximum heap size becasue the smallest acceptable value for the maximum heap size is 125 percent of the garbage collected heap size.
Because you can specify a larger value for the maximum heap size without affecting performance, IBM recommends that you set the largest possible value based on the resource restrictions of the JVM or the limitations of your system configuration.
After you determine an appropriate value for the maximum heap size, you might need to set up or adjust the pool in which the JVM runs. By default, WebSphere Application Server jobs run in the base system pool (storage pool 2 as shown by WRKSYSSTS), but you can specify a different pool. The maximum heap size should not be set larger than 125 percent of the size of the pool in which the JVM is running. It is recommended that you run the JVM in its own memory pool with the memory permanently assigned to that pool, if possible.
If the performance adjuster is set to adjust the memory pools (that is, the system value QPFRADJ is set to a value other than 0), IBM recommends that you specify a minimum size for the pool using WRKSHRPOOL. The minimum size should be approximately equal to your garbage collected heap working set size. Setting a correct maximum heap size and properly configuring the memory pool can prevent a JVM with a memory leak from consuming system resources, but still offers excellent performance.
When a JVM must run in a shared pool, it is more difficult to determine an appropriate value for the maximum heap size. Other jobs running in the pool can cause the garbage collected heap pages to be aged out of the pool. If the garbage collected heap pages are aged out of the pool, the garbage collector must fault the pages back into the pool on the next garbage collection cycle because it needs to access all of the pages in the garbage collected heap. Because the i5/OS JVM does not stop all of the JVM threads to clean the heap, excessive page faulting causes the garbage collector to slow down and the garbage collected heap to grow. Instead the size of the heap is increased, and threads continue to run. This heap growth is an artificial inflation of the garbage collected heap working set size, and must be considered if you want to specify a maximum heap value. When a small amount of artificial inflation occurs, the garbage collector reduces the size of the heap over time if the space remains unused and the activity in the pool returns to a steady state. However, in a shared pool, you might experience problems if the maximum heap size is not set correctly:
- If the maximum heap size is too small, artificial inflation can result in severe performance degradation or system failure if the JVM throws an out-of-memory error.
- If the maximum heap size is set too large, the garbage collector might reach a point where it is unable to recover the artificial inflation of the garbage collected heap. In this case, performance is also negatively affected. A value that is too large might also keep the garbage collector from preventing a JVM failure. However, the garbage collector is still able to prevent a run-away JVM from consuming excessive amounts of system resources.
If set the maximum heap size to guarantee that the heap size does not exceed a given level, specify an initial heap size that is 80-90% smaller than the maximum heap size. However, the value specified should be large enough to not negatively affect performance. The JVM has thresholds it uses to manage the JVM's storage. When the thresholds are reached, the garbage collector gets invoked to free up unused storage. Therefore, garbage collection can cause significant degradation of Java performance. Before changing the initial and maximum heap sizes, you should consider the following information:
- In the majority of cases you should set the maximum JVM heap size to value higher than the initial JVM heap size. This allows for the JVM to operate efficiently during normal, steady state periods within the confines of the initial heap but also to operate effectively during periods of high transaction volume by expanding the heap up to the maximum JVM heap size. In some rare cases where absolute optimal performance is required you might want to specify the same value for both the initial and maximum heap size. This will eliminate some overhead that occurs when the JVM needs to expand or contract the size of the JVM heap. Make sure the region is large enough to hold the specified JVM heap.
- Beware of making the Initial Heap Size too large. While it initially improves performance by delaying garbage collection, it ultimately affects response time when garbage collection eventually kicks in (because it runs for a longer time).
The IBM Developer Kit and Runtime Environment, Java2 Technology Edition, V5.0 Diagnostics Guide, that is available on the developerWorks Web site, provides additional information on tuning the heap size.
To use the administrative console to configure the heap size:
- In the administrative console, click Servers > Application Servers > server.
- Under Server Infrastructure, click Java and Process Management > Process Definition > Java Virtual Machine.
- Specify a new value in either the Initial heap size or the Maximum
heap size field.
You can also specify values for both fields if you need to adjust both settings.
For performance analysis, the initial and maximum heap sizes should be equal.
The Initial heap size setting specifies, in megabytes, how often garbage collection runs. The Maximum heap size setting specifies how often garbage collection runs. Both of these settings have a significant effect on performance.
When tuning a production system where the working set size of the Java application is not understood, IBM recommends that you set the initial heap size to 96MB per processor. The total heap size in an i5/OS JVM can be approximated as the sum of the amount of live (in use) heap space at the end of the last garbage collection plus the initial heap size.
The illustration represents three CPU profiles, each running a fixed workload with varying Java heap settings. In the middle profile, the initial and maximum heap sizes are set to 128MB. Four garbage collections occur. The total time in garbage collection is about 15% of the total run. When the heap parameters are doubled to 256MB, as in the top profile, the length of the work time increases between garbage collections. Only three garbage collections occur, but the length of each garbage collection is also increased. In the third profile, the heap size is reduced to 64MB and exhibits the opposite effect. With a smaller heap size, both the time between garbage collections and the time for each garbage collection are shorter. For all three configurations, the total time in garbage collection is approximately 15%. This example illustrates an important concept about the Java heap and its relationship to object utilization. There is always a cost for garbage collection in Java applications. Run a series of test experiments that vary the Java heap settings. For example, run experiments with 128MB, 192MB, 256MB, and 320MB. During each experiment, monitor the total memory usage. If you expand the heap too aggressively, paging can occur. If paging occurs, reduce the size of the heap or add more memory to the system. When all the runs are finished, compare the following statistics:
- Number of garbage collection calls
- Average duration of a single garbage collection call
- Ratio between the length of a single garbage collection call and the average time between calls
If the application is not over utilizing objects and has no memory leaks, the state of steady memory utilization is reached. Garbage collection also occurs less frequently and for short duration.
Unlike other JVM implementations, a large amount of heap free space is not generally a concern for the i5/OS JVM.
The default maximum heap size is 0, which indicates that there is no maximum value. It is recommended that you do not change the maximum heap size. When the maximum heap size triggers a garbage collection cycle, the i5/OS JVM's garbage collection stops operating asynchronously. When this happens, the application server cannot process user threads until the garbage collection cycle ends, which significantly lowers performance. See the topic Tuning Garbage Collection for Java and WebSphere on iSeries in the i5/OS Information Center for more information on initial and maximum heap sizes.
- Click Apply or OK.
- Save your changes to the master configuration.
- Stop and restart the application server.
You can also use the following command line parameters to adjust these settings. These parameters apply to all supported JVMs and are used to adjust the minimum and maximum heap size for each application server or application server instance.
- -Xms
This setting controls the initial size of the Java heap. Properly tuning this parameter reduces the overhead of garbage collection, which improves server response time and throughput. For some applications, the default setting for this option might be too low, which causes a high number of minor garbage collections.
Default: 50MB. This default value applies for both 31-bit and 64-bit configurations. Recommended: Workload specific, but higher than the default. Usage: -Xms256m sets the initial heap size to 256 megabytes. - -Xmx
This setting controls the maximum size of the Java heap. Increasing this parameter increases the memory available to the application server, and reduces the frequency of garbage collection. Increasing this setting can improve server response time and throughput. However, increasing this setting also increases the duration of a garbage collection when it does occur. This setting should never be increased above the system memory available for the application server instance. Increasing the setting above the available system memory can cause system paging and a significant decrease in performance.
Default: 256MB. This default value applies for both 31-bit and 64-bit configurations. Recommended: Workload specific, but higher than the default, depending on the amount of available physical memory. Usage: -Xmx512m sets the maximum heap size to 512 megabytes.
- The application is not over utilizing objects
- The application is not leaking objects
- The Java heap parameters are set properly to handle a given object usage pattern
- Check for over-utilization of objects.
You can use the Tivoli Performance Viewer to check if the application is overusing objects, by observing the counters for the JVM runtime. You have to set the -XrunpmiJvmpiProfiler command line option, as well as the JVM module maximum level in order to enable the Java virtual machine profiler interface (JVMPI) counters. See Enabling the Java virtual machine profiler data for more information about JVMPI counters. You can also use the following tools to monitor JVM object creation:
- The DMPJVM command. The DMPJVM (Dump Java Virtual Machine) command dumps JVM information for a specific job.
- The ANZJVM command. The ANZJVM (Analyze Java Virtual Machine) command collects information about the Java Virtual Machine (JVM) for a specified job. This command is available in i5/OS V5R2 and higher.
- The Performance Trace Data Visualizer (PTDV)
The best result for the average time between garbage collections is at least 5-6 times the average duration of a single garbage collection. If you do not achieve this number, the application is spending more than 15% of its time in garbage collection.
If the information indicates a garbage collection bottleneck, there are two ways to clear the bottleneck. The most cost-effective way to optimize the application is to implement object caches and pools. Use a Java profiler to determine which objects to target. If you can not optimize the application, adding memory, processors and clones might help. Additional memory allows each clone to maintain a reasonable heap size. Additional processors allow the clones to run in parallel.
- Test for memory leaks
Memory leaks in the Java language are a dangerous contributor to garbage collection bottlenecks. Memory leaks are more damaging than memory overuse, because a memory leak ultimately leads to system instability. Over time, garbage collection occurs more frequently until the heap is exhausted and the Java code fails with a fatal out-of-memory exception. Memory leaks occur when an unused object has references that are never freed. Memory leaks most commonly occur in collection classes, such as Hashtable because the table always has a reference to the object, even after real references are deleted.
High workload often causes applications to crash immediately after deployment in the production environment. This is especially true for leaking applications where the high workload accelerates the magnification of the leakage and a memory allocation failure occurs. The goal of memory leak testing is to magnify numbers. Memory leaks are measured in terms of the amount of bytes or kilobytes that cannot be garbage collected. The delicate task is to differentiate these amounts between expected sizes of useful and unusable memory. This task is achieved more easily if the numbers are magnified, resulting in larger gaps and easier identification of inconsistencies. The following list contains important conclusions about memory leaks:
- Long-running test
Memory leak problems can manifest only after a period of time, therefore, memory leaks are found easily during long-running tests. Short running tests can lead to false alarms. It is sometimes difficult to know when a memory leak is occurring in the Java language, especially when memory usage has seemingly increased either abruptly or monotonically in a given period of time. The reason it is hard to detect a memory leak is that these kinds of increases can be valid or might be the intention of the developer. You can learn how to differentiate the delayed use of objects from completely unused objects by running applications for a longer period of time. Long-running application testing gives you higher confidence for whether the delayed use of objects is actually occurring.
- Repetitive test
In many cases, memory leak problems occur by successive repetitions of the same test case. The goal of memory leak testing is to establish a big gap between unusable memory and used memory in terms of their relative sizes. By repeating the same scenario over and over again, the gap is multiplied in a very progressive way. This testing helps if the number of leaks caused by the execution of a test case is so minimal that it is hardly noticeable in one run.
You can use repetitive tests at the system level or module level. The advantage with modular testing is better control. When a module is designed to keep the private module without creating external side effects such as memory usage, testing for memory leaks is easier. First, the memory usage before running the module is recorded. Then, a fixed set of test cases are run repeatedly. At the end of the test run, the current memory usage is recorded and checked for significant changes. Remember, garbage collection must be suggested when recording the actual memory usage by inserting System.gc() in the module where you want garbage collection to occur, or using a profiling tool, to force the event to occur.
- Concurrency test
Some memory leak problems can occur only when there are several threads running in the application. Unfortunately, synchronization points are very susceptible to memory leaks because of the added complication in the program logic. Careless programming can lead to kept or unreleased references. The incident of memory leaks is often facilitated or accelerated by increased concurrency in the system. The most common way to increase concurrency is to increase the number of clients in the test driver. Consider the following points when choosing which test cases to use for memory leak testing:
- A good test case exercises areas of the application where objects are created. Most of the time, knowledge of the application is required. A description of the scenario can suggest creation of data spaces, such as adding a new record, creating an HTTP session, performing a transaction and searching a record.
- Look at areas where collections of objects are used. Typically, memory leaks are composed of objects within the same class. Also, collection classes such as Vector and Hashtable are common places where references to objects are implicitly stored by calling corresponding insertion methods. For example, the get method of a Hashtable object does not remove its reference to the retrieved object.
You can use these tools to detect memory leaks:
- Tivoli Performance Viewer. For more information and examples of using the Tivoli Performance Viewer to detect memory leaks, see the topic "Tuning Garbage Collection for Java(TM) and WebSphere on iSeries." in the i5/OS Information Center.
- The DMPJVM command. The DMPJVM (Dump Java Virtual Machine) command dumps JVM information for a specific job.
- The ANZJVM command. The ANZJVM (Analyze Java Virtual Machine) command collects information about the Java Virtual Machine (JVM) for a specified job. This command is available in i5/OS V5R2 and higher.
- The Heap Analysis Tools for Java. This tool is a component of the iDoctor for iSeries suite of performance monitoring tools. The Heap Analysis Tools component performs Java application heap analysis and object create profiling (size and identification) over time. This tool is sometimes called Java Watcher or Heap Analyzer.
For the best results, repeat experiments with increasing duration, like 1000, 2000, and 4000 page requests. The Tivoli Performance Viewer graph of used memory should have a sawtooth shape. Each drop on the graph corresponds to a garbage collection. There is a memory leak if one of the following occurs:
- The amount of memory used immediately after each garbage collection increases significantly. The sawtooth pattern looks more like a staircase.
- The sawtooth pattern has an irregular shape.
Also, look at the difference between the number of objects allocated and the number of objects freed. If the gap between the two increases over time, there is a memory leak.
Heap consumption indicating a possible leak during a heavy workload (the application server is consistently near 100% CPU utilization), yet appearing to recover during a subsequent lighter or near-idle workload, is an indication of heap fragmentation. Heap fragmentation can occur when the JVM can free sufficient objects to satisfy memory allocation requests during garbage collection cycles, but the JVM does not have the time to compact small free memory areas in the heap to larger contiguous spaces.
Another form of heap fragmentation occurs when small objects (less than 512 bytes) are freed. The objects are freed, but the storage is not recovered, resulting in memory fragmentation until a heap compaction has been run.
- Long-running test
The i5/OS JVM uses concurrent (asynchronous) garbage collection. This type of garbage collection results in shorter pause times and allows application threads to continue processing requests during the garbage collection cycle.
Examining Java garbage collection gives insight to how the application is utilizing memory. Garbage collection is a Java strength. By taking the burden of memory management away from the application writer, Java applications are more robust than applications written in languages that do not provide garbage collection. This robustness applies as long as the application is not abusing objects. Garbage collection normally consumes from 5% to 20% of total execution time of a properly functioning application. If not managed, garbage collection is one of the biggest bottlenecks for an application.
Monitoring garbage collection during the execution of a fixed workload, enables you to gain insight as to whether the application is over-utilizing objects. Garbage collection can even detect the presence of memory leaks.
You can use JVM settings to configure the type and behavior of garbage collection. When the JVM cannot allocate an object from the current heap because of lack of contiguous space, the garbage collector is invoked to reclaim memory from Java objects that are no longer being used. Each JVM vendor provides unique garbage collector policies and tuning parameters.
You can use the Verbose garbage collection setting in the administrative console to enable garbage collection monitoring. The output from this setting includes class garbage collection statistics. The format of the generated report is not standardized between different JVMs or release levels.
To ensure meaningful statistics, run a fixed workload until the application state is steady. It usually takes several minutes to reach a steady state.
You can also use object statistics in the Tivoli Performance Viewer to monitor garbage collection statistics. For more information about monitoring garbage collection, see:
- Performance: Resources for learning for a description of the IBM verbose:gc output
- The description of the DMPJVM command in the i5/OS Information Center. This command dumps JVM information for a specific job.
- The iDoctor for iSeries for a description of the Heap Analysis Tools for Java. This tool is a component of the iDoctor for iSeries suite of performance monitoring tools. The Heap Analysis Tools component performs Java application heap analysis and object create profiling (size and identification) over time. This tool is sometimes called Java Watcher or Heap Analyzer.
- The description of Performance Explorer (PEX) contained in the i5/OS Information Center topic "Tuning Garbage Collection for Java(TM) and WebSphere on iSeries." You can use a Performance Explorer (PEX) trace to determine how much CPU is being used by the garbage collector.
To adjust your JVM garbage collection settings:
- In the administrative console, click Servers > Application Servers > server.
- Under Server Infrastructure, click Java and Process Management > Process Definition > Java Virtual Machine.
- Enter the –X option you want to change in the Generic JVM arguments field.
- Click OK.
- Save your changes to the master configuration.
- Stop and restart the application server.
The following list describes the –X options for the different JVM garbage collectors.
- The IBM JVM garbage collector.
-
A complete guide to the IBM Java garbage collector is provided in the IBM
Developer Kit and Runtime Environment, Java2 Technology Edition, V5.0
Diagnostics Guide. This document is available on the developerWorks Web site.
- -Xnoclassgc
By default, the JVM unloads a class from memory whenever there are no live instances of that class left. Therefore, class unloading can degrade performance.
You can use the -Xnoclassgc argument to disable class garbage collection so that your applications can reuse classes more easily. Turning off class garbage collection eliminates the overhead of loading and unloading the same class multiple times.
Default: Class garbage collection is enabled. Recommended: Do not disable class garbage collection.
Usage: Xnoclassgc disables class garbage collection.
- -Xnoclassgc
- The more application servers or clusters there are defined in cell, the longer it takes to save a configuration change.
- The more applications there are deployed in a cell, the longer it takes to save a configuration change.
If the amount of time required to change a configuration change is unsatisfactory, you can add the config_consistency_check custom property to your JVM settings and set the value of this property to false.
- In the administrative console, click System administration > Deployment manager.
- Under Server Infrastructure, select Java and Process Management, and then click Process Definition.
- Under Additional Properties, click Java Virtual Machine > Custom Properties > New.
- Enter config_consistency_check in the Name field and false in the Value field.
- Click OK and then Save to apply these changes to the master configuration.
- Restart the server.