Application server dies or hangs

 

+

Search Tips   |   Advanced Search

 

If an appserver dies or freezes...

1. Isolate the problem by installing Web modules on different servers, if possible.

2. Use the performance viewer to determine which resources have reached their maximum capacity, such as Java heap memory (indicating a possible memory leak) and database connections.

   If a particular resource appears to have reached its maximum capacity, review the application code for a possible cause...

   • If database connections are used and never freed, ensure that application code performs a close() on any opened Connection object within a finally{} block.

   • If there is a steady increase in servlet engine threads in use, review application synchronized code blocks for possible deadlock conditions.

   • If there is a steady increase in a JVM heap size, review application code for memory leak opportunities.

3. As an alternative to using the performance viewer to detect memory leak problems, enable verbose garbage collection (verbose:gc) in the Java Virtual Machine.

   Servers | Application Servers | server | Process Definition | Java Virtual Machine

   ...and then cycle the server.

   Browse the log files, looking for statements beginning with "allocation failure", which indicates that a need for memory allocation has triggered a garbage collection. Allocation failures themselves are normal and not necessarily indicative of a problem.

   If there is a steady increase in the total amount of free and used memory the JVM allocates for itself, or if the JVM becomes unable to allocate as much memory as it needs (indicated by the bytes needed statement), there might be a memory leak.

4. If either the performance viewer or verbose garbage collection output indicates that the appserver is running out of memory there might be a memory leak in your application code. To pinpoint the cause of a memory leak, enable the RunHProf function in...

   Servers | Application Servers | server_name | Process Definition | Java Virtual Machine

   Set the HProf Arguments field to a value similar to...

   depth=20,file=heapdmp.txt

   ...This value shows exception stacks to a maximum of 20 levels, and saves the heapdump output to...

   $WAS_HOME/bin/heapdmp.txt file.

   Save the settings and cycle the appserver.

   Re-enact the scenario or access the resource that causes the hang or crash, if possible. Stop the appserver. If this is not possible, wait until the hang or crash happens again and stop the application server.

   Examine heapdmp.txt and search for the string, "SITES BEGIN". This finds the location of a list of Java objects in memory, which shows the amount of memory allocated to the objects.

   The list of Java objects occurs each time there was a memory allocation in the JVM. There is a record of what type of object the memory instantiated and an identifier of a trace stack, listed elsewhere in the dump, that shows the Java method that made the allocation.

   The list of Java object is in descending order by number of bytes allocated. Depending on the nature of the leak, the problem class should show up near the top of the list, but this is not always the case. Look throughout the list for large amounts of memory or frequent instances of the same class being instantiated. In the latter case, use the ID in the trace stack column to identify allocations occurring repeatedly in the same class and method.

5. The default maximum heap size of the appserver needs to be increased.

6. If an appserver spontaneously dies, look for a Java thread dump file. The JVM creates the file in the product directory structure, with a name like javacore[number].txt.

   To force an application to create a thread dump (or javacore):

   • Using the wsadmin command prompt, get a handle to the problem application server:

     wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=server,*]

   • Generate the thread dump:

     wsadmin>$AdminControl invoke $jvm dumpThreads

   • Look for an output file in the installation root directory with a name like javacore.date.time.id.txt.

   If the JVM creates the thread dump as it closes (the thread dump is not manually forced), there might be "error" or "exception information" strings at the beginning of the file. These strings indicate the thread that caused the appserver to die.

   The thread dump contains a snapshot of each thread in the process, starting in the section labeled "Full thread dump."

   • Look for threads with a description that contains "state:R". Such threads are active and running when the dump is forced, or the process exited.

   • Look for multiple threads in the same Java application code source location. Multiple threads from the same location might indicate a deadlock condition (multiple threads waiting on a monitor) or an infinite loop, and help identify the application code with the problem.

If these steps do not fix your problem, search to see if the problem is known and documented, using the methods identified in the available online support (hints and tips, technotes, and fixes) topic. If you find that your problem is not known, contact IBM support to report it.

For current information available from IBM Support on known problems and their resolution, see IBM Support page.

IBM Support has documents that can save you time gathering information needed to resolve this problem. Before opening a PMR, see IBM Support page.

 

See Also

Troubleshooting by component: What is not working?

 

---