Fraska Portal

Exploring the WebSphere Commerce world

How to start a troubleshooting investigation for Deadlocks and Hungs threads on WebSphere Application Server

Posted by on in Performance
  • Font size: Larger Smaller
  • Hits: 6222
  • 23 Comments
  • Subscribe to this entry
  • Print

Suggested reading: Definitions of Thread, Pool, DeadLock and Hung in WebSphere Application Server 

How to investigate deadlock issues

  1. Collect javacores;
  2. Look for deadlock messages
  3. Look for threads in waiting status
  4. Look for threads owns monitor lock
  5. Compare threads and compare monitors

 

Step 1: Collect javacores 

For a typical hung- possible caused by a deadlock- the collection should take at least 3 dumps (javacore) with 5 minutes interval between one and the next one. The best way to get mentioned data is follow the instructions of WAS MustGather tech document. For example, WAS hung issue on AIX:
http://www-01.ibm.com/support/docview.wss?uid=swg21052641

 

Step 2: Look for deadlock messages

Open the javacores with any text file word processor. Look for the string "Deadlock detected". 

  1. If the warning message appears you should review the code reported by the Thread's stack trace
  2. If not, it means the built-in hung thread detection monitor did not find any deadlock; in other words it does not mean a deadlock is not happening but it means it has not been automatically detected. So, the investigation should go on with the step 3.

 

Step 3: Look for threads in waiting status

Open the javacores with the IBM tool "Thread and Monitor Dump Analyzer". Find the Thread Status Analysis in the report generated for the 1st javacore:


keep particular attention on the number of "Waiting on condition" and "Blocked" threads. At this point it will be useful review the details of the waiting/blocked threads looking their stack trace in order to understand what and why their are waiting. Point of investigation are:

  • custom code involved;
  • external resources threads could wait on;
  • threads status during different javacores;
  • lock data.

The lock information is the key in the quest of finding a deadlock. The lock is a resource that can only be owned by one thread at a time. Other threads waiting for that lock are blocked until the thread that owns it releases it.

 

Step 4: Look for threads owns monitor lock

Thread Monitor and dump analyzer shows the threads are keeping the lock on resources. They are identified with special icons:


As you can see the icon is the Monitor image just beside the name of the Thread. As soon as you select the thread you can see what it's locking.


So, for example this Thread owns Monitor Lock on com/ibm/ws/util/BoundedBuffer$GetQueueLock@0x0...
The investigation should continue looking for threads are locking Monitor for long time and if there is a relevant number of threads waiting/or blocked by these threads. 

 

Step 5: compare Threads and Monitors

A possible further step in the investigation is compare the threads and the monitors during different javacores. Thread Monitor and dump analyzer could help to accomplish this task without any effort but just selecting the option "Analysis > Compare threads" and "Analysis > Compare monitors" (after the selection of the javacores collected).

 

Compare threads with Thread Monitor and dump analyzer

This feature offers a view of the threads status during the time reported by the javacore collected (that's why it's important collect more javacores!!!). The best way to describe this feature is show some example:

 

Above image shows 4 running threads and their status during 4 different javacores (the columns). The border indicate the status (green is Running) and the background color (in this case red) indicate a suspect thread. Different conclusions can be done looking- for example- above image: in fact, the WebContainer is blocked in the first two javacore but than just waiting on condition in the next two javacore for different tasks (you can understand the tasks were different looking at the stack trace of the thread reported by Thread Monitor and dump analyzer on the right side of thewindow); since the task requested by the thread is different (in different javacore) this thread (or better pool) should not be a suspected thread. So the analysis should be on looking for threads (in particular WebContainer) in wait status or blocked from the first javacore till the last one requesting the same operation.

Compare monitors with Thread Monitor and dump analyzer

At this point the investigation could continue or have confirmation on what we have found previously using the comparison monitors feature:

The above image shows as the tool detects a suspect Thread- DRSTHreadPool: DMN0- locks specific Monitor for the time of the last three javacores. In the Waiting Threads tab there is the list of threads in waiting for current thread to unblock the lock. So, this Thread could be a good candidate to investigate on looking at the stack trace and try to understand what it's doing and why it does not release the lock for the time of the last 3 javacores.

Summary

The detection of an hung or deadlock issue is unfortunately a task could requires time to investigate on. The most important things are:

  • detect the suspect threads (using the steps reported above);
  • investigate what those threads are doing (looking at the stack trace);
  • find the resources involved and understand why those threads are causing a possible issue.

 

Source:

Rate this blog entry:
0
Tagged in: Deadlock Javacore Troubleshooting WebSphere Application Server WebSphere Commerce

Comments

Leave your comment

Guest
Guest Friday, 17 August 2018

Most Popular Post

WebSphere Commerce, the SOLR extension index
Administration
Rate this blog entry:
4
43 comment(s) 14382 hits Read More
on 09 July 2014
WebSphere Commerce, Data Load and SOLR Delta Index
Data Load
Rate this blog entry:
0
116 comment(s) 13127 hits Read More
on 04 June 2013
WebSphere Commerce, FEP 7, SOLR index pre-process error
Administration
Rate this blog entry:
0
14 comment(s) 12120 hits Read More
on 02 May 2014

Tag Cloud

Validation Search Expression Processor Store Thank You Precision Marketing Landing page Inventory Subcore FEP7 Content Core Dynacache REST Data Model Congratulations Tables delta Admin UI Performances DataLoad WebSphere Commerce Developer Web Activity ATP Deadlock Delta ClassNotFoundException Search Navigation Flow Search Expression Provider Coupons JR47273 RAD Quality WAS Plugin non-ATP Management Center Data cache WebSphere Commerce Deploy European stores CommandLevelAuthorizationCacheCmdImpl WCUserDistributedMapCache Migration WebSphere Application Server Index Cores Marketing Troubleshooting Quality Assurance Team Toolkit SOLR Javacore Promotions preprocess InitializationServlet SoapUI

Categories

Latest Blogs

WebSphere Commerce, CommandLevelAuthorizationCache
Cache
Rate this blog entry:
0
Comments (24) Hits (1699)
Posted by Francesco Schettini on 08 December 2016
WebSphere Commerce v8, toolkit exception, ClassNotFound db2
Administration
Rate this blog entry:
1
Comments (26) Hits (3239)
Posted by Francesco Schettini on 18 April 2016
WebSphere Commerce, ATP migration
Store
Rate this blog entry:
0
Comments (27) Hits (4495)
Posted by Francesco Schettini on 28 March 2016
WebSphere Commerce, the curious life of a front-end catalog request
Core
Rate this blog entry:
5
Comments (25) Hits (11858)
Posted by Francesco Schettini on 08 October 2014
WebSphere Commerce, Performance analysis of few European stores
Performance
Rate this blog entry:
0
Comments (18) Hits (4004)
Posted by Francesco Schettini on 22 July 2014