In this blog post, I am going to explain How to Find which Java Thread Consumes high CPU..
Frequently we are facing a
strange issue with one of our JBOSS server in Production. The monitors that we
configured on these always showed degraded performance. We started to analyze
the performance of the server by using the basic top command which gave us the
highest CPU and Memory. High CPU may cause System Crash or Server Crash or Slow
Responsiveness of Server.
High CPU utilization is a very
very common thing for an Application Server. It is very important for an Administrator
to find out what is causing the High CPU.
Fallow the below steps to find out The High CPU consumed thread.
1. We
use top to find out the process id (PID) of the java process consuming most of
the CPU.
top -n1 | grep -m1 java | perl -pe
's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' '
|
2. Next step is we started to find out the Thread
ID that is causing the HIGH CPU usage in this process which gave us this.
printf "%x" $(top -n1 -H | grep -m1
java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' ')
|
3. So
we have a process ID of our busy JVM and Linux thread ID (most likely from that
process) consuming our CPU. By using jstack we can be obtained Thread dump. Theses dumps/
Thread stack traces re very useful for find out root causes of issues.
$JAVA_HOME/bin/jstack <PID> | grep -A500 <ThreadID> | grep
-m1 "^$" -B 500
|
The above command simply dumps
the JVM stack trace of the given PID and filters out (using grep) the thread
which has matching Thread ID.
Shell script :
I have written shell scripts as
below with all above commands. This
script make easy to find out High CPU consumed JAVA PID & Thread ID and simply
dumps the JVM stack trace.
#!/bin/bash
JAVA_HOME=/usr/java/jdk1.8.0_121/bin
PID=$(top -n1 | grep -m1 java | perl -pe
's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' ')
ThreadID=$(printf "%x" $(top -n1 -H |
grep -m1 java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' '))
EAP=`ps -ef | grep java |grep -v grep | awk
'{print $2 "," $9}' | grep $PID | awk -F, '{ print $2}' | sed -e
's/^-D//' | tr -d '[]' | sed -e 's/Server://'`
if [ -z
"$PID" ] || [ -z "$NID" ] || [ -z "$EAP" ];
then
echo ""
echo
"No High CPU consumption Threads at this moment"
exit
1
else
echo
""
echo
-e
"****************************************************************"
echo
" High CPU consumed PID :-
" $PID
echo
" High CPU consumed EAP Name :-
" $EAP
echo
" High CPU consumed Thread ID :- " $ThreadID
echo
-e
"****************************************************************"
echo
""
echo
" High CPU consumption Thread Stack Trace"
echo"########################################################################"
echo
""
$JAVA_HOME/bin/jstack $PID | grep -A500 $ThreadID | grep -m1
"^$" -B 500
echo
""
echo"########################################################################"
fi
|
Copy the
above Code and Save this with .sh extension and run it with sufficient permissions.
Let see how this script will work
[jboss@middleware ~]$ ./High_CPU_Consumption_Threads.sh
****************************************************************
High CPU consumed PID
:- 72520
High CPU consumed EAP Name
:- Server-One
High CPU consumed Thread ID :-
6d08
****************************************************************
High CPU consumption Thread Stack Trace
################################################################
"HttpManagementService-threads - 2288" #10447 prio=5
os_prio=0 tid=0x0000000000891000 nid=0x6d08 waiting on condition
[0x00007f80ba35a000]
java.lang.Thread.State:
TIMED_WAITING (parking)
at
sun.misc.Unsafe.park(Native Method)
- parking to wait
for <0x00000000e0461778> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
java.lang.Thread.run(Thread.java:745)
at
org.jboss.threads.JBossThread.run(JBossThread.java:122)
################################################################
|
Running the script multiple times (or with watch, see below) will
capture Busy thread in different places..
Hope this helps
No comments:
Post a Comment