Identify Java code consuming high CPU in Linux (linking JVM thread and Linux PID)

Applications Manager | February 9, 2011 | 4 min read

You can easily identify problematic Java code that is leading to high CPU utilization in Linux. Let me explain the steps with the following example:

  1.  package test;
  3.  public class ThreadDumpTest {
  4.          public void test(){
  5.                  for (int i = 0; i < 10 ; i++) {
  6.                          Thread th=new Thread(new TR(i));
  7.                          th.setName(“MyThread-“+(1000+i));
  8.                          th.start();
  9.                  }
  10.          }
  11.          public static void main(String[] args) {
  12.                  ThreadDumpTest t=new ThreadDumpTest();
  13.                  t.test();
  14.          }
  15.          private class TR implements Runnable{
  16.                  int ins=0;
  17.                  TR(int i){
  18.                          ins=i;
  19.                  }
  20.                  public void run(){
  21.                          while (true) {
  22.                                  if(ins!=5) {
  23.                                          try {
  24.                                                  Thread.sleep(10000);
  25.                                          } catch (Exception e) {
  26.                                                  e.printStackTrace();
  27.                                          }
  28.                                  }
  29.                          }
  30.                  }
  31.          }
  32.  }

In the above example, all the threads are in the while loop. Except for the MyThread-1005 thread, all other threads will sleep 10 seconds inside the loop. The MyThread-1005 thread will not enter the sleep part, so it will run in the while loop without sleep. Due to the while loop, the MyThread-1005 thread will lead to high CPU utilization.

How do you identify high CPU utilization?

Step 1:

Execute the top command on the console. You can see the java command with PID 7074 is utilizing 97% of the CPU.

Step 2:

The top command displays the process list like the above image. Press Shift + h and wait a few seconds. You can see the Show threads on message in the top console. Now, you can see thread-level details like CPU and memory utilization. You’ll notice that a java command thread with PID 7087 is utilizing 94% of the CPU.

Step 3:

The identified problematic thread, PID 7087, is in decimal format. Convert it into hexadecimal format. The respective hexadecimal for “7087” is “1BAF”. Convert it into lowercase: 1baf.

Step 4:

Take a thread dump and search for the converted hexadecimal PID (1baf) in the thread dump. You can find the hex PID as nid (with ‘0x’) of the thread trace. From the trace, you can find that MyThread-1005 is the problematic thread in the test program. And using the thread trace, you can identify the problematic code and fix the problem.
  1. [rameshj@rameshj thread]$ java test/ThreadDumpTest
  2. ^\Full thread dump Java HotSpot(TM) Client VM (1.5.0_06-b05 mixed mode):
  3. “DestroyJavaVM” prio=1 tid=0x083d2d78 nid=0x1ba2 waiting on condition [0x00000000..0xbff4bf60]
  4. “MyThread-1009” prio=1 tid=0x08510a88 nid=0x1bb3 waiting on condition [0xa9cca000..0xa9ccb020]
  5.         at java.lang.Thread.sleep(Native Method)
  6.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)          &nbs
    p;
  7.         at java.lang.Thread.run(Thread.java:595)
  8. “MyThread-1008” prio=1 tid=0x08510318 nid=0x1bb2 waiting on condition [0xa9d4b000..0xa9d4bea0]
  9.         at java.lang.Thread.sleep(Native Method)
  10.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  11.         at java.lang.Thread.run(Thread.java:595)
  12. “MyThread-1007” prio=1 tid=0x0850f298 nid=0x1bb1 waiting on condition [0xa9dcc000..0xa9dccf20]
  13.         at java.lang.Thread.sleep(Native Method)
  14.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  15.         at java.lang.Thread.run(Thread.java:595)
  16. “MyThread-1006” prio=1 tid=0x0850d818 nid=0x1bb0 waiting on condition [0xa9e4d000..0xa9e4dda0]
  17.         at java.lang.Thread.sleep(Native Method)
  18.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  19.         at java.lang.Thread.run(Thread.java:595)
  20. “MyThread-1005” prio=1 tid=0x0850c9f8 nid=0x1baf runnable [0xa9ece000..0xa9ecee20]
  21.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:43)
  22.         at java.lang.Thread.run(Thread.java:595)
  23. “MyThread-1004” prio=1 tid=0x0850b8e8 nid=0x1bae waiting on condition [0xa9f4f000..0xa9f500a0]
  24.         at java.lang.Thread.sleep(Native Method)
  25.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  26.         at java.lang.Thread.run(Thread.java:595)
  27. “MyThread-1003” prio=1 tid=0x0850aac8 nid=0x1bad waiting on condition [0xa9fd0000..0xa9fd1120]
  28.         at java.lang.Thread.sleep(Native Method)
  29.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  30.         at java.lang.Thread.run(Thread.java:595)
  31. “MyThread-1002” prio=1 tid=0x085099b8 nid=0x1bac waiting on condition [0xaa051000..0xaa051fa0]
  32.         at java.lang.Thread.sleep(Native Method)
  33.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  34.         at java.lang.Thread.run(Thread.java:595)
  35. “MyThread-1001” prio=1 tid=0x08508a28 nid=0x1bab waiting
    on condition [0xaa0d2000..0xaa0d3020]
  36.         at java.lang.Thread.sleep(Native Method)
  37.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  38.         at java.lang.Thread.run(Thread.java:595)
  39. “MyThread-1000” prio=1 tid=0x08508520 nid=0x1baa waiting on condition [0xaa153000..0xaa153ea0]
  40.         at java.lang.Thread.sleep(Native Method)
  41.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  42.         at java.lang.Thread.run(Thread.java:595)
  43. “Low Memory Detector” daemon prio=1 tid=0x0843cf58 nid=0x1ba8 runnable [0x00000000..0x00000000]
  44. “CompilerThread0” daemon prio=1 tid=0x0843b930 nid=0x1ba7 waiting on condition [0x00000000..0xaa3eb808]
  45. “Signal Dispatcher” daemon prio=1 tid=0x0843a9d0 nid=0x1ba6 waiting on condition [0x00000000..0x00000000]
  46. “Finalizer” daemon prio=1 tid=0x08430740 nid=0x1ba5 in Object.wait() [0xaa52c000..0xaa52d120]
  47.         at java.lang.Object.wait(Native Method)
  48.         – waiting on <0xaa6b0848> (a java.lang.ref.ReferenceQueue$Lock)
  49.         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
  50.         – locked <0xaa6b0848> (a java.lang.ref.ReferenceQueue$Lock)
  51.         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
  52.         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
  53. “Reference Handler” daemon prio=1 tid=0x0842e898 nid=0x1ba4 in Object.wait() [0xaa5ad000..0xaa5adfa0]
  54.         at java.lang.Object.wait(Native Method)
  55.         – waiting on <0xaa6b0758> (a java.lang.ref.Reference$Lock)
  56.         at java.lang.Object.wait(Object.java:474)
  57.         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
  58.         – locked <0xaa6b0758> (a java.lang.ref.Reference$Lock)
  59. “VM Thread” prio=1 tid=0x0842bb68 nid=0x1ba3 runnable
  60. “VM Periodic Task Thread” prio=1 tid=0x0843e538 nid=0x1ba9 waiting on condition

Update: You can automatically take a thread dump of a Java application when a high CPU threshold is violated by using Applications Manager. Learn more, watch a related video, or download Applications Manager now.

rameshj@zohocorp.com

  1. Reply
    jeck

    Easily understand code….well done
    Excellent!
    Thank you.

  2. Reply
    Rajesh

    Do you know any windows equivalent command to see the native light weight process ids (native thread id’s)

  3. Reply
    Rajesh

    Do you know any windows equivalent commands for observing the native light weight processes (thread native ids)

  4. Reply
    Steephen

    top -Hp will list the threads running under process with PID

  6. Reply
    toddkaufmann

    Why not just use the Java VisualVM ?
    It has come with Sun (Oracle) JDK since at least 1.5.. find it in your bin directory where java is as jvisualvm (Linux & windows), /usr/share/java/Tools/Java VisualVM.app on OSX.

    You can also connect to remote processes on other machines, see which classes have the most instances, etc.

    more info at: http://visualvm.java.net/

  7. Reply
    Ron

    for step 3 you can run: echo “obase=16;” | bc | tr [:upper:] [:lower:]

  8. Reply
    Java Coder

    Excellent Bro !

    Many Many thx for this tip

  9. Reply
    Ketan

    Thanks Ramesh. This was helpful. We had a similar problem in our production envionment where GC was consuming high CPU. Was able to identify the rogue GC thread which was consuming high CPU.

  10. Reply
    Aaron

    while (true) is a busy-wait and will run a single-threaded process to max CPU in any language, on any platform.

  11. Reply

    No. If you press ‘Shift + h’, it will show thread level usage until you press ‘Shift + h’ again.

  12. Reply
    non

    Press ‘Shift + h’ and wait few secs. You can see ‘Show threads on’ message in the top console.
    It only shows thread about 1 sec, then back to process, too short to find the java threads, how to make top console keep the threads view?

  13. Reply
    Ramesh

    Just press “Ctrl+|” on the java process running console. If the java process running in the background, execute “kill -3 [PID]” command in any console. You can find the “PID” using “ps -aef | grep java | grep -v grep” command. The second column of the output is the ‘PID” of the java process.

  14. Reply
    Arnold

    In Step 4, you said “Take thread dump”. How do I do that?