Identify Java code consuming high CPU in Linux (linking JVM thread and Linux PID)

We can easily identify a problematic java code which leads to high CPU utilization in Linux. Let me explain the steps with the following example,

  1.  package test;
  2.  public class ThreadDumpTest {
  3.          public void test(){
  4.                  for (int i = 0; i < 10 ; i++) {
  5.                          Thread th=new Thread(new TR(i));
  6.                          th.setName(“MyThread-”+(1000+i));
  7.                          th.start();
  8.                  }
  9.          }
  10.          public static void main(String[] args) {
  11.                  ThreadDumpTest t=new ThreadDumpTest();
  12.                  t.test();
  13.          }
  14.          private class TR implements Runnable{
  15.                  int ins=0;
  16.                  TR(int i){
  17.                          ins=i;
  18.                  }
  19.                  public void run(){
  20.                          while (true) {
  21.                                  if(ins!=5) {
  22.                                          try {
  23.                                                  Thread.sleep(10000);
  24.                                          } catch (Exception e) {
  25.                                                  e.printStackTrace();
  26.                                          }
  27.                                  }
  28.                          }
  29.                  }
  30.          }
  31.  }

In the above example, all the threads are in while loop. Except ‘MyThread-1005‘ thread, all other threads will sleep 10 secs inside the loop. The ‘MyThread-1005‘ thread will not enter sleep part, so it will run in while loop without sleep. Due to while loop, the ‘MyThread-1005‘ thread will leads to high CPU utilization.

How to identify ?

Step 1 :

Execute ‘top‘ command on the console. You can see the ‘java’ command with PID 7074 utilized 97% of CPU.

Step 2 :

The top command displays the process list like the above image. Press ‘Shift + h‘ and wait few secs. You can see ‘Show threads on‘ message in the top console. Now, you can see thread level details like CPU/Memory utilization. You can see a ‘java’ command thread with PID 7087 utilized 94% of CPU.

Step 3:

The identified problematic thread PID ( 7087 ) is in decimal format. Convert it into hexadecimal format. The respective hexadecimal for 7087 is 1BAF. And convert it into lowercase (1baf ).

Step 4:

Take thread dump and search the converted hexadecimal PID ( 1baf ) in the thread dump. You can find the hex PID as ‘nid‘ (with ’0x’) of the thread trace. From the trace, you can find ‘MyThread-1005‘ is the problematic thread in the test program. And using the thread trace you can identify the problematic code and fix the problem.

  1. [rameshj@rameshj thread]$ java test/ThreadDumpTest
  2. ^\Full thread dump Java HotSpot(TM) Client VM (1.5.0_06-b05 mixed mode):
  3. “DestroyJavaVM” prio=1 tid=0x083d2d78 nid=0x1ba2 waiting on condition [0x00000000..0xbff4bf60]
  4. “MyThread-1009″ prio=1 tid=0x08510a88 nid=0x1bb3 waiting on condition [0xa9cca000..0xa9ccb020]
  5.         at java.lang.Thread.sleep(Native Method)                                             
  6.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)          &nbs
    p;                     
  7.         at java.lang.Thread.run(Thread.java:595)                                             
  8. “MyThread-1008″ prio=1 tid=0×08510318 nid=0x1bb2 waiting on condition [0xa9d4b000..0xa9d4bea0]
  9.         at java.lang.Thread.sleep(Native Method)                                             
  10.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)                                
  11.         at java.lang.Thread.run(Thread.java:595)                                             
  12. “MyThread-1007″ prio=1 tid=0x0850f298 nid=0x1bb1 waiting on condition [0xa9dcc000..0xa9dccf20]
  13.         at java.lang.Thread.sleep(Native Method)                                             
  14.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)                                
  15.         at java.lang.Thread.run(Thread.java:595)                                             
  16. “MyThread-1006″ prio=1 tid=0x0850d818 nid=0x1bb0 waiting on condition [0xa9e4d000..0xa9e4dda0]
  17.         at java.lang.Thread.sleep(Native Method)                                             
  18.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)                                
  19.         at java.lang.Thread.run(Thread.java:595)                                             
  20. MyThread-1005” prio=1 tid=0x0850c9f8 nid=0x1baf runnable [0xa9ece000..0xa9ecee20]
  21.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:43)                    
  22.         at java.lang.Thread.run(Thread.java:595)                                 
  23. “MyThread-1004″ prio=1 tid=0x0850b8e8 nid=0x1bae waiting on condition [0xa9f4f000..0xa9f500a0]
  24.         at java.lang.Thread.sleep(Native Method)                                             
  25.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)                                
  26.         at java.lang.Thread.run(Thread.java:595)                                             
  27. “MyThread-1003″ prio=1 tid=0x0850aac8 nid=0x1bad waiting on condition [0xa9fd0000..0xa9fd1120]
  28.         at java.lang.Thread.sleep(Native Method)                                             
  29.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)                                
  30.         at java.lang.Thread.run(Thread.java:595)                                             
  31. “MyThread-1002″ prio=1 tid=0x085099b8 nid=0x1bac waiting on condition [0xaa051000..0xaa051fa0]
  32.         at java.lang.Thread.sleep(Native Method)                                             
  33.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)                                
  34.         at java.lang.Thread.run(Thread.java:595)                                             
  35. “MyThread-1001″ prio=1 tid=0x08508a28 nid=0x1bab waiting
    on condition [0xaa0d2000..0xaa0d3020]
  36.         at java.lang.Thread.sleep(Native Method)                                             
  37.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  38.         at java.lang.Thread.run(Thread.java:595)
  39. “MyThread-1000″ prio=1 tid=0×08508520 nid=0x1baa waiting on condition [0xaa153000..0xaa153ea0]
  40.         at java.lang.Thread.sleep(Native Method)
  41.         at test.ThreadDumpTest$TR.run(ThreadDumpTest.java:45)
  42.         at java.lang.Thread.run(Thread.java:595)
  43. “Low Memory Detector” daemon prio=1 tid=0x0843cf58 nid=0x1ba8 runnable [0x00000000..0x00000000]
  44. “CompilerThread0″ daemon prio=1 tid=0x0843b930 nid=0x1ba7 waiting on condition [0x00000000..0xaa3eb808]
  45. “Signal Dispatcher” daemon prio=1 tid=0x0843a9d0 nid=0x1ba6 waiting on condition [0x00000000..0x00000000]
  46. “Finalizer” daemon prio=1 tid=0×08430740 nid=0x1ba5 in Object.wait() [0xaa52c000..0xaa52d120]
  47.         at java.lang.Object.wait(Native Method)
  48.         – waiting on <0xaa6b0848> (a java.lang.ref.ReferenceQueue$Lock)
  49.         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
  50.         – locked <0xaa6b0848> (a java.lang.ref.ReferenceQueue$Lock)
  51.         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
  52.         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
  53. “Reference Handler” daemon prio=1 tid=0x0842e898 nid=0x1ba4 in Object.wait() [0xaa5ad000..0xaa5adfa0]
  54.         at java.lang.Object.wait(Native Method)
  55.         – waiting on <0xaa6b0758> (a java.lang.ref.Reference$Lock)
  56.         at java.lang.Object.wait(Object.java:474)
  57.         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
  58.         – locked <0xaa6b0758> (a java.lang.ref.Reference$Lock)
  59. “VM Thread” prio=1 tid=0x0842bb68 nid=0x1ba3 runnable
  60. “VM Periodic Task Thread” prio=1 tid=0x0843e538 nid=0x1ba9 waiting on condition

Update : FYI : You can automatically take a thread dump of a Java application when a high CPU threshold is violated by using Applications Manager.  Learn more or watch a related video or download Applications Manager now.

- Ramesh

10 comments on “Identify Java code consuming high CPU in Linux (linking JVM thread and Linux PID)

  1. Jerry Lee on said:

    excellent step-by-step explanation.

    PS:
    https://github.com/oldratlee/useful-shells/blob/master/show-busy-java-threads.sh
    this script automate the step of finding busy thread.

    usage:
    $ show-busy-java-threads.sh
    or
    $ show-busy-java-threads.sh -c

  2. toddkaufmann on said:

    Why not just use the Java VisualVM ?
    It has come with Sun (Oracle) JDK since at least 1.5.. find it in your bin directory where java is as jvisualvm (Linux & windows), /usr/share/java/Tools/Java VisualVM.app on OSX.

    You can also connect to remote processes on other machines, see which classes have the most instances, etc.

    more info at: http://visualvm.java.net/

  3. Ron on said:

    for step 3 you can run: echo “obase=16;” | bc | tr [:upper:] [:lower:]

  4. Java Coder on said:

    Excellent Bro !

    Many Many thx for this tip

  5. Ketan on said:

    Thanks Ramesh. This was helpful. We had a similar problem in our production envionment where GC was consuming high CPU. Was able to identify the rogue GC thread which was consuming high CPU.

  6. Aaron on said:

    while (true) is a busy-wait and will run a single-threaded process to max CPU in any language, on any platform.

  7. Ramesh on said:

    No. If you press ‘Shift + h’, it will show thread level usage until you press ‘Shift + h’ again.

  8. non on said:

    Press ‘Shift + h’ and wait few secs. You can see ‘Show threads on’ message in the top console.
    It only shows thread about 1 sec, then back to process, too short to find the java threads, how to make top console keep the threads view?

  9. Ramesh on said:

    Just press “Ctrl+|” on the java process running console. If the java process running in the background, execute “kill -3 [PID]” command in any console. You can find the “PID” using “ps -aef | grep java | grep -v grep” command. The second column of the output is the ‘PID” of the java process.

  10. Arnold on said:

    In Step 4, you said “Take thread dump”. How do I do that?

Leave a Reply

Your email address will not be published. Required fields are marked *

*

HTML tags are not allowed.