Why aren’t your threads behaving? Although most developers are confident about handling better-known issues such as deadlocks and simple hangs, many people still come across liveness issues that are not always easy to identify and resolve. From complete hangs that bring your application to a halt, to scalability issues, there are all sorts of things that go bump in the night. This session explores the most common types of things that can prevent your Java threads from getting work done and shows how to address them.
3. JVM Sustaining Engineer
OpenJDK Updates Project
Maintainer
JavaOne Rock Star
Co-author of Oracle WebLogic
Server 11g 構築・運用ガイド
@DavidBuckJP
https://blogs.oracle.com/buck/
Who am I? David Buck (left)
5. Java 1.0 1996
Threading as a First Class Member of the Language
Java 1.2 1998
Deprecation of asynchronous Thread methods
Java 1.4 2002
NIO (non-blocking IO)
Java 5 2004
Java.util.concurrent
Improved Memory Model
Java 7 2011
ForkJoin
Java 8 2014
Lambda Expressions
21. Java Monitor Limitations
No way to check status of a lock
No timeout
No way to cancel
Must be recursive
No reader / writer locking
21
22. Java Monitor Limitations
No way to check status of a lock
No timeout
No way to cancel
Must be recursive
No reader / writer locking
Security Issues
22
28. JavaThreads
Run Java (Byte)code
Stop during STW
Always show up in thread dump
Many aliases:
Execution Thread
Mutator Thread
Other Threads
Runs non-Java code
May run during STW
May not show up in thread dump
JVM internal threads
GC Worker Threads
JIT Compiler Threads
Non-JVM threads
Primordial thread
Other user threads (JNI application)
29. Primordial Thread
First JVM thread created by OS
Poor control over stack setup (size, guard pages, etc.)
Poor visibility of stack
Should not be used to run Java Code
Java launcher “abandons” primordial thread (JDK >= 6)
JNI invocation code should do the same
30. Thread Dumps
A snapshot of all Java Threads (and some JVM Threads)
Stop-the-World operation
Some non-thread info thrown in for good measure
Many ways to collect
Slowly evolving “format”
Many tools to help with analysis
31. Snapshots
One dump alone can not show
“motion” or change
Each dump is mostly self-
consistent
Stop-the-World pause during
collection
By FlanellKamerasFilm - Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=68147463
32. Is this train moving?
By Shiodome - ja:image:Mono-shibaura.JPG, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1157483
33. Thread Uncertainty Principle
You can tell where a thread is (at what safepoint)
You can tell how fast a thread is “moving” (CPU consumption)
You can’t tell both at the same time
But you don’t really need to
Collect CPU consumption data along with thread dumps
Collect multiple thread dumps
34. Safepoints
Stop-the-World implementation
Guarantees Java state of thread is known
All threads executing Java code must stop
JNI code can continue
Cooperative suspension
Polling based
By YPLeroux at English Wikipedia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=17734859
35. Safepoints
Stop-the-World implementation
Guarantees Java state of thread is known
All threads executing Java code must stop
JNI code can continue
Cooperative suspension
Polling based
By YPLeroux at English Wikipedia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=17734859
36. Safepoint Polling
Interpreter can stop instantly
template dispatch table replaced
“polling” is essentially free
Explicit poll inserted into C1/C2 JIT output
Global safepoint polling page in address space
Poll implemented by attempted read of this address
Safepoint triggered (SIGSEGV) by change in page permissions
37. JIT Safepoint Polling
On return from method call
Backend branches in most loops
Any transition to native code (JVM / JNI)
38. Not quite a snapshot
Each thread stops at its next safepoint
Not all threads will stop at the same time
A critical section might not even have a safepoint
Not unusual to see thread blocking for a lock no one holds
More like a “panorama mode” picture: expect inconsistencies
39. Ways to Get a Thread Dump
SIGQUIT / CTL-BRK handler
Diagnostic command (jcmd)
Java Flight recording periodic event
jstack tool
40. jstack
Force thread dump (might work)
Mixed mode (might work)
Thread dump from process or core file
Integrated into servicabililty agent as of JDK 9
42. Thread Dump Output
Header
Timestamp
JVM version information
safe memory reclamation (SMR) data
Thread List
All attached Java Threads
Most Native threads
Footer
JNI global references
Memory usage statistics
44. Version Information
Full thread dump Java HotSpot(TM) 64-Bit Server VM
(13+33 mixed mode, sharing)
Similar to “java –version” output
Sharing -> class data sharing enabled
JIT compiler state
mixed mode
interpreted mode (-Xint)
compiled mode (-Xcomp)
45. Version Information
Full thread dump Java HotSpot(TM) 64-Bit Server VM
(25.221-b11 mixed mode):
JDK 8 and earlier use HotSpot Express version number
HotSpot == 25.x -> JDK 8
HotSpot <= 24 -> Google it…
46. Safe Memory Reclamation
Threads class SMR info:
_java_thread_list=0x00007f4e842e9a60, length=6,
elements={
0x00007f4e84017800, 0x00007f4e84232800,
0x00007f4e84236000, 0x00007f4e84257000,
0x00007f4e842da800, 0x00007f4e842e8000
}
tid of each JavaThread
JVM internal information (can be ignored)
Added in JDK 9
47. JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
48. JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
49. What’s in a name?
Arbitrary string
Thread.setName();
Can be changed as often as you want
Name can be used as a debugging tool
A thread by any other name...
By Photo: JLPC / Wikimedia Commons, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=30151561
50. What’s in a name?
"[STUCK] ExecuteThread: '139' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10
tid=0x00007fd2a00b6000 nid=0x690b runnable [0x00007fd0ecbc7000]
java.lang.Thread.State: RUNNABLE
at java.io.ObjectOutputStream$BlockDataOutputStream.getUTFLength(ObjectOutputStream.java:2151)
51. What’s in a name?
"[STUCK] ExecuteThread: '139' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10
tid=0x00007fd2a00b6000 nid=0x690b runnable [0x00007fd0ecbc7000]
java.lang.Thread.State: RUNNABLE
at java.io.ObjectOutputStream$BlockDataOutputStream.getUTFLength(ObjectOutputStream.java:2151)
52. What’s in a name?
theThread.setName("[really cool] " + theThread.getName());
System.out.println("I made your thread cool!");
< do cool stuff >
theThread.setName(myThread.getName().substring(14));
System.out.println("Your thread is no longer cool!");
57. JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
73. nid
Linux: kernel thread id from syscall gettid
Solaris: thread library identity from thr_create() or thr_self()
Windows: thread id from _beginthreadex
macOS: kernel thread id from
pthread_mach_thread_np(pthread_self());
Other BSD: kernel (?) thread id from syscall thr_self or getthrid
AIX: thread library identity from pthread_create() or pthread_self()
74. JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
76. JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
77. Last Java Frame
Stack address of most recent Java frame on stack
Stack pointer (sp) may be much farther down
Null if there are no Java frames
"C1 CompilerThread0" #8 daemon prio=9 os_prio=0
cpu=12.22ms elapsed=2.44s tid=0x00007f60781fd800
nid=0x37dc waiting on condition [0x0000000000000000]
78. Last Java Frame
Stack address of most recent Java frame on stack
Stack pointer (sp) may be much farther down
Null if there are no Java frames
"C1 CompilerThread0" #8 daemon prio=9 os_prio=0
cpu=12.22ms elapsed=2.44s tid=0x00007f60781fd800
nid=0x37dc waiting on condition [0x0000000000000000]
79. JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
80. java.lang.Thread.State
NEW
A thread that has not yet started is in this state.
RUNNABLE
A thread executing in the Java virtual machine is in this state.
BLOCKED
A thread that is blocked waiting for a monitor lock is in this state.
WAITING
A thread that is waiting indefinitely for another thread to perform a particular action is in this
state.
TIMED_WAITING
A thread that is waiting for another thread to perform an action for up to a specified waiting
time is in this state.
TERMINATED
A thread that has exited is in this state.
81. JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
82. Non-Java Thread
"VM Thread" os_prio=0 cpu=2.63ms elapsed=2.47s
tid=0x00007f0e841ef800 nid=0x3c41 runnable
No Java thread ID (e.g. #42)
No Java priority (e.g. prio=5)
No last Java frame (e.g. [0x00007f4e8c82c000])
No java.lang.Thread.State
No Java stack
83. JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
84. JNI Global References
JNI global refs: 6, weak refs: 0
JNI global reference leaks are a common bug
Investigate any significant growth post app warmup
By John Newton and William Palmer, - A terrestial globe on which the tracts and discoveries are laid down from the accurate observations made by Captains Cook, Furneux, Phipps, published May 9th, 1782 globe by John Newton, cartography by William Palmer, Sate Library of New South
Wales https://primo-slnsw.hosted.exlibrisgroup.com/primo-explore/fulldisplay?docid=ADLIB110577047&context=L&vid=SLNSW&search_scope=EEA&tab=default_tab&lang=en_US, Public Domain, https://commons.wikimedia.org/w/index.php?curid=55030411
85. JNI Global References
JNI global refs: 6, weak refs: 0
JNI global reference leaks are a common bug
Investigate any significant growth post app warmup
Weak Global Reference breakout is new (JDK 11)
Even if referent is collected, memory is leaked unless DeleteWeakGlobalRef
is called
86. Heap Stats
Heap
garbage-first heap total 256000K, used 776K [0x0000000709200000, 0x0000000800000000)
region size 1024K, 1 young (1024K), 0 survivors (0K)
Metaspace used 161K, capacity 4486K, committed 4864K, reserved 1056768K
class space used 7K, capacity 386K, committed 512K, reserved 1048576K
88. Runnable
"Thread-0" #12 prio=5 os_prio=0 cpu=1489.99ms elapsed=1.49s tid=0x00007f5c78262000
nid=0xa6f runnable [0x00007f5c497f4000]
java.lang.Thread.State: RUNNABLE
at LockTest$1.run(LockTest.java:6)
at java.lang.Thread.run(java.base@13/Thread.java:830)
89. Blocked on a Monitor
"Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000
nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker1.run(LockChain.java:28)
- waiting to lock <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
90. Blocked on a Monitor
"Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000
nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker1.run(LockChain.java:28)
- waiting to lock <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
91. Blocked on a Monitor
"Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000
nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker1.run(LockChain.java:28)
- waiting to lock <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
92. Who has my lock?
"Thread-5" #17 prio=5 os_prio=0 cpu=0.85ms elapsed=4.58s tid=0x00007fe2dc26c800
nid=0x239 waiting for monitor entry [0x00007fe2acff5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker2.run(LockChain.java:39)
- waiting to lock <0x0000000718948ef0> (a java.lang.Object)
- locked <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
93. The plot thickens...
"Thread-5" #17 prio=5 os_prio=0 cpu=0.85ms elapsed=4.58s tid=0x00007fe2dc26c800
nid=0x239 waiting for monitor entry [0x00007fe2acff5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker2.run(LockChain.java:39)
- waiting to lock <0x0000000718948ef0> (a java.lang.Object)
- locked <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
94. End of the road...
"Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000
nid=0x23b waiting on condition [0x00007fe2acdf3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at LockChain$Worker3.run(LockChain.java:51)
- locked <0x0000000718948ef0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
97. Thread A Thread B
Holds Lock 1
Waiting on Lock 2
Holds Lock 2
Waiting on Lock 1
98. Deadlock
Restarting JVM only way to recover
All threads should acquire locks in well-defined order
Most JVMs / tools can identify trivial deadlocks
99. Java stack information for the threads listed above:
===================================================
"Thread-0":
at DL$Thread1.run(DL.java:18)
- waiting to lock <0x0000000718949078> (a java.lang.Object)
- locked <0x0000000718949068> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
"Thread-1":
at DL$Thread2.run(DL.java:30)
- waiting to lock <0x0000000718949068> (a java.lang.Object)
- locked <0x0000000718949078> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Found 1 deadlock.
100. Intrinsic lock (monitor) issues
Easy to identify in thread dumps
Deep understanding of application code not needed for
identification
Amenable to automation (thread dump analysis tools)
101. Waiting on a monitor
"Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800
nid=0xba2 in Object.wait() [0x00007f670d9d3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718949e30> (a Drop)
at java.lang.Object.wait(java.base@13/Object.java:326)
at Drop.take(Drop.java:47)
- locked <0x0000000718949e30> (a Drop)
at Consumer.run(Consumer.java:45)
at java.lang.Thread.run(java.base@13/Thread.java:830)
102. Wait and Notify
Used to signal other threads to wake up and work
Producer / Consumer is the textbook use-case
Thread must hold the lock before calling wait or notify
Otherwise IllegalMonitorStateException thrown
103. Wait and Notify
Thread A locks object
Thread A calls Object.wait()
Thread B locks object
Thread B calls Object.notify()
Thread B releases lock
Thread A wakes up and continues working
104. Wait and Notify
Thread A locks object
Thread A calls Object.wait() (lock released!)
Thread B locks object
Thread B calls Object.notify()
Thread B releases lock
Thread A wakes up and continues working
105. Waiting on a monitor
"Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800
nid=0xba2 in Object.wait() [0x00007f670d9d3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718949e30> (a Drop)
at java.lang.Object.wait(java.base@13/Object.java:326)
at Drop.take(Drop.java:47)
- locked <0x0000000718949e30> (a Drop)
at Consumer.run(Consumer.java:45)
at java.lang.Thread.run(java.base@13/Thread.java:830)
106. Waiting on a monitor
"Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800
nid=0xba2 in Object.wait() [0x00007f670d9d3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718949e30> (a Drop)
at java.lang.Object.wait(java.base@13/Object.java:326)
at Drop.take(Drop.java:47)
- locked <0x0000000718949e30> (a Drop)
at Consumer.run(Consumer.java:45)
at java.lang.Thread.run(java.base@13/Thread.java:830)
107. Waiting on a monitor
"Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800
nid=0xba2 in Object.wait() [0x00007f670d9d3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718949e30> (a Drop)
at java.lang.Object.wait(java.base@13/Object.java:326)
at Drop.take(Drop.java:47)
- locked <0x0000000718949e30> (a Drop)
at Consumer.run(Consumer.java:45)
at java.lang.Thread.run(java.base@13/Thread.java:830)
108. Wait / Notify
Intrinsic lock troubleshooting pattern not very helpful
Knowledge of application design needed to know what thread
should call notify()
Analysis not amenable to automation
Having a bunch of threads waiting might be normal under low load
112. Intrinsic locks
"Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000
nid=0x23b waiting on condition [0x00007fe2acdf3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at LockChain$Worker3.run(LockChain.java:51)
- locked <0x0000000718948ef0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
113. java.util.concurrent
Lock does not necessarily belong to a frame or even a thread
A lock might be taken and released by different methods
A lock might be taken and released by different threads
114. Intrinsic locks
"Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000
nid=0x23b waiting on condition [0x00007fe2acdf3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at LockChain$Worker3.run(LockChain.java:51)
- locked <0x0000000718948ef0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
117. private static class Worker implements Runnable {
public void run() {
getLockAndSleep();
}
}
118. public static void main(String[] args) throws InterruptedException {
Thread childThread = new Thread(new Worker());
childThread.start();
Thread.sleep(1000 * 3); // 3 seconds
System.out.println("main thread will try to get lock...");
getLockAndSleep();
}
119. "main" #1 prio=5 os_prio=0 cpu=45.91ms elapsed=7.66s tid=0x00007ff0d4017800 nid=0x1b1f waiting
on condition [0x00007ff0dbf02000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@13/Native Method)
- parking to wait for <0x0000000718953ed8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@13/LockSupport.java:194)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@13/Abstrac
tQueuedSynchronizer.java:885)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@13/AbstractQueuedS
ynchronizer.java:917)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@13/AbstractQueuedSynchro
nizer.java:1240)
at java.util.concurrent.locks.ReentrantLock.lock(java.base@13/ReentrantLock.java:267)
at JUCLExample.getLockAndSleep(JUCLExample.java:19)
at JUCLExample.main(JUCLExample.java:15)
120. Thread that “holds” the lock
"Thread-0" #12 prio=5 os_prio=0 cpu=0.39ms elapsed=7.62s tid=0x00007ff0d4264000
nid=0x1b2f waiting on condition [0x00007ff0a45cc000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at JUCLExample.getLockAndSleep(JUCLExample.java:21)
at JUCLExample.access$100(JUCLExample.java:3)
at JUCLExample$Worker.run(JUCLExample.java:31)
at java.lang.Thread.run(java.base@13/Thread.java:830)
121. With great power comes great...
No thread really holds the lock
Only the developer knows what thread should unlock it
Thread that do the unlock may not even exist yet
122. That said...
Most designs do use j.u.c locks in a “symmetric” way
Often the thread that takes the lock is the one that releases it
We can keep track of which thread took each lock:
-XX:+PrintConcurrentLocks
123. "Thread-0" #12 prio=5 os_prio=0 cpu=0.40ms elapsed=18.75s tid=0x00007fc7b425c000
nid=0x1e65 waiting on condition [0x00007fc784ce5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at JUCLExample.getLockAndSleep(JUCLExample.java:21)
at JUCLExample.access$100(JUCLExample.java:3)
at JUCLExample$Worker.run(JUCLExample.java:31)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Locked ownable synchronizers:
- <0x0000000718953e98> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
124. PrintConcurrentLocks
Performance impact (examine heap on thread dump)
Tells you what thread took the lock, which may not be the thread
that will release it
Customer synchronizers built from j.u.c primitives may not be
tracked (should extend AbstractOwnableSynchronizer)
Can also be retrieved on-demand:
jhsdb jstack --pid <pid> --locks
130. ThreadLogic
Tailored for WebLogic and Oracle
Fusion Middleware
Automatically identifies many
known issues
Based on Eclipse and TDA (Thread
Dump Analyzer) projects
Open Source
132. Service Thread
"Service Thread" #9 daemon prio=9 os_prio=0
tid=0x00007fe4ac13f800 nid=0x3d3a runnable
[0x0000000000000000]
Since JDK 7
Handles
JVMTI events
Low Memory Detection
JMX event handling for diagnostic commands
Etc.
133. JIT Compiler Thread
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0
tid=0x00007fe4ac12c000 nid=0x3d39 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
• Does not run Bytecode
• Is a JavaThread for mostly historical reasons
134. Signal Dispatcher
"Signal Dispatcher" #4 daemon prio=9 os_prio=0
tid=0x00007fe4ac116800 nid=0x3d35 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
• sun.misc.SignalHandler
• Not for direct use
By Mark Wolfe - This image is from the FEMA Photo Library., Public Domain, https://commons.wikimedia.org/w/index.php?curid=8075137
136. Finalizer
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f056c0ec000 nid=0x460e in Object.wait()
[0x00007f054d466000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076db88ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x000000076db88ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
137. Secondary finalizer
"Secondary finalizer" #10 prio=5 os_prio=0 tid=0x00007f908c154800
nid=0x4ff4 waiting on condition [0x00007f9074dc2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at FinStop.finalize(FinStop.java:22)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$2.run(Finalizer.java:161)
at java.lang.Thread.run(Thread.java:748)
138. Reference Handler
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f908c0e7800
nid=0x4feb in Object.wait() [0x00007f90756cb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c9200ae0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006c9200ae0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
By Giuseppe Arcimboldo - LSH 97136 (sm_dig12951), Public Domain, https://commons.wikimedia.org/w/index.php?curid=36899673
139. “main”
"main" #1 prio=5 os_prio=0 tid=0x00007ff04804b000
nid=0x6eca waiting on condition [0x00007ff04fbfd000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Sleeper.main(Sleeper.java:5)
141. VM Periodic Task Thread
"VM Periodic Task Thread" os_prio=0
tid=0x00007ff04813a000 nid=0x6edc waiting on
condition
Timer based events
Profiling
Sampling
By Ambrogio Lorenzetti - The Yorck Project (2002) 10.000 Meisterwerke der Malerei (DVD-ROM), distributed by DIRECTMEDIA Publishing GmbH. ISBN:
3936122202., Public Domain, https://commons.wikimedia.org/w/index.php?curid=3499217
142. GC Worker Threads
"GC task thread#0 (ParallelGC)" os_prio=0
tid=0x00007ff04805e000 nid=0x6ecb runnable
Depend on the collector currently in use
Number of threads may be chosen by ergonomics
Many different types
143. Common-Cleaner
"Common-Cleaner" #11 daemon prio=8 os_prio=0 cpu=0.13ms elapsed=2.49s tid=0x00007f8224254000
nid=0x766a in Object.wait() [0x00007f81f58f6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718945448> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@13/ReferenceQueue.java:155)
- locked <0x0000000718945448> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base@13/CleanerImpl.java:148)
at java.lang.Thread.run(java.base@13/Thread.java:830)
at jdk.internal.misc.InnocuousThread.run(java.base@13/InnocuousThread.java:134)
JDK 9 and later
Shared java.lang.ref.Cleaner thread
https://bugs.openjdk.java.net/browse/JDK-8145041
By CurranH (talk) (Uploads) - Own work, Public Domain, https://en.wikipedia.org/w/index.php?curid=11838492
144. Destroyer of Worlds
"DestroyJavaVM" #13 prio=5 os_prio=0 cpu=63.15ms
elapsed=3.19s tid=0x00007f951c017800 nid=0x4cfc
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Main has exited, but non-Deamon threads remain
By Jbuzbee - Own work, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=4835571
145. Takaways
Collect at least 3 dumps
Collect CPU usage with OS-level tool
top –H
Process Explorer
Enable j.u.concurrent lock output
146. Java Concurrency in Practice
Java Concurrency Bible
Covers JSR-133 / JSR-166