SlideShare a Scribd company logo
1 of 147
Download to read offline
Hangs, Slowdowns, Starvation—Oh My!
A Deep Dive into the Life of a Java Thread
[DEV4356]
Principal Member of Technical Staff
Java Platform Group
September 17, 2019
David Buck
Copyright © 2019 Oracle and/or its affiliates.
The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor
Copyright © 2019 Oracle and/or its affiliates.
JVM Sustaining Engineer
OpenJDK Updates Project
Maintainer
JavaOne Rock Star
Co-author of Oracle WebLogic
Server 11g 構築・運用ガイド
@DavidBuckJP
https://blogs.oracle.com/buck/
Who am I? David Buck (left)
History
Java 1.0 1996
Threading as a First Class Member of the Language
Java 1.2 1998
Deprecation of asynchronous Thread methods
Java 1.4 2002
NIO (non-blocking IO)
Java 5 2004
Java.util.concurrent
Improved Memory Model
Java 7 2011
ForkJoin
Java 8 2014
Lambda Expressions
Java 9 2017
Variable Handles
CompletableFuture API improvements
Future
Project Loom
Multithreading as Part of the Language
7
So, what exactly is a Monitor?
8
Mutual Exclusion
9
Mutual Exclusion
Mut ex
10
Mutual Exclusion
Mutex
11
condition variable
12
"Puffin crossing, London, UK" by secretlondon is licensed under CC BY-SA 3.0
Monitor = Mutex + Condition Variable
13
Monitor = Mutex + Condition Variable
Mutex
synchronized keyword
14
Monitor = Mutex + Condition Variable
Mutex
synchronized keyword
Condition Variable
Object.wait()
Object.notify()
Object.notifyAll()
15
Java Monitor Limitations
16
Java Monitor Limitations
No way to check status of a lock
17
Java Monitor Limitations
No way to check status of a lock
No timeout
18
Java Monitor Limitations
No way to check status of a lock
No timeout
No way to cancel
19
Java Monitor Limitations
No way to check status of a lock
No timeout
No way to cancel
Must be recursive
20
Java Monitor Limitations
No way to check status of a lock
No timeout
No way to cancel
Must be recursive
No reader / writer locking
21
Java Monitor Limitations
No way to check status of a lock
No timeout
No way to cancel
Must be recursive
No reader / writer locking
Security Issues
22
java.util.concurrent
23
Threads
Green Threads (Pre-HotSpot)
Modern Threads (HotSpot)
1 to 1 (Everyone else)Many to Many (Solaris)
Threads JavaThreads
JavaThreads
Run Java (Byte)code
Stop during STW
Always show up in thread dump
Many aliases:
Execution Thread
Mutator Thread
Other Threads
Runs non-Java code
May run during STW
May not show up in thread dump
JVM internal threads
GC Worker Threads
JIT Compiler Threads
Non-JVM threads
Primordial thread
Other user threads (JNI application)
Primordial Thread
First JVM thread created by OS
Poor control over stack setup (size, guard pages, etc.)
Poor visibility of stack
Should not be used to run Java Code
Java launcher “abandons” primordial thread (JDK >= 6)
JNI invocation code should do the same
Thread Dumps
A snapshot of all Java Threads (and some JVM Threads)
Stop-the-World operation
Some non-thread info thrown in for good measure
Many ways to collect
Slowly evolving “format”
Many tools to help with analysis
Snapshots
One dump alone can not show
“motion” or change
Each dump is mostly self-
consistent
Stop-the-World pause during
collection
By FlanellKamerasFilm - Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=68147463
Is this train moving?
By Shiodome - ja:image:Mono-shibaura.JPG, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1157483
Thread Uncertainty Principle
You can tell where a thread is (at what safepoint)
You can tell how fast a thread is “moving” (CPU consumption)
You can’t tell both at the same time
But you don’t really need to
Collect CPU consumption data along with thread dumps
Collect multiple thread dumps
Safepoints
Stop-the-World implementation
Guarantees Java state of thread is known
All threads executing Java code must stop
JNI code can continue
Cooperative suspension
Polling based
By YPLeroux at English Wikipedia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=17734859
Safepoints
Stop-the-World implementation
Guarantees Java state of thread is known
All threads executing Java code must stop
JNI code can continue
Cooperative suspension
Polling based
By YPLeroux at English Wikipedia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=17734859
Safepoint Polling
Interpreter can stop instantly
template dispatch table replaced
“polling” is essentially free
Explicit poll inserted into C1/C2 JIT output
Global safepoint polling page in address space
Poll implemented by attempted read of this address
Safepoint triggered (SIGSEGV) by change in page permissions
JIT Safepoint Polling
On return from method call
Backend branches in most loops
Any transition to native code (JVM / JNI)
Not quite a snapshot
Each thread stops at its next safepoint
Not all threads will stop at the same time
A critical section might not even have a safepoint
Not unusual to see thread blocking for a lock no one holds
More like a “panorama mode” picture: expect inconsistencies
Ways to Get a Thread Dump
SIGQUIT / CTL-BRK handler
Diagnostic command (jcmd)
Java Flight recording periodic event
jstack tool
jstack
Force thread dump (might work)
Mixed mode (might work)
Thread dump from process or core file
Integrated into servicabililty agent as of JDK 9
Thread Dump Output
Thread Dump Output
Header
Timestamp
JVM version information
safe memory reclamation (SMR) data
Thread List
All attached Java Threads
Most Native threads
Footer
JNI global references
Memory usage statistics
Timestamp
2019-09-07 03:00:18
Useful to sanity check timing dump was collected
Makes interpreting per-thread stats much easier
Version Information
Full thread dump Java HotSpot(TM) 64-Bit Server VM
(13+33 mixed mode, sharing)
Similar to “java –version” output
Sharing -> class data sharing enabled
JIT compiler state
mixed mode
interpreted mode (-Xint)
compiled mode (-Xcomp)
Version Information
Full thread dump Java HotSpot(TM) 64-Bit Server VM
(25.221-b11 mixed mode):
JDK 8 and earlier use HotSpot Express version number
HotSpot == 25.x -> JDK 8
HotSpot <= 24 -> Google it…
Safe Memory Reclamation
Threads class SMR info:
_java_thread_list=0x00007f4e842e9a60, length=6,
elements={
0x00007f4e84017800, 0x00007f4e84232800,
0x00007f4e84236000, 0x00007f4e84257000,
0x00007f4e842da800, 0x00007f4e842e8000
}
tid of each JavaThread
JVM internal information (can be ignored)
Added in JDK 9
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
What’s in a name?
Arbitrary string
Thread.setName();
Can be changed as often as you want
Name can be used as a debugging tool
A thread by any other name...
By Photo: JLPC / Wikimedia Commons, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=30151561
What’s in a name?
"[STUCK] ExecuteThread: '139' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10
tid=0x00007fd2a00b6000 nid=0x690b runnable [0x00007fd0ecbc7000]
java.lang.Thread.State: RUNNABLE
at java.io.ObjectOutputStream$BlockDataOutputStream.getUTFLength(ObjectOutputStream.java:2151)
What’s in a name?
"[STUCK] ExecuteThread: '139' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10
tid=0x00007fd2a00b6000 nid=0x690b runnable [0x00007fd0ecbc7000]
java.lang.Thread.State: RUNNABLE
at java.io.ObjectOutputStream$BlockDataOutputStream.getUTFLength(ObjectOutputStream.java:2151)
What’s in a name?
theThread.setName("[really cool] " + theThread.getName());
System.out.println("I made your thread cool!");
< do cool stuff >
theThread.setName(myThread.getName().substring(14));
System.out.println("Your thread is no longer cool!");
Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [Code One 2019]
Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [Code One 2019]
Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [Code One 2019]
Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [Code One 2019]
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
Thread ID
Thread.getID()
Automatically assigned by JVM during thread creation
Read-Only
Consistent throughout thread lifetime
Daemon Thread
"child thread" #12 daemon prio=5 os_prio=0 cpu=0.27ms
elapsed=1.98s tid=0x00007fec48278800 nid=0x179
waiting on condition [0x00007fec18df2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
Java Thread Priority
Thread.getPriority()
Thread.setPriority(int)
Min is usually 1
Max is usually 10
Ignored by many JVMs
10
9
8
7
3
2
1
4
5
6
Java Thread Priority
Thread.getPriority()
Thread.setPriority(int)
Min is usually 1
Max is usually 10
Ignored by many JVMs
10
9
8
7
3
2
1
4
5
6
Turtles all the way down!!!
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
OS Thread Priority
Not directly user controllable
Interpretation is platform dependent
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
-XX:+PrintExtendedThreadInfo
"main" #1 prio=5 os_prio=0 cpu=58.42ms elapsed=2.60s
allocated=289K defined_classes=4 tid=0x00007f8138017800
nid=0x344f waiting on condition [0x00007f8141890000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
tid
Address of HotSpot data structure used to track thread
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
Safe Memory Reclamation
Threads class SMR info:
_java_thread_list=0x00007f4e842e9a60, length=6,
elements={
0x00007f4e84017800, 0x00007f4e84232800,
0x00007f4e84236000, 0x00007f4e84257000,
0x00007f4e842da800, 0x00007f4e842e8000
}
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
nid
“native” thread identifier
Useful for using OS tools (e.g. gdb, top -H)
TD output in Hexadecimal, many OS tools use Decimal
nid
Linux: kernel thread id from syscall gettid
Solaris: thread library identity from thr_create() or thr_self()
Windows: thread id from _beginthreadex
macOS: kernel thread id from
pthread_mach_thread_np(pthread_self());
Other BSD: kernel (?) thread id from syscall thr_self or getthrid
AIX: thread library identity from pthread_create() or pthread_self()
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
Thread state (native)
runnable
waiting for monitor entry
waiting on condition
in Object.wait()
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
Last Java Frame
Stack address of most recent Java frame on stack
Stack pointer (sp) may be much farther down
Null if there are no Java frames
"C1 CompilerThread0" #8 daemon prio=9 os_prio=0
cpu=12.22ms elapsed=2.44s tid=0x00007f60781fd800
nid=0x37dc waiting on condition [0x0000000000000000]
Last Java Frame
Stack address of most recent Java frame on stack
Stack pointer (sp) may be much farther down
Null if there are no Java frames
"C1 CompilerThread0" #8 daemon prio=9 os_prio=0
cpu=12.22ms elapsed=2.44s tid=0x00007f60781fd800
nid=0x37dc waiting on condition [0x0000000000000000]
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
java.lang.Thread.State
NEW
A thread that has not yet started is in this state.
RUNNABLE
A thread executing in the Java virtual machine is in this state.
BLOCKED
A thread that is blocked waiting for a monitor lock is in this state.
WAITING
A thread that is waiting indefinitely for another thread to perform a particular action is in this
state.
TIMED_WAITING
A thread that is waiting for another thread to perform an action for up to a specified waiting
time is in this state.
TERMINATED
A thread that has exited is in this state.
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
Non-Java Thread
"VM Thread" os_prio=0 cpu=2.63ms elapsed=2.47s
tid=0x00007f0e841ef800 nid=0x3c41 runnable
No Java thread ID (e.g. #42)
No Java priority (e.g. prio=5)
No last Java frame (e.g. [0x00007f4e8c82c000])
No java.lang.Thread.State
No Java stack
JavaThread
"main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s
tid=0x00007f4e84017800 nid=0x7e4d waiting on condition
[0x00007f4e8c82c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at Sleeper.main(Sleeper.java:5)
JNI Global References
JNI global refs: 6, weak refs: 0
JNI global reference leaks are a common bug
Investigate any significant growth post app warmup
By John Newton and William Palmer, - A terrestial globe on which the tracts and discoveries are laid down from the accurate observations made by Captains Cook, Furneux, Phipps, published May 9th, 1782 globe by John Newton, cartography by William Palmer, Sate Library of New South
Wales https://primo-slnsw.hosted.exlibrisgroup.com/primo-explore/fulldisplay?docid=ADLIB110577047&context=L&vid=SLNSW&search_scope=EEA&tab=default_tab&lang=en_US, Public Domain, https://commons.wikimedia.org/w/index.php?curid=55030411
JNI Global References
JNI global refs: 6, weak refs: 0
JNI global reference leaks are a common bug
Investigate any significant growth post app warmup
Weak Global Reference breakout is new (JDK 11)
Even if referent is collected, memory is leaked unless DeleteWeakGlobalRef
is called
Heap Stats
Heap
garbage-first heap total 256000K, used 776K [0x0000000709200000, 0x0000000800000000)
region size 1024K, 1 young (1024K), 0 survivors (0K)
Metaspace used 161K, capacity 4486K, committed 4864K, reserved 1056768K
class space used 7K, capacity 386K, committed 512K, reserved 1048576K
Thread States
Runnable
"Thread-0" #12 prio=5 os_prio=0 cpu=1489.99ms elapsed=1.49s tid=0x00007f5c78262000
nid=0xa6f runnable [0x00007f5c497f4000]
java.lang.Thread.State: RUNNABLE
at LockTest$1.run(LockTest.java:6)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Blocked on a Monitor
"Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000
nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker1.run(LockChain.java:28)
- waiting to lock <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Blocked on a Monitor
"Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000
nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker1.run(LockChain.java:28)
- waiting to lock <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Blocked on a Monitor
"Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000
nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker1.run(LockChain.java:28)
- waiting to lock <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Who has my lock?
"Thread-5" #17 prio=5 os_prio=0 cpu=0.85ms elapsed=4.58s tid=0x00007fe2dc26c800
nid=0x239 waiting for monitor entry [0x00007fe2acff5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker2.run(LockChain.java:39)
- waiting to lock <0x0000000718948ef0> (a java.lang.Object)
- locked <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
The plot thickens...
"Thread-5" #17 prio=5 os_prio=0 cpu=0.85ms elapsed=4.58s tid=0x00007fe2dc26c800
nid=0x239 waiting for monitor entry [0x00007fe2acff5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at LockChain$Worker2.run(LockChain.java:39)
- waiting to lock <0x0000000718948ef0> (a java.lang.Object)
- locked <0x0000000718948ee0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
End of the road...
"Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000
nid=0x23b waiting on condition [0x00007fe2acdf3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at LockChain$Worker3.run(LockChain.java:51)
- locked <0x0000000718948ef0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Thread A
Thread E
Thread D
Thread B Thread C
Thread A
Thread F
Thread D
Thread B Thread C
Thread E Thread G
Thread A Thread B
Holds Lock 1
Waiting on Lock 2
Holds Lock 2
Waiting on Lock 1
Deadlock
Restarting JVM only way to recover
All threads should acquire locks in well-defined order
Most JVMs / tools can identify trivial deadlocks
Java stack information for the threads listed above:
===================================================
"Thread-0":
at DL$Thread1.run(DL.java:18)
- waiting to lock <0x0000000718949078> (a java.lang.Object)
- locked <0x0000000718949068> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
"Thread-1":
at DL$Thread2.run(DL.java:30)
- waiting to lock <0x0000000718949068> (a java.lang.Object)
- locked <0x0000000718949078> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Found 1 deadlock.
Intrinsic lock (monitor) issues
Easy to identify in thread dumps
Deep understanding of application code not needed for
identification
Amenable to automation (thread dump analysis tools)
Waiting on a monitor
"Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800
nid=0xba2 in Object.wait() [0x00007f670d9d3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718949e30> (a Drop)
at java.lang.Object.wait(java.base@13/Object.java:326)
at Drop.take(Drop.java:47)
- locked <0x0000000718949e30> (a Drop)
at Consumer.run(Consumer.java:45)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Wait and Notify
Used to signal other threads to wake up and work
Producer / Consumer is the textbook use-case
Thread must hold the lock before calling wait or notify
Otherwise IllegalMonitorStateException thrown
Wait and Notify
Thread A locks object
Thread A calls Object.wait()
Thread B locks object
Thread B calls Object.notify()
Thread B releases lock
Thread A wakes up and continues working
Wait and Notify
Thread A locks object
Thread A calls Object.wait() (lock released!)
Thread B locks object
Thread B calls Object.notify()
Thread B releases lock
Thread A wakes up and continues working
Waiting on a monitor
"Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800
nid=0xba2 in Object.wait() [0x00007f670d9d3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718949e30> (a Drop)
at java.lang.Object.wait(java.base@13/Object.java:326)
at Drop.take(Drop.java:47)
- locked <0x0000000718949e30> (a Drop)
at Consumer.run(Consumer.java:45)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Waiting on a monitor
"Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800
nid=0xba2 in Object.wait() [0x00007f670d9d3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718949e30> (a Drop)
at java.lang.Object.wait(java.base@13/Object.java:326)
at Drop.take(Drop.java:47)
- locked <0x0000000718949e30> (a Drop)
at Consumer.run(Consumer.java:45)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Waiting on a monitor
"Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800
nid=0xba2 in Object.wait() [0x00007f670d9d3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718949e30> (a Drop)
at java.lang.Object.wait(java.base@13/Object.java:326)
at Drop.take(Drop.java:47)
- locked <0x0000000718949e30> (a Drop)
at Consumer.run(Consumer.java:45)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Wait / Notify
Intrinsic lock troubleshooting pattern not very helpful
Knowledge of application design needed to know what thread
should call notify()
Analysis not amenable to automation
Having a bunch of threads waiting might be normal under low load
java.util.concurrent
More flexible than intrinsic locks (monitors)
Harder to debug
Plenty of rope to hang yourself
Intrinsic locks
void methodA() {
synchronized(mySpiffyObject) {
<do stuff>
methodB();
}
}
Intrinsic locks
void synchronized methodA() {
methodB();
}
Intrinsic locks
"Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000
nid=0x23b waiting on condition [0x00007fe2acdf3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at LockChain$Worker3.run(LockChain.java:51)
- locked <0x0000000718948ef0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
java.util.concurrent
Lock does not necessarily belong to a frame or even a thread
A lock might be taken and released by different methods
A lock might be taken and released by different threads
Intrinsic locks
"Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000
nid=0x23b waiting on condition [0x00007fe2acdf3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at LockChain$Worker3.run(LockChain.java:51)
- locked <0x0000000718948ef0> (a java.lang.Object)
at java.lang.Thread.run(java.base@13/Thread.java:830)
import java.util.concurrent.locks.ReentrantLock;
<…>
private static final ReentrantLock lock = new ReentrantLock();
private static void getLockAndSleep() {
lock.lock();
try {
Thread.sleep(1000 * 1000);
} catch (Exception e) {
} finally {
lock.unlock();
}
}
private static class Worker implements Runnable {
public void run() {
getLockAndSleep();
}
}
public static void main(String[] args) throws InterruptedException {
Thread childThread = new Thread(new Worker());
childThread.start();
Thread.sleep(1000 * 3); // 3 seconds
System.out.println("main thread will try to get lock...");
getLockAndSleep();
}
"main" #1 prio=5 os_prio=0 cpu=45.91ms elapsed=7.66s tid=0x00007ff0d4017800 nid=0x1b1f waiting
on condition [0x00007ff0dbf02000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@13/Native Method)
- parking to wait for <0x0000000718953ed8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@13/LockSupport.java:194)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@13/Abstrac
tQueuedSynchronizer.java:885)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@13/AbstractQueuedS
ynchronizer.java:917)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@13/AbstractQueuedSynchro
nizer.java:1240)
at java.util.concurrent.locks.ReentrantLock.lock(java.base@13/ReentrantLock.java:267)
at JUCLExample.getLockAndSleep(JUCLExample.java:19)
at JUCLExample.main(JUCLExample.java:15)
Thread that “holds” the lock
"Thread-0" #12 prio=5 os_prio=0 cpu=0.39ms elapsed=7.62s tid=0x00007ff0d4264000
nid=0x1b2f waiting on condition [0x00007ff0a45cc000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at JUCLExample.getLockAndSleep(JUCLExample.java:21)
at JUCLExample.access$100(JUCLExample.java:3)
at JUCLExample$Worker.run(JUCLExample.java:31)
at java.lang.Thread.run(java.base@13/Thread.java:830)
With great power comes great...
No thread really holds the lock
Only the developer knows what thread should unlock it
Thread that do the unlock may not even exist yet
That said...
Most designs do use j.u.c locks in a “symmetric” way
Often the thread that takes the lock is the one that releases it
We can keep track of which thread took each lock:
-XX:+PrintConcurrentLocks
"Thread-0" #12 prio=5 os_prio=0 cpu=0.40ms elapsed=18.75s tid=0x00007fc7b425c000
nid=0x1e65 waiting on condition [0x00007fc784ce5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@13/Native Method)
at JUCLExample.getLockAndSleep(JUCLExample.java:21)
at JUCLExample.access$100(JUCLExample.java:3)
at JUCLExample$Worker.run(JUCLExample.java:31)
at java.lang.Thread.run(java.base@13/Thread.java:830)
Locked ownable synchronizers:
- <0x0000000718953e98> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
PrintConcurrentLocks
Performance impact (examine heap on thread dump)
Tells you what thread took the lock, which may not be the thread
that will release it
Customer synchronizers built from j.u.c primitives may not be
tracked (should extend AbstractOwnableSynchronizer)
Can also be retrieved on-demand:
jhsdb jstack --pid <pid> --locks
Dump Analysis Tools
Samurai
Samurai
Lightweight
Software Stack Independent
Open Source
Spotify Online Java Thread Dump
Analyzer
Runs in your browser (no upload)
No install needed
Open Source
ThreadLogic
ThreadLogic
Tailored for WebLogic and Oracle
Fusion Middleware
Automatically identifies many
known issues
Based on Eclipse and TDA (Thread
Dump Analyzer) projects
Open Source
The Usual Suspects
Service Thread
"Service Thread" #9 daemon prio=9 os_prio=0
tid=0x00007fe4ac13f800 nid=0x3d3a runnable
[0x0000000000000000]
Since JDK 7
Handles
JVMTI events
Low Memory Detection
JMX event handling for diagnostic commands
Etc.
JIT Compiler Thread
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0
tid=0x00007fe4ac12c000 nid=0x3d39 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
• Does not run Bytecode
• Is a JavaThread for mostly historical reasons
Signal Dispatcher
"Signal Dispatcher" #4 daemon prio=9 os_prio=0
tid=0x00007fe4ac116800 nid=0x3d35 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
• sun.misc.SignalHandler
• Not for direct use
By Mark Wolfe - This image is from the FEMA Photo Library., Public Domain, https://commons.wikimedia.org/w/index.php?curid=8075137
Attach Listener
"Attach Listener" #10 daemon prio=9 os_prio=0
tid=0x00007f0520001000 nid=0x4660 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Finalizer
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f056c0ec000 nid=0x460e in Object.wait()
[0x00007f054d466000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076db88ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x000000076db88ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
Secondary finalizer
"Secondary finalizer" #10 prio=5 os_prio=0 tid=0x00007f908c154800
nid=0x4ff4 waiting on condition [0x00007f9074dc2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at FinStop.finalize(FinStop.java:22)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$2.run(Finalizer.java:161)
at java.lang.Thread.run(Thread.java:748)
Reference Handler
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f908c0e7800
nid=0x4feb in Object.wait() [0x00007f90756cb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c9200ae0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006c9200ae0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
By Giuseppe Arcimboldo - LSH 97136 (sm_dig12951), Public Domain, https://commons.wikimedia.org/w/index.php?curid=36899673
“main”
"main" #1 prio=5 os_prio=0 tid=0x00007ff04804b000
nid=0x6eca waiting on condition [0x00007ff04fbfd000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Sleeper.main(Sleeper.java:5)
VM Thread
"VM Thread" os_prio=0 tid=0x00007ff0480dd800
nid=0x6ed3 runnable
Stop-the-World thread
GC
Thread dumps
Bias revocation
VM Periodic Task Thread
"VM Periodic Task Thread" os_prio=0
tid=0x00007ff04813a000 nid=0x6edc waiting on
condition
Timer based events
Profiling
Sampling
By Ambrogio Lorenzetti - The Yorck Project (2002) 10.000 Meisterwerke der Malerei (DVD-ROM), distributed by DIRECTMEDIA Publishing GmbH. ISBN:
3936122202., Public Domain, https://commons.wikimedia.org/w/index.php?curid=3499217
GC Worker Threads
"GC task thread#0 (ParallelGC)" os_prio=0
tid=0x00007ff04805e000 nid=0x6ecb runnable
Depend on the collector currently in use
Number of threads may be chosen by ergonomics
Many different types
Common-Cleaner
"Common-Cleaner" #11 daemon prio=8 os_prio=0 cpu=0.13ms elapsed=2.49s tid=0x00007f8224254000
nid=0x766a in Object.wait() [0x00007f81f58f6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@13/Native Method)
- waiting on <0x0000000718945448> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@13/ReferenceQueue.java:155)
- locked <0x0000000718945448> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base@13/CleanerImpl.java:148)
at java.lang.Thread.run(java.base@13/Thread.java:830)
at jdk.internal.misc.InnocuousThread.run(java.base@13/InnocuousThread.java:134)
JDK 9 and later
Shared java.lang.ref.Cleaner thread
https://bugs.openjdk.java.net/browse/JDK-8145041
By CurranH (talk) (Uploads) - Own work, Public Domain, https://en.wikipedia.org/w/index.php?curid=11838492
Destroyer of Worlds
"DestroyJavaVM" #13 prio=5 os_prio=0 cpu=63.15ms
elapsed=3.19s tid=0x00007f951c017800 nid=0x4cfc
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Main has exited, but non-Deamon threads remain
By Jbuzbee - Own work, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=4835571
Takaways
Collect at least 3 dumps
Collect CPU usage with OS-level tool
top –H
Process Explorer
Enable j.u.concurrent lock output
Java Concurrency in Practice
Java Concurrency Bible
Covers JSR-133 / JSR-166
The preceding is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor
Copyright © 2019 Oracle and/or its affiliates.

More Related Content

What's hot

Future of Java EE with Java SE 8
Future of Java EE with Java SE 8Future of Java EE with Java SE 8
Future of Java EE with Java SE 8Hirofumi Iwasaki
 
Java EE 7: Boosting Productivity and Embracing HTML5
Java EE 7: Boosting Productivity and Embracing HTML5Java EE 7: Boosting Productivity and Embracing HTML5
Java EE 7: Boosting Productivity and Embracing HTML5Arun Gupta
 
Compile ahead of time. It's fine?
Compile ahead of time. It's fine?Compile ahead of time. It's fine?
Compile ahead of time. It's fine?Dmitry Chuyko
 
Head toward Java 14 and Java 15 #LINE_DM
Head toward Java 14 and Java 15 #LINE_DMHead toward Java 14 and Java 15 #LINE_DM
Head toward Java 14 and Java 15 #LINE_DMYuji Kubota
 
JavaOne Shanghai 2013 - Servlet 3.1 (JSR 340)
JavaOne Shanghai 2013 - Servlet 3.1 (JSR 340)JavaOne Shanghai 2013 - Servlet 3.1 (JSR 340)
JavaOne Shanghai 2013 - Servlet 3.1 (JSR 340)Shing Wai Chan
 
Flavors of Concurrency in Java
Flavors of Concurrency in JavaFlavors of Concurrency in Java
Flavors of Concurrency in JavaJavaDayUA
 
Managing Change
Managing ChangeManaging Change
Managing ChangeMirko Jahn
 
Head toward Java 14 and Java 15
Head toward Java 14 and Java 15Head toward Java 14 and Java 15
Head toward Java 14 and Java 15Yuji Kubota
 
Java 10 New Features
Java 10 New FeaturesJava 10 New Features
Java 10 New FeaturesAli BAKAN
 
The Java EE 7 Platform: Productivity & HTML5 at JavaOne Latin America 2012
The Java EE 7 Platform: Productivity & HTML5 at JavaOne Latin America 2012The Java EE 7 Platform: Productivity & HTML5 at JavaOne Latin America 2012
The Java EE 7 Platform: Productivity & HTML5 at JavaOne Latin America 2012Arun Gupta
 
CompletableFuture уже здесь
CompletableFuture уже здесьCompletableFuture уже здесь
CompletableFuture уже здесьDmitry Chuyko
 
Building Java Desktop Apps with JavaFX 8 and Java EE 7
Building Java Desktop Apps with JavaFX 8 and Java EE 7Building Java Desktop Apps with JavaFX 8 and Java EE 7
Building Java Desktop Apps with JavaFX 8 and Java EE 7Bruno Borges
 
The Web on OSGi: Here's How
The Web on OSGi: Here's HowThe Web on OSGi: Here's How
The Web on OSGi: Here's Howmrdon
 
You're Off the Hook: Blinding Security Software
You're Off the Hook: Blinding Security SoftwareYou're Off the Hook: Blinding Security Software
You're Off the Hook: Blinding Security SoftwareCylance
 
Challenge for GlassFish Builpack
Challenge for GlassFish BuilpackChallenge for GlassFish Builpack
Challenge for GlassFish BuilpackKenji Kazumura
 

What's hot (20)

Future of Java EE with Java SE 8
Future of Java EE with Java SE 8Future of Java EE with Java SE 8
Future of Java EE with Java SE 8
 
Java EE 7: Boosting Productivity and Embracing HTML5
Java EE 7: Boosting Productivity and Embracing HTML5Java EE 7: Boosting Productivity and Embracing HTML5
Java EE 7: Boosting Productivity and Embracing HTML5
 
Compile ahead of time. It's fine?
Compile ahead of time. It's fine?Compile ahead of time. It's fine?
Compile ahead of time. It's fine?
 
Head toward Java 14 and Java 15 #LINE_DM
Head toward Java 14 and Java 15 #LINE_DMHead toward Java 14 and Java 15 #LINE_DM
Head toward Java 14 and Java 15 #LINE_DM
 
Hotspot & AOT
Hotspot & AOTHotspot & AOT
Hotspot & AOT
 
What's new in Java 11
What's new in Java 11What's new in Java 11
What's new in Java 11
 
JavaOne Shanghai 2013 - Servlet 3.1 (JSR 340)
JavaOne Shanghai 2013 - Servlet 3.1 (JSR 340)JavaOne Shanghai 2013 - Servlet 3.1 (JSR 340)
JavaOne Shanghai 2013 - Servlet 3.1 (JSR 340)
 
Javaee6 Overview
Javaee6 OverviewJavaee6 Overview
Javaee6 Overview
 
Flavors of Concurrency in Java
Flavors of Concurrency in JavaFlavors of Concurrency in Java
Flavors of Concurrency in Java
 
Managing Change
Managing ChangeManaging Change
Managing Change
 
HotSpotコトハジメ
HotSpotコトハジメHotSpotコトハジメ
HotSpotコトハジメ
 
Head toward Java 14 and Java 15
Head toward Java 14 and Java 15Head toward Java 14 and Java 15
Head toward Java 14 and Java 15
 
Java 10 New Features
Java 10 New FeaturesJava 10 New Features
Java 10 New Features
 
The Java EE 7 Platform: Productivity & HTML5 at JavaOne Latin America 2012
The Java EE 7 Platform: Productivity & HTML5 at JavaOne Latin America 2012The Java EE 7 Platform: Productivity & HTML5 at JavaOne Latin America 2012
The Java EE 7 Platform: Productivity & HTML5 at JavaOne Latin America 2012
 
CompletableFuture уже здесь
CompletableFuture уже здесьCompletableFuture уже здесь
CompletableFuture уже здесь
 
Java 9 preview
Java 9 previewJava 9 preview
Java 9 preview
 
Building Java Desktop Apps with JavaFX 8 and Java EE 7
Building Java Desktop Apps with JavaFX 8 and Java EE 7Building Java Desktop Apps with JavaFX 8 and Java EE 7
Building Java Desktop Apps with JavaFX 8 and Java EE 7
 
The Web on OSGi: Here's How
The Web on OSGi: Here's HowThe Web on OSGi: Here's How
The Web on OSGi: Here's How
 
You're Off the Hook: Blinding Security Software
You're Off the Hook: Blinding Security SoftwareYou're Off the Hook: Blinding Security Software
You're Off the Hook: Blinding Security Software
 
Challenge for GlassFish Builpack
Challenge for GlassFish BuilpackChallenge for GlassFish Builpack
Challenge for GlassFish Builpack
 

Similar to Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [Code One 2019]

Java code coverage with JCov. Implementation details and use cases.
Java code coverage with JCov. Implementation details and use cases.Java code coverage with JCov. Implementation details and use cases.
Java code coverage with JCov. Implementation details and use cases.Alexandre (Shura) Iline
 
Shopzilla On Concurrency
Shopzilla On ConcurrencyShopzilla On Concurrency
Shopzilla On ConcurrencyRodney Barlow
 
Shopzilla On Concurrency
Shopzilla On ConcurrencyShopzilla On Concurrency
Shopzilla On ConcurrencyWill Gage
 
WebSockets: The Current State of the Most Valuable HTML5 API for Java Developers
WebSockets: The Current State of the Most Valuable HTML5 API for Java DevelopersWebSockets: The Current State of the Most Valuable HTML5 API for Java Developers
WebSockets: The Current State of the Most Valuable HTML5 API for Java DevelopersViktor Gamov
 
JVMs in Containers - Best Practices
JVMs in Containers - Best PracticesJVMs in Containers - Best Practices
JVMs in Containers - Best PracticesDavid Delabassee
 
20160221 va interconnect_pub
20160221 va interconnect_pub20160221 va interconnect_pub
20160221 va interconnect_pubCanturk Isci
 
Java EE 6 & GlassFish v3: Paving path for the future
Java EE 6 & GlassFish v3: Paving path for the futureJava EE 6 & GlassFish v3: Paving path for the future
Java EE 6 & GlassFish v3: Paving path for the futureArun Gupta
 
Java is Container Ready - Vaibhav - Container Conference 2018
Java is Container Ready - Vaibhav - Container Conference 2018Java is Container Ready - Vaibhav - Container Conference 2018
Java is Container Ready - Vaibhav - Container Conference 2018CodeOps Technologies LLP
 
Java è il linguaggio dell’IoT - Weaver
Java è il linguaggio dell’IoT - WeaverJava è il linguaggio dell’IoT - Weaver
Java è il linguaggio dell’IoT - WeaverCodemotion
 
NYC Java Meetup - Profiling and Performance
NYC Java Meetup - Profiling and PerformanceNYC Java Meetup - Profiling and Performance
NYC Java Meetup - Profiling and PerformanceJason Shao
 
Software Profiling: Understanding Java Performance and how to profile in Java
Software Profiling: Understanding Java Performance and how to profile in JavaSoftware Profiling: Understanding Java Performance and how to profile in Java
Software Profiling: Understanding Java Performance and how to profile in JavaIsuru Perera
 
Java API for WebSocket 1.0: Java EE 7 and GlassFish
Java API for WebSocket 1.0: Java EE 7 and GlassFishJava API for WebSocket 1.0: Java EE 7 and GlassFish
Java API for WebSocket 1.0: Java EE 7 and GlassFishArun Gupta
 
Case Studies and Lessons Learned from SSL/TLS Certificate Verification Vulner...
Case Studies and Lessons Learned from SSL/TLS Certificate Verification Vulner...Case Studies and Lessons Learned from SSL/TLS Certificate Verification Vulner...
Case Studies and Lessons Learned from SSL/TLS Certificate Verification Vulner...JPCERT Coordination Center
 
HTTP Security Headers Every Java Developer Must Know
HTTP Security Headers Every Java Developer Must KnowHTTP Security Headers Every Java Developer Must Know
HTTP Security Headers Every Java Developer Must KnowAyoma Wijethunga
 
Getting Started with WebSocket and Server-Sent Events using Java by Arun Gupta
Getting Started with WebSocket and Server-Sent Events using Java by Arun GuptaGetting Started with WebSocket and Server-Sent Events using Java by Arun Gupta
Getting Started with WebSocket and Server-Sent Events using Java by Arun GuptaCodemotion
 
Java EE 7 - Novidades e Mudanças
Java EE 7 - Novidades e MudançasJava EE 7 - Novidades e Mudanças
Java EE 7 - Novidades e MudançasBruno Borges
 

Similar to Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [Code One 2019] (20)

Java code coverage with JCov. Implementation details and use cases.
Java code coverage with JCov. Implementation details and use cases.Java code coverage with JCov. Implementation details and use cases.
Java code coverage with JCov. Implementation details and use cases.
 
Shopzilla On Concurrency
Shopzilla On ConcurrencyShopzilla On Concurrency
Shopzilla On Concurrency
 
Shopzilla On Concurrency
Shopzilla On ConcurrencyShopzilla On Concurrency
Shopzilla On Concurrency
 
WebSockets: The Current State of the Most Valuable HTML5 API for Java Developers
WebSockets: The Current State of the Most Valuable HTML5 API for Java DevelopersWebSockets: The Current State of the Most Valuable HTML5 API for Java Developers
WebSockets: The Current State of the Most Valuable HTML5 API for Java Developers
 
JVMs in Containers - Best Practices
JVMs in Containers - Best PracticesJVMs in Containers - Best Practices
JVMs in Containers - Best Practices
 
20160221 va interconnect_pub
20160221 va interconnect_pub20160221 va interconnect_pub
20160221 va interconnect_pub
 
Java EE 6 & GlassFish v3: Paving path for the future
Java EE 6 & GlassFish v3: Paving path for the futureJava EE 6 & GlassFish v3: Paving path for the future
Java EE 6 & GlassFish v3: Paving path for the future
 
Java is Container Ready - Vaibhav - Container Conference 2018
Java is Container Ready - Vaibhav - Container Conference 2018Java is Container Ready - Vaibhav - Container Conference 2018
Java is Container Ready - Vaibhav - Container Conference 2018
 
JVMs in Containers
JVMs in ContainersJVMs in Containers
JVMs in Containers
 
Java Cloud and Container Ready
Java Cloud and Container ReadyJava Cloud and Container Ready
Java Cloud and Container Ready
 
Java è il linguaggio dell’IoT - Weaver
Java è il linguaggio dell’IoT - WeaverJava è il linguaggio dell’IoT - Weaver
Java è il linguaggio dell’IoT - Weaver
 
NYC Java Meetup - Profiling and Performance
NYC Java Meetup - Profiling and PerformanceNYC Java Meetup - Profiling and Performance
NYC Java Meetup - Profiling and Performance
 
Software Profiling: Understanding Java Performance and how to profile in Java
Software Profiling: Understanding Java Performance and how to profile in JavaSoftware Profiling: Understanding Java Performance and how to profile in Java
Software Profiling: Understanding Java Performance and how to profile in Java
 
NIO-Grizly.pdf
NIO-Grizly.pdfNIO-Grizly.pdf
NIO-Grizly.pdf
 
Java API for WebSocket 1.0: Java EE 7 and GlassFish
Java API for WebSocket 1.0: Java EE 7 and GlassFishJava API for WebSocket 1.0: Java EE 7 and GlassFish
Java API for WebSocket 1.0: Java EE 7 and GlassFish
 
Case Studies and Lessons Learned from SSL/TLS Certificate Verification Vulner...
Case Studies and Lessons Learned from SSL/TLS Certificate Verification Vulner...Case Studies and Lessons Learned from SSL/TLS Certificate Verification Vulner...
Case Studies and Lessons Learned from SSL/TLS Certificate Verification Vulner...
 
HTTP Security Headers Every Java Developer Must Know
HTTP Security Headers Every Java Developer Must KnowHTTP Security Headers Every Java Developer Must Know
HTTP Security Headers Every Java Developer Must Know
 
Ah java-ppt1
Ah java-ppt1Ah java-ppt1
Ah java-ppt1
 
Getting Started with WebSocket and Server-Sent Events using Java by Arun Gupta
Getting Started with WebSocket and Server-Sent Events using Java by Arun GuptaGetting Started with WebSocket and Server-Sent Events using Java by Arun Gupta
Getting Started with WebSocket and Server-Sent Events using Java by Arun Gupta
 
Java EE 7 - Novidades e Mudanças
Java EE 7 - Novidades e MudançasJava EE 7 - Novidades e Mudanças
Java EE 7 - Novidades e Mudanças
 

More from David Buck

JDK 13 New Features [MeetUp with Java Experts! @Gaienmae/Dojima 2019]
JDK 13 New Features [MeetUp with Java Experts! @Gaienmae/Dojima 2019]JDK 13 New Features [MeetUp with Java Experts! @Gaienmae/Dojima 2019]
JDK 13 New Features [MeetUp with Java Experts! @Gaienmae/Dojima 2019]David Buck
 
JDK Mission Control: Where We Are, Where We Are Going [Groundbreakers APAC 20...
JDK Mission Control: Where We Are, Where We Are Going [Groundbreakers APAC 20...JDK Mission Control: Where We Are, Where We Are Going [Groundbreakers APAC 20...
JDK Mission Control: Where We Are, Where We Are Going [Groundbreakers APAC 20...David Buck
 
Z Garbage Collector
Z Garbage CollectorZ Garbage Collector
Z Garbage CollectorDavid Buck
 
Valhalla Update JJUG CCC Spring 2019
Valhalla Update JJUG CCC Spring 2019Valhalla Update JJUG CCC Spring 2019
Valhalla Update JJUG CCC Spring 2019David Buck
 
Var handles jjug_ccc_spring_2018
Var handles jjug_ccc_spring_2018Var handles jjug_ccc_spring_2018
Var handles jjug_ccc_spring_2018David Buck
 
JDK 10 へようこそ
JDK 10 へようこそJDK 10 へようこそ
JDK 10 へようこそDavid Buck
 
Java SE 8におけるHotSpotの進化 [Java Day Tokyo 2014 C-2]
Java SE 8におけるHotSpotの進化 [Java Day Tokyo 2014 C-2]Java SE 8におけるHotSpotの進化 [Java Day Tokyo 2014 C-2]
Java SE 8におけるHotSpotの進化 [Java Day Tokyo 2014 C-2]David Buck
 
HotSpot のロック: A Peek Under the Hood [JJUG ナイトセミナ JVM 特集 2015年8月]
HotSpot のロック: A Peek Under the Hood [JJUG ナイトセミナ  JVM 特集  2015年8月]HotSpot のロック: A Peek Under the Hood [JJUG ナイトセミナ  JVM 特集  2015年8月]
HotSpot のロック: A Peek Under the Hood [JJUG ナイトセミナ JVM 特集 2015年8月]David Buck
 
Java Concurrency, A(nother) Peek Under the Hood [JavaOne 2016 CON1497]
Java Concurrency, A(nother) Peek Under the Hood [JavaOne 2016 CON1497]Java Concurrency, A(nother) Peek Under the Hood [JavaOne 2016 CON1497]
Java Concurrency, A(nother) Peek Under the Hood [JavaOne 2016 CON1497]David Buck
 
Bytecode Verification, the Hero That Java Needs [JavaOne 2016 CON1500]
Bytecode Verification, the Hero That Java Needs [JavaOne 2016 CON1500]Bytecode Verification, the Hero That Java Needs [JavaOne 2016 CON1500]
Bytecode Verification, the Hero That Java Needs [JavaOne 2016 CON1500]David Buck
 
Java Debuggers: A Peek Under the Hood [JavaOne 2016 CON1503]
Java Debuggers: A Peek Under the Hood [JavaOne 2016 CON1503]Java Debuggers: A Peek Under the Hood [JavaOne 2016 CON1503]
Java Debuggers: A Peek Under the Hood [JavaOne 2016 CON1503]David Buck
 
Lambda: A Peek Under The Hood [Java Day Tokyo 2015 6-3]
Lambda: A Peek Under The Hood [Java Day Tokyo 2015 6-3]Lambda: A Peek Under The Hood [Java Day Tokyo 2015 6-3]
Lambda: A Peek Under The Hood [Java Day Tokyo 2015 6-3]David Buck
 
Java Concurrency, A(nother) Peek Under the Hood [Java Day Tokyo 2016 3-C]
Java Concurrency, A(nother) Peek Under the Hood [Java Day Tokyo 2016 3-C]Java Concurrency, A(nother) Peek Under the Hood [Java Day Tokyo 2016 3-C]
Java Concurrency, A(nother) Peek Under the Hood [Java Day Tokyo 2016 3-C]David Buck
 
Ahead-of-Time Compilation with JDK 9 [Java Day Tokyo 2017 D1-A1]
Ahead-of-Time Compilation with JDK 9 [Java Day Tokyo 2017 D1-A1]Ahead-of-Time Compilation with JDK 9 [Java Day Tokyo 2017 D1-A1]
Ahead-of-Time Compilation with JDK 9 [Java Day Tokyo 2017 D1-A1]David Buck
 
InvokeDynamic for Mere Mortals [JavaOne 2015 CON7682]
InvokeDynamic for Mere Mortals [JavaOne 2015 CON7682]InvokeDynamic for Mere Mortals [JavaOne 2015 CON7682]
InvokeDynamic for Mere Mortals [JavaOne 2015 CON7682]David Buck
 
HotSpot Synchronization, A Peek Under the Hood [JavaOne 2015 CON7570]
HotSpot Synchronization, A Peek Under the Hood [JavaOne 2015 CON7570]HotSpot Synchronization, A Peek Under the Hood [JavaOne 2015 CON7570]
HotSpot Synchronization, A Peek Under the Hood [JavaOne 2015 CON7570]David Buck
 
Let’s Write Our Own Chip-8 Interpreter! [JavaOne 2017 CON3584]
Let’s Write Our Own Chip-8 Interpreter! [JavaOne 2017 CON3584] Let’s Write Our Own Chip-8 Interpreter! [JavaOne 2017 CON3584]
Let’s Write Our Own Chip-8 Interpreter! [JavaOne 2017 CON3584] David Buck
 
Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...
Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...
Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...David Buck
 
Full Speed Ahead! (Ahead-of-Time Compilation for Java SE) [JavaOne 2017 CON3738]
Full Speed Ahead! (Ahead-of-Time Compilation for Java SE) [JavaOne 2017 CON3738]Full Speed Ahead! (Ahead-of-Time Compilation for Java SE) [JavaOne 2017 CON3738]
Full Speed Ahead! (Ahead-of-Time Compilation for Java SE) [JavaOne 2017 CON3738]David Buck
 
OpenJDK: How to Join In on All the Fun [JavaOne 2017 CON3667]
OpenJDK: How to Join In on All the Fun [JavaOne 2017 CON3667]OpenJDK: How to Join In on All the Fun [JavaOne 2017 CON3667]
OpenJDK: How to Join In on All the Fun [JavaOne 2017 CON3667]David Buck
 

More from David Buck (20)

JDK 13 New Features [MeetUp with Java Experts! @Gaienmae/Dojima 2019]
JDK 13 New Features [MeetUp with Java Experts! @Gaienmae/Dojima 2019]JDK 13 New Features [MeetUp with Java Experts! @Gaienmae/Dojima 2019]
JDK 13 New Features [MeetUp with Java Experts! @Gaienmae/Dojima 2019]
 
JDK Mission Control: Where We Are, Where We Are Going [Groundbreakers APAC 20...
JDK Mission Control: Where We Are, Where We Are Going [Groundbreakers APAC 20...JDK Mission Control: Where We Are, Where We Are Going [Groundbreakers APAC 20...
JDK Mission Control: Where We Are, Where We Are Going [Groundbreakers APAC 20...
 
Z Garbage Collector
Z Garbage CollectorZ Garbage Collector
Z Garbage Collector
 
Valhalla Update JJUG CCC Spring 2019
Valhalla Update JJUG CCC Spring 2019Valhalla Update JJUG CCC Spring 2019
Valhalla Update JJUG CCC Spring 2019
 
Var handles jjug_ccc_spring_2018
Var handles jjug_ccc_spring_2018Var handles jjug_ccc_spring_2018
Var handles jjug_ccc_spring_2018
 
JDK 10 へようこそ
JDK 10 へようこそJDK 10 へようこそ
JDK 10 へようこそ
 
Java SE 8におけるHotSpotの進化 [Java Day Tokyo 2014 C-2]
Java SE 8におけるHotSpotの進化 [Java Day Tokyo 2014 C-2]Java SE 8におけるHotSpotの進化 [Java Day Tokyo 2014 C-2]
Java SE 8におけるHotSpotの進化 [Java Day Tokyo 2014 C-2]
 
HotSpot のロック: A Peek Under the Hood [JJUG ナイトセミナ JVM 特集 2015年8月]
HotSpot のロック: A Peek Under the Hood [JJUG ナイトセミナ  JVM 特集  2015年8月]HotSpot のロック: A Peek Under the Hood [JJUG ナイトセミナ  JVM 特集  2015年8月]
HotSpot のロック: A Peek Under the Hood [JJUG ナイトセミナ JVM 特集 2015年8月]
 
Java Concurrency, A(nother) Peek Under the Hood [JavaOne 2016 CON1497]
Java Concurrency, A(nother) Peek Under the Hood [JavaOne 2016 CON1497]Java Concurrency, A(nother) Peek Under the Hood [JavaOne 2016 CON1497]
Java Concurrency, A(nother) Peek Under the Hood [JavaOne 2016 CON1497]
 
Bytecode Verification, the Hero That Java Needs [JavaOne 2016 CON1500]
Bytecode Verification, the Hero That Java Needs [JavaOne 2016 CON1500]Bytecode Verification, the Hero That Java Needs [JavaOne 2016 CON1500]
Bytecode Verification, the Hero That Java Needs [JavaOne 2016 CON1500]
 
Java Debuggers: A Peek Under the Hood [JavaOne 2016 CON1503]
Java Debuggers: A Peek Under the Hood [JavaOne 2016 CON1503]Java Debuggers: A Peek Under the Hood [JavaOne 2016 CON1503]
Java Debuggers: A Peek Under the Hood [JavaOne 2016 CON1503]
 
Lambda: A Peek Under The Hood [Java Day Tokyo 2015 6-3]
Lambda: A Peek Under The Hood [Java Day Tokyo 2015 6-3]Lambda: A Peek Under The Hood [Java Day Tokyo 2015 6-3]
Lambda: A Peek Under The Hood [Java Day Tokyo 2015 6-3]
 
Java Concurrency, A(nother) Peek Under the Hood [Java Day Tokyo 2016 3-C]
Java Concurrency, A(nother) Peek Under the Hood [Java Day Tokyo 2016 3-C]Java Concurrency, A(nother) Peek Under the Hood [Java Day Tokyo 2016 3-C]
Java Concurrency, A(nother) Peek Under the Hood [Java Day Tokyo 2016 3-C]
 
Ahead-of-Time Compilation with JDK 9 [Java Day Tokyo 2017 D1-A1]
Ahead-of-Time Compilation with JDK 9 [Java Day Tokyo 2017 D1-A1]Ahead-of-Time Compilation with JDK 9 [Java Day Tokyo 2017 D1-A1]
Ahead-of-Time Compilation with JDK 9 [Java Day Tokyo 2017 D1-A1]
 
InvokeDynamic for Mere Mortals [JavaOne 2015 CON7682]
InvokeDynamic for Mere Mortals [JavaOne 2015 CON7682]InvokeDynamic for Mere Mortals [JavaOne 2015 CON7682]
InvokeDynamic for Mere Mortals [JavaOne 2015 CON7682]
 
HotSpot Synchronization, A Peek Under the Hood [JavaOne 2015 CON7570]
HotSpot Synchronization, A Peek Under the Hood [JavaOne 2015 CON7570]HotSpot Synchronization, A Peek Under the Hood [JavaOne 2015 CON7570]
HotSpot Synchronization, A Peek Under the Hood [JavaOne 2015 CON7570]
 
Let’s Write Our Own Chip-8 Interpreter! [JavaOne 2017 CON3584]
Let’s Write Our Own Chip-8 Interpreter! [JavaOne 2017 CON3584] Let’s Write Our Own Chip-8 Interpreter! [JavaOne 2017 CON3584]
Let’s Write Our Own Chip-8 Interpreter! [JavaOne 2017 CON3584]
 
Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...
Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...
Everything You Wanted to Know About JIT Compilation but Were Afraid to Ask [J...
 
Full Speed Ahead! (Ahead-of-Time Compilation for Java SE) [JavaOne 2017 CON3738]
Full Speed Ahead! (Ahead-of-Time Compilation for Java SE) [JavaOne 2017 CON3738]Full Speed Ahead! (Ahead-of-Time Compilation for Java SE) [JavaOne 2017 CON3738]
Full Speed Ahead! (Ahead-of-Time Compilation for Java SE) [JavaOne 2017 CON3738]
 
OpenJDK: How to Join In on All the Fun [JavaOne 2017 CON3667]
OpenJDK: How to Join In on All the Fun [JavaOne 2017 CON3667]OpenJDK: How to Join In on All the Fun [JavaOne 2017 CON3667]
OpenJDK: How to Join In on All the Fun [JavaOne 2017 CON3667]
 

Recently uploaded

Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
Vectors are the new JSON in PostgreSQL (SCaLE 21x)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)Vectors are the new JSON in PostgreSQL (SCaLE 21x)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)Jonathan Katz
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Understanding Native Mobile App Development
Understanding Native Mobile App DevelopmentUnderstanding Native Mobile App Development
Understanding Native Mobile App DevelopmentMobulous Technologies
 
New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024ThousandEyes
 
React 19: Revolutionizing Web Development
React 19: Revolutionizing Web DevelopmentReact 19: Revolutionizing Web Development
React 19: Revolutionizing Web DevelopmentBOSC Tech Labs
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 

Recently uploaded (20)

Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
Vectors are the new JSON in PostgreSQL (SCaLE 21x)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)Vectors are the new JSON in PostgreSQL (SCaLE 21x)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Understanding Native Mobile App Development
Understanding Native Mobile App DevelopmentUnderstanding Native Mobile App Development
Understanding Native Mobile App Development
 
New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024
 
React 19: Revolutionizing Web Development
React 19: Revolutionizing Web DevelopmentReact 19: Revolutionizing Web Development
React 19: Revolutionizing Web Development
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 

Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [Code One 2019]

  • 1. Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [DEV4356] Principal Member of Technical Staff Java Platform Group September 17, 2019 David Buck Copyright © 2019 Oracle and/or its affiliates.
  • 2. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
  • 3. JVM Sustaining Engineer OpenJDK Updates Project Maintainer JavaOne Rock Star Co-author of Oracle WebLogic Server 11g 構築・運用ガイド @DavidBuckJP https://blogs.oracle.com/buck/ Who am I? David Buck (left)
  • 5. Java 1.0 1996 Threading as a First Class Member of the Language Java 1.2 1998 Deprecation of asynchronous Thread methods Java 1.4 2002 NIO (non-blocking IO) Java 5 2004 Java.util.concurrent Improved Memory Model Java 7 2011 ForkJoin Java 8 2014 Lambda Expressions
  • 6. Java 9 2017 Variable Handles CompletableFuture API improvements Future Project Loom
  • 7. Multithreading as Part of the Language 7
  • 8. So, what exactly is a Monitor? 8
  • 12. condition variable 12 "Puffin crossing, London, UK" by secretlondon is licensed under CC BY-SA 3.0
  • 13. Monitor = Mutex + Condition Variable 13
  • 14. Monitor = Mutex + Condition Variable Mutex synchronized keyword 14
  • 15. Monitor = Mutex + Condition Variable Mutex synchronized keyword Condition Variable Object.wait() Object.notify() Object.notifyAll() 15
  • 17. Java Monitor Limitations No way to check status of a lock 17
  • 18. Java Monitor Limitations No way to check status of a lock No timeout 18
  • 19. Java Monitor Limitations No way to check status of a lock No timeout No way to cancel 19
  • 20. Java Monitor Limitations No way to check status of a lock No timeout No way to cancel Must be recursive 20
  • 21. Java Monitor Limitations No way to check status of a lock No timeout No way to cancel Must be recursive No reader / writer locking 21
  • 22. Java Monitor Limitations No way to check status of a lock No timeout No way to cancel Must be recursive No reader / writer locking Security Issues 22
  • 26. Modern Threads (HotSpot) 1 to 1 (Everyone else)Many to Many (Solaris)
  • 28. JavaThreads Run Java (Byte)code Stop during STW Always show up in thread dump Many aliases: Execution Thread Mutator Thread Other Threads Runs non-Java code May run during STW May not show up in thread dump JVM internal threads GC Worker Threads JIT Compiler Threads Non-JVM threads Primordial thread Other user threads (JNI application)
  • 29. Primordial Thread First JVM thread created by OS Poor control over stack setup (size, guard pages, etc.) Poor visibility of stack Should not be used to run Java Code Java launcher “abandons” primordial thread (JDK >= 6) JNI invocation code should do the same
  • 30. Thread Dumps A snapshot of all Java Threads (and some JVM Threads) Stop-the-World operation Some non-thread info thrown in for good measure Many ways to collect Slowly evolving “format” Many tools to help with analysis
  • 31. Snapshots One dump alone can not show “motion” or change Each dump is mostly self- consistent Stop-the-World pause during collection By FlanellKamerasFilm - Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=68147463
  • 32. Is this train moving? By Shiodome - ja:image:Mono-shibaura.JPG, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1157483
  • 33. Thread Uncertainty Principle You can tell where a thread is (at what safepoint) You can tell how fast a thread is “moving” (CPU consumption) You can’t tell both at the same time But you don’t really need to Collect CPU consumption data along with thread dumps Collect multiple thread dumps
  • 34. Safepoints Stop-the-World implementation Guarantees Java state of thread is known All threads executing Java code must stop JNI code can continue Cooperative suspension Polling based By YPLeroux at English Wikipedia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=17734859
  • 35. Safepoints Stop-the-World implementation Guarantees Java state of thread is known All threads executing Java code must stop JNI code can continue Cooperative suspension Polling based By YPLeroux at English Wikipedia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=17734859
  • 36. Safepoint Polling Interpreter can stop instantly template dispatch table replaced “polling” is essentially free Explicit poll inserted into C1/C2 JIT output Global safepoint polling page in address space Poll implemented by attempted read of this address Safepoint triggered (SIGSEGV) by change in page permissions
  • 37. JIT Safepoint Polling On return from method call Backend branches in most loops Any transition to native code (JVM / JNI)
  • 38. Not quite a snapshot Each thread stops at its next safepoint Not all threads will stop at the same time A critical section might not even have a safepoint Not unusual to see thread blocking for a lock no one holds More like a “panorama mode” picture: expect inconsistencies
  • 39. Ways to Get a Thread Dump SIGQUIT / CTL-BRK handler Diagnostic command (jcmd) Java Flight recording periodic event jstack tool
  • 40. jstack Force thread dump (might work) Mixed mode (might work) Thread dump from process or core file Integrated into servicabililty agent as of JDK 9
  • 42. Thread Dump Output Header Timestamp JVM version information safe memory reclamation (SMR) data Thread List All attached Java Threads Most Native threads Footer JNI global references Memory usage statistics
  • 43. Timestamp 2019-09-07 03:00:18 Useful to sanity check timing dump was collected Makes interpreting per-thread stats much easier
  • 44. Version Information Full thread dump Java HotSpot(TM) 64-Bit Server VM (13+33 mixed mode, sharing) Similar to “java –version” output Sharing -> class data sharing enabled JIT compiler state mixed mode interpreted mode (-Xint) compiled mode (-Xcomp)
  • 45. Version Information Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.221-b11 mixed mode): JDK 8 and earlier use HotSpot Express version number HotSpot == 25.x -> JDK 8 HotSpot <= 24 -> Google it…
  • 46. Safe Memory Reclamation Threads class SMR info: _java_thread_list=0x00007f4e842e9a60, length=6, elements={ 0x00007f4e84017800, 0x00007f4e84232800, 0x00007f4e84236000, 0x00007f4e84257000, 0x00007f4e842da800, 0x00007f4e842e8000 } tid of each JavaThread JVM internal information (can be ignored) Added in JDK 9
  • 47. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 48. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 49. What’s in a name? Arbitrary string Thread.setName(); Can be changed as often as you want Name can be used as a debugging tool A thread by any other name... By Photo: JLPC / Wikimedia Commons, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=30151561
  • 50. What’s in a name? "[STUCK] ExecuteThread: '139' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fd2a00b6000 nid=0x690b runnable [0x00007fd0ecbc7000] java.lang.Thread.State: RUNNABLE at java.io.ObjectOutputStream$BlockDataOutputStream.getUTFLength(ObjectOutputStream.java:2151)
  • 51. What’s in a name? "[STUCK] ExecuteThread: '139' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fd2a00b6000 nid=0x690b runnable [0x00007fd0ecbc7000] java.lang.Thread.State: RUNNABLE at java.io.ObjectOutputStream$BlockDataOutputStream.getUTFLength(ObjectOutputStream.java:2151)
  • 52. What’s in a name? theThread.setName("[really cool] " + theThread.getName()); System.out.println("I made your thread cool!"); < do cool stuff > theThread.setName(myThread.getName().substring(14)); System.out.println("Your thread is no longer cool!");
  • 57. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 58. Thread ID Thread.getID() Automatically assigned by JVM during thread creation Read-Only Consistent throughout thread lifetime
  • 59. Daemon Thread "child thread" #12 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=1.98s tid=0x00007fec48278800 nid=0x179 waiting on condition [0x00007fec18df2000] java.lang.Thread.State: TIMED_WAITING (sleeping)
  • 60. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 61. Java Thread Priority Thread.getPriority() Thread.setPriority(int) Min is usually 1 Max is usually 10 Ignored by many JVMs 10 9 8 7 3 2 1 4 5 6
  • 62. Java Thread Priority Thread.getPriority() Thread.setPriority(int) Min is usually 1 Max is usually 10 Ignored by many JVMs 10 9 8 7 3 2 1 4 5 6 Turtles all the way down!!!
  • 63. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 64. OS Thread Priority Not directly user controllable Interpretation is platform dependent
  • 65. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 66. -XX:+PrintExtendedThreadInfo "main" #1 prio=5 os_prio=0 cpu=58.42ms elapsed=2.60s allocated=289K defined_classes=4 tid=0x00007f8138017800 nid=0x344f waiting on condition [0x00007f8141890000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 67. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 68. tid Address of HotSpot data structure used to track thread
  • 69. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 70. Safe Memory Reclamation Threads class SMR info: _java_thread_list=0x00007f4e842e9a60, length=6, elements={ 0x00007f4e84017800, 0x00007f4e84232800, 0x00007f4e84236000, 0x00007f4e84257000, 0x00007f4e842da800, 0x00007f4e842e8000 }
  • 71. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 72. nid “native” thread identifier Useful for using OS tools (e.g. gdb, top -H) TD output in Hexadecimal, many OS tools use Decimal
  • 73. nid Linux: kernel thread id from syscall gettid Solaris: thread library identity from thr_create() or thr_self() Windows: thread id from _beginthreadex macOS: kernel thread id from pthread_mach_thread_np(pthread_self()); Other BSD: kernel (?) thread id from syscall thr_self or getthrid AIX: thread library identity from pthread_create() or pthread_self()
  • 74. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 75. Thread state (native) runnable waiting for monitor entry waiting on condition in Object.wait()
  • 76. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 77. Last Java Frame Stack address of most recent Java frame on stack Stack pointer (sp) may be much farther down Null if there are no Java frames "C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=12.22ms elapsed=2.44s tid=0x00007f60781fd800 nid=0x37dc waiting on condition [0x0000000000000000]
  • 78. Last Java Frame Stack address of most recent Java frame on stack Stack pointer (sp) may be much farther down Null if there are no Java frames "C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=12.22ms elapsed=2.44s tid=0x00007f60781fd800 nid=0x37dc waiting on condition [0x0000000000000000]
  • 79. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 80. java.lang.Thread.State NEW A thread that has not yet started is in this state. RUNNABLE A thread executing in the Java virtual machine is in this state. BLOCKED A thread that is blocked waiting for a monitor lock is in this state. WAITING A thread that is waiting indefinitely for another thread to perform a particular action is in this state. TIMED_WAITING A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state. TERMINATED A thread that has exited is in this state.
  • 81. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 82. Non-Java Thread "VM Thread" os_prio=0 cpu=2.63ms elapsed=2.47s tid=0x00007f0e841ef800 nid=0x3c41 runnable No Java thread ID (e.g. #42) No Java priority (e.g. prio=5) No last Java frame (e.g. [0x00007f4e8c82c000]) No java.lang.Thread.State No Java stack
  • 83. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  • 84. JNI Global References JNI global refs: 6, weak refs: 0 JNI global reference leaks are a common bug Investigate any significant growth post app warmup By John Newton and William Palmer, - A terrestial globe on which the tracts and discoveries are laid down from the accurate observations made by Captains Cook, Furneux, Phipps, published May 9th, 1782 globe by John Newton, cartography by William Palmer, Sate Library of New South Wales https://primo-slnsw.hosted.exlibrisgroup.com/primo-explore/fulldisplay?docid=ADLIB110577047&context=L&vid=SLNSW&search_scope=EEA&tab=default_tab&lang=en_US, Public Domain, https://commons.wikimedia.org/w/index.php?curid=55030411
  • 85. JNI Global References JNI global refs: 6, weak refs: 0 JNI global reference leaks are a common bug Investigate any significant growth post app warmup Weak Global Reference breakout is new (JDK 11) Even if referent is collected, memory is leaked unless DeleteWeakGlobalRef is called
  • 86. Heap Stats Heap garbage-first heap total 256000K, used 776K [0x0000000709200000, 0x0000000800000000) region size 1024K, 1 young (1024K), 0 survivors (0K) Metaspace used 161K, capacity 4486K, committed 4864K, reserved 1056768K class space used 7K, capacity 386K, committed 512K, reserved 1048576K
  • 88. Runnable "Thread-0" #12 prio=5 os_prio=0 cpu=1489.99ms elapsed=1.49s tid=0x00007f5c78262000 nid=0xa6f runnable [0x00007f5c497f4000] java.lang.Thread.State: RUNNABLE at LockTest$1.run(LockTest.java:6) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 89. Blocked on a Monitor "Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000 nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker1.run(LockChain.java:28) - waiting to lock <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 90. Blocked on a Monitor "Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000 nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker1.run(LockChain.java:28) - waiting to lock <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 91. Blocked on a Monitor "Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000 nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker1.run(LockChain.java:28) - waiting to lock <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 92. Who has my lock? "Thread-5" #17 prio=5 os_prio=0 cpu=0.85ms elapsed=4.58s tid=0x00007fe2dc26c800 nid=0x239 waiting for monitor entry [0x00007fe2acff5000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker2.run(LockChain.java:39) - waiting to lock <0x0000000718948ef0> (a java.lang.Object) - locked <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 93. The plot thickens... "Thread-5" #17 prio=5 os_prio=0 cpu=0.85ms elapsed=4.58s tid=0x00007fe2dc26c800 nid=0x239 waiting for monitor entry [0x00007fe2acff5000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker2.run(LockChain.java:39) - waiting to lock <0x0000000718948ef0> (a java.lang.Object) - locked <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 94. End of the road... "Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000 nid=0x23b waiting on condition [0x00007fe2acdf3000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at LockChain$Worker3.run(LockChain.java:51) - locked <0x0000000718948ef0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 95. Thread A Thread E Thread D Thread B Thread C
  • 96. Thread A Thread F Thread D Thread B Thread C Thread E Thread G
  • 97. Thread A Thread B Holds Lock 1 Waiting on Lock 2 Holds Lock 2 Waiting on Lock 1
  • 98. Deadlock Restarting JVM only way to recover All threads should acquire locks in well-defined order Most JVMs / tools can identify trivial deadlocks
  • 99. Java stack information for the threads listed above: =================================================== "Thread-0": at DL$Thread1.run(DL.java:18) - waiting to lock <0x0000000718949078> (a java.lang.Object) - locked <0x0000000718949068> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830) "Thread-1": at DL$Thread2.run(DL.java:30) - waiting to lock <0x0000000718949068> (a java.lang.Object) - locked <0x0000000718949078> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830) Found 1 deadlock.
  • 100. Intrinsic lock (monitor) issues Easy to identify in thread dumps Deep understanding of application code not needed for identification Amenable to automation (thread dump analysis tools)
  • 101. Waiting on a monitor "Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800 nid=0xba2 in Object.wait() [0x00007f670d9d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718949e30> (a Drop) at java.lang.Object.wait(java.base@13/Object.java:326) at Drop.take(Drop.java:47) - locked <0x0000000718949e30> (a Drop) at Consumer.run(Consumer.java:45) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 102. Wait and Notify Used to signal other threads to wake up and work Producer / Consumer is the textbook use-case Thread must hold the lock before calling wait or notify Otherwise IllegalMonitorStateException thrown
  • 103. Wait and Notify Thread A locks object Thread A calls Object.wait() Thread B locks object Thread B calls Object.notify() Thread B releases lock Thread A wakes up and continues working
  • 104. Wait and Notify Thread A locks object Thread A calls Object.wait() (lock released!) Thread B locks object Thread B calls Object.notify() Thread B releases lock Thread A wakes up and continues working
  • 105. Waiting on a monitor "Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800 nid=0xba2 in Object.wait() [0x00007f670d9d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718949e30> (a Drop) at java.lang.Object.wait(java.base@13/Object.java:326) at Drop.take(Drop.java:47) - locked <0x0000000718949e30> (a Drop) at Consumer.run(Consumer.java:45) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 106. Waiting on a monitor "Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800 nid=0xba2 in Object.wait() [0x00007f670d9d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718949e30> (a Drop) at java.lang.Object.wait(java.base@13/Object.java:326) at Drop.take(Drop.java:47) - locked <0x0000000718949e30> (a Drop) at Consumer.run(Consumer.java:45) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 107. Waiting on a monitor "Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800 nid=0xba2 in Object.wait() [0x00007f670d9d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718949e30> (a Drop) at java.lang.Object.wait(java.base@13/Object.java:326) at Drop.take(Drop.java:47) - locked <0x0000000718949e30> (a Drop) at Consumer.run(Consumer.java:45) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 108. Wait / Notify Intrinsic lock troubleshooting pattern not very helpful Knowledge of application design needed to know what thread should call notify() Analysis not amenable to automation Having a bunch of threads waiting might be normal under low load
  • 109. java.util.concurrent More flexible than intrinsic locks (monitors) Harder to debug Plenty of rope to hang yourself
  • 110. Intrinsic locks void methodA() { synchronized(mySpiffyObject) { <do stuff> methodB(); } }
  • 111. Intrinsic locks void synchronized methodA() { methodB(); }
  • 112. Intrinsic locks "Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000 nid=0x23b waiting on condition [0x00007fe2acdf3000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at LockChain$Worker3.run(LockChain.java:51) - locked <0x0000000718948ef0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 113. java.util.concurrent Lock does not necessarily belong to a frame or even a thread A lock might be taken and released by different methods A lock might be taken and released by different threads
  • 114. Intrinsic locks "Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000 nid=0x23b waiting on condition [0x00007fe2acdf3000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at LockChain$Worker3.run(LockChain.java:51) - locked <0x0000000718948ef0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 115. import java.util.concurrent.locks.ReentrantLock; <…> private static final ReentrantLock lock = new ReentrantLock();
  • 116. private static void getLockAndSleep() { lock.lock(); try { Thread.sleep(1000 * 1000); } catch (Exception e) { } finally { lock.unlock(); } }
  • 117. private static class Worker implements Runnable { public void run() { getLockAndSleep(); } }
  • 118. public static void main(String[] args) throws InterruptedException { Thread childThread = new Thread(new Worker()); childThread.start(); Thread.sleep(1000 * 3); // 3 seconds System.out.println("main thread will try to get lock..."); getLockAndSleep(); }
  • 119. "main" #1 prio=5 os_prio=0 cpu=45.91ms elapsed=7.66s tid=0x00007ff0d4017800 nid=0x1b1f waiting on condition [0x00007ff0dbf02000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@13/Native Method) - parking to wait for <0x0000000718953ed8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(java.base@13/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@13/Abstrac tQueuedSynchronizer.java:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@13/AbstractQueuedS ynchronizer.java:917) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@13/AbstractQueuedSynchro nizer.java:1240) at java.util.concurrent.locks.ReentrantLock.lock(java.base@13/ReentrantLock.java:267) at JUCLExample.getLockAndSleep(JUCLExample.java:19) at JUCLExample.main(JUCLExample.java:15)
  • 120. Thread that “holds” the lock "Thread-0" #12 prio=5 os_prio=0 cpu=0.39ms elapsed=7.62s tid=0x00007ff0d4264000 nid=0x1b2f waiting on condition [0x00007ff0a45cc000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at JUCLExample.getLockAndSleep(JUCLExample.java:21) at JUCLExample.access$100(JUCLExample.java:3) at JUCLExample$Worker.run(JUCLExample.java:31) at java.lang.Thread.run(java.base@13/Thread.java:830)
  • 121. With great power comes great... No thread really holds the lock Only the developer knows what thread should unlock it Thread that do the unlock may not even exist yet
  • 122. That said... Most designs do use j.u.c locks in a “symmetric” way Often the thread that takes the lock is the one that releases it We can keep track of which thread took each lock: -XX:+PrintConcurrentLocks
  • 123. "Thread-0" #12 prio=5 os_prio=0 cpu=0.40ms elapsed=18.75s tid=0x00007fc7b425c000 nid=0x1e65 waiting on condition [0x00007fc784ce5000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at JUCLExample.getLockAndSleep(JUCLExample.java:21) at JUCLExample.access$100(JUCLExample.java:3) at JUCLExample$Worker.run(JUCLExample.java:31) at java.lang.Thread.run(java.base@13/Thread.java:830) Locked ownable synchronizers: - <0x0000000718953e98> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  • 124. PrintConcurrentLocks Performance impact (examine heap on thread dump) Tells you what thread took the lock, which may not be the thread that will release it Customer synchronizers built from j.u.c primitives may not be tracked (should extend AbstractOwnableSynchronizer) Can also be retrieved on-demand: jhsdb jstack --pid <pid> --locks
  • 128. Spotify Online Java Thread Dump Analyzer Runs in your browser (no upload) No install needed Open Source
  • 130. ThreadLogic Tailored for WebLogic and Oracle Fusion Middleware Automatically identifies many known issues Based on Eclipse and TDA (Thread Dump Analyzer) projects Open Source
  • 132. Service Thread "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fe4ac13f800 nid=0x3d3a runnable [0x0000000000000000] Since JDK 7 Handles JVMTI events Low Memory Detection JMX event handling for diagnostic commands Etc.
  • 133. JIT Compiler Thread "C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007fe4ac12c000 nid=0x3d39 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE • Does not run Bytecode • Is a JavaThread for mostly historical reasons
  • 134. Signal Dispatcher "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fe4ac116800 nid=0x3d35 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE • sun.misc.SignalHandler • Not for direct use By Mark Wolfe - This image is from the FEMA Photo Library., Public Domain, https://commons.wikimedia.org/w/index.php?curid=8075137
  • 135. Attach Listener "Attach Listener" #10 daemon prio=9 os_prio=0 tid=0x00007f0520001000 nid=0x4660 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE
  • 136. Finalizer "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f056c0ec000 nid=0x460e in Object.wait() [0x00007f054d466000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x000000076db88ed8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked <0x000000076db88ed8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
  • 137. Secondary finalizer "Secondary finalizer" #10 prio=5 os_prio=0 tid=0x00007f908c154800 nid=0x4ff4 waiting on condition [0x00007f9074dc2000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at FinStop.finalize(FinStop.java:22) at java.lang.System$2.invokeFinalize(System.java:1270) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102) at java.lang.ref.Finalizer.access$100(Finalizer.java:34) at java.lang.ref.Finalizer$2.run(Finalizer.java:161) at java.lang.Thread.run(Thread.java:748)
  • 138. Reference Handler "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f908c0e7800 nid=0x4feb in Object.wait() [0x00007f90756cb000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000006c9200ae0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x00000006c9200ae0> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) By Giuseppe Arcimboldo - LSH 97136 (sm_dig12951), Public Domain, https://commons.wikimedia.org/w/index.php?curid=36899673
  • 139. “main” "main" #1 prio=5 os_prio=0 tid=0x00007ff04804b000 nid=0x6eca waiting on condition [0x00007ff04fbfd000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at Sleeper.main(Sleeper.java:5)
  • 140. VM Thread "VM Thread" os_prio=0 tid=0x00007ff0480dd800 nid=0x6ed3 runnable Stop-the-World thread GC Thread dumps Bias revocation
  • 141. VM Periodic Task Thread "VM Periodic Task Thread" os_prio=0 tid=0x00007ff04813a000 nid=0x6edc waiting on condition Timer based events Profiling Sampling By Ambrogio Lorenzetti - The Yorck Project (2002) 10.000 Meisterwerke der Malerei (DVD-ROM), distributed by DIRECTMEDIA Publishing GmbH. ISBN: 3936122202., Public Domain, https://commons.wikimedia.org/w/index.php?curid=3499217
  • 142. GC Worker Threads "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ff04805e000 nid=0x6ecb runnable Depend on the collector currently in use Number of threads may be chosen by ergonomics Many different types
  • 143. Common-Cleaner "Common-Cleaner" #11 daemon prio=8 os_prio=0 cpu=0.13ms elapsed=2.49s tid=0x00007f8224254000 nid=0x766a in Object.wait() [0x00007f81f58f6000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718945448> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@13/ReferenceQueue.java:155) - locked <0x0000000718945448> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run(java.base@13/CleanerImpl.java:148) at java.lang.Thread.run(java.base@13/Thread.java:830) at jdk.internal.misc.InnocuousThread.run(java.base@13/InnocuousThread.java:134) JDK 9 and later Shared java.lang.ref.Cleaner thread https://bugs.openjdk.java.net/browse/JDK-8145041 By CurranH (talk) (Uploads) - Own work, Public Domain, https://en.wikipedia.org/w/index.php?curid=11838492
  • 144. Destroyer of Worlds "DestroyJavaVM" #13 prio=5 os_prio=0 cpu=63.15ms elapsed=3.19s tid=0x00007f951c017800 nid=0x4cfc waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Main has exited, but non-Deamon threads remain By Jbuzbee - Own work, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=4835571
  • 145. Takaways Collect at least 3 dumps Collect CPU usage with OS-level tool top –H Process Explorer Enable j.u.concurrent lock output
  • 146. Java Concurrency in Practice Java Concurrency Bible Covers JSR-133 / JSR-166
  • 147. The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.