Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [Code One 2019]

Why aren’t your threads behaving? Although most developers are confident about handling better-known issues such as deadlocks and simple hangs, many people still come across liveness issues that are not always easy to identify and resolve. From complete hangs that bring your application to a halt, to scalability issues, there are all sorts of things that go bump in the night. This session explores the most common types of things that can prevent your Java threads from getting work done and shows how to address them.

  • Login to see the comments

Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [Code One 2019]

  1. 1. Hangs, Slowdowns, Starvation—Oh My! A Deep Dive into the Life of a Java Thread [DEV4356] Principal Member of Technical Staff Java Platform Group September 17, 2019 David Buck Copyright © 2019 Oracle and/or its affiliates.
  2. 2. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
  3. 3. JVM Sustaining Engineer OpenJDK Updates Project Maintainer JavaOne Rock Star Co-author of Oracle WebLogic Server 11g 構築・運用ガイド @DavidBuckJP https://blogs.oracle.com/buck/ Who am I? David Buck (left)
  4. 4. History
  5. 5. Java 1.0 1996 Threading as a First Class Member of the Language Java 1.2 1998 Deprecation of asynchronous Thread methods Java 1.4 2002 NIO (non-blocking IO) Java 5 2004 Java.util.concurrent Improved Memory Model Java 7 2011 ForkJoin Java 8 2014 Lambda Expressions
  6. 6. Java 9 2017 Variable Handles CompletableFuture API improvements Future Project Loom
  7. 7. Multithreading as Part of the Language 7
  8. 8. So, what exactly is a Monitor? 8
  9. 9. Mutual Exclusion 9
  10. 10. Mutual Exclusion Mut ex 10
  11. 11. Mutual Exclusion Mutex 11
  12. 12. condition variable 12 "Puffin crossing, London, UK" by secretlondon is licensed under CC BY-SA 3.0
  13. 13. Monitor = Mutex + Condition Variable 13
  14. 14. Monitor = Mutex + Condition Variable Mutex synchronized keyword 14
  15. 15. Monitor = Mutex + Condition Variable Mutex synchronized keyword Condition Variable Object.wait() Object.notify() Object.notifyAll() 15
  16. 16. Java Monitor Limitations 16
  17. 17. Java Monitor Limitations No way to check status of a lock 17
  18. 18. Java Monitor Limitations No way to check status of a lock No timeout 18
  19. 19. Java Monitor Limitations No way to check status of a lock No timeout No way to cancel 19
  20. 20. Java Monitor Limitations No way to check status of a lock No timeout No way to cancel Must be recursive 20
  21. 21. Java Monitor Limitations No way to check status of a lock No timeout No way to cancel Must be recursive No reader / writer locking 21
  22. 22. Java Monitor Limitations No way to check status of a lock No timeout No way to cancel Must be recursive No reader / writer locking Security Issues 22
  23. 23. java.util.concurrent 23
  24. 24. Threads
  25. 25. Green Threads (Pre-HotSpot)
  26. 26. Modern Threads (HotSpot) 1 to 1 (Everyone else)Many to Many (Solaris)
  27. 27. Threads JavaThreads
  28. 28. JavaThreads Run Java (Byte)code Stop during STW Always show up in thread dump Many aliases: Execution Thread Mutator Thread Other Threads Runs non-Java code May run during STW May not show up in thread dump JVM internal threads GC Worker Threads JIT Compiler Threads Non-JVM threads Primordial thread Other user threads (JNI application)
  29. 29. Primordial Thread First JVM thread created by OS Poor control over stack setup (size, guard pages, etc.) Poor visibility of stack Should not be used to run Java Code Java launcher “abandons” primordial thread (JDK >= 6) JNI invocation code should do the same
  30. 30. Thread Dumps A snapshot of all Java Threads (and some JVM Threads) Stop-the-World operation Some non-thread info thrown in for good measure Many ways to collect Slowly evolving “format” Many tools to help with analysis
  31. 31. Snapshots One dump alone can not show “motion” or change Each dump is mostly self- consistent Stop-the-World pause during collection By FlanellKamerasFilm - Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=68147463
  32. 32. Is this train moving? By Shiodome - ja:image:Mono-shibaura.JPG, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1157483
  33. 33. Thread Uncertainty Principle You can tell where a thread is (at what safepoint) You can tell how fast a thread is “moving” (CPU consumption) You can’t tell both at the same time But you don’t really need to Collect CPU consumption data along with thread dumps Collect multiple thread dumps
  34. 34. Safepoints Stop-the-World implementation Guarantees Java state of thread is known All threads executing Java code must stop JNI code can continue Cooperative suspension Polling based By YPLeroux at English Wikipedia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=17734859
  35. 35. Safepoints Stop-the-World implementation Guarantees Java state of thread is known All threads executing Java code must stop JNI code can continue Cooperative suspension Polling based By YPLeroux at English Wikipedia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=17734859
  36. 36. Safepoint Polling Interpreter can stop instantly template dispatch table replaced “polling” is essentially free Explicit poll inserted into C1/C2 JIT output Global safepoint polling page in address space Poll implemented by attempted read of this address Safepoint triggered (SIGSEGV) by change in page permissions
  37. 37. JIT Safepoint Polling On return from method call Backend branches in most loops Any transition to native code (JVM / JNI)
  38. 38. Not quite a snapshot Each thread stops at its next safepoint Not all threads will stop at the same time A critical section might not even have a safepoint Not unusual to see thread blocking for a lock no one holds More like a “panorama mode” picture: expect inconsistencies
  39. 39. Ways to Get a Thread Dump SIGQUIT / CTL-BRK handler Diagnostic command (jcmd) Java Flight recording periodic event jstack tool
  40. 40. jstack Force thread dump (might work) Mixed mode (might work) Thread dump from process or core file Integrated into servicabililty agent as of JDK 9
  41. 41. Thread Dump Output
  42. 42. Thread Dump Output Header Timestamp JVM version information safe memory reclamation (SMR) data Thread List All attached Java Threads Most Native threads Footer JNI global references Memory usage statistics
  43. 43. Timestamp 2019-09-07 03:00:18 Useful to sanity check timing dump was collected Makes interpreting per-thread stats much easier
  44. 44. Version Information Full thread dump Java HotSpot(TM) 64-Bit Server VM (13+33 mixed mode, sharing) Similar to “java –version” output Sharing -> class data sharing enabled JIT compiler state mixed mode interpreted mode (-Xint) compiled mode (-Xcomp)
  45. 45. Version Information Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.221-b11 mixed mode): JDK 8 and earlier use HotSpot Express version number HotSpot == 25.x -> JDK 8 HotSpot <= 24 -> Google it…
  46. 46. Safe Memory Reclamation Threads class SMR info: _java_thread_list=0x00007f4e842e9a60, length=6, elements={ 0x00007f4e84017800, 0x00007f4e84232800, 0x00007f4e84236000, 0x00007f4e84257000, 0x00007f4e842da800, 0x00007f4e842e8000 } tid of each JavaThread JVM internal information (can be ignored) Added in JDK 9
  47. 47. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  48. 48. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  49. 49. What’s in a name? Arbitrary string Thread.setName(); Can be changed as often as you want Name can be used as a debugging tool A thread by any other name... By Photo: JLPC / Wikimedia Commons, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=30151561
  50. 50. What’s in a name? "[STUCK] ExecuteThread: '139' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fd2a00b6000 nid=0x690b runnable [0x00007fd0ecbc7000] java.lang.Thread.State: RUNNABLE at java.io.ObjectOutputStream$BlockDataOutputStream.getUTFLength(ObjectOutputStream.java:2151)
  51. 51. What’s in a name? "[STUCK] ExecuteThread: '139' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fd2a00b6000 nid=0x690b runnable [0x00007fd0ecbc7000] java.lang.Thread.State: RUNNABLE at java.io.ObjectOutputStream$BlockDataOutputStream.getUTFLength(ObjectOutputStream.java:2151)
  52. 52. What’s in a name? theThread.setName("[really cool] " + theThread.getName()); System.out.println("I made your thread cool!"); < do cool stuff > theThread.setName(myThread.getName().substring(14)); System.out.println("Your thread is no longer cool!");
  53. 53. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  54. 54. Thread ID Thread.getID() Automatically assigned by JVM during thread creation Read-Only Consistent throughout thread lifetime
  55. 55. Daemon Thread "child thread" #12 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=1.98s tid=0x00007fec48278800 nid=0x179 waiting on condition [0x00007fec18df2000] java.lang.Thread.State: TIMED_WAITING (sleeping)
  56. 56. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  57. 57. Java Thread Priority Thread.getPriority() Thread.setPriority(int) Min is usually 1 Max is usually 10 Ignored by many JVMs 10 9 8 7 3 2 1 4 5 6
  58. 58. Java Thread Priority Thread.getPriority() Thread.setPriority(int) Min is usually 1 Max is usually 10 Ignored by many JVMs 10 9 8 7 3 2 1 4 5 6 Turtles all the way down!!!
  59. 59. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  60. 60. OS Thread Priority Not directly user controllable Interpretation is platform dependent
  61. 61. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  62. 62. -XX:+PrintExtendedThreadInfo "main" #1 prio=5 os_prio=0 cpu=58.42ms elapsed=2.60s allocated=289K defined_classes=4 tid=0x00007f8138017800 nid=0x344f waiting on condition [0x00007f8141890000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  63. 63. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  64. 64. tid Address of HotSpot data structure used to track thread
  65. 65. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  66. 66. Safe Memory Reclamation Threads class SMR info: _java_thread_list=0x00007f4e842e9a60, length=6, elements={ 0x00007f4e84017800, 0x00007f4e84232800, 0x00007f4e84236000, 0x00007f4e84257000, 0x00007f4e842da800, 0x00007f4e842e8000 }
  67. 67. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  68. 68. nid “native” thread identifier Useful for using OS tools (e.g. gdb, top -H) TD output in Hexadecimal, many OS tools use Decimal
  69. 69. nid Linux: kernel thread id from syscall gettid Solaris: thread library identity from thr_create() or thr_self() Windows: thread id from _beginthreadex macOS: kernel thread id from pthread_mach_thread_np(pthread_self()); Other BSD: kernel (?) thread id from syscall thr_self or getthrid AIX: thread library identity from pthread_create() or pthread_self()
  70. 70. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  71. 71. Thread state (native) runnable waiting for monitor entry waiting on condition in Object.wait()
  72. 72. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  73. 73. Last Java Frame Stack address of most recent Java frame on stack Stack pointer (sp) may be much farther down Null if there are no Java frames "C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=12.22ms elapsed=2.44s tid=0x00007f60781fd800 nid=0x37dc waiting on condition [0x0000000000000000]
  74. 74. Last Java Frame Stack address of most recent Java frame on stack Stack pointer (sp) may be much farther down Null if there are no Java frames "C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=12.22ms elapsed=2.44s tid=0x00007f60781fd800 nid=0x37dc waiting on condition [0x0000000000000000]
  75. 75. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  76. 76. java.lang.Thread.State NEW A thread that has not yet started is in this state. RUNNABLE A thread executing in the Java virtual machine is in this state. BLOCKED A thread that is blocked waiting for a monitor lock is in this state. WAITING A thread that is waiting indefinitely for another thread to perform a particular action is in this state. TIMED_WAITING A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state. TERMINATED A thread that has exited is in this state.
  77. 77. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  78. 78. Non-Java Thread "VM Thread" os_prio=0 cpu=2.63ms elapsed=2.47s tid=0x00007f0e841ef800 nid=0x3c41 runnable No Java thread ID (e.g. #42) No Java priority (e.g. prio=5) No last Java frame (e.g. [0x00007f4e8c82c000]) No java.lang.Thread.State No Java stack
  79. 79. JavaThread "main" #1 prio=5 os_prio=0 cpu=93.83ms elapsed=2.62s tid=0x00007f4e84017800 nid=0x7e4d waiting on condition [0x00007f4e8c82c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at Sleeper.main(Sleeper.java:5)
  80. 80. JNI Global References JNI global refs: 6, weak refs: 0 JNI global reference leaks are a common bug Investigate any significant growth post app warmup By John Newton and William Palmer, - A terrestial globe on which the tracts and discoveries are laid down from the accurate observations made by Captains Cook, Furneux, Phipps, published May 9th, 1782 globe by John Newton, cartography by William Palmer, Sate Library of New South Wales https://primo-slnsw.hosted.exlibrisgroup.com/primo-explore/fulldisplay?docid=ADLIB110577047&context=L&vid=SLNSW&search_scope=EEA&tab=default_tab&lang=en_US, Public Domain, https://commons.wikimedia.org/w/index.php?curid=55030411
  81. 81. JNI Global References JNI global refs: 6, weak refs: 0 JNI global reference leaks are a common bug Investigate any significant growth post app warmup Weak Global Reference breakout is new (JDK 11) Even if referent is collected, memory is leaked unless DeleteWeakGlobalRef is called
  82. 82. Heap Stats Heap garbage-first heap total 256000K, used 776K [0x0000000709200000, 0x0000000800000000) region size 1024K, 1 young (1024K), 0 survivors (0K) Metaspace used 161K, capacity 4486K, committed 4864K, reserved 1056768K class space used 7K, capacity 386K, committed 512K, reserved 1048576K
  83. 83. Thread States
  84. 84. Runnable "Thread-0" #12 prio=5 os_prio=0 cpu=1489.99ms elapsed=1.49s tid=0x00007f5c78262000 nid=0xa6f runnable [0x00007f5c497f4000] java.lang.Thread.State: RUNNABLE at LockTest$1.run(LockTest.java:6) at java.lang.Thread.run(java.base@13/Thread.java:830)
  85. 85. Blocked on a Monitor "Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000 nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker1.run(LockChain.java:28) - waiting to lock <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  86. 86. Blocked on a Monitor "Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000 nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker1.run(LockChain.java:28) - waiting to lock <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  87. 87. Blocked on a Monitor "Thread-0" #12 prio=5 os_prio=0 cpu=7.75ms elapsed=4.58s tid=0x00007fe2dc26b000 nid=0x238 waiting for monitor entry [0x00007fe2ad0f6000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker1.run(LockChain.java:28) - waiting to lock <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  88. 88. Who has my lock? "Thread-5" #17 prio=5 os_prio=0 cpu=0.85ms elapsed=4.58s tid=0x00007fe2dc26c800 nid=0x239 waiting for monitor entry [0x00007fe2acff5000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker2.run(LockChain.java:39) - waiting to lock <0x0000000718948ef0> (a java.lang.Object) - locked <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  89. 89. The plot thickens... "Thread-5" #17 prio=5 os_prio=0 cpu=0.85ms elapsed=4.58s tid=0x00007fe2dc26c800 nid=0x239 waiting for monitor entry [0x00007fe2acff5000] java.lang.Thread.State: BLOCKED (on object monitor) at LockChain$Worker2.run(LockChain.java:39) - waiting to lock <0x0000000718948ef0> (a java.lang.Object) - locked <0x0000000718948ee0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  90. 90. End of the road... "Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000 nid=0x23b waiting on condition [0x00007fe2acdf3000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at LockChain$Worker3.run(LockChain.java:51) - locked <0x0000000718948ef0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  91. 91. Thread A Thread E Thread D Thread B Thread C
  92. 92. Thread A Thread F Thread D Thread B Thread C Thread E Thread G
  93. 93. Thread A Thread B Holds Lock 1 Waiting on Lock 2 Holds Lock 2 Waiting on Lock 1
  94. 94. Deadlock Restarting JVM only way to recover All threads should acquire locks in well-defined order Most JVMs / tools can identify trivial deadlocks
  95. 95. Java stack information for the threads listed above: =================================================== "Thread-0": at DL$Thread1.run(DL.java:18) - waiting to lock <0x0000000718949078> (a java.lang.Object) - locked <0x0000000718949068> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830) "Thread-1": at DL$Thread2.run(DL.java:30) - waiting to lock <0x0000000718949068> (a java.lang.Object) - locked <0x0000000718949078> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830) Found 1 deadlock.
  96. 96. Intrinsic lock (monitor) issues Easy to identify in thread dumps Deep understanding of application code not needed for identification Amenable to automation (thread dump analysis tools)
  97. 97. Waiting on a monitor "Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800 nid=0xba2 in Object.wait() [0x00007f670d9d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718949e30> (a Drop) at java.lang.Object.wait(java.base@13/Object.java:326) at Drop.take(Drop.java:47) - locked <0x0000000718949e30> (a Drop) at Consumer.run(Consumer.java:45) at java.lang.Thread.run(java.base@13/Thread.java:830)
  98. 98. Wait and Notify Used to signal other threads to wake up and work Producer / Consumer is the textbook use-case Thread must hold the lock before calling wait or notify Otherwise IllegalMonitorStateException thrown
  99. 99. Wait and Notify Thread A locks object Thread A calls Object.wait() Thread B locks object Thread B calls Object.notify() Thread B releases lock Thread A wakes up and continues working
  100. 100. Wait and Notify Thread A locks object Thread A calls Object.wait() (lock released!) Thread B locks object Thread B calls Object.notify() Thread B releases lock Thread A wakes up and continues working
  101. 101. Waiting on a monitor "Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800 nid=0xba2 in Object.wait() [0x00007f670d9d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718949e30> (a Drop) at java.lang.Object.wait(java.base@13/Object.java:326) at Drop.take(Drop.java:47) - locked <0x0000000718949e30> (a Drop) at Consumer.run(Consumer.java:45) at java.lang.Thread.run(java.base@13/Thread.java:830)
  102. 102. Waiting on a monitor "Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800 nid=0xba2 in Object.wait() [0x00007f670d9d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718949e30> (a Drop) at java.lang.Object.wait(java.base@13/Object.java:326) at Drop.take(Drop.java:47) - locked <0x0000000718949e30> (a Drop) at Consumer.run(Consumer.java:45) at java.lang.Thread.run(java.base@13/Thread.java:830)
  103. 103. Waiting on a monitor "Thread-1" #13 prio=5 os_prio=0 cpu=11.25ms elapsed=12.71s tid=0x00007f675c264800 nid=0xba2 in Object.wait() [0x00007f670d9d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718949e30> (a Drop) at java.lang.Object.wait(java.base@13/Object.java:326) at Drop.take(Drop.java:47) - locked <0x0000000718949e30> (a Drop) at Consumer.run(Consumer.java:45) at java.lang.Thread.run(java.base@13/Thread.java:830)
  104. 104. Wait / Notify Intrinsic lock troubleshooting pattern not very helpful Knowledge of application design needed to know what thread should call notify() Analysis not amenable to automation Having a bunch of threads waiting might be normal under low load
  105. 105. java.util.concurrent More flexible than intrinsic locks (monitors) Harder to debug Plenty of rope to hang yourself
  106. 106. Intrinsic locks void methodA() { synchronized(mySpiffyObject) { <do stuff> methodB(); } }
  107. 107. Intrinsic locks void synchronized methodA() { methodB(); }
  108. 108. Intrinsic locks "Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000 nid=0x23b waiting on condition [0x00007fe2acdf3000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at LockChain$Worker3.run(LockChain.java:51) - locked <0x0000000718948ef0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  109. 109. java.util.concurrent Lock does not necessarily belong to a frame or even a thread A lock might be taken and released by different methods A lock might be taken and released by different threads
  110. 110. Intrinsic locks "Thread-6" #18 prio=5 os_prio=0 cpu=0.26ms elapsed=4.58s tid=0x00007fe2dc270000 nid=0x23b waiting on condition [0x00007fe2acdf3000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at LockChain$Worker3.run(LockChain.java:51) - locked <0x0000000718948ef0> (a java.lang.Object) at java.lang.Thread.run(java.base@13/Thread.java:830)
  111. 111. import java.util.concurrent.locks.ReentrantLock; <…> private static final ReentrantLock lock = new ReentrantLock();
  112. 112. private static void getLockAndSleep() { lock.lock(); try { Thread.sleep(1000 * 1000); } catch (Exception e) { } finally { lock.unlock(); } }
  113. 113. private static class Worker implements Runnable { public void run() { getLockAndSleep(); } }
  114. 114. public static void main(String[] args) throws InterruptedException { Thread childThread = new Thread(new Worker()); childThread.start(); Thread.sleep(1000 * 3); // 3 seconds System.out.println("main thread will try to get lock..."); getLockAndSleep(); }
  115. 115. "main" #1 prio=5 os_prio=0 cpu=45.91ms elapsed=7.66s tid=0x00007ff0d4017800 nid=0x1b1f waiting on condition [0x00007ff0dbf02000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@13/Native Method) - parking to wait for <0x0000000718953ed8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(java.base@13/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@13/Abstrac tQueuedSynchronizer.java:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@13/AbstractQueuedS ynchronizer.java:917) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@13/AbstractQueuedSynchro nizer.java:1240) at java.util.concurrent.locks.ReentrantLock.lock(java.base@13/ReentrantLock.java:267) at JUCLExample.getLockAndSleep(JUCLExample.java:19) at JUCLExample.main(JUCLExample.java:15)
  116. 116. Thread that “holds” the lock "Thread-0" #12 prio=5 os_prio=0 cpu=0.39ms elapsed=7.62s tid=0x00007ff0d4264000 nid=0x1b2f waiting on condition [0x00007ff0a45cc000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at JUCLExample.getLockAndSleep(JUCLExample.java:21) at JUCLExample.access$100(JUCLExample.java:3) at JUCLExample$Worker.run(JUCLExample.java:31) at java.lang.Thread.run(java.base@13/Thread.java:830)
  117. 117. With great power comes great... No thread really holds the lock Only the developer knows what thread should unlock it Thread that do the unlock may not even exist yet
  118. 118. That said... Most designs do use j.u.c locks in a “symmetric” way Often the thread that takes the lock is the one that releases it We can keep track of which thread took each lock: -XX:+PrintConcurrentLocks
  119. 119. "Thread-0" #12 prio=5 os_prio=0 cpu=0.40ms elapsed=18.75s tid=0x00007fc7b425c000 nid=0x1e65 waiting on condition [0x00007fc784ce5000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(java.base@13/Native Method) at JUCLExample.getLockAndSleep(JUCLExample.java:21) at JUCLExample.access$100(JUCLExample.java:3) at JUCLExample$Worker.run(JUCLExample.java:31) at java.lang.Thread.run(java.base@13/Thread.java:830) Locked ownable synchronizers: - <0x0000000718953e98> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  120. 120. PrintConcurrentLocks Performance impact (examine heap on thread dump) Tells you what thread took the lock, which may not be the thread that will release it Customer synchronizers built from j.u.c primitives may not be tracked (should extend AbstractOwnableSynchronizer) Can also be retrieved on-demand: jhsdb jstack --pid <pid> --locks
  121. 121. Dump Analysis Tools
  122. 122. Samurai
  123. 123. Samurai Lightweight Software Stack Independent Open Source
  124. 124. Spotify Online Java Thread Dump Analyzer Runs in your browser (no upload) No install needed Open Source
  125. 125. ThreadLogic
  126. 126. ThreadLogic Tailored for WebLogic and Oracle Fusion Middleware Automatically identifies many known issues Based on Eclipse and TDA (Thread Dump Analyzer) projects Open Source
  127. 127. The Usual Suspects
  128. 128. Service Thread "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fe4ac13f800 nid=0x3d3a runnable [0x0000000000000000] Since JDK 7 Handles JVMTI events Low Memory Detection JMX event handling for diagnostic commands Etc.
  129. 129. JIT Compiler Thread "C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007fe4ac12c000 nid=0x3d39 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE • Does not run Bytecode • Is a JavaThread for mostly historical reasons
  130. 130. Signal Dispatcher "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fe4ac116800 nid=0x3d35 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE • sun.misc.SignalHandler • Not for direct use By Mark Wolfe - This image is from the FEMA Photo Library., Public Domain, https://commons.wikimedia.org/w/index.php?curid=8075137
  131. 131. Attach Listener "Attach Listener" #10 daemon prio=9 os_prio=0 tid=0x00007f0520001000 nid=0x4660 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE
  132. 132. Finalizer "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f056c0ec000 nid=0x460e in Object.wait() [0x00007f054d466000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x000000076db88ed8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked <0x000000076db88ed8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
  133. 133. Secondary finalizer "Secondary finalizer" #10 prio=5 os_prio=0 tid=0x00007f908c154800 nid=0x4ff4 waiting on condition [0x00007f9074dc2000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at FinStop.finalize(FinStop.java:22) at java.lang.System$2.invokeFinalize(System.java:1270) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102) at java.lang.ref.Finalizer.access$100(Finalizer.java:34) at java.lang.ref.Finalizer$2.run(Finalizer.java:161) at java.lang.Thread.run(Thread.java:748)
  134. 134. Reference Handler "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f908c0e7800 nid=0x4feb in Object.wait() [0x00007f90756cb000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000006c9200ae0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x00000006c9200ae0> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) By Giuseppe Arcimboldo - LSH 97136 (sm_dig12951), Public Domain, https://commons.wikimedia.org/w/index.php?curid=36899673
  135. 135. “main” "main" #1 prio=5 os_prio=0 tid=0x00007ff04804b000 nid=0x6eca waiting on condition [0x00007ff04fbfd000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at Sleeper.main(Sleeper.java:5)
  136. 136. VM Thread "VM Thread" os_prio=0 tid=0x00007ff0480dd800 nid=0x6ed3 runnable Stop-the-World thread GC Thread dumps Bias revocation
  137. 137. VM Periodic Task Thread "VM Periodic Task Thread" os_prio=0 tid=0x00007ff04813a000 nid=0x6edc waiting on condition Timer based events Profiling Sampling By Ambrogio Lorenzetti - The Yorck Project (2002) 10.000 Meisterwerke der Malerei (DVD-ROM), distributed by DIRECTMEDIA Publishing GmbH. ISBN: 3936122202., Public Domain, https://commons.wikimedia.org/w/index.php?curid=3499217
  138. 138. GC Worker Threads "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ff04805e000 nid=0x6ecb runnable Depend on the collector currently in use Number of threads may be chosen by ergonomics Many different types
  139. 139. Common-Cleaner "Common-Cleaner" #11 daemon prio=8 os_prio=0 cpu=0.13ms elapsed=2.49s tid=0x00007f8224254000 nid=0x766a in Object.wait() [0x00007f81f58f6000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@13/Native Method) - waiting on <0x0000000718945448> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@13/ReferenceQueue.java:155) - locked <0x0000000718945448> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run(java.base@13/CleanerImpl.java:148) at java.lang.Thread.run(java.base@13/Thread.java:830) at jdk.internal.misc.InnocuousThread.run(java.base@13/InnocuousThread.java:134) JDK 9 and later Shared java.lang.ref.Cleaner thread https://bugs.openjdk.java.net/browse/JDK-8145041 By CurranH (talk) (Uploads) - Own work, Public Domain, https://en.wikipedia.org/w/index.php?curid=11838492
  140. 140. Destroyer of Worlds "DestroyJavaVM" #13 prio=5 os_prio=0 cpu=63.15ms elapsed=3.19s tid=0x00007f951c017800 nid=0x4cfc waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Main has exited, but non-Deamon threads remain By Jbuzbee - Own work, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=4835571
  141. 141. Takaways Collect at least 3 dumps Collect CPU usage with OS-level tool top –H Process Explorer Enable j.u.concurrent lock output
  142. 142. Java Concurrency in Practice Java Concurrency Bible Covers JSR-133 / JSR-166
  143. 143. The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.

×