Profilers find performance bottlenecks in your app but provide confusing information. Let's give you insights into how your profiler and your app are really interacting. What profiling APIs are available, how they work, and what their implementation on the JVM (OpenJDK) side looks like: Stack sampling profilers: stop motion view of your app GetCallTrace(JVisualVM case study): The official stack sampling API Safepoints and safepoint sampling bias AsyncGetCallTrace(Honest Profiler Case Study): The unofficial API JVM Profilers vs System Profilers: No API needed?

1. JVM Profiling
Under da Hood
Richard Warburton - @RichardWarburto
Nitsan Wakart - @nitsanw

2. Why Profile?
Lies, Damn Lies and Statistical Profiling
Under the Hood
Conclusion

5. Measure data from your application

6. Exploratory Profiling

7. Execution Profiling
=
Where in code is my application
spending time?

8. CPU Profiling Limitations
● Finds CPU bound bottlenecks
● Many problems not CPU Bound
○ Networking
○ Database or External Service
○ I/O
○ Garbage Collection
○ Insufficient Parallelism
○ Blocking & Queuing Effects

9. Why Profile?
Lies, Damn Lies and Statistical Profiling
Under the Hood
Conclusion

11. Different Execution Profilers
● Instrumenting
○ Adds timing code to application
● Sampling
○ Collects thread dumps periodically

12. Sampling Profilers
WebServerThread.run()
Controller.doSomething() Controller.next()
Repo.readPerson()
new Person()
View.printHtml()

13. Periodicity Bias
● Bias from sampling at a fixed interval
● Periodic operations with the same frequency
as the samples
● Timed operations

14. Periodicity Bias
a() ??? a() ??? a() ??? a() ???

15. Stack Trace Sampling
● JVMTI interface: GetCallTrace
○ Trigger a global safepoint(not on Zing)
○ Collect stack trace
● Large impact on application
● Samples only at safepoints

16. Example
private static void outer()
{
for (int i = 0; i < OUTER; i++)
{
hotMethod(i);
}
}
// https://github.com/RichardWarburton/profiling-samples

17. Example (2)
private static void hotMethod(final int i)
{
for (int k = 0; k < N; k++)
{
final int[] array = SafePointBias. array;
final int index = i % SIZE;
for (int j = index; j < SIZE; j++)
{
array[index] += array[j];
}
}
}

19. -XX:+PrintSafepointStatistics
ThreadDump 48
Maximum sync time 985 ms

20. Whats a safepoint?
● Java threads poll global flag
○ At ‘uncounted’ loops back edge
○ At method exit/enter
● A safepoint poll can be delayed by:
○ Large methods
○ Long running ‘counted’ loops
○ BONUS: Page faults/thread suspension

22. Safepoint Bias
WebServerThread.run()
Controller.doSomething() Controller.next()
Repo.readPerson()
new Person()
View.printHtml() ???

24. Let sleeping dogs lie?
● ‘GetCallTrace’ profilers will sample ALL
threads
● Even sleeping threads...

25. This Application Mostly Sleeps
JVisualVM snapshot

26. No CPU? No profile!
JMC profile

27. Why Profile?
Lies, Damn Lies and Statistical Profiling
Under the Hood
Conclusion

28. Honest Profiler
https://github.com/richardwarburton/honest-profiler

30. AsyncGetCallTrace
● Used by Oracle Solaris Studio
● Adapted to open source prototype by
Google’s Jeremy Manson
● Unsupported, Undocumented …
Underestimated

31. SIGPROF - Interrupt Handlers
● OS Managed timing based interrupt
● Interrupts the thread and directly calls an
event handler
● Used by profilers we’ll be talking about

32. Design
Log File
Processor
Thread Graphical UI
Console UI
Signal
Handler
Signal
Handler
Os Timer Thread

33. “You are in a maze of twisty little stack frames,
all alike”

34. AsyncGetCallTrace under the hood
● A Java thread is ‘possessed’
● You have the PC/FP/SP
● What is the call trace?
○ jmethodId - Java Method Identifier
○ bci - Byte Code Index -> used to find line number

35. Where Am I?
● Given a PC what is the current method?
● Is this a Java method?
○ Each method ‘lives’ in a range of addresses
● If not, what do we do?

36. Java Method? Which line?
● Given a PC, what is the current line?
○ Not all instructions map directly to a source line
● Given super-scalar CPUs what does PC
mean?
● What are the limits of PC accuracy?

37. “> I think Andi mentioned this to me last year --
> that instruction profiling was no longer reliable.
It never was.”
http://permalink.gmane.org/gmane.linux.kernel.perf.user/1948
Exchange between Brenden Gregg and Andi Kleen

38. Skid
● PC indicated will be >= to PC at sample time
● Known limitation of instruction profiling
● Leads to harder ‘blame analysis’

39. Limits of line number accuracy:
Line number (derived from BCI) is the closest
attributable BCI to the PC (-XX:+DebugNonSafepoint)
The PC itself is within some skid distance from
actual sampled instruction

40. ● Divided into frames
○ frame { sender*, stack*, pc }
● A single linked list:
root(null, s0, pc1) <- call1 (root, s1, pc2) <- call2(call1, s2, pc2)
● Convert to: (jmethodId,lineno)
The Stack

41. A typical stack
● JVM Thread runner infra:
○ JavaThread::run to JavaCalls::call_helper
● Interleaved Java frames:
○ Interpreted
○ Compiled
○ Java to Native and back
● Top frame may be Java or Native

42. Native frames
● Ignored, but need to navigate through
● Use a dedicated FP register to find sender
● But only if compiled to do so…
● Use a last remembered Java frame instead
See: http://duartes.org/gustavo/blog/post/journey-to-the-stack/

43. Java Compiled Frames
● C1/C2 produce native code
● No FP register: use set frame size
● Challenge: methods can move (GC)
● Challenge: methods can get recompiled

44. Java Interpreter frames
● Separately managed by the runtime
● Make an effort to look like normal frames
● Challenge: may be interrupted half-way
through construction...

45. Virtual Frames
● C1/C2 inline code (intrinsics/other methods)
● No data on stack
● Must use JVM debug info

46. AsyncGetCallTrace Limitations
● Only profiles running threads
● Accuracy of line info limited by reality
● Only reports Java frames/threads
● Must lookup debug info during call

47. Compilers: Friend or Fiend?
void safe_reset(void *start, size_t size) {
char *base = reinterpret_cast<char *>(start);
char *end = base + size;
for (char *p = base; p < end; p++) {
*p = 0;
}
}

48. Compilers: Friend or Fiend?
safe_reset(void*, unsigned long):
lea rdx, [rdi+rsi]
cmp rdi, rdx
jae .L3
sub rdx, rdi
xor esi, esi
jmp memset
.L3:
rep ret

49. Concurrency Bug
● Even simple concurrency bugs are hard to
spot
● Unspotted race condition in the ring buffer
● Spotted thanks to open source & Rajiv
Signal

50. Writer
Reader

51. Writer
Reader

52. Extra Credit!

53. Native Profiling Tools
● Profile native methods
● Profile at the instruction level
● Profile hardware counters

54. Perf
● A Linux profiling tool
● Can be made to work with Java
● JMH integration
● Ongoing integration efforts

55. Solaris Studio
● Works on Linux!
● Secret Weapon!
● Give it a go!

56. ZVision
● Works for Zing
● No HWC support
● Very informative

57. Why Profile?
Lies, Damn Lies and Statistical Profiling
Under the Hood
Conclusion

58. What did we cover?
● Biases in Profilers
● More accurate sampling
● Alternative Profiling Approaches

59. Don’t just blindly trust your tooling.

60. Test your measuring instruments

61. Open Source enables implementation review

62. Q & A
@nitsanw
psy-lob-saw.blogspot.co.uk
@richardwarburto
insightfullogic.com
java8training.com
www.pluralsight.
com/author/richard-
warburton

63. Slides after here just for reference,
don’t delete or show