Bjørn Vårdal, J9VM Software Developer, IBM, @bvaardal New language runtimes appear all the time, but most of them die young. Failure can be attributed to different reasons, but an important factor is that lack of support can limit the community’s and industry’s willingness to adopt the new language. Quicker development and improved serviceability allows emerging languages to overcome this obstacle. By building on the proven technology available in Eclipse OMR, language developers can get more than performance and stability; you also get tools that help you quickly debug your language runtime, allowing you to provide competitive serviceability. From this presentation, you will learn how to enable Eclipse OMR’s mature debugging features in your language runtime, and also how Eclipse OMR can assist with development and debugging

1. J9, OMR and beyond
Serviceable Runtimes

2. • Bjørn Vårdal
• IBM Runtimes Developer
• J9 JVM
• @bvaardal
• MethodHandles
• Lambdas
• VarHandles
• Project Valhalla
Bjørn Vårdal IBM Runtime Technologies 2
Introduction

3. Bjørn Vårdal IBM Runtime Technologies 3
Most die young_______runtimes
unless you service them.

4. The information contained in this presentation is provided for informational purposes only.
Whilst efforts were made to verify the completeness and accuracy of the information contained in this
presentation, it is provided “as is”, without warranty of any kind, express or implied.
All performance data included in this presentation have been gathered in a controlled environment. Your own test
results may vary based on hardware, software or infrastructure differences.
All data included in this presentation are meant to be used only as a guide.
In addition, the information contained in this presentation is based on IBM’s current product plans and strategy,
which are subject to change by IBM, without notice.
IBM and its affiliated companies shall not be responsible for any damages arising out of the use of, or otherwise
related to, this presentation or any other documentation.
Nothing contained in this presentation is intended to, or shall have the effect of:
- creating any warrant or representation from IBM, its affiliated companies or its or their suppliers and/or licensors
Bjørn Vårdal IBM Runtime Technologies 4
Disclaimer

5. • Advanced core dump analysis for J9
• Used by the J9 service team
• Used by J9 developers
• Front-end: DDR Interactive
Bjørn Vårdal IBM Runtime Technologies 5
Direct Dump Reader

6. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.12345
Bjørn Vårdal IBM Runtime Technologies 6
DDR Interactive
Ships with J9

7. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.12345
Bjørn Vårdal IBM Runtime Technologies 7
DDR Interactive

8. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.12345
*0 : PID: 2824; !j9javavm 0x7f27480eef0
Run !j9help to see a list of commands
>
Bjørn Vårdal IBM Runtime Technologies 8
DDR Interactive

9. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.12345
*0 : PID: 2824; !j9javavm 0x7f27480eef0
Run !j9help to see a list of commands
>
Bjørn Vårdal IBM Runtime Technologies 9
DDR Interactive

10. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.12345
*0 : PID: 2824; !j9javavm 0x7f27480eef0
Run !j9help to see a list of commands
> !j9help
Bjørn Vårdal IBM Runtime Technologies 10
DDR Interactive

11. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.12345
*0 : PID: 2824; !j9javavm 0x7f27480eef0
Run !j9help to see a list of commands
> !j9help
threads Lists VM threads
stack Walks the Java stack for <thread>
stackslots Walks the Java stack (including objects) for <thread>
.
.
.
classforname Find the class corresponding to name (with wildcards)
dumpromclass Dump the specified J9ROMClass. Wild cards are allowed in class name.
findpattern Search memory for a specific pattern
whatis Determine the type of structure at a given address
Bjørn Vårdal IBM Runtime Technologies 11
DDR - !j9help

12. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.20170315.010552.2824.01.dmp
*0 : PID: 2824; !j9javavm 0x7f27480eef0
Run !j9help to see a list of commands
> !threads
Bjørn Vårdal IBM Runtime Technologies 12
DDR - !threads

13. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.20170315.010552.2824.01.dmp
*0 : PID: 2824; !j9javavm 0x7f27480eef0
Run !j9help to see a list of commands
> !threads
!stack 0x0228d70 !j9vmthread 0x0228d70 !j9thread 0x07f274807d70 tid 0x4b (75) // (main)
!stack 0x02294f0 !j9vmthread 0x02294f0 !j9thread 0x7f27480b0bf8 tid 0x4e (78) // (JIT Compilation Thread-1)
!stack 0x022af20 !j9vmthread 0x022af20 !j9thread 0x7f27480bfcb0 tid 0x55 (85) // (JIT-SamplerThread)
!stack 0x022b2e0 !j9vmthread 0x022b2e0 !j9thread 0x7f27480c0228 tid 0x56 (86) // (IProfiler)
!stack 0x0238620 !j9vmthread 0x0238620 !j9thread 0x7f2748238170 tid 0x57 (87) // (Common-Cleaner)
!stack 0x023f480 !j9vmthread 0x023f480 !j9thread 0x7f27482386e8 tid 0x58 (88) // (Signal Dispatcher)
!stack 0x023f840 !j9vmthread 0x023f840 !j9thread 0x7f27483d01f8 tid 0x5a (90) // (Concurrent Mark Helper)
!stack 0x0023fc0 !j9vmthread 0x0023fc0 !j9thread 0x7f27483d0c90 tid 0x5b (91) // (GC Slave)
!stack 0x0002420 !j9vmthread 0x0002420 !j9thread 0x007f26e084d0 tid 0x5f (95) // (Attach API wait loop)
!stack 0x0240730 !j9vmthread 0x0240730 !j9thread 0x7f27483d2218 tid 0x60 (96) // (Finalizer thread)
>
Bjørn Vårdal IBM Runtime Technologies 13
DDR - !threads

14. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.20170315.010552.2824.01.dmp
*0 : PID: 2824; !j9javavm 0x7f27480eef0
Run !j9help to see a list of commands
> !threads
!stack 0x0228d70 !j9vmthread 0x0228d70 !j9thread 0x07f274807d70 tid 0x4b (75) // (main)
!stack 0x02294f0 !j9vmthread 0x02294f0 !j9thread 0x7f27480b0bf8 tid 0x4e (78) // (JIT Compilation Thread-1)
!stack 0x022af20 !j9vmthread 0x022af20 !j9thread 0x7f27480bfcb0 tid 0x55 (85) // (JIT-SamplerThread)
!stack 0x022b2e0 !j9vmthread 0x022b2e0 !j9thread 0x7f27480c0228 tid 0x56 (86) // (IProfiler)
!stack 0x0238620 !j9vmthread 0x0238620 !j9thread 0x7f2748238170 tid 0x57 (87) // (Common-Cleaner)
!stack 0x023f480 !j9vmthread 0x023f480 !j9thread 0x7f27482386e8 tid 0x58 (88) // (Signal Dispatcher)
!stack 0x023f840 !j9vmthread 0x023f840 !j9thread 0x7f27483d01f8 tid 0x5a (90) // (Concurrent Mark Helper)
!stack 0x0023fc0 !j9vmthread 0x0023fc0 !j9thread 0x7f27483d0c90 tid 0x5b (91) // (GC Slave)
!stack 0x0002420 !j9vmthread 0x0002420 !j9thread 0x007f26e084d0 tid 0x5f (95) // (Attach API wait loop)
!stack 0x0240730 !j9vmthread 0x0240730 !j9thread 0x7f27483d2218 tid 0x60 (96) // (Finalizer thread)
>
Bjørn Vårdal IBM Runtime Technologies 14
DDR - !threads

15. $ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.20170315.010552.2824.01.dmp
*0 : PID: 2824; !j9javavm 0x7f27480eef0
Run !j9help to see a list of commands
> !threads
!stack 0x0228d70 !j9vmthread 0x0228d70 !j9thread 0x07f274807d70 tid 0x4b (75) // (main)
!stack 0x02294f0 !j9vmthread 0x02294f0 !j9thread 0x7f27480b0bf8 tid 0x4e (78) // (JIT Compilation Thread-1)
!stack 0x022af20 !j9vmthread 0x022af20 !j9thread 0x7f27480bfcb0 tid 0x55 (85) // (JIT-SamplerThread)
!stack 0x022b2e0 !j9vmthread 0x022b2e0 !j9thread 0x7f27480c0228 tid 0x56 (86) // (IProfiler)
!stack 0x0238620 !j9vmthread 0x0238620 !j9thread 0x7f2748238170 tid 0x57 (87) // (Common-Cleaner)
!stack 0x023f480 !j9vmthread 0x023f480 !j9thread 0x7f27482386e8 tid 0x58 (88) // (Signal Dispatcher)
!stack 0x023f840 !j9vmthread 0x023f840 !j9thread 0x7f27483d01f8 tid 0x5a (90) // (Concurrent Mark Helper)
!stack 0x0023fc0 !j9vmthread 0x0023fc0 !j9thread 0x7f27483d0c90 tid 0x5b (91) // (GC Slave)
!stack 0x0002420 !j9vmthread 0x0002420 !j9thread 0x007f26e084d0 tid 0x5f (95) // (Attach API wait loop)
!stack 0x0240730 !j9vmthread 0x0240730 !j9thread 0x7f27483d2218 tid 0x60 (96) // (Finalizer thread)
>
Bjørn Vårdal IBM Runtime Technologies 15
DDR - !threads

16. > !stack 0x0228d70
<1a2970> Generic special frame
<1a2970> !j9method 0x01B9A718 X.print(Ljava/lang/String;)V
<1a2970> !j9method 0x01B9A6F8 X.m(I)V
<1a2970> !j9method 0x01B9A6D8 X.main([Ljava/lang/String;)V
<1a2970> JNI call-in frame
<1a2970> Native method frame
>
Bjørn Vårdal IBM Runtime Technologies 16
DDR - !stack

17. > !stack 0x0228d70
<1a2970> Generic special frame
<1a2970> !j9method 0x01B9A718 X.print(Ljava/lang/String;)V
<1a2970> !j9method 0x01B9A6F8 X.m(I)V
<1a2970> !j9method 0x01B9A6D8 X.main([Ljava/lang/String;)V
<1a2970> JNI call-in frame
<1a2970> Native method frame
>
Bjørn Vårdal IBM Runtime Technologies 17
DDR - !stack

18. > !stackslots 0x0228d70
<1a2970> *** BEGIN STACK WALK, flags = 0401 walkThread = 0x01A2970 ***
<1a2970> ITERATE_O_SLOTS
<1a2970> RECORD_BYTECODE_PC_OFFSET
<1a2970> Initial values: walkSP = 0x01AE6D60, PC = 0x01, literals = 0x0, A0 = 0x01AE6D78, j2iFrame = 0x0, ...
<1a2970> Generic special frame: bp = 0x01AE6D78, sp = 0x01AE6D60, pc = 0x01, cp = 0x0, arg0EA = 0x01AE ...
<1a2970> Bytecode frame: bp = 0x01AE6D90, sp = 0x01AE6D80, pc = 0x07FDCB43A6344, cp = 0x01B9A580, arg0EA = 0x01AE6D98, ...
<1a2970> Method: X.print(Ljava/lang/String;)V !j9method 0x01B9A718
<1a2970> Bytecode index = 16
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6D98 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6D98] = 0x0FFEF46D0
<1a2970> Bytecode frame: bp = 0x01AE6DB0, sp = 0x01AE6DA0, pc = 0x07FDCB43A630A, cp = 0x01B9A580, arg0EA = 0x01AE6DB8, ...
<1a2970> Method: X.m(I)V !j9method 0x01B9A6F8
<1a2970> Bytecode index = 6
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DB8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234
<1a2970> Bytecode frame: bp = 0x01AE6DD0, sp = 0x01AE6DC0, pc = 0x07FDCB43A62DB, cp = 0x1B9A580, arg0EA = 0x1AE6DD8, ...
<1a2970> Method: X.main([Ljava/lang/String;)V !j9method 0x01B9A6D8
<1a2970> Bytecode index = 3
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DD8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DD8] = 0x0FFEE5668
<1a2970> JNI call-in frame: bp = 0x01AE6E0, sp = 0x01AE6DE0, pc = 0x07FDCB96DE314, cp = 0x0, arg0EA = 0x01AE6E0, ...
<1a2970> New ELS = 0x0
<1a2970> JNI native method frame: bp = 0x01AE6E88, sp = 0x01AE6E08, pc = 0x07, cp = 0x0, arg0EA = 0x01AE6E88, ...
<1a2970> Object pushes starting at 0x01AE6E08 for 12 slots
Bjørn Vårdal IBM Runtime Technologies 18
DDR - !stackslots

19. > !stackslots 0x0228d70
<1a2970> *** BEGIN STACK WALK, flags = 0401 walkThread = 0x01A2970 ***
<1a2970> ITERATE_O_SLOTS
<1a2970> RECORD_BYTECODE_PC_OFFSET
<1a2970> Initial values: walkSP = 0x01AE6D60, PC = 0x01, literals = 0x0, A0 = 0x01AE6D78, j2iFrame = 0x0, ...
<1a2970> Generic special frame: bp = 0x01AE6D78, sp = 0x01AE6D60, pc = 0x01, cp = 0x0, arg0EA = 0x01AE ...
<1a2970> Bytecode frame: bp = 0x01AE6D90, sp = 0x01AE6D80, pc = 0x07FDCB43A6344, cp = 0x01B9A580, arg0EA = 0x01AE6D98, ...
<1a2970> Method: X.print(Ljava/lang/String;)V !j9method 0x01B9A718
<1a2970> Bytecode index = 16
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6D98 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6D98] = 0x0FFEF46D0
<1a2970> Bytecode frame: bp = 0x01AE6DB0, sp = 0x01AE6DA0, pc = 0x07FDCB43A630A, cp = 0x01B9A580, arg0EA = 0x01AE6DB8, ...
<1a2970> Method: X.m(I)V !j9method 0x01B9A6F8
<1a2970> Bytecode index = 6
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DB8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234
<1a2970> Bytecode frame: bp = 0x01AE6DD0, sp = 0x01AE6DC0, pc = 0x07FDCB43A62DB, cp = 0x1B9A580, arg0EA = 0x1AE6DD8, ...
<1a2970> Method: X.main([Ljava/lang/String;)V !j9method 0x01B9A6D8
<1a2970> Bytecode index = 3
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DD8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DD8] = 0x0FFEE5668
<1a2970> JNI call-in frame: bp = 0x01AE6E0, sp = 0x01AE6DE0, pc = 0x07FDCB96DE314, cp = 0x0, arg0EA = 0x01AE6E0, ...
<1a2970> New ELS = 0x0
<1a2970> JNI native method frame: bp = 0x01AE6E88, sp = 0x01AE6E08, pc = 0x07, cp = 0x0, arg0EA = 0x01AE6E88, ...
<1a2970> Object pushes starting at 0x01AE6E08 for 12 slots
Bjørn Vårdal IBM Runtime Technologies 19
DDR - !stackslots

20. > !stackslots 0x0228d70
<1a2970> *** BEGIN STACK WALK, flags = 0401 walkThread = 0x01A2970 ***
<1a2970> ITERATE_O_SLOTS
<1a2970> RECORD_BYTECODE_PC_OFFSET
<1a2970> Initial values: walkSP = 0x01AE6D60, PC = 0x01, literals = 0x0, A0 = 0x01AE6D78, j2iFrame = 0x0, ...
<1a2970> Generic special frame: bp = 0x01AE6D78, sp = 0x01AE6D60, pc = 0x01, cp = 0x0, arg0EA = 0x01AE ...
<1a2970> Bytecode frame: bp = 0x01AE6D90, sp = 0x01AE6D80, pc = 0x07FDCB43A6344, cp = 0x01B9A580, arg0EA = 0x01AE6D98, ...
<1a2970> Method: X.print(Ljava/lang/String;)V !j9method 0x01B9A718
<1a2970> Bytecode index = 16
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6D98 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6D98] = 0x0FFEF46D0
<1a2970> Bytecode frame: bp = 0x01AE6DB0, sp = 0x01AE6DA0, pc = 0x07FDCB43A630A, cp = 0x01B9A580, arg0EA = 0x01AE6DB8, ...
<1a2970> Method: X.m(I)V !j9method 0x01B9A6F8
<1a2970> Bytecode index = 6
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DB8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234
<1a2970> Bytecode frame: bp = 0x01AE6DD0, sp = 0x01AE6DC0, pc = 0x07FDCB43A62DB, cp = 0x1B9A580, arg0EA = 0x1AE6DD8, ...
<1a2970> Method: X.main([Ljava/lang/String;)V !j9method 0x01B9A6D8
<1a2970> Bytecode index = 3
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DD8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DD8] = 0x0FFEE5668
<1a2970> JNI call-in frame: bp = 0x01AE6E0, sp = 0x01AE6DE0, pc = 0x07FDCB96DE314, cp = 0x0, arg0EA = 0x01AE6E0, ...
<1a2970> New ELS = 0x0
<1a2970> JNI native method frame: bp = 0x01AE6E88, sp = 0x01AE6E08, pc = 0x07, cp = 0x0, arg0EA = 0x01AE6E88, ...
<1a2970> Object pushes starting at 0x01AE6E08 for 12 slots
Bjørn Vårdal IBM Runtime Technologies 20
DDR - !stackslots

21. > !stackslots 0x0228d70
<1a2970> *** BEGIN STACK WALK, flags = 0401 walkThread = 0x01A2970 ***
<1a2970> ITERATE_O_SLOTS
<1a2970> RECORD_BYTECODE_PC_OFFSET
<1a2970> Initial values: walkSP = 0x01AE6D60, PC = 0x01, literals = 0x0, A0 = 0x01AE6D78, j2iFrame = 0x0, ...
<1a2970> Generic special frame: bp = 0x01AE6D78, sp = 0x01AE6D60, pc = 0x01, cp = 0x0, arg0EA = 0x01AE ...
<1a2970> Bytecode frame: bp = 0x01AE6D90, sp = 0x01AE6D80, pc = 0x07FDCB43A6344, cp = 0x01B9A580, arg0EA = 0x01AE6D98, ...
<1a2970> Method: X.print(Ljava/lang/String;)V !j9method 0x01B9A718
<1a2970> Bytecode index = 16
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6D98 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6D98] = 0x0FFEF46D0
<1a2970> Bytecode frame: bp = 0x01AE6DB0, sp = 0x01AE6DA0, pc = 0x07FDCB43A630A, cp = 0x01B9A580, arg0EA = 0x01AE6DB8, ...
<1a2970> Method: X.m(I)V !j9method 0x01B9A6F8
<1a2970> Bytecode index = 6
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DB8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234
<1a2970> Bytecode frame: bp = 0x01AE6DD0, sp = 0x01AE6DC0, pc = 0x07FDCB43A62DB, cp = 0x1B9A580, arg0EA = 0x1AE6DD8, ...
<1a2970> Method: X.main([Ljava/lang/String;)V !j9method 0x01B9A6D8
<1a2970> Bytecode index = 3
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DD8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DD8] = 0x0FFEE5668
<1a2970> JNI call-in frame: bp = 0x01AE6E0, sp = 0x01AE6DE0, pc = 0x07FDCB96DE314, cp = 0x0, arg0EA = 0x01AE6E0, ...
<1a2970> New ELS = 0x0
<1a2970> JNI native method frame: bp = 0x01AE6E88, sp = 0x01AE6E08, pc = 0x07, cp = 0x0, arg0EA = 0x01AE6E88, ...
<1a2970> Object pushes starting at 0x01AE6E08 for 12 slots
Bjørn Vårdal IBM Runtime Technologies 21
DDR - !stackslots

22. > !j9object 0x0FFEF46D0
J9VMJavaLangString at 0x00000000FFEF46D0 {
struct J9Class* clazz = !j9class 0x1A5B500 // java/lang/String
Object flags = 0x00000000;
[B value = !fj9object 0x86f36828 (offset=0) (java/lang/String)
I count = 0x00000007 (offset=4) (java/lang/String)
I hashCode = 0x00000000 (offset=8) (java/lang/String)
"foo4660"
}
Bjørn Vårdal IBM Runtime Technologies 22
DDR - !j9object

23. > !j9object 0x0FFEF46D0
J9VMJavaLangString at 0x00000000FFEF46D0 {
struct J9Class* clazz = !j9class 0x1A5B500 // java/lang/String
Object flags = 0x00000000;
[B value = !fj9object 0x86f36828 (offset=0) (java/lang/String)
I count = 0x00000007 (offset=4) (java/lang/String)
I hashCode = 0x00000000 (offset=8) (java/lang/String)
"foo4660"
}
Bjørn Vårdal IBM Runtime Technologies 23
DDR - !j9object

24. GDB
• Requires intimate knowledge
about the JVM internal structures
• Requires debug symbols
• Source must match
• Platform specific
DDR
• Has intimate knowledge about
the JVM internal structures
• Doesn’t require debug symbols
• Cross JVM version
• Cross platform
Bjørn Vårdal IBM Runtime Technologies 24
GDB vs DDR

25. > !stackslots 0x0228d70
<1a2970> *** BEGIN STACK WALK, flags = 0401 walkThread = 0x01A2970 ***
<1a2970> ITERATE_O_SLOTS
<1a2970> RECORD_BYTECODE_PC_OFFSET
<1a2970> Initial values: walkSP = 0x01AE6D60, A0 = 0x01AE6D78,
<1a2970> Generic special frame: bp = 0x01AE6D78, sp = 0x01AE6D60,
<1a2970> Bytecode frame: bp = 0x01AE6D90, sp = 0x01AE6D80, arg0EA =
<1a2970> Method: X.print(Ljava/lang/String;)V !j9method 0x01B9A718
<1a2970> Bytecode index = 16
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6D98 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6D98] = 0x0FFEF46D0
<1a2970> Bytecode frame: bp = 0x01AE6DB0, sp = 0x01AE6DA0, pc =
<1a2970> Method: X.m(I)V !j9method 0x01B9A6F8
<1a2970> Bytecode index = 6
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DB8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234
<1a2970> Bytecode frame: bp = 0x01AE6DD0, arg0EA = 0x1AE6DD8
<1a2970> Method: X.main([Ljava/lang/String;)V !j9method 0x01B9A6D8
<1a2970> Bytecode index = 3
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6DD8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DD8] = 0x0FFEE5668
<1a2970> JNI call-in frame: sp = 0x01AE6DE0, pc = 0x07FDCB96DE314
<1a2970> JNI native method frame: cp = 0x0, arg0EA = 0x01AE6E88, ...
<1a2970> Object pushes starting at 0x01AE6E08 for 12 slots
(gdb) x/200 0x0000000001AE6D60
0x1ae6d60: 0x00000000 0x00000000 0x01b9a718 0x00000000
0x1ae6d70: 0xb43a6344 0x00007fdc 0x01ae6d9a 0x00000000
0x1ae6d80: 0x01b9a6f8 0x00000000 0xb43a630a 0x00007fdc
0x1ae6d90: 0x01ae6db8 0x00000000 0xffef46d0 0x00000000
0x1ae6da0: 0x01b9a6d8 0x00000000 0xb43a62db 0x00007fdc
0x1ae6db0: 0x01ae6dd8 0x00000000 0x00001234 0x00000000
0x1ae6dc0: 0x00000000 0x00000000 0xb96de314 0x00007fdc
0x1ae6dd0: 0x01ae6e00 0x00000000 0xffee5668 0x00000000
0x1ae6de0: 0x00000000 0x00000000 0x00020000 0x00000000
0x1ae6df0: 0x00000060 0x00000000 0x00000007 0x00000000
0x1ae6e00: 0x01ae6e8a 0x00000000 0xffee5668 0x00000000
0x1ae6e10: 0x86ef0828 0x00000000 0x86f317d0 0x00000000
0x1ae6e20: 0x86f317d0 0x00000000 0x00000000 0x00000000
0x1ae6e30: 0x00000000 0x00000000 0x86f2ed50 0x00000000
0x1ae6e40: 0x86ef3548 0x00000000 0x86ef34b8 0x00000000
0x1ae6e50: 0x86ef0240 0x00000000 0x86ef0fb0 0x00000000
DDR
Bjørn Vårdal IBM Runtime Technologies 25
GDB vs DDR
GDB

26. Bjørn Vårdal IBM Runtime Technologies 26
DDR Interactive
”The tool that I use the most and that saves me the most time.”
J9 Service Team Lead
“We can implement commands to look for known symptoms,
which drastically reduces our triage work.”
J9 Service Team Member

27. • We are open-sourcing the J9 JVM
• DDR coming along for the ride 
• http://openj9.org
Bjørn Vårdal IBM Runtime Technologies 27
OpenJ9

28. Reliable Shared Components for High Performance Language Runtimes
http://www.eclipse.org/omr
https://github.com/eclipse/omr
https://developer.ibm.com/open/omr
Bjørn Vårdal IBM Runtime Technologies 28
Eclipse OMR

29. Bjørn Vårdal IBM Runtime Technologies 29
Agenda
• DDR in J9
• Capabilities
• How DDR is implemented
• OMR ddrgen
• Generating the DDR resources
• DDR in a new runtime
• Adding DDR capabilities to Base9
• Loading a Base9 core into DDR Interactive

30. • Print structure at address
!j9object 0x12345678
!j9method 0xABCDEF00
• Other printing commands
!stackslots 0xABCDEF0
• Utility functions
!classforname
!findpattern
!whatis
• Data analysis
gccheck
deadlock
Bjørn Vårdal IBM Runtime Technologies 30
What can DDR do?
>

31. • Print structure at address
!j9object 0x12345678
!j9method 0xABCDEF00
• Other printing commands
!stackslots 0xABCDEF0
• Utility functions
!classforname
!findpattern
!whatis
• Data analysis
gccheck
deadlock
Bjørn Vårdal IBM Runtime Technologies 31
What can DDR do?
> gccheck
Starting GC Check
Checking HEAP...done (919 ms).
Checking CLASS HEAP...done (296 ms).
Checking REMEMBERED SET...done (54 ms).
Checking UNFINALIZED...done (3 ms).
Checking FINALIZABLE...done (3 ms).
Checking OWNABLE_SYNCHRONIZER...done (1 ms).
Checking STRING TABLE...done (516 ms).
Checking CLASS LOADERS...done (4 ms).
Checking JNI GLOBAL REFS...done (1 ms).
Checking JNI WEAK GLOBAL REFS...done (1 ms).
Checking JVMTI OBJECT TAG TABLES...done (3 ms).
Checking VM CLASS SLOTS...done (1 ms).
Checking MONITOR TABLE...done (7 ms).
Checking VM THREAD SLOTS...done (208 ms).
Checking THREAD STACKS...done (25 ms).
Done (2058ms)

32. • Structure layout information in
the process memory
• DDR data generator generates
blob and code
• Detectable RAS structure in
memory at runtime
Bjørn Vårdal IBM Runtime Technologies 32
How does DDR work?
Runtime
DDR InteractiveDDR data
generator
DDR
blob
Core dump
DDR
blob
Generated
code
RAS RAS

33. Bjørn Vårdal IBM Runtime Technologies 33
DDR Components
OpenJ9Eclipse OMR
J9 JVM
DDR data
generator
DDR Interactive
RAS

34. Bjørn Vårdal IBM Runtime Technologies 34
RAS Structure
Runtime memory
VM structure
Threads
Port library
etc.

35. Bjørn Vårdal IBM Runtime Technologies 35
RAS Structure
Runtime memory
RAS Structure
Eye catcher: “J9VMRAS”
Bit pattern: 0xAA55AA55
...
DDR Structure blob
…
VM
…
VM structure
Threads
Port library
etc.

36. Bjørn Vårdal IBM Runtime Technologies 36
RAS Structure
Runtime memory
DDR blob
Structure layouts
Field offsets
Build flags
RAS Structure
Eye catcher: “J9VMRAS”
Bit pattern: 0xAA55AA55
...
DDR Structure blob
…
VM
…
VM structure
Threads
Port library
etc.

37. Bjørn Vårdal IBM Runtime Technologies 37
DDR Structure Blob
Runtime memory
DDR blob
Structure layouts
Field offsets
Build flags
RAS Structure
Eye catcher: “J9VMRAS”
Bit pattern: 0xAA55AA55
...
DDR Structure blob
…
VM
…
VM structure
Threads
Port library
etc.

38. Bjørn Vårdal IBM Runtime Technologies 38
OMR ddrgen
Compile
with debug
symbols
ddrgen
blob.dat
Load into
memory at
runtime
Scan source
files for
build flags
./getMacros.sh .
libdwarf

39. Bjørn Vårdal IBM Runtime Technologies 39
DDR Class Generation
superset.out
DDR Interactive
(j9ddr.jar)
Generate Java
structure classes
superset.out
Generate Java
pointer classes

40. Bjørn Vårdal IBM Runtime Technologies 40
DDR Interactive startup
Generated
Java classes
DDR Interactive
DDR blob
from core file
DDR
ClassLoader
Template Field offsets
Load structure
classes

41. • DDR capabilities – Analyzing the JVM in a core file
• RAS structure – A detectable starting point
• OMR ddrgen – Creating the DDR blob and superset
• DDR Interactive – Using the blob and superset
Bjørn Vårdal IBM Runtime Technologies 41
Summary

42. Bjørn Vårdal IBM Runtime Technologies 42
DDR in a new Runtime

43. • Small tutorial runtime with ~10 opcodes
• For illustrating basic OMR JITBuilder functionality
• For demonstrating basic DDR functionality?
• My fork: https://github.com/bjornvar/Base9/tree/ddr
Bjørn Vårdal IBM Runtime Technologies 43
Base9

44. • Quick prototype
• ~1 hour of hacking
• Mostly copied from J9 JVM
Bjørn Vårdal IBM Runtime Technologies 44
DDR in Base9

45. Bjørn Vårdal IBM Runtime Technologies 45
RAS Structure in J9
JVM runtime memory
DDR blob
Structure layouts
Build flags
RAS Structure
Eye catcher: “J9VMRAS”
Bit pattern: 0xAA55AA55
...
DDR Structure blob
…
JavaVM
…
JavaVM structure
Threads
Port library
etc.

46. Bjørn Vårdal IBM Runtime Technologies 46
RAS Structure in Base9
Base9 runtime memory
DDR blob
Structure layouts
Build flags
Generated by running
ddrgen on Base9
RAS Structure
Eye catcher: “J9VMRAS”
Bit pattern: 0xAA55AA55
...
DDR Structure blob
…
ExecutionContext
Copied the structure
from the J9 source
ExecutionContext
Stack
String table
etc.

47. Bjørn Vårdal IBM Runtime Technologies 47
Base9 Code Changes

48. Bjørn Vårdal IBM Runtime Technologies 48
Base9 Code Changes

49. Bjørn Vårdal IBM Runtime Technologies 49
Base9 Code Changes

50. Bjørn Vårdal IBM Runtime Technologies 50
RAS Structure in Base9
Base9 runtime memory
DDR blob
Structure layouts
Build flags
RAS Structure
Eye catcher: “J9VMRAS”
Bit pattern: 0xAA55AA55
...
DDR Structure blob
…
ExecutionContext
ExecutionContext
Stack
String table
etc.

51. Bjørn Vårdal IBM Runtime Technologies 51
DDR Interactive
Some changes were required:
• Disabled all J9 specific functionality (!stackslots, etc.)
• Told it to look for B9RAS instead of J9RAS
• Added build flags constants

52. $
Bjørn Vårdal IBM Runtimes 52
DDR Interactive

53. Bjørn Vårdal IBM Runtime Technologies 53
What’s next?
• Make DDR Interactive plug-and-play
• Always support structure commands on DDR enabled core files
• Easy DDR generation process
• Release DDR Interactive and DDR generators
• OMR
• OpenJ9

54. Bjørn Vårdal IBM Runtime Technologies 54
Conclusion
• DDR in J9
• Capabilities
• How DDR is implemented
• OMR ddrgen
• Turning debug symbols into blob and superset
• DDR in Base9
• Adding DDR capabilities to a new runtime
• Loading a Base9 core into DDR Interactive

55. Bjørn Vårdal IBM Runtime Technologies 55
Q & A