Native out-of-memory errors happen when a Java application runs out of memory, not in the Java object heap but outside it. The cause may be memory use for native libraries, class loading, multithreading, working data for the Java VM, backing storage for Java objects, or other reasons. No single tool can give you all the answers, and we need to cross-reference information from multiple sources to isolate a problem. Operating system tools, Java dumps, logs, and debuggers all provide useful perspectives, and your challenge is to line them up to see the whole picture. This session works through the tools and data available on the main server platforms to give you a repeatable framework for native out-of-memory error debug.

1. Matthew Kilner – IBM Java L3 Service – Core team lead
23rd September 2013
Debugging Native Heap OOM - Tools &
Techniques
© 2013 IBM Corporation

2. Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT
PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE
USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR
SUPPLIERS AND/OR LICENSORS
2
© 2013 IBM Corporation

3. About me
 Matthew Kilner

 Work for IBM
– 13 years working on IBM Java
• Memory Management
• Class Sharing
• RAS
– Currently leading the Core customer support team.

 Contact info
– kilnerm@uk.ibm.com
– Twitter: @IBMJTC
– Youtube: IBM_JTC

 Visit the IBM booth #5112 and meet other IBM developers
at JavaOne 2013
3
© 2013 IBM Corporation

4. What should you get from this talk?
 An understanding of what we mean by the Native Heap.

 A clear problem determination path for Native Heap OOM Errors:
– How to determine you have a native heap OOM.
– An outline process for determining what is causing it.
© 2013 IBM Corporation

5. The Java process
 All applications run within the bounds of an operating system process
– Java is no exception
–
 The JVM is subject to the same restrictions as any other application, the most pertinent being:
• Addressing
• OS Memory Model
© 2013 IBM Corporation

6. What do we mean by addressing restrictions?
 Every process has a finite address space which is dictated by the architecture it runs on.
–
 A 32bit architecture has an addressable range of:
– 2^32
–0x00000000 – 0xFFFFFFFF
–which is 4GB
–
 A 64bit architecture has an addressable range of:
–2^64
–0x0000000000000000 – 0xFFFFFFFFFFFFFFFF
–which is 16 EiB
© 2013 IBM Corporation

7. What do we mean by Memory Model Restrictions?
 Not all addressable memory is available to a process.
 The operating system has its own requirements such as:
–The kernel
–The runtime support libraries
 Requirements vary by Operating System both in terms of:
– How much memory is needed, and
– Where that memory is located
 The addressable memory remaining is often referred to as “User Space”
© 2013 IBM Corporation

8. A view by platform
 The chart shows default maximum user space available on common 32-bit platforms
9
8
7
6
5
GiB
4
3
2
1
0
Windows 32
Windows 32 /3GB
Linux 32 bit
Linux 32 bit
Hugemem
Operating System
zLinux 31
AIX 32
zOS
Kernel Space
User Space
© 2013 IBM Corporation

9. What goes in the “User Space”
Kernel Space
 Java Heap
 Just In Time (JIT) Data
–Runtime data & executable code
 Virtual Machine Resources
–RAS engines & GC Infrastructure
 Native & JNI Allocations
 Resources to underpin Java Objects
–Classes and ClassLoaders
–Threads
–Direct java.nio.ByteBuffers
–Sockets
Java Heap
User Space
JIT Data
VM Resources
Native & JNI
Allocations
Java Libraries
© 2013 IBM Corporation

10. The Native heap
 We define the native heap as:
–
–
Native Heap = “User Space” - Maximum Heap Size
–
 It is the total “User Space” not reserved for backing the Java Heap.


© 2013 IBM Corporation

11. Not quite the whole story
 “User Space” availability is not our only consideration when looking at native memory shortage.

 Machines have to be able to back addressable memory with physical memory.

 The total physical memory available on a machine is

Physical RAM + Swap Space

 Problem symptoms vary based on which resource runs out.
© 2013 IBM Corporation

12. Failure Symptoms
 The chart shows some of the symptoms you see when a particular resource is exhausted
Resource
Address Space
OutOfMemoryError
Symptoms
Physical RAM
OutOfMemoryError
Memory Pages
Console Messages
Crash/Other?
Physical + Swap
(Virtual Memory)
Unresponsive apps
Linux: OOM Killer
Win/Sol: alloc's fail
© 2013 IBM Corporation

13. How do I know if I have a problem?
 Detecting a problem is easy if you hit one of the symptoms described previously.
– OutOfMemoryError's should be fatal to your application.
– Paging will cause obvious unresponsiveness or slowdown.

 Early detection is possible if you monitor the size of your process.
– Monitoring is also an important part of understanding any native memory issue.
© 2013 IBM Corporation

14. Monitoring the size of your process
 Process sizes are reported in two ways across all platforms:
– Resident Size
– Virtual Size
 Each platform has its own methods for obtaining this information:
– Windows:
Performance Monitor
– Linux & z/OS:
ps
– AIX:
svmon
 The IBM Garbage Collection & Memory Visualizer (GCMV) tool provides scripts and instructions
in its help documentation for gathering the necessary data
© 2013 IBM Corporation

15. Plotting the size of your process
 Analysis of the process size is best done visually.

 GCMV and Performance Monitor will plot the raw data for you.

 A persistent growth in the virtual size of your process may indicate an issue.


© 2013 IBM Corporation

16. How do I identify a Native OOM from a javacore?
 When any OOME occurs the IBM JVM's default configuration will write a javacore file.

 This file provides several pointers to the fact you have an OOM related to the native heap:
– A header that includes information on which resource cannot be allocated
– Details of the current memory usage on the Java heap
– A record of recent Garbage Collection activity
– The stack of the thread encountering the problem



© 2013 IBM Corporation

17. The javacore header
 At the top of each javacore is its header which tells you what event caused the file to be written.

 Under certain conditions additional information is written at the head of the javacore when the
OOME is triggered:
1TISIGINFO
Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal
-1073741830, errno 11" received

 On a Java Heap OOM you will see:
1TISIGINFO
Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Java heap space" received
© 2013 IBM Corporation

18. The javacore heap usage summary
 Within the javacore you will find the MEMINFO section.

0SECTION
MEMINFO subcomponent dump routine
NULL
=================================
1STHEAPFREE Bytes of Heap Space Free: 3B3AE8
1STHEAPALLOC Bytes of Heap Space Allocated: 400000

 If you see a large value for bytes free then it is a good indicator that you are experiencing a native
heap OOME.


© 2013 IBM Corporation

19. The javacore GC history
The javacore file also contains a snapshot of the most recent GC activity.
Where an OOM is due to a java heap allocation failing you will see this in the data:

1STGCHTYPE
3STHSTTYPE
GC History
14:12:41:476340000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 8000024 bytes requested for
object of class 00007FD4801D6E10 from memory space 'Flat'
id=00007FD480046EE8
If this entry is not present it is another good indicator you are experiencing a native heap
OOME.
© 2013 IBM Corporation

20. The javacore current thread
The current thread within the javacore is the thread which triggered the OOME
– The top stack frame contains the interesting data
You can identify whether the frame is native:
3XMTHREADINFO
...............
3XMTHREADINFO3
4XESTACKTRACE
4XESTACKTRACE
"main" J9VMThread:0xB8D1C600, j9thread_t:0xB8D019E4, java/lang/Thread:0x98B01960, state:R, prio=5
Java callstack:
at java/lang/Thread.startImpl(Native Method)
at java/lang/Thread.start(Thread.java:891)
Or java:
3XMTHREADINFO
...............
3XMTHREADINFO3
4XESTACKTRACE
"main" J9VMThread:0x00007FD480043D00, j9thread_t:0x00007FD4800079B0,
Java callstack:
at StringOOM.main(StringOOM.java:11)
If it is native then this is another good indicator you are experiencing a native heap OOME.
© 2013 IBM Corporation

21. How do I find out what is causing my native OOME?
 It can be tricky to attribute a root cause to a Native OOME
– Debug capabilities vary by platform.

 The fundamental approach is the same irrespective of platform:
• 1) Understand the rate of Native Memory Growth
• 2) Capture multiple snapshots of data over time.
• 3) Compare the snapshots and attribute growth to components.
© 2013 IBM Corporation

22. Understanding the rate of memory growth
The rate of memory growth can be determined from the size of your process.

Calculate the delta in virtual size of your process between data snapshots.

Other data snapshots will identify different areas of native memory growth, understanding
the proportion each area contributes to the total growth is key to identifying a root cause.
© 2013 IBM Corporation

23. What other data is needed?
Some data is common across platforms, other data is platform specific

Common data:
– Javacore files taken at regular intervals
– Core files taken at regular intervals (optional)

Platform specific data:
– Windows:
UMDH tracing, Debug Diag tracing, VMMAP tracing
– Linux:
No recommended tools
– AIX:
Debug malloc tracing
© 2013 IBM Corporation

24. The javacore NATIVEMEMINFO section
From the J9 2.4 JVM the javacore file contains a NATIVEMEMINFO section.
NATIVEMEMINFO subcomponent dump routine
=======================================
JRE: 555,698,264 bytes / 1208 allocations
+--VM: 552,977,664 bytes / 856 allocations
| +--Classes: 1,949,664 bytes / 92 allocations
| +--Memory Manager (GC): 547,705,848 bytes / 146 allocations
| | +--Java Heap: 536,875,008 bytes / 1 allocation
| | +--Other: 10,830,840 bytes / 145 allocations
| +--Threads: 2,660,804 bytes / 104 allocations
| | +--Java Stack: 64,944 bytes / 9 allocations
| | +--Native Stack: 2,523,136 bytes / 11 allocations
| | +--Other: 72,724 bytes / 84 allocations
| +--Trace: 92,464 bytes / 208 allocations
| +--JVMTI: 17,328 bytes / 13 allocations
| +--JNI: 15,944 bytes / 32 allocations
| +--Port Library: 6,824 bytes / 56 allocations
| +--Other: 528,788 bytes / 205 allocations
+--JIT: 1,748,808 bytes / 82 allocations
| +--JIT Code Cache: 524,320 bytes / 1 allocation
| +--JIT Data Cache: 524,336 bytes / 1 allocation
| +--Other: 700,152 bytes / 80 allocations
+--Class Libraries: 971,792 bytes / 270 allocations
| +--Harmony Class Libraries: 1,024 bytes / 1 allocation
| +--VM Class Libraries: 970,768 bytes / 276 allocations
| | +--sun.misc.Unsafe: 69,688 bytes / 1 allocation
| | +--Other: 901,080 bytes / 275 allocations
Comparing the output from multiple javacores can identify areas of growth in the JVM
If an identified area is a significant portion of the total memory growth it is likely the cause
of the problem.
© 2013 IBM Corporation

25. Javacores from earlier JDK versions
If your JDK is based on a JVM earlier than the J9 2.4 JVM the javacore file doe not have a
NATIVEMEMINFO section.

They do still contain valuable insight, although a little more work is required to obtain it.
– The “MEMINFO subcomponent dump routine” lists a summary of memory blocks the
JDK has allocated for various purposes
– The “Classes subcomponent dump routine” lists a summary of classloaders and loaded
classes.

By parsing and comparing this information across multiple javacore files you can
determine you have any signs of a memory area growth or classloader leak.
© 2013 IBM Corporation

26. What core files offer
Binary core files provide the same view as the javacore but require processing with
external tools:
– Interactive Diagnostic Dump Explorer (IDDE).
– On earlier JDK versions they provide a more accurate summary of JDK memory
allocations the the javacore file.

They also provide a complete image of the process, which means:
– You can inspect the content of memory
– You can inspect free memory blocks (subject to platform)
– You can inspect allocated memory blocks (subject to platform)

These advantages are offset by the size of the files and additional overhead of processing
them.
© 2013 IBM Corporation

27. Windows tooling
Windows offers three excellent options for understanding your native memory growth:
– UMDH
– DebugDiag
– VMMAP

Each has distinct usage characteristics:
– UMDH is command line driven.
– DebugDiag injects a tracking library into the process and parses core dumps.
– VMMAP launches the application you wish to track and is GUI based.
© 2013 IBM Corporation

28. Linux tooling
Commercial tools are available but carry a license fee

Free tools also exist but carry a large performance overhead

We have custom built tooling that logs all calls to allocate and free memory
– Building your own is possible.
© 2013 IBM Corporation

29. AIX tooling
 AIX provides a debug extension directly into the malloc subsystem
– MALLOCDEBUG
–
 Enables tracing in the allocation subroutines

 At termination of the process a report is generated detailing all allocations that were not freed
– Some additional parsing is needed
© 2013 IBM Corporation

30. What the platform tooling tells you
While each platform has different tools, the end result from them is largely the same

The tools give you one or more stack traces that relate to memory allocations that have
not been freed.

You are looking for the stacks that demonstrate the same or similar rates of growth as the
total process size between snapshots of data.
© 2013 IBM Corporation

31. What next?
The next step depends on the stack that has been identified as the root cause.
–
– If it is native code you own:
• Check to see you are releasing the memory your are allocating
–
–If it is native code relating to a java class:
• Check that you don't have an on heap leak of the related object type
• Check for known issues
• Contact the JDK vendor for assistance
–
–If it is third party native code:
• Check for any known issues
• Contact the vendor for assistance
© 2013 IBM Corporation

32. In Summary
The process for diagnosis and root cause determination for a native OOME is as follows:
1) Understand the limitations of the platform
2) Monitor the size of the process to understand the rate of memory growth
3) Use a combination of JDK and platform diagnostics to determine the area or stack
driving the growth
© 2013 IBM Corporation

33. I would like to know more
 Visit the IBM booth #5112

 BOF 4159 - The Most Useful Tools for Debugging on Windows
– Today: 9/23/13 (Monday) 7:30 PM - Hilton - Continental Ballroom 6

 “Thanks for the memory”
– http://www.ibm.com/developerworks/java/library/j-nativememory-linux/
– https://www.ibm.com/developerworks/java/library/j-nativememory-aix/
–

© 2013 IBM Corporation

34. Questions?
–

© 2013 IBM Corporation

35. IBM@JavaOne
http://ibm.co/JavaOne2013
35
IBM Confidential
© 2013 IBM Corporation