SAP JVM Analysis of permanent generation memory

usage using jvmmon

Skip to end of metadata

Attachments:13
Added by Patrick Whitty, last edited by Patrick Whitty on Oct 17, 2012 (view change)
show comment
Go to start of metadata
Purpose
Outline the use of the SAP JVM Monitoring Tool 'jvmmon' to retrieve information about memory consumption in the
permanent generation (perm space) of SAP JVM
Overview
In depth analysis of memory consumption in the JVM heap space is made easy using the powerful heap dump analysis
features of the Eclipse Memory Analyzer Tool. However, heap dumps contain comparatively little information about perm
space, and as a result, the amount of the information about perm space memory consumption that can be gleaned from a
heap dump is limited. This document describes how one can use the SAP JVM Monitoring Tool 'jvmmon' and in certain
cases, both 'jvmmon' and Eclipse Memory Analyzer Tool together, to get detailed information about what is stored in perm
space memory

Getting information about perm space from a java heap dump
When faced with high memory consumption in the heap space (i.e. new and tenured generations) of a Hotspot VM, a typical
analysis approach is to trigger a heap dump and open the heap dump file with the Eclipse Memory Analyzer . One can then
use some of the many features of the Memory Analyzer to get information about the objects in memory, check for memory
leaks and determine the reason for the high memory footprint

Example GC output showing high memory consumption in heap space. The output of one minor GC and one full GC is
shown. Note that in both new generation and tenured generation the used memory before GC equals used memory after
GC indicating that no memory was freed up by the GC and that used memory after GC is almost equal to total available
memory for each generation, indicating that the heap memory is almost exhausted. This is also evident from the output
labelled 'Heap Space' where the total available memory, 'Total Mem' is calculated as the Max Heap Size minus one of the
survivor spaces. In contrast it is evident from the output labelled 'Perm Space' that some perm space memory was freed
during the Full GC and that only approximately 50% of total available perm space memory is used


In contrast to the new and tenured generation which contain the java objects in memory, the permanent generation (perm
space) contains class and method data as well as interned strings. Memory exhaustion in perm space can be caused by too
many classes loaded into memory, leaking class loaders or too many interned strings.



Example GC output showing high memory consumption in perm space. The output of two full GCs is shown. From the
output labelled 'Tenured Gen' one can see that prior to and after the GC cycle the amount of used memory in the Tenured
Generation is just 67% of total available memory in that generation. From the output labelled 'Heap Space' one can see that
prior to and after the GC cycle the amount of used memory is just 59% of total available in the entire heap space minus one
of the survivor spaces. However it is evident from the output labelled 'Perm Space' that the used memory before and after
the GC is almost equal to the total available memory, indicating that there is almost no free memory in the Permanent
Generation. Furthermore the output shows that no memory was freed in perm space during the last two GC cycles
suggesting that an OutOfMemoryError is likely to occur due to perm space exhaustion


You can use the Eclipse Memory Analyzer to get some information about perm space from the heap dump e.g. to list the
class names, to check for duplicate classes and leaking class loaders, however the heap dump provides no information
about interned strings. For more information refer to the article The Unknown Generation: Perm by Andreas Buchen.
To get a more in-depth look at whats stored in perm space one can use jvmmon.
Getting information about perm space using jvmmon
If you would like to acquire some statistics about what is stored in perm space, for example perhaps you suspect excessive
memory consumption in perm space based on garbage collection (GC) output analysis or memory profiling, then you can
use SAP JVM Monitoring Tool jvmmon to do so
jvmmon is an SAP JVM specific tool for a wide range of monitoring tasks. It is part of the SAP JVM delivery and can be
found in the same directory as the VM itself under the bin folder.
1. Start the graphical user interface jvmmon-gui by executing jvmmon -gui from a console or using jvmmon-gui.exe

2. Select the VM that you would like to acquire statistics on for example the process exhibiting excessive memory
consumption.
3. In the Dump menu there are three dump commands that output information about perm space; Dump class statistic, Dump
detailed class statistic and Dump symbol
info.

4. The output of these commands is written to the standard out of the process, which for applications running on the AS Java is
the std_<process><n>.out file in the work directory of the instance (usr/sap/<SID>/<InstanceID>/work). The class statistic
summarizes the objects found in the generations of the Java heap. It can display the number and the cumulated size of the
objects in specific generations.
Class statistic

Choose Dump detailed class stastic and in the output three collections of statistics concerning perm space can be found
under the headings:Class statistic for perm generation, Class meta data statistic, Class loader statistic
Class statistic for perm generation
This is a list of classes ordered by their size in bytes. This allows you to determine the largest memory consumers
| Class statistic for perm generation at GC Nr.
| Rank | # objects | Size in bytes | Class
|----------------------------------------------------------------------------------------------------------------
| 1 | 187315 | 31820816 | {constMethod}
| 2 | 187315 | 23993552 | {method}
| 3 | 17266 | 21755656 | {constant pool}
| 4 | 323584 | 18179136 | {symbol}
| 5 | 17266 | 13911408 | {instance class}
| 6 | 14044 | 12209216 | {constant pool cache}
| 7 | 9912 | 5229688 | {method data}
| 8 | 54379 | 4125456 | char[]
| 9 | 20884 | 3078096 | byte[]
| 10 | 42195 | 2580792 | int[]
| 11 | 18322 | 1905488 | java.lang.Class
| 12 | 54236 | 1735552 | java.lang.String
| 13 | 20776 | 1583824 | short[]
| 14 | 26566 | 1344472 | java.lang.Object[]
| 15 | 1027 | 616200 | {object array class}
| 16 | 170 | 5440 | {compiledICHolder}
| 17 | 8 | 4800 | {type array class}
| 18 | 12 | 2496 | {other class}
| 19 | 15 | 480 | java.lang.OutOfMemoryError
| 20 | 10 | 240 | com.sap.jvm.impl.monitor.vm.natives.NativeOutOfMemoryContext
| 21 | 1 | 32 | java.lang.ref.Reference
| 22 | 1 | 32 | java.lang.ArithmeticException
| 23 | 1 | 32 | java.lang.VirtualMachineError
| 24 | 1 | 32 | java.lang.NullPointerException
| 25 | 1 | 16 | java.lang.Object
|----------------------------------------------------------------------------------------------------------------
| | 995307 | 144082952 | *all classes*

As is evident from 'Class statistic for perm generation' the permanent generation contains normal Java objects like character
arrays or java.lang.String objects and also some preallocated exceptions which are allocated in advance to make it possible
to throw them without having to allocate memory on the Java heap.
In addition there are internal objects used by the JVM to store class and method data, indentified by the curly braces in
which they are written. A brief description of some of these internal objects follows:
The 'instance class' object stores the basic data for each non-array class loaded by the VM
The 'type array class' object stores the basic data for each one-dimensional primitive array
The 'object array class' object stores the basic data for each non-primitive array and multi-dimensional arrays of a primitive
type.
The 'constant pool' is an object which represents the constant pool of a class.
When an entry from the constant pool is first used by the VM it is said to be resolved. The 'constant pool cache' holds the
resolved result
The 'method' objects store basic information about each method of a class, like access flags or the maximum number of
local variables
The 'const method' objects store the most important properties of a method
The 'method data' object is associated with a 'method' object and contains profiling information gathered during execution
of the method
The 'symbol' object contains an UTF-8 string. 'symbol' objects are mostly used by the constant pool objects to store strings.
Note that 'symbol' objects are unique, meaning that there are no two 'symbol' objects which represent the same string. In
this regard they are similar to a interned string, but instead of being areal java.lang.String object they are more like a C
string

Class meta data statistic
In Class meta data statistic classes are listed ordered by their meta data size with each classs class loader ID and
number of metadata objects also shown. The meta-data of a class are the internal objects which can be attributed to that
class. For a normal class like java.lang.String this would be their 'instance class', the associated 'constant pool' and
'constant pool cache' and the 'method', 'method data' and 'const method' objects of that class and all the primitive arrays
these refer too. What's not included are the ''symbol' objects and interned strings, since these might be shared between
many classes.
| Class meta data statistic at GC Nr.
| Rank | # meta data | size meta data | Class Loader Id | Class Id | Class
|----------------------------------------------------------------------------------------------------------------
| 1 | 2529 | 400336 | 0 | 1d52b |
com.sun.corba.se.impl.logging.ORBUtilSystemException
| 2 | 269 | 266288 | 452e | b00b | com.sap.sql.sqlparser.SQLParser
| 3 | 13 | 228992 | d26e | 20fa2 |
com.sap.sld.api.wbem.sap.SLDElementNames
| 4 | 521 | 223584 | 73e34 | 74404 |
com.sap.workflow.worklist.ejb.EWorkListServerBean
| 5 | 1127 | 194928 | d351 | 1792c |
com.sap.tc.lm.nzdm.crr.srv.util.sapControlWS.ObjectFactory
| 6 | 1127 | 194928 | 4510b | 4525f | sapcontrol.ObjectFactory
| 7 | 1146 | 183688 | d284 | 3a670 | com.sap.lcr.api.cim.CIMFactory
Class loader statistic
In Class loader statistic class loaders are listed in the order of number of classes defined. Each class is defined by exactly
one class loader and this is normally the class loader that loaded the definition of the class. Class loader statistic allows one
to do a mapping between the class loader IDs shown in Class meta data statistic and the class loader names. The overall
number of meta-data objects attributed to all the classes defined by the respective loader and the overall size of these meta-
data objects is also shown
| Class loader statistic at GC Nr.
| Rank | # classes | # meta data | size meta data | Class Loader Id | Class loader
|----------------------------------------------------------------------------------------------------------------
| 1 | 4118 | 120857 | 24501640 | 0 | <bootstrap>
| 2 | 3366 | 127559 | 21265496 | 4f566 | sap.com/com.sap.xi.repository
| 3 | 1702 | 57010 | 12557856 | 5a82a | sap.com/tc~voear
| 4 | 1425 | 35702 | 6733784 | 911a7 | sap.com/tc~dtr~enterpriseapp
| 5 | 1393 | 43795 | 8514776 | 1d5d | library:core_lib
| 6 | 1188 | 24255 | 3658136 | 6c307 | test.sap.com/caf~uicoupl~testappl~v2~ear
| 7 | 971 | 18730 | 2788464 | d351 | service:tc~lm~nzdm~crrsrv
| 8 | 921 | 25124 | 5312208 | d27f | library:tc~je~j2eedescriptors~lib
| 9 | 883 | 16418 | 2325096 | 4510b | sap.com/tc~lm~ctc~util~wsclient~proxy_ear
Class loader statistic is particularly useful if you are trying to track down leaking class loaders. If there is a certain class
loader that is causing the problem it will typically appear near the top of this list.
Consider the following example output showing multiple copies of a classloader WCE_PRIVATE_CL_JavaEE/leakproducer,
all with different classloader IDs. This indicates that there are leaking classloaders
| 385 | 2 | 72 | 9096 | d125 | library:sqlmapper
| 386 | 2 | 56 | 8968 | d0fe | library:tc?je?exprlang?plb
| 387 | 2 | 30 | 7224 | 57dfe | WCE_PRIVATE_CL_JavaEE/leakproducer
(disabled)
| 388 | 2 | 30 | 7224 | 582db | WCE_PRIVATE_CL_JavaEE/leakproducer
| 389 | 2 | 30 | 7224 | 54730 | WCE_PRIVATE_CL_JavaEE/leakproducer
(disabled)
| 390 | 2 | 30 | 7224 | 582c5 | WCE_PRIVATE_CL_JavaEE/leakproducer
(disabled)
| 391 | 2 | 51 | 6456 | 4ba6b |
sap.com/com.sap.portal.runtime.system.clusterinformation
| 392 | 2 | 47 | 6392 | d0fc | interface:endpoint_api
A possible next investigative step is to acquire a heap dump and parse it with a Memory Analyser Tool that includes the SAP
Netweaver extension.
The Leaking Loaders feature of MAT allows you to list the leaking loaders and check why they are still held in memory. A
possible cause is running threads not stopped during removal of an application or in a servlets destroy method. When a
thread has not died it holds a reference to its context classloader and the classloader cannot be collected. When the
classloader cannot be collected, it keeps resources in Perm space. Another possible cause is neglecting to unregister
objects held by a classloader from a HashMap held by another classloader. In that case the object remaining in the
datastructure will keep the reference to its classloader and prevent it from being freed.
Select Leaking loaders

Click on any of the leaked classloaders and right button click to show the context menu. There choose List objects -> with
incoming references

A number of still active threads are displayed preventing their classloaders from being collected.

Symbol Info
The other jvmmon command that provides information about perm space is Dump symbol info. It outputs a list of the top 10
interned strings and symbols

This can be particularly useful if based on the output of Dump class statistic" or "Dump detailed class statistic it appears that
"symbols" consume most of the memory then you can use the option "Dump symbol info" to get more information about the
strings and symbols. Often the string content itself will hint at who is responsible for the internment of the string.
String Table Dump:
0: Interned String ( 2267): 'Test a long intern string. Test a long intern string. Test a long intern string. Test
a long intern
1: Interned String ( 1440): ',ADAND,AEARE,AFAFG,AGATG,AIAIA,ALALB,AMARM,ANANT,AOAGO,AQATA,ARARG,
2: Interned String ( 1316): '

3: Interned String ( 1128):
',aaaar,ababk,aeave,afafr,akaka,amamh,anarg,arara,asasm,avava,ayaym,azaze,babak,bebel,
4: Interned String ( 56): '%(\d+\$)?([-#+ 0,(\<]*)?(\d+)?(\.\d+)?([tT])?([a-zA-Z%])'
5: Interned String ( 48): 'Java(TM) 2 Runtime Environment, Standard Edition'
6: Interned String ( 37): 'sun.lang.ClassLoader.allowArraySyntax'
7: Interned String ( 37): 'com.sap.jvm.UseUCPCacheForGetResource'
8: Interned String ( 36): 'Ljava/security/PermissionCollection;'
9: Interned String ( 35): 'com.sap.vmc.sharedSystemClassLoader'

Symbol Table Dump:
0: Symbol ( 2267): 'Test a long intern string. Test a long intern string. Test a long intern string. Test a long
intern stri.
1: Symbol ( 1440): ',ADAND,AEARE,AFAFG,AGATG,AIAIA,ALALB,AMARM,ANANT,AOAGO,AQATA,ARARG,ASA
2: Symbol ( 1316): '
3: Symbol ( 1128): ',aaaar,ababk,aeave,afafr,akaka,amamh,anarg,arara,asasm,avava,ayaym,azaze,babak,bebel,bgbul,
4: Symbol ( 188):
'(Ljava/lang/String;Ljava/util/Map<Ljava/lang/String;Ljava/lang/String;>;Ljava/util/Map<Ljava/lang/Stri
5: Symbol ( 172): '(Lcom/sap/vmc/core/impl/sapjvm/sharing/NativeSharedClassLoaderImpl;Lcom/sap/vmc/core/impl/s
6: Symbol ( 171):
'<U:Ljava/lang/Object;W:Ljava/lang/Object;>(Ljava/lang/Class<TU;>;Ljava/lang/Class<TW;>;Ljava/l
7: Symbol ( 166):
'(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/
8: Symbol ( 165):
'(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ILjava/lang
9: Symbol ( 161):
'(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/
Related Content
Related Documents
The SAP Java VM GC Configuration And Troubleshooting Guide by Ralf Schmelter
SAP JVM - SAP Documentation
Configuration and Setup of SAP JVM
Related Notes