agent/doc/transported_core.html

Mon, 28 Jul 2014 15:06:38 -0700

author
fzhinkin
date
Mon, 28 Jul 2014 15:06:38 -0700
changeset 6997
dbb05f6d93c4
parent 0
f90c822e73f8
permissions
-rw-r--r--

8051344: JVM crashed in Compile::start() during method parsing w/ UseRTMDeopt turned on
Summary: call rtm_deopt() only if there were no compilation bailouts before.
Reviewed-by: kvn

     1 <html>
     2 <head>
     3 <title>
     4 Debugging transported core dumps
     5 </title>
     6 </head>
     7 <body>
     8 <h1>Debugging transported core dumps</h1>
    10 <p>
    11 When a core dump is moved to a machine different from the one where it was
    12 produced ("transported core dump"), debuggers (dbx, gdb, windbg or SA) do not
    13 always successfully open the dump. This is due to kernel, library (shared
    14 objects or DLLs) mismatch between core dump machine and debugger machine.
    15 </p>
    17 <p>
    18 In most platforms, core dumps do not contain text (a.k.a) Code pages.
    19 There pages are to be read from executable and shared objects (or DLLs).
    20 Therefore it is important to have matching executable and shared object
    21 files in debugger machine. 
    22 </p>
    24 <h3>Solaris transported core dumps</h3>
    26 <p>
    27 Debuggers on Solaris (and Linux) use two addtional shared objects
    28 <b>rtld_db.so</b> and <b>libthread_db.so</b>. rtld_db.so is used to
    29 read information on shared objects from the core dump. libthread_db.so
    30 is used to get information on threads from the core dump. rtld_db.so
    31 evolves along with rtld.so (the runtime linker library) and libthread_db.so
    32 evolves along with libthread.so (user land multithreading library). 
    33 Hence, debugger machine should have right version of rtld_db.so and
    34 libthread_db.so to open the core dump successfully. More details on
    35 these debugger libraries can be found in 
    36 <a href="http://docs.sun.com/app/docs/doc/817-1984/">
    37 Solaris Linkers and Libraries Guide - 817-1984</a>
    38 </p>
    40 <h3>Solaris SA against transported core dumps</h3>
    42 <p>
    43 With transported core dumps, you may get "rtld_db failures" or
    44 "libthread_db failures" or SA may just throw some other error
    45 (hotspot symbol is missing) when opening the core dump. 
    46 Enviroment variable <b>LIBSAPROC_DEBUG</b> may be set to any value
    47 to debug such scenarios. With this env. var set, SA prints many
    48 messages in standard error which can be useful for further debugging.
    49 SA on Solaris uses <b>libproc.so</b> library. This library also
    50 prints debug messages with env. var <b>LIBPROC_DEBUG</b>. But,
    51 setting LIBSAPROC_DEBUG results in setting LIBPROC_DEBUG as well.
    52 </p>
    53 <p>
    54 The best possible way to debug a transported core dump is to match the
    55 debugger machine to that of core dump machine. i.e., have same Kernel
    56 and libthread patch level between the machines. mdb (Solaris modular
    57 debugger) may be used to find the Kernel patch level of core dump
    58 machine and debugger machine may be brought to the same level.
    59 </p>
    60 <p>
    61 If the matching machine is "far off" in your network, then
    62 <ul>
    63 <li>consider using rlogin and <a href="clhsdb.html">CLHSDB - SA command line HSDB interface</a> or
    64 <li>use SA remote debugging and debug the core from core machine remotely.
    65 </ul>
    66 </p>
    68 <p>
    69 But, it may not be feasible to find matching machine to debug. 
    70 If so, you can copy all application shared objects (and libthread_db.so, if needed) from the core dump 
    71 machine into your debugger machine's directory, say, /export/applibs. Now, set <b>SA_ALTROOT</b> 
    72 environment variable to point to /export/applibs directory. Note that /export/applibs should either 
    73 contain matching 'full path' of libraries. i.e., /usr/lib/libthread_db.so from core 
    74 machine should be under /export/applibs/use/lib directory and /use/java/jre/lib/sparc/client/libjvm.so 
    75 from core machine should be under /export/applibs/use/java/jre/lib/sparc/client so on or /export/applibs 
    76 should just contain libthread_db.so, libjvm.so etc. directly. 
    77 </p>
    79 <p>
    80 Support for transported core dumps is <b>not</b> built into the standard version of libproc.so. You need to
    81 set <b>LD_LIBRARY_PATH</b> env var to point to the path of a specially built version of libproc.so. 
    82 Note that this version of libproc.so has a special symbol to support transported core dump debugging. 
    83 In future, we may get this feature built into standard libproc.so -- if that happens, this step (of 
    84 setting LD_LIBRARY_PATH) can be skipped.
    85 </p>
    87 <h3>Ignoring libthread_db.so failures</h3>
    88 <p>
    89 If you are okay with missing thread related information, you can set 
    90 <b>SA_IGNORE_THREADDB</b> environment variable to any value. With this
    91 set, SA ignores libthread_db failure, but you won't be able to get any
    92 thread related information. But, you would be able to use SA and get
    93 other information.
    94 </p>
    96 <h3>Linux SA against transported core dumps</h3>
    97 <p>
    98 On Linux, SA parses core and shared library ELF files. SA <b>does not</b> use
    99 libthread_db.so or rtld_db.so for core dump debugging (although 
   100 libthread_db.so is used for live process debugging). But, you
   101 may still face problems with transported core dumps, because matching shared
   102 objects may not be in the path(s) specified in core dump file. To
   103 workaround this, you can define environment variable <b>SA_ALTROOT</b>
   104 to be the directory where shared libraries are kept. The semantics of
   105 this env. variable is same as that for Solaris (please refer above).
   106 </p>
   109 </body>
   110 </html>

mercurial