Mon, 28 Feb 2011 22:35:01 -0800
Merge
aoqi@0 | 1 | <HTML> |
aoqi@0 | 2 | |
aoqi@0 | 3 | |
aoqi@0 | 4 | <HEAD> |
aoqi@0 | 5 | <TITLE> |
aoqi@0 | 6 | Using The HotSpot Serviceability Agent |
aoqi@0 | 7 | </TITLE> |
aoqi@0 | 8 | </HEAD> |
aoqi@0 | 9 | |
aoqi@0 | 10 | <BODY TEXT="#000000" BGCOLOR="#FFFFFF"> |
aoqi@0 | 11 | <H1> |
aoqi@0 | 12 | Using The HotSpot Serviceability Agent |
aoqi@0 | 13 | </H1> |
aoqi@0 | 14 | |
aoqi@0 | 15 | <P> |
aoqi@0 | 16 | <H2> |
aoqi@0 | 17 | Contents |
aoqi@0 | 18 | </H2> |
aoqi@0 | 19 | </P> |
aoqi@0 | 20 | |
aoqi@0 | 21 | <UL> |
aoqi@0 | 22 | <LI> <A HREF = "#INTRODUCTION">Introduction</A> |
aoqi@0 | 23 | <LI> <A HREF = "#SOURCES">Organization of the sources</A> |
aoqi@0 | 24 | <LI> <A HREF = "#RUNNING">Running HSDB</A> |
aoqi@0 | 25 | <LI> <A HREF = "#NOTES">Notes</A> |
aoqi@0 | 26 | </UL> |
aoqi@0 | 27 | |
aoqi@0 | 28 | <H2> |
aoqi@0 | 29 | <A NAME="INTRODUCTION"> |
aoqi@0 | 30 | Introduction |
aoqi@0 | 31 | </A> |
aoqi@0 | 32 | </H2> |
aoqi@0 | 33 | |
aoqi@0 | 34 | <P> |
aoqi@0 | 35 | The HotSpot Serviceability Agent (SA) is a set of Java APIs which |
aoqi@0 | 36 | mirror the internal APIs of the HotSpot VM and which can be used to |
aoqi@0 | 37 | examine the state of a HotSpot VM. |
aoqi@0 | 38 | </P> |
aoqi@0 | 39 | |
aoqi@0 | 40 | <P> |
aoqi@0 | 41 | The system understands the layout of certain VM data structures and is |
aoqi@0 | 42 | able to traverse these structures in an examination-only fashion; that |
aoqi@0 | 43 | is, it does not rely on being able to run code in the target VM. For |
aoqi@0 | 44 | this reason it transparently works with either a running VM or a core |
aoqi@0 | 45 | file. |
aoqi@0 | 46 | </P> |
aoqi@0 | 47 | |
aoqi@0 | 48 | <P> |
aoqi@0 | 49 | The system can reconstruct information about Java frames on the stack |
aoqi@0 | 50 | and objects in the heap. Many of the important data structures in the |
aoqi@0 | 51 | VM like the CodeCache, Universe, StubQueue, Frames, and VFrames have |
aoqi@0 | 52 | been exposed and have relatively complete (or at least useful) |
aoqi@0 | 53 | implementations. |
aoqi@0 | 54 | </P> |
aoqi@0 | 55 | |
aoqi@0 | 56 | <P> |
aoqi@0 | 57 | A small graphical debugger called HSDB (the "HotSpot Debugger") has |
aoqi@0 | 58 | been written using these APIs. It provides stack memory dumps |
aoqi@0 | 59 | annotated with method invocations, compiled-code inlining (if |
aoqi@0 | 60 | present), interpreted vs. compiled code, interpreter codelets (if |
aoqi@0 | 61 | interpreted), and live oops from oop-map information. It also provides |
aoqi@0 | 62 | a tree-based oop inspector. More information will be added as |
aoqi@0 | 63 | necessary; please <A HREF = "mailto:kenneth.russell@eng">send |
aoqi@0 | 64 | email</A> with suggestions on what would be useful. |
aoqi@0 | 65 | </P> |
aoqi@0 | 66 | |
aoqi@0 | 67 | <P> |
aoqi@0 | 68 | The SA currently only works on Solaris. It uses dbx to connect to the |
aoqi@0 | 69 | remote process or core file and communicates with a small piece of |
aoqi@0 | 70 | code (an "import module") loaded into the debugger. |
aoqi@0 | 71 | </P> |
aoqi@0 | 72 | |
aoqi@0 | 73 | <H2> |
aoqi@0 | 74 | <A NAME="SOURCES"> |
aoqi@0 | 75 | Organization of the sources |
aoqi@0 | 76 | </A> |
aoqi@0 | 77 | </H2> |
aoqi@0 | 78 | |
aoqi@0 | 79 | <P> |
aoqi@0 | 80 | The Java-side source code, which is the bulk of the SA, is in |
aoqi@0 | 81 | src/share/vm/agent. The organization of the sun.jvm.hotspot package |
aoqi@0 | 82 | hierarchy mirrors the organization in the VM. This should allow |
aoqi@0 | 83 | engineers familiar with the HotSpot sources to quickly understand how |
aoqi@0 | 84 | the SA works and to make modifications if necessary. To build these |
aoqi@0 | 85 | sources, cd to src/share/vm/agent and type "make". |
aoqi@0 | 86 | </P> |
aoqi@0 | 87 | |
aoqi@0 | 88 | <P> |
aoqi@0 | 89 | |
aoqi@0 | 90 | The SA on Solaris works by communicating with a small piece of code |
aoqi@0 | 91 | (an "import module") loaded into dbx. The source code for this import |
aoqi@0 | 92 | module is in src/os/solaris/agent. To build this library, cd to |
aoqi@0 | 93 | src/os/solaris/agent and type "make 32bit" or "make 64bit". The |
aoqi@0 | 94 | distinction is necessary because the SPARC version of dbx ships with |
aoqi@0 | 95 | two versions of its executable, and depending on which architecture |
aoqi@0 | 96 | (v8 or v9) the debugger is running on selects the appropriate |
aoqi@0 | 97 | executable. The SA tries the v8 version first, but if you are running |
aoqi@0 | 98 | on a v9 machine you must provide both versions to the SA. |
aoqi@0 | 99 | </P> |
aoqi@0 | 100 | |
aoqi@0 | 101 | <P> |
aoqi@0 | 102 | |
aoqi@0 | 103 | The system is currently hardwired to look on jano for its dbx |
aoqi@0 | 104 | executable and import module. The relevant directory structure looks |
aoqi@0 | 105 | like this: |
aoqi@0 | 106 | |
aoqi@0 | 107 | <UL> |
aoqi@0 | 108 | <LI> .../hotspot/sa/ |
aoqi@0 | 109 | <UL> |
aoqi@0 | 110 | <LI> solaris/ |
aoqi@0 | 111 | <UL> |
aoqi@0 | 112 | <LI> sparc/ |
aoqi@0 | 113 | <UL> |
aoqi@0 | 114 | <LI> bin/ |
aoqi@0 | 115 | <UL> |
aoqi@0 | 116 | <LI> dbx: symlink to (v8) dbx 7.0 executable |
aoqi@0 | 117 | </UL> |
aoqi@0 | 118 | </UL> |
aoqi@0 | 119 | <UL> |
aoqi@0 | 120 | <LI> lib/ |
aoqi@0 | 121 | <UL> |
aoqi@0 | 122 | <LI> libsvc_agent_dbx.so: 32-bit version of import module |
aoqi@0 | 123 | </UL> |
aoqi@0 | 124 | </UL> |
aoqi@0 | 125 | <LI> sparcv9/ |
aoqi@0 | 126 | <UL> |
aoqi@0 | 127 | <LI> lib/ |
aoqi@0 | 128 | <UL> |
aoqi@0 | 129 | <LI> libsvc_agent_dbx.so: 32-bit version of import module |
aoqi@0 | 130 | </UL> |
aoqi@0 | 131 | </UL> |
aoqi@0 | 132 | </UL> |
aoqi@0 | 133 | </UL> |
aoqi@0 | 134 | </UL> |
aoqi@0 | 135 | </P> |
aoqi@0 | 136 | |
aoqi@0 | 137 | <P> |
aoqi@0 | 138 | The code which builds up path names to these executables is contained |
aoqi@0 | 139 | in sun.jvm.hotspot.HotSpotAgent.java. There are hardcoded paths in |
aoqi@0 | 140 | this file to jano, but the rest of the system is isolated from this. |
aoqi@0 | 141 | </P> |
aoqi@0 | 142 | |
aoqi@0 | 143 | <P> |
aoqi@0 | 144 | (It would be nice to have a panel in the debugger allowing |
aoqi@0 | 145 | configuration of all of the known operating systems' options; right |
aoqi@0 | 146 | now Solaris is the only supported OS, making that easier.) |
aoqi@0 | 147 | </P> |
aoqi@0 | 148 | |
aoqi@0 | 149 | <H2> |
aoqi@0 | 150 | <A NAME="RUNNING"> |
aoqi@0 | 151 | Running HSDB |
aoqi@0 | 152 | </A> |
aoqi@0 | 153 | </H2> |
aoqi@0 | 154 | |
aoqi@0 | 155 | <P> |
aoqi@0 | 156 | An installation of HSDB has been placed on jano. To access it, add the |
aoqi@0 | 157 | following directory to your PATH: |
aoqi@0 | 158 | </P> |
aoqi@0 | 159 | |
aoqi@0 | 160 | <P> |
aoqi@0 | 161 | <PRE> |
aoqi@0 | 162 | /net/jano/export/disk05/hotspot/sa/bin/common |
aoqi@0 | 163 | </PRE> |
aoqi@0 | 164 | </P> |
aoqi@0 | 165 | |
aoqi@0 | 166 | <P> |
aoqi@0 | 167 | To start the debugger, type "hsdb". |
aoqi@0 | 168 | </P> |
aoqi@0 | 169 | |
aoqi@0 | 170 | <P> |
aoqi@0 | 171 | Alternatively, you can start a local build of the debugger by building |
aoqi@0 | 172 | the sources in src/share/vm/agent, cd'ing to that directory, and |
aoqi@0 | 173 | typing "java sun.jvm.hotspot.HSDB". |
aoqi@0 | 174 | </P> |
aoqi@0 | 175 | |
aoqi@0 | 176 | <P> |
aoqi@0 | 177 | There are three modes for the debugger: attaching to a local process, |
aoqi@0 | 178 | opening a core file, and attaching to a remote "debug server". The |
aoqi@0 | 179 | remote case requires two programs to be running on the remote machine: |
aoqi@0 | 180 | the rmiregistry (see the script "start-rmiregistry" in this directory; |
aoqi@0 | 181 | run this in the background) and the debug server (see the script |
aoqi@0 | 182 | "start-debug-server"), in that order. start-rmiregistry takes no |
aoqi@0 | 183 | arguments; start-debug-server takes as argument the process ID or the |
aoqi@0 | 184 | executable and core file names to allow remote debugging of. Make sure |
aoqi@0 | 185 | you do NOT have a CLASSPATH environment variable set when you run |
aoqi@0 | 186 | these scripts. (The classes put into the rmiregistry are in sun.*, and |
aoqi@0 | 187 | there are permissions problems if they aren't placed on the boot |
aoqi@0 | 188 | classpath.) |
aoqi@0 | 189 | </P> |
aoqi@0 | 190 | |
aoqi@0 | 191 | <P> |
aoqi@0 | 192 | NOTE that the SA currently only works against VMs on Solaris/SPARC. |
aoqi@0 | 193 | Remote debugging of Solaris/SPARC VMs on arbitrary platforms is |
aoqi@0 | 194 | possible using the debug server; select "Connect to debug server..." |
aoqi@0 | 195 | in HSDB. |
aoqi@0 | 196 | </P> |
aoqi@0 | 197 | |
aoqi@0 | 198 | <P> |
aoqi@0 | 199 | Once the debugger has been launched, the threads list is displayed. |
aoqi@0 | 200 | The current set of functionality allows: |
aoqi@0 | 201 | </P> |
aoqi@0 | 202 | |
aoqi@0 | 203 | <UL> |
aoqi@0 | 204 | <LI> Browsing of the annotated stack memory ("Stack Memory" button). It |
aoqi@0 | 205 | is currently annotated with the following information: |
aoqi@0 | 206 | <UL> |
aoqi@0 | 207 | <LI> Method names of the Java frames and their extents (supporting |
aoqi@0 | 208 | inlined compiled methods) |
aoqi@0 | 209 | <LI> Locations and types of oops, found using the oop map information |
aoqi@0 | 210 | from compiled methods (interpreter oop maps coming soon) |
aoqi@0 | 211 | <LI> If a Java frame was interrupted by a signal (e.g., because of a |
aoqi@0 | 212 | crash), annotates the frame with the signal name and number |
aoqi@0 | 213 | <LI> Interpreter codelet descriptions for interpreted frames |
aoqi@0 | 214 | </UL> |
aoqi@0 | 215 | <LI> Finding which thread or threads caused a crash (currently |
aoqi@0 | 216 | identified by the presence of a signal handler frame) |
aoqi@0 | 217 | <LI> Browsing of oops using the Oop Inspector. |
aoqi@0 | 218 | <LI> Browsing of the java.lang.Thread object's oop. |
aoqi@0 | 219 | <LI> Object histogram and inspection of objects therein. |
aoqi@0 | 220 | </UL> |
aoqi@0 | 221 | </P> |
aoqi@0 | 222 | |
aoqi@0 | 223 | <P> |
aoqi@0 | 224 | More functionality is planned. Please <A HREF = |
aoqi@0 | 225 | "mailto:kenneth.russell@eng">send email</A> with suggestions on what |
aoqi@0 | 226 | would be useful, with any questions or comments, or if the debugger |
aoqi@0 | 227 | crashes. |
aoqi@0 | 228 | </P> |
aoqi@0 | 229 | |
aoqi@0 | 230 | <H2> |
aoqi@0 | 231 | <A NAME="NOTES"> |
aoqi@0 | 232 | Notes |
aoqi@0 | 233 | </A> |
aoqi@0 | 234 | </H2> |
aoqi@0 | 235 | |
aoqi@0 | 236 | <P> |
aoqi@0 | 237 | HSDB does not suspend the system at a safepoint, but at an arbitrary |
aoqi@0 | 238 | point. This means that many of the invariants in the VM's code are not |
aoqi@0 | 239 | followed. |
aoqi@0 | 240 | </P> |
aoqi@0 | 241 | |
aoqi@0 | 242 | <P> |
aoqi@0 | 243 | As it happens, most of the places where the code ported over from the |
aoqi@0 | 244 | VM has failed involve the topmost frame on the stack. Some |
aoqi@0 | 245 | modifications have been made to allow the system to recognize |
aoqi@0 | 246 | problematic situations. |
aoqi@0 | 247 | </P> |
aoqi@0 | 248 | |
aoqi@0 | 249 | <P> |
aoqi@0 | 250 | Certainly, not all of the failure modes of the debugger have been |
aoqi@0 | 251 | found. Please <A HREF = "mailto:kenneth.russell@eng">send email</A> if |
aoqi@0 | 252 | HSDB throws an exception. The best debugging aid in these situations |
aoqi@0 | 253 | is a core file since it is a static view of the VM to which we can |
aoqi@0 | 254 | then adapt the debugger code, as opposed to having to try to suspend |
aoqi@0 | 255 | the VM over and over to reproduce the failure. gcore (1) is a useful |
aoqi@0 | 256 | tool. (NOTE: do not try gcore with any application using the DGA X |
aoqi@0 | 257 | server extension (example: Java2Demo); the kernel will panic. See bug |
aoqi@0 | 258 | 4343237.) |
aoqi@0 | 259 | </P> |
aoqi@0 | 260 | |
aoqi@0 | 261 | </BODY> |
aoqi@0 | 262 | </HTML> |