Linux - lldb 환경에서 sos 확장 명령어를 이용한 닷넷 프로세스 디버깅
윈도우에 windbg가 있다면, 리눅스에는 gdb/lldb가 있습니다. 단지, .NET Core의 sos 확장이 gdb를 (아직인지는 모르겠지만) 지원하지 않으므로 lldb가 닷넷 개발자에게는 더 의미가 있습니다.
Debugging .NET Core on Linux with LLDB
; https://www.raydbg.com/2018/Debugging-Net-Core-on-Linux-with-LLDB/
lldb 설치 방법에 대해서는 다음의 문서에 상세하게 나오는데,
.NET Core Diagnostics Repo - Getting lldb
; https://github.com/dotnet/diagnostics/blob/master/README.md#getting-lldb
; https://github.com/dotnet/diagnostics/blob/master/documentation/lldb/linux-instructions.md
우분투 18.04의 경우에는 매우 간단하게 설치할 수 있습니다.
[설치]
$ sudo apt-get install lldb-3.9 llvm-3.9 python-lldb-3.9
[실행]
$ lldb-3.9
반면 CentOS 7은 직접 빌드해야 하는데 이를 위해 다음의 구성 요소를 설치하고,
sudo yum install centos-release-SCL epel-release
sudo yum install cmake cmake3 gcc gcc-c++ git libicu libunwind make python27 tar wget which zip
sudo yum install doxygen libedit-devel libxml2-devel python-argparse python-devel readline-devel swig xz
빌드 스크립트를 내려받아 실행하면,
cd $HOME
git clone https://github.com/dotnet/diagnostics.git
$HOME/diagnostics/documentation/lldb/centos7/build-install-lldb.sh
제법 한 참의 시간이 흐른 후에야 lldb-3.9.1 바이너리가 생성됩니다. (참고로 빌드를 위해 1GB 넘는 공간이 필요합니다.)
$ /usr/local/bin/lldb-3.9.1
일단 실행 파일을 얻은 후에는 빌드를 위해 설치한 구성 요소를 다시 지워도 무방합니다.
sudo yum remove doxygen libedit-devel libxml2-devel python-argparse python-devel readline-devel swig xz
sudo yum clean all
자, 이제 준비가 되었으니 ^^ lldb를 실행하고,
[Centos 7]
$ /usr/local/bin/lldb-3.9.1
[Ubuntu 18.04]
$ lldb-3.9
libsosplugin.so 플러그인을 이용하면 windbg의 경험 그대로 sos 확장 명령어를 통해 닷넷 프로세스를 분석할 수 있습니다.
예를 들기 위해, 다음과 같은 .NET Core 프로그램을 하나 빌드하고,
using System;
using System.Threading;
class Program
{
static void Main(string[] args)
{
while (true)
{
Console.WriteLine(DateTime.Now);
Thread.Sleep(1000 * 10);
}
}
}
실행을 해보겠습니다.
$ dotnet /home/tusr/temp/bin/Debug/netcoreapp2.0/temp.dll
이 프로세스를 lldb에서 연결하기 위해 우선 대상 프로세스의 pid를 알아낸 후,
// # apt install -y procps
$ ps -Af | grep dotnet
tusr 123855 62261 1 11:21 pts/4 00:00:00 dotnet ./bin/Debug/netcoreapp2.0/temp.dll
root 123868 102608 0 11:21 pts/2 00:00:00 grep --color=auto dotnet
lldb 실행 환경에서,
# lldb-3.9
(lldb)
"process attach" 명령을 이용하면 됩니다.
(lldb) process attach -p 123855
windbg의 경우 그다음에는 sos.dll 확장 모듈을 로드했는데, 이 작업을 lldb 환경에서는 .NET Core 런타임의 libsosplugin.so를 로드하는 것으로 대신합니다. 이때, 대상 프로세스가 "dotnet" 실행 모듈을 기반으로 temp.dll을 실행 중이므로 당연히 바로 그 "dotnet"의 런타임과 그 하위에 있는 libsosplugin.so 모듈의 위치를 알아야 합니다. 이런 작업은 간단하게 which와 find 명령어로,
$ ls /usr/bin/dotnet
/usr/bin/dotnet
$ ls /usr/bin/dotnet -l
lrwxrwxrwx 1 root root 22 11월 12 23:51 /usr/bin/dotnet -> ../share/dotnet/dotnet
$ readlink -f $(which dotnet)
/usr/share/dotnet/dotnet
$ find /usr/share/dotnet -name libsosplugin.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.7/libsosplugin.so
$ find $(dirname $(readlink -f $(which dotnet))) -name libsosplugin.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.7/libsosplugin.so
알아낸 것을 사용하면 됩니다.
(lldb) plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.7/libsosplugin.so
마지막으로 .NET Core의 런타임 DAC에 해당하는 libmscordaccore.so 파일의 위치를 알려야 하는데, 이 역시 "libsosplugin.so" 모듈의 런타임 경로에 위치하고 있으므로 다음과 같이 지정하면 됩니다.
(lldb) setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.7/
Set load path for sos/dac/dbi to '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.7/'
(그러니까, setclrpath는
windbg에서 ".cordll -lp [...path...]"의 역할과 동일합니다.) 이후 다음과 같이 libsosplugin.so 확장에서 제공하는 명령어를
windbg/sos.dll에서의 경험을 살려 닷넷 프로세스를 분석하면 됩니다.
(lldb) clrstack
OS Thread Id: 0x1e3cf (1)
Child SP IP Call Site
00007FFD564D1088 00007f6bad85ced9 [HelperMethodFrame: 00007ffd564d1088] System.Threading.Thread.SleepInternal(Int32)
00007FFD564D11D0 00007F6B32D45ABB System.Threading.Thread.Sleep(Int32)
00007FFD564D11E0 00007F6B32BE1E3B Program.Main(System.String[]) [/home/tusr/temp/Program.cs @ 11]
00007FFD564D14C8 00007f6babda717f [GCFrame: 00007ffd564d14c8]
00007FFD564D18D0 00007f6babda717f [GCFrame: 00007ffd564d18d0]
보는 바와 같이, 진입 문턱만 살짝 넘어간다면 .NET 프로세스의 분석은 윈도우에서의 느낌 그대로 진행할 수 있습니다.
.NET Core 런타임이 다중으로 설치된 경우라면,
$ find /usr/share/dotnet -name libsosplugin.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libsosplugin.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.11/libsosplugin.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.7/libsosplugin.so
이런 경우에는 차라리
lsof를 이용해 대상 프로세스에 로드된 (libcoreclr.so 등의) 모듈 경로를 사용하는 것이 편합니다.
$ lsof -p 123855 | grep libcoreclr.so
dotnet 88956 tusr mem REG 8,1 8511288 397640 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.7/libcoreclr.so
참고로, 프로그램을 새로 명령행에서 실행할 수 있다면 lldb의 명령행 인자를 통해 단계를 좀 더 줄일 수 있습니다.
$ lldb-3.9 -o "plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.7/libsosplugin.so" -o "setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.7/" -o "process launch -s" dotnet /home/tusr/temp/bin/Debug/netcoreapp2.0/temp.dll
이후 프로세스 실행을 resume/suspend할 수 있고,
(lldb) process continue
Process 123855 resuming
error: Process is running. Use 'process interrupt' to pause execution.
(lldb) process interrupt
(lldb) Process 123855 stopped
* thread #1: tid = 123788, 0x00007ffff7bc3ed9 libpthread.so.0`__pthread_cond_timedwait + 649, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007ffff7bc3ed9 libpthread.so.0`__pthread_cond_timedwait + 649
libpthread.so.0`__pthread_cond_timedwait:
-> 0x7ffff7bc3ed9 <+649>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7ffff7bc3edf <+655>: ja 0x7ffff7bc41c0 ; <+1392>
0x7ffff7bc3ee5 <+661>: movl %r8d, %edi
0x7ffff7bc3ee8 <+664>: callq 0x7ffff7bc7060 ; __pthread_disable_asynccancel
당연히, 프로세스가 멈춘 시점에 한해 lldb 및 sos 확장 명령어를 사용할 수 있습니다.
그 외 다음의 항목도 읽어보시고,
Debugging CoreCLR on Linux and macOS
; https://github.com/dotnet/coreclr/blob/master/Documentation/building/debugging-instructions.md#debugging-coreclr-on-linux-and-macos
The libsosplugin.so built for lldb 3.9 does work with lldb 4.0, 5.0 and 6.0 but most of the testing has been on lldb 3.9.
lldb 관련한 명령어 사용법도 한 번쯤 훑어보시고,
An introduction to Debugging (in C and lldb), Part- I
; https://towardsdatascience.com/an-introduction-to-debugging-in-c-and-lldb-part-i-e3c51991f83a
[Xcode][LLDB]Debugging With Xcode and LLDB
; http://minsone.github.io/ios/mac/xcode-lldb-debugging-with-xcode-and-lldb
디버깅 포스트모템 - 박살난 소멸자
LLDB를 이용해서 버퍼 오버플로우 찾기
; https://libsora.so/posts/buffer-overflow-broken-dtor-debugging-post-mortem/
마지막으로 libsosplugin.so의 자세한 사용법은 .NET Core 런타임(또는 publish한) 경로에 있는 "sosdocsunix.txt" 파일에 있으니 읽어보시면 됩니다. 아래는 그 파일의 내용을 그대로 실은 것입니다.
-------------------------------------------------------------------------------
NOTE: THIS FILE CONTAINS SOS DOCUMENTATION. THE FORMAT OF THE FILE IS:
<optional comments>
COMMAND: <cmd name, all lower case>
<descriptive text of the command>
\\ <these are two backslashes, immediately followed by a newline>
<repeat the sequence above>
The first command is "contents" which is the general help screen. The rest
correspond to SOS command names. This file is embedded as a resource in the SOS
binary. Be sure to list any new commands here.
-------------------------------------------------------------------------------
COMMAND: contents.
SOS is a debugger extension DLL designed to aid in the debugging of managed
programs. Functions are listed by category, then roughly in order of
importance. Shortcut names for popular functions are listed in parenthesis.
Type "soshelp <functionname>" for detailed info on that function.
Object Inspection Examining code and stacks
----------------------------- -----------------------------
DumpObj (dumpobj) Threads (clrthreads)
DumpArray ThreadState
DumpStackObjects (dso) IP2MD (ip2md)
DumpHeap (dumpheap) u (clru)
DumpVC DumpStack (dumpstack)
GCRoot (gcroot) EEStack (eestack)
PrintException (pe) ClrStack (clrstack)
GCInfo
EHInfo
bpmd (bpmd)
Examining CLR data structures Diagnostic Utilities
----------------------------- -----------------------------
DumpDomain VerifyHeap
EEHeap (eeheap) FindAppDomain
Name2EE (name2ee) DumpLog (dumplog)
DumpMT (dumpmt)
DumpClass (dumpclass)
DumpMD (dumpmd)
Token2EE
DumpModule (dumpmodule)
DumpAssembly
DumpRuntimeTypes
DumpIL (dumpil)
DumpSig
DumpSigElem
Examining the GC history Other
----------------------------- -----------------------------
HistInit (histinit) FAQ
HistRoot (histroot) CreateDump (createdump)
HistObj (histobj) Help (soshelp)
HistObjFind (histobjfind)
HistClear (histclear)
\\
COMMAND: faq.
>> Where can I get the right version of SOS for my build?
If you are running a xplat version of coreclr, the sos module (exact name
is platform dependent) is installed in the same directory as the main coreclr
module. There is also an lldb sos plugin command that allows the path where
the sos, dac and dbi modules are loaded:
"setsospath /home/user/coreclr/bin/Product/Linux.x64.Debug""
If you are using a dump file created on another machine, it is a little bit
more complex. You need to make sure the dac module that came with that install
is in the directory set with the above command.
>> I have a chicken and egg problem. I want to use SOS commands, but the CLR
isn't loaded yet. What can I do?
TBD
>> I got the following error message. Now what?
(lldb) sos DumpStackObjects
The coreclr module is not loaded yet in the target process
(lldb)
This means that the clr is not loaded yet, or has been unloaded. You need to
wait until your managed program is running in order to use these commands. If
you have just started the program a good way to do this is to type
breakpoint set coreclr`EEStartup
in the debugger, and let it run. After the function EEStartup is finished,
there will be a minimal managed environment for executing SOS commands.
\\
COMMAND: dumpobj.
DumpObj [-nofields] <object address>
This command allows you to examine the fields of an object, as well as learn
important properties of the object such as the EEClass, the MethodTable, and
the size.
You might find an object pointer by running DumpStackObjects and choosing
from the resultant list. Here is a simple object:
(lldb) dumpobj a79d40
Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes
(/home/user/pub/unittest)
Fields:
MT Field Offset Type VT Attr Value Name
009038ec 4000008 4 Customer 0 instance 00a79ce4 name
009038ec 4000009 8 Bank 0 instance 00a79d2c bank
Note that fields of type Customer and Bank are themselves objects, and you can
run DumpObj on them too. You could look at the field directly in memory using
the offset given. "dd a79d40+8 l1" would allow you to look at the bank field
directly. Be careful about using this to set memory breakpoints, since objects
can move around in the garbage collected heap.
What else can you do with an object? You might run GCRoot, to determine what
roots are keeping it alive. Or you can find all objects of that type with
"dumpheap -type Customer".
The column VT contains the value 1 if the field is a valuetype structure, and
0 if the field contains a pointer to another object. For valuetypes, you can
take the MethodTable pointer in the MT column, and the Value and pass them to
the command DumpVC.
The arguments in detail:
-nofields: do not print fields of the object, useful for objects like String
\\
COMMAND: dumparray.
DumpArray
[-start <startIndex>]
[-length <length>]
[-details]
[-nofields]
<array object address>
This command allows you to examine elements of an array object.
The arguments in detail:
-start <startIndex>: optional, only supported for single dimension array.
Specify from which index the command shows the elements.
-length <length>: optional, only supported for single dimension array.
Specify how many elements to show.
-details: optional. Ask the command to print out details
of the element using DumpObj and DumpVC format.
-nofields: optional, only takes effect when -details is used. Do
not print fields of the elements. Useful for arrays of
objects like String
Example output:
(lldb) sos DumpArray -start 2 -length 3 -details 00ad28d0
Name: Value[]
MethodTable: 03e41044
EEClass: 03e40fc0
Size: 132(0x84) bytes
Array: Rank 1, Number of elements 10, Type VALUETYPE
Element Type: Value
[2] 00ad28f0
Name: Value
MethodTable 03e40f4c
EEClass: 03ef1698
Size: 20(0x14) bytes
(/home/user/bugs/225271/arraytest)
Fields:
MT Field Offset Type Attr Value Name
5b9a628c 4000001 0 System.Int32 instance 2 x
5b9a628c 4000002 4 System.Int32 instance 4 y
5b9a628c 4000003 8 System.Int32 instance 6 z
[3] 00ad28fc
Name: Value
MethodTable 03e40f4c
EEClass: 03ef1698
Size: 20(0x14) bytes
(/home/user/bugs/225271/arraytest)
Fields:
MT Field Offset Type Attr Value Name
5b9a628c 4000001 0 System.Int32 instance 3 x
5b9a628c 4000002 4 System.Int32 instance 6 y
5b9a628c 4000003 8 System.Int32 instance 9 z
[4] 00ad2908
Name: Value
MethodTable 03e40f4c
EEClass: 03ef1698
Size: 20(0x14) bytes
(/home/user/bugs/225271/arraytest.exe)
Fields:
MT Field Offset Type Attr Value Name
5b9a628c 4000001 0 System.Int32 instance 4 x
5b9a628c 4000002 4 System.Int32 instance 8 y
5b9a628c 4000003 8 System.Int32 instance 12 z
\\
COMMAND: dumpstackobjects.
DumpStackObjects [-verify] [top stack [bottom stack]]
This command will display any managed objects it finds within the bounds of
the current stack. Combined with the stack tracing commands like K and
CLRStack, it is a good aid to determining the values of locals and
parameters.
If you use the -verify option, each non-static CLASS field of an object
candidate is validated. This helps to eliminate false positives. It is not
on by default because very often in a debugging scenario, you are
interested in objects with invalid fields.
The abbreviation dso can be used for brevity.
\\
COMMAND: dumpheap.
DumpHeap [-stat]
[-strings]
[-short]
[-min <size>]
[-max <size>]
[-live]
[-dead]
[-thinlock]
[-startAtLowerBound]
[-mt <MethodTable address>]
[-type <partial type name>]
[start [end]]
DumpHeap is a powerful command that traverses the garbage collected heap,
collection statistics about objects. With it's various options, it can look for
particular types, restrict to a range, or look for ThinLocks (see SyncBlk
documentation). Finally, it will provide a warning if it detects excessive
fragmentation in the GC heap.
When called without options, the output is first a list of objects in the heap,
followed by a report listing all the types found, their size and number:
(lldb) dumpheap
Address MT Size
00a71000 0015cde8 12 Free
00a7100c 0015cde8 12 Free
00a71018 0015cde8 12 Free
00a71024 5ba58328 68
00a71068 5ba58380 68
00a710ac 5ba58430 68
00a710f0 5ba5dba4 68
...
total 619 objects
Statistics:
MT Count TotalSize Class Name
5ba7607c 1 12 System.Security.Permissions.HostProtectionResource
5ba75d54 1 12 System.Security.Permissions.SecurityPermissionFlag
5ba61f18 1 12 System.Collections.CaseInsensitiveComparer
...
0015cde8 6 10260 Free
5ba57bf8 318 18136 System.String
...
"Free" objects are simply regions of space the garbage collector can use later.
If 30% or more of the heap contains "Free" objects, the process may suffer from
heap fragmentation. This is usually caused by pinning objects for a long time
combined with a high rate of allocation. Here is example output where DumpHeap
provides a warning about fragmentation:
<After the Statistics section>
Fragmented blocks larger than 1MB:
Addr Size Followed by
00a780c0 1.5MB 00bec800 System.Byte[]
00da4e38 1.2MB 00ed2c00 System.Byte[]
00f16df0 1.2MB 01044338 System.Byte[]
The arguments in detail:
-stat Restrict the output to the statistical type summary
-strings Restrict the output to a statistical string value summary
-short Limits output to just the address of each object. This allows you
to easily pipe output from the command to another debugger
command for automation.
-min Ignore objects less than the size given in bytes
-max Ignore objects larger than the size given in bytes
-live Only print live objects
-dead Only print dead objects (objects which will be collected in the
next full GC)
-thinlock Report on any ThinLocks (an efficient locking scheme, see SyncBlk
documentation for more info)
-startAtLowerBound
Force heap walk to begin at lower bound of a supplied address range.
(During plan phase, the heap is often not walkable because objects
are being moved. In this case, DumpHeap may report spurious errors,
in particular bad objects. It may be possible to traverse more of
the heap after the reported bad object. Even if you specify an
address range, DumpHeap will start its walk from the beginning of
the heap by default. If it finds a bad object before the specified
range, it will stop before displaying the part of the heap in which
you are interested. This switch will force DumpHeap to begin its
walk at the specified lower bound. You must supply the address of a
good object as the lower bound for this to work. Display memory at
the address of the bad object to manually find the next method
table (use DumpMT to verify). If the GC is currently in a call to
memcopy, You may also be able to find the next object's address by
adding the size to the start address given as parameters.)
-mt List only those objects with the MethodTable given
-type List only those objects whose type name is a substring match of the
string provided.
start Begin listing from this address
end Stop listing at this address
A special note about -type: Often, you'd like to find not only Strings, but
System.Object arrays that are constrained to contain Strings. ("new
String[100]" actually creates a System.Object array, but it can only hold
System.String object pointers). You can use -type in a special way to find
these arrays. Just pass "-type System.String[]" and those Object arrays will
be returned. More generally, "-type <Substring of interesting type>[]".
The start/end parameters can be obtained from the output of eeheap -gc. For
example, if you only want to list objects in the large heap segment:
(lldb) eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00c32754
generation 1 starts at 0x00c32748
generation 2 starts at 0x00a71000
segment begin allocated size
00a70000 00a71000 010443a8 005d33a8(6108072)
Large object heap starts at 0x01a71000
segment begin allocated size
01a70000 01a71000 01a75000 0x00004000(16384)
Total Size 0x5d73a8(6124456)
------------------------------
GC Heap Size 0x5d73a8(6124456)
(lldb) dumpheap 1a71000 1a75000
Address MT Size
01a71000 5ba88bd8 2064
01a71810 0019fe48 2032 Free
01a72000 5ba88bd8 4096
01a73000 0019fe48 4096 Free
01a74000 5ba88bd8 4096
total 5 objects
Statistics:
MT Count TotalSize Class Name
0019fe48 2 6128 Free
5ba88bd8 3 10256 System.Object[]
Total 5 objects
Finally, if GC heap corruption is present, you may see an error like this:
(lldb) dumpheap -stat
object 00a73d24: does not have valid MT
curr_object : 00a73d24
Last good object: 00a73d14
----------------
That indicates a serious problem. See the help for VerifyHeap for more
information on diagnosing the cause.
\\
COMMAND: dumpvc.
DumpVC <MethodTable address> <Address>
DumpVC allows you to examine the fields of a value class. In C#, this is a
struct, and lives on the stack or within an Object on the GC heap. You need
to know the MethodTable address to tell SOS how to interpret the fields, as
a value class is not a first-class object with it's own MethodTable as the
first field. For example:
(lldb) sos DumpObj a79d98
Name: Mainy
MethodTable: 009032d8
EEClass: 03ee1424
Size: 28(0x1c) bytes
(/home/user/pub/unittest)
Fields:
MT Field Offset Type Attr Value Name
0090320c 4000010 4 VALUETYPE instance 00a79d9c m_valuetype
009032d8 400000f 4 CLASS static 00a79d54 m_sExcep
m_valuetype is a value type. The value in the MT column (0090320c) is the
MethodTable for it, and the Value column provides the start address:
(lldb) sos DumpVC 0090320c 00a79d9c
Name: Funny
MethodTable 0090320c
EEClass: 03ee14b8
Size: 28(0x1c) bytes
(/home/user/pub/unittest)
Fields:
MT Field Offset Type Attr Value Name
0090320c 4000001 0 CLASS instance 00a743d8 signature
0090320c 4000002 8 System.Int32 instance 2345 m1
0090320c 4000003 10 System.Boolean instance 1 b1
0090320c 4000004 c System.Int32 instance 1234 m2
0090320c 4000005 4 CLASS instance 00a79d98 backpointer
DumpVC is quite a specialized function. Some managed programs make heavy use
of value classes, while others do not.
\\
COMMAND: gcroot.
GCRoot [-nostacks] <Object address>
GCRoot looks for references (or roots) to an object. These can exist in four
places:
1. On the stack
2. Within a GC Handle
3. In an object ready for finalization
4. As a member of an object found in 1, 2 or 3 above.
First, all stacks will be searched for roots, then handle tables, and finally
the freachable queue of the finalizer. Some caution about the stack roots:
GCRoot doesn't attempt to determine if a stack root it encountered is valid
or is old (discarded) data. You would have to use CLRStack and U to
disassemble the frame that the local or argument value belongs to in order to
determine if it is still in use.
Because people often want to restrict the search to gc handles and freachable
objects, there is a -nostacks option.
\\
COMMAND: pe.
COMMAND: printexception.
PrintException [-nested] [-lines] [-ccw] [<Exception object address>] [<CCW pointer>]
This will format fields of any object derived from System.Exception. One of the
more useful aspects is that it will format the _stackTrace field, which is a
binary array. If _stackTraceString field is not filled in, that can be helpful
for debugging. You can of course use DumpObj on the same exception object to
explore more fields.
If called with no parameters, PrintException will look for the last outstanding
exception on the current thread and print it. This will be the same exception
that shows up in a run of clrthreads.
PrintException will notify you if there are any nested exceptions on the
current managed thread. (A nested exception occurs when you throw another
exception within a catch handler already being called for another exception).
If there are nested exceptions, you can re-run PrintException with the
"-nested" option to get full details on the nested exception objects. The
clrthreads command will also tell you which threads have nested exceptions.
PrintException can display source information if available, by specifying the
-lines command line argument.
PrintException prints the exception object corresponding to a given CCW pointer,
which can be specified using the -ccw option.
The abbreviation 'pe' can be used for brevity.
\\
COMMAND: threadstate.
ThreadState value
The clrthreads command outputs, among other things, the state of the thread.
This is a bit field which corresponds to various states the thread is in.
To check the state of the thread, simply pass that bit field from the
output of clrthreads into ThreadState.
Example:
(lldb) clrthreads
ThreadCount: 2
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 250 0019b068 a020 Disabled 02349668:02349fe8 0015def0 0 MTA
2 2 944 001a6020 b220 Enabled 00000000:00000000 0015def0 0 MTA (Finalizer)
0:003> sos ThreadState b220
Legal to Join
Background
CLR Owns
CoInitialized
In Multi Threaded Apartment
Possible thread states:
Thread Abort Requested
GC Suspend Pending
User Suspend Pending
Debug Suspend Pending
GC On Transitions
Legal to Join
Yield Requested
Hijacked by the GC
Blocking GC for Stack Overflow
Background
Unstarted
Dead
CLR Owns
CoInitialized
In Single Threaded Apartment
In Multi Threaded Apartment
Reported Dead
Fully initialized
Task Reset
Sync Suspended
Debug Will Sync
Stack Crawl Needed
Suspend Unstarted
Aborted
Thread Pool Worker Thread
Interruptible
Interrupted
Completion Port Thread
Abort Initiated
Finalized
Failed to Start
Detached
\\
COMMAND: threads.
COMMAND: clrthreads.
Threads [-live] [-special]
Threads (clrthreads) lists all the mananaged threads in the process.
-live: optional. Only print threads associated with a live thread.
-special: optional. With this switch, the command will display all the special
threads created by CLR. Those threads might not be managed threads
so they might not be shown in the first part of the command's
output. Example of special threads include: GC threads (in
concurrent GC and server GC), Debugger helper threads, Finalizer
threads, AppDomain Unload threads, and Threadpool timer threads.
Each thread has many attributes, many of which can be ignored. The important
ones are discussed below:
There are three ID columns:
1) The debugger shorthand ID (When the runtime is hosted this column might
display the special string "<<<<" when this internal thread object is not
associated with any physical thread - this may happen when the host reuses
the runtime internal thread object)
2) The CLR Thread ID
3) The OS thread ID.
If PreEmptiveGC is enabled for a thread, then a garbage collection
can occur while that thread is running. For example, if you break in while
a managed thread is making a PInvoke call to a Win32 function, that thread
will be in PreEmptive GC mode.
The Domain column indicates what AppDomain the thread is currently executing
in. You can pass this value to DumpDomain to find out more.
The APT column gives the COM apartment mode.
Exception will list the last thrown exception (if any) for the thread. More
details can be obtained by passing the pointer value to PrintException. If
you get the notation "(nested exceptions)", you can get details on those
exceptions by switching to the thread in question, and running
"PrintException -nested".
\\
COMMAND: clrstack.
CLRStack [-a] [-l] [-p] [-n] [-f]
CLRStack [-a] [-l] [-p] [-i] [variable name] [frame]
CLRStack attempts to provide a true stack trace for managed code only. It is
handy for clean, simple traces when debugging straightforward managed
programs. The -p parameter will show arguments to the managed function. The
-l parameter can be used to show information on local variables in a frame.
SOS can't retrieve local names at this time, so the output for locals is in
the format <local address> = <value>. The -a (all) parameter is a short-cut
for -l and -p combined.
The -f option (full mode) displays the native frames intermixing them with
the managed frames and the assembly name and function offset for the managed
frames.
If the debugger has the option SYMOPT_LOAD_LINES specified (either by the
.lines or .symopt commands), SOS will look up the symbols for every managed
frame and if successful will display the corresponding source file name and
line number. The -n (No line numbers) parameter can be specified to disable
this behavior.
When you see methods with the name "[Frame:...", that indicates a transition
between managed and unmanaged code. You could run IP2MD on the return
addresses in the call stack to get more information on each managed method.
On x64 platforms, Transition Frames are not displayed at this time. To avoid
heavy optimization of parameters and locals one can request the JIT compiler
to not optimize functions in the managed app by creating a file myapp.ini
(if your program is myapp.exe) in the same directory. Put the following lines
in myapp.ini and re-run:
[.NET Framework Debugging Control]
GenerateTrackingInfo=1
AllowOptimize=0
The -i option is a new EXPERIMENTAL addition to CLRStack and will use the ICorDebug
interfaces to display the managed stack and variables. With this option you can also
view and expand arrays and fields for managed variables. If a stack frame number is
specified in the command line, CLRStack will show you the parameters and/or locals
only for that frame (provided you specify -l or -p or -a of course). If a variable
name and a stack frame number are specified in the command line, CLRStack will show
you the parameters and/or locals for that frame, and will also show you the fields
for that variable name you specified. Here are some examples:
clrstack -i -a : This will show you all parameters and locals for all frames
clrstack -i -a 3 : This will show you all parameters and locals, for frame 3
clrstack -i var1 0 : This will show you the fields of 'var1' for frame 0
clrstack -i var1.abc 2 : This will show you the fields of 'var1', and expand
'var1.abc' to show you the fields of the 'abc' field,
for frame 2.
clrstack -i var1.[basetype] 0 : This will show you the fields of 'var1', and
expand the base type of 'var1' to show you its
fields.
clrstack -i var1.[6] 0 : If 'var1' is an array, this will show you the element
at index 6 in the array, along with its fields
The -i options uses DML output for a better debugging experience, so typically you
should only need to execute "clrstack -i", and from there, click on the DML
hyperlinks to inspect the different managed stack frames and managed variables.
\\
COMMAND: createdump.
createdump [options] [dumpFileName]
-n - create minidump.
-h - create minidump with heap (default).
-t - create triage minidump.
-f - create full core dump (everything).
-d - enable diagnostic messages.
Creates a platform (ELF core on Linux, etc.) minidump. The pid can be placed in the dump
file name with %d. The default is '/tmp/coredump.%d'.
\\
COMMAND: ip2md.
IP2MD <Code address>
Given an address in managed JITTED code, IP2MD attempts to find the MethodDesc
associated with it. For example, this output from K:
(lldb) bt
...
frame #9: 0x00007fffffffbf60 0x00007ffff61c6d89 libcoreclr.so`MethodDesc::DoPrestub(this=0x00007ffff041f870, pDispatchingMT=0x0000000000000000) + 3001 at prestub.cpp:1490
frame #10: 0x00007fffffffc140 0x00007ffff61c5f17 libcoreclr.so`::PreStubWorker(pTransitionBlock=0x00007fffffffc9a8, pMD=0x00007ffff041f870) + 1399 at prestub.cpp:1037
frame #11: 0x00007fffffffc920 0x00007ffff5f5238c libcoreclr.so`ThePreStub + 92 at theprestubamd64.S:800
frame #12: 0x00007fffffffca10 0x00007ffff04981cc
frame #13: 0x00007fffffffca30 0x00007ffff049773c
frame #14: 0x00007fffffffca80 0x00007ffff04975ad
...
frame #22: 0x00007fffffffcc90 0x00007ffff5f51a0f libcoreclr.so`CallDescrWorkerInternal + 124 at calldescrworkeramd64.S:863
frame #23: 0x00007fffffffccb0 0x00007ffff5d6d6dc libcoreclr.so`CallDescrWorkerWithHandler(pCallDescrData=0x00007fffffffce80, fCriticalCall=0) + 476 at callhelpers.cpp:88
frame #24: 0x00007fffffffcd00 0x00007ffff5d6eb38 libcoreclr.so`MethodDescCallSite::CallTargetWorker(this=0x00007fffffffd0c8, pArguments=0x00007fffffffd048) + 2504 at callhelpers.cpp:633
(lldb) ip2md 0x00007ffff049773c
MethodDesc: 00007ffff7f71920
Method Name: Microsoft.Win32.SafeHandles.SafeFileHandle.Open(System.Func`1<Int32>)
Class: 00007ffff0494bf8
MethodTable: 00007ffff7f71a58
mdToken: 0000000006000008
Module: 00007ffff7f6b938
IsJitted: yes
CodeAddr: 00007ffff04976c0
Transparency: Critical
We have taken a return address into Mainy.Main, and discovered information
about that method. You could run U, DumpMT, DumpClass, DumpMD, or
DumpModule on the fields listed to learn more.
The "Source line" output will only be present if the debugger can find the
symbols for the managed module containing the given <code address>, and if the
debugger is configured to load line number information.
\\
COMMAND: clru.
COMMAND: u.
U [-gcinfo] [-ehinfo] [-n] [-o] <MethodDesc address> | <Code address>
Presents an annotated disassembly of a managed method when given a MethodDesc
pointer for the method, or a code address within the method body. Unlike the
debugger "U" function, the entire method from start to finish is printed,
with annotations that convert metadata tokens to names.
<example output>
...
03ef015d b901000000 mov ecx,0x1
03ef0162 ff156477a25b call dword ptr [mscorlib_dll+0x3c7764 (5ba27764)] (System.Console.InitializeStdOutError(Boolean), mdToken: 06000713)
03ef0168 a17c20a701 mov eax,[01a7207c] (Object: SyncTextWriter)
03ef016d 89442414 mov [esp+0x14],eax
If you pass the -gcinfo flag, you'll get inline display of the GCInfo for
the method. You can also obtain this information with the GCInfo command.
If you pass the -ehinfo flag, you'll get inline display of exception info
for the method. (Beginning and end of try/finally/catch handlers, etc.).
You can also obtain this information with the EHInfo command.
If you pass the -o flag, the byte offset of each instruction from the
beginning of the method will be printed in addition to the absolute address of
the instruction.
If the debugger has the option SYMOPT_LOAD_LINES specified (either by the
.lines or .symopt commands), and if symbols are available for the managed
module containing the method being examined, the output of the command will
include the source file name and line number corresponding to the
disassembly. The -n (No line numbers) flag can be specified to disable this
behavior.
<example output>
...
c:\Code\prj.mini\exc.cs @ 38:
001b00b0 8b0d3020ab03 mov ecx,dword ptr ds:[3AB2030h] ("Break in debugger. When done type <Enter> to continue: ")
001b00b6 e8d5355951 call mscorlib_ni+0x8b3690 (51743690) (System.Console.Write(System.String), mdToken: 0600091b)
001b00bb 90 nop
c:\Code\prj.mini\exc.cs @ 39:
001b00bc e863cdc651 call mscorlib_ni+0xf8ce24 (51e1ce24) (System.Console.ReadLine(), mdToken: 060008f6)
>>> 001b00c1 90 nop
...
\\
COMMAND: dumpstack.
DumpStack [-EE] [-n] [top stack [bottom stack]]
[x86 and x64 documentation]
This command provides a verbose stack trace obtained by "scraping." Therefore
the output is very noisy and potentially confusing. The command is good for
viewing the complete call stack when "kb" gets confused. For best results,
make sure you have valid symbols.
-EE will only show managed functions.
If the debugger has the option SYMOPT_LOAD_LINES specified (either by the
.lines or .symopt commands), SOS will look up the symbols for every managed
frame and if successful will display the corresponding source file name and
line number. The -n (No line numbers) parameter can be specified to disable
this behavior.
You can also pass a stack range to limit the output.
\\
COMMAND: eestack.
EEStack [-short] [-EE]
This command runs DumpStack on all threads in the process. The -EE option is
passed directly to DumpStack. The -short option tries to narrow down the
output to "interesting" threads only, which is defined by
1) The thread has taken a lock.
2) The thread has been "hijacked" in order to allow a garbage collection.
3) The thread is currently in managed code.
See the documentation for DumpStack for more info.
\\
COMMAND: ehinfo.
EHInfo (<MethodDesc address> | <Code address>)
EHInfo shows the exception handling blocks in a jitted method. For each
handler, it shows the type, including code addresses and offsets for the clause
block and the handler block. For a TYPED handler, this would be the "try" and
"catch" blocks respectively.
Sample output:
(lldb) sos EHInfo 33bbd3a
MethodDesc: 03310f68
Method Name: MainClass.Main()
Class: 03571358
MethodTable: 0331121c
mdToken: 0600000b
Module: 001e2fd8
IsJitted: yes
CodeAddr: 033bbca0
Transparency: Critical
EHHandler 0: TYPED catch(System.IO.FileNotFoundException)
Clause: [033bbd2b, 033bbd3c] [8b, 9c]
Handler: [033bbd3c, 033bbd50] [9c, b0]
EHHandler 1: FINALLY
Clause: [033bbd83, 033bbda3] [e3, 103]
Handler: [033bbda3, 033bbdc5] [103, 125]
EHHandler 2: TYPED catch(System.Exception)
Clause: [033bbd7a, 033bbdc5] [da, 125]
Handler: [033bbdc5, 033bbdd6] [125, 136]
\\
COMMAND: gcinfo.
GCInfo (<MethodDesc address> | <Code address>)
GCInfo is especially useful for CLR Devs who are trying to determine if there
is a bug in the JIT Compiler. It parses the GCEncoding for a method, which is a
compressed stream of data indicating when registers or stack locations contain
managed objects. It is important to keep track of this information, because if
a garbage collection occurs, the collector needs to know where roots are so it
can update them with new object pointer values.
Here is sample output where you can see the change in register state. Normally
you would print this output out and read it alongside a disassembly of the
method. For example, the notation "reg EDI becoming live" at offset 0x11 of the
method might correspond to a "mov edi,ecx" statement.
(lldb) sos GCInfo 5b68dbb8 (5b68dbb8 is the start of a JITTED method)
entry point 5b68dbb8
preJIT generated code
GC info 5b9f2f09
Method info block:
method size = 0036
prolog size = 19
epilog size = 8
epilog count = 1
epilog end = yes
saved reg. mask = 000B
ebp frame = yes
fully interruptible=yes
double align = no
security check = no
exception handlers = no
local alloc = no
edit & continue = no
varargs = no
argument count = 4
stack frame size = 1
untracked count = 5
var ptr tab count = 0
epilog at 002E
36 D4 8C C7 AA |
93 F3 40 05 |
Pointer table:
14 | [EBP+14H] an untracked local
10 | [EBP+10H] an untracked local
0C | [EBP+0CH] an untracked local
08 | [EBP+08H] an untracked local
44 | [EBP-04H] an untracked local
F1 79 | 0011 reg EDI becoming live
72 | 0013 reg ESI becoming live
83 | 0016 push ptr 0
8B | 0019 push ptr 1
93 | 001C push ptr 2
9B | 001F push ptr 3
56 | 0025 reg EDX becoming live
4A | 0027 reg ECX becoming live
0E | 002D reg ECX becoming dead
10 | 002D reg EDX becoming dead
E0 | 002D pop 4 ptrs
F0 31 | 0036 reg ESI becoming dead
38 | 0036 reg EDI becoming dead
FF |
This function is important for CLR Devs, but very difficult for anyone else to
make sense of it. You would usually come to use it if you suspect a gc heap
corruption bug caused by invalid GCEncoding for a particular method.
\\
COMMAND: bpmd.
bpmd [-nofuturemodule] <module name> <method name> [<il offset>]
bpmd <source file name>:<line number>
bpmd -md <MethodDesc>
bpmd -list
bpmd -clear <pending breakpoint number>
bpmd -clearall
bpmd provides managed breakpoint support. If it can resolve the method name
to a loaded, jitted or ngen'd function it will create a breakpoint with "bp".
If not then either the module that contains the method hasn't been loaded yet
or the module is loaded, but the function is not jitted yet. In these cases,
bpmd asks the Windows Debugger to receive CLR Notifications, and waits to
receive news of module loads and JITs, at which time it will try to resolve
the function to a breakpoint. -nofuturemodule can be used to suppress
creating a breakpoint against a module that has not yet been loaded.
Management of the list of pending breakpoints can be done via bpmd -list,
bpmd -clear, and bpmd -clearall commands. bpmd -list generates a list of
all of the pending breakpoints. If the pending breakpoint has a non-zero
module id, then that pending breakpoint is specific to function in that
particular loaded module. If the pending breakpoint has a zero module id, then
the breakpoint applies to modules that have not yet been loaded. Use
bpmd -clear or bpmd -clearall to remove pending breakpoints from the list.
This brings up a good question: "I want to set a breakpoint on the main
method of my application. How can I do this?"
1) Stop after coreclr is loaded - TBD
2) Add the breakpoint with command such as:
bpmd myapp.exe MyApp.Main
3) g
4) You will stop at the start of MyApp.Main. If you type "bl" you will
see the breakpoint listed.
To correctly specify explicitly implemented methods make sure to retrieve the
method name from the metadata, or from the output of the "dumpmt -md" command.
For example:
public interface I1
{
void M1();
}
public class ExplicitItfImpl : I1
{
...
void I1.M1() // this method's name is 'I1.M1'
{ ... }
}
bpmd myapp.exe ExplicitItfImpl.I1.M1
bpmd works equally well with generic types. Adding a breakpoint on a generic
type sets breakpoints on all already JIT-ted generic methods and sets a pending
breakpoint for any instantiation that will be JIT-ted in the future.
Example for generics:
Given the following two classes:
class G3<T1, T2, T3>
{
...
public void F(T1 p1, T2 p2, T3 p3)
{ ... }
}
public class G1<T> {
// static method
static public void G<W>(W w)
{ ... }
}
One would issue the following commands to set breapoints on G3.F() and
G1.G():
bpmd myapp.exe G3`3.F
bpmd myapp.exe G1`1.G
And for explicitly implemented methods on generic interfaces:
public interface IT1<T>
{
void M1(T t);
}
public class ExplicitItfImpl<U> : IT1<U>
{
...
void IT1<U>.M1(U u) // this method's name is 'IT1<U>.M1'
{ ... }
}
bpmd bpmd.exe ExplicitItfImpl`1.IT1<U>.M1
Additional examples:
If IT1 and ExplicitItfImpl are types declared inside another class,
Outer, the bpmd command would become:
bpmd bpmd.exe Outer+ExplicitItfImpl`1.Outer.IT1<U>.M1
(note that the fully qualified type name for ExplicitItfImpl became
Outer+ExplicitItfImpl, using the '+' separator, while the method name
is Outer.IT1<U>.M1, using a '.' as the separator)
Furthermore, if the Outer class resides in a namespace, NS, the bpmd
command to use becomes:
bpmd bpmd.exe NS.Outer+ExplicitItfImpl`1.NS.Outer.IT1<U>.M1
bpmd does not accept offsets nor parameters in the method name. You can add
an IL offset as an optional parameter seperate from the name. If there are overloaded
methods, bpmd will set a breakpoint for all of them.
In the case of hosted environments such as SQL, the module name may be
complex, like 'price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.
For this case, just be sure to surround the module name with single quotes,
like:
bpmd 'price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' Price.M2
\\
COMMAND: dumpdomain.
DumpDomain [<Domain address>]
When called with no parameters, DumpDomain will list all the AppDomains in the
process. It enumerates each Assembly loaded into those AppDomains as well.
In addition to your application domain, and any domains it might create, there
are two special domains: the Shared Domain and the System Domain.
Any Assembly pointer in the output can be passed to DumpAssembly. Any Module
pointer in the output can be passed to DumpModule. Any AppDomain pointer can
be passed to DumpDomain to limit output only to that AppDomain. Other
functions provide an AppDomain pointer as well, such as clrthreads where it lists
the current AppDomain for each thread.
\\
COMMAND: eeheap.
EEHeap [-gc] [-loader]
EEHeap enumerates process memory consumed by internal CLR data structures. You
can limit the output by passing "-gc" or "-loader". All information will be
displayed otherwise.
The information for the Garbage Collector lists the ranges of each Segment in
the managed heap. This can be useful if you believe you have an object pointer.
If the pointer falls within a segment range given by "eeheap -gc", then you do
have an object pointer, and can attempt to run "dumpobj" on it.
Here is output for a simple program:
(lldb) eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00a71018
generation 1 starts at 0x00a7100c
generation 2 starts at 0x00a71000
segment begin allocated size
00a70000 00a71000 00a7e01c 0000d01c(53276)
Large object heap starts at 0x01a71000
segment begin allocated size
01a70000 01a71000 01a76000 0x00005000(20480)
Total Size 0x1201c(73756)
------------------------------
GC Heap Size 0x1201c(73756)
So the total size of the GC Heap is only 72K. On a large web server, with
multiple processors, you can expect to see a GC Heap of 400MB or more. The
Garbage Collector attempts to collect and reclaim memory only when required to
by memory pressure for better performance. You can also see the notion of
"generations," wherein the youngest objects live in generation 0, and
long-lived objects eventually get "promoted" to generation 2.
The loader output lists various private heaps associated with AppDomains. It
also lists heaps associated with the JIT compiler, and heaps associated with
Modules. For example:
(lldb) eeheap -loader
Loader Heap:
--------------------------------------
System Domain: 5e0662a0
LowFrequencyHeap:008f0000(00002000:00001000) Size: 0x00001000 bytes.
HighFrequencyHeap:008f2000(00008000:00001000) Size: 0x00001000 bytes.
StubHeap:008fa000(00002000:00001000) Size: 0x00001000 bytes.
Total size: 0x3000(12288)bytes
--------------------------------------
Shared Domain: 5e066970
LowFrequencyHeap:00920000(00002000:00001000) 03e30000(00010000:00003000) Size: 0x00004000 bytes.
Wasted: 0x00001000 bytes.
HighFrequencyHeap:00922000(00008000:00001000) Size: 0x00001000 bytes.
StubHeap:0092a000(00002000:00001000) Size: 0x00001000 bytes.
Total size: 0x6000(24576)bytes
--------------------------------------
Domain 1: 14f000
LowFrequencyHeap:00900000(00002000:00001000) 03ee0000(00010000:00003000) Size: 0x00004000 bytes.
Wasted: 0x00001000 bytes.
HighFrequencyHeap:00902000(00008000:00003000) Size: 0x00003000 bytes.
StubHeap:0090a000(00002000:00001000) Size: 0x00001000 bytes.
Total size: 0x8000(32768)bytes
--------------------------------------
Jit code heap:
Normal JIT:03ef0000(00010000:00002000) Size: 0x00002000 bytes.
Total size: 0x2000(8192)bytes
--------------------------------------
Module Thunk heaps:
Module 5ba22410: Size: 0x00000000 bytes.
Module 001c1320: Size: 0x00000000 bytes.
Module 001c03f0: Size: 0x00000000 bytes.
Module 001caa38: Size: 0x00000000 bytes.
Total size: 0x0(0)bytes
--------------------------------------
Module Lookup Table heaps:
Module 5ba22410:Size: 0x00000000 bytes.
Module 001c1320:Size: 0x00000000 bytes.
Module 001c03f0:Size: 0x00000000 bytes.
Module 001caa38:03ec0000(00010000:00002000) Size: 0x00002000 bytes.
Total size: 0x2000(8192)bytes
--------------------------------------
Total LoaderHeap size: 0x15000(86016)bytes
=======================================
By using eeheap to keep track of the growth of these private heaps, we are
able to rule out or include them as a source of a memory leak.
\\
COMMAND: name2ee.
Name2EE <module name> <type or method name>
Name2EE <module name>!<type or method name>
This function allows you to turn a class name into a MethodTable and EEClass.
It turns a method name into a MethodDesc. Here is an example for a method:
(lldb) name2ee unittest.exe MainClass.Main
Module: 001caa38
Token: 0x0600000d
MethodDesc: 00902f40
Name: MainClass.Main()
JITTED Code Address: 03ef00b8
and for a class:
(lldb) name2ee unittest!MainClass
Module: 001caa38
Token: 0x02000005
MethodTable: 009032d8
EEClass: 03ee1424
Name: MainClass
The module you are "browsing" with Name2EE needs to be loaded in the process.
To get a type name exactly right, first browse the module with ILDASM. You
can also pass * as the <module name> to search all loaded managed modules.
<module name> can also be the debugger's name for a module, such as
mscorlib or image00400000.
The <module>!<type> syntax is also supported. You can use an asterisk on the
left of the !, but the type on the right side needs to be fully qualified.
If you are looking for a way to display a static field of a class (and you
don't have an instance of the class, so dumpobj won't help you), note that
once you have the EEClass, you can run DumpClass, which will display the
value of all static fields.
There is yet one more way to specify a module name. In the case of modules
loaded from an assembly store (such as a SQL db) rather than disk, the
module name will look like this:
price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
For this kind of module, simply use price as the module name:
0:044> name2ee price Price
Module: 10f028b0 (price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)
Token: 0x02000002
MethodTable: 11a47ae0
EEClass: 11a538c8
Name: Price
Where are we getting these module names from? Run DumpDomain to see a list of
all loaded modules in all domains. And remember that you can browse all the
types in a module with DumpModule -mt <module pointer>.
\\
COMMAND: dumpmt.
DumpMT [-MD] <MethodTable address>
Examine a MethodTable. Each managed object has a MethodTable pointer at the
start. If you pass the "-MD" flag, you'll also see a list of all the methods
defined on the object.
\\
COMMAND: dumpclass.
DumpClass <EEClass address>
The EEClass is a data structure associated with an object type. DumpClass
will show attributes, as well as list the fields of the type. The output is
similar to DumpObj. Although static field values will be displayed,
non-static values won't because you need an instance of an object for that.
You can get an EEClass to look at from DumpMT, DumpObj, Name2EE, and
Token2EE among others.
\\
COMMAND: dumpmd.
DumpMD <MethodDesc address>
This command lists information about a MethodDesc. You can use ip2md to turn
a code address in a managed function into a MethodDesc:
(lldb) dumpmd 902f40
Method Name: Mainy.Main()
Class: 03ee1424
MethodTable: 009032d8
mdToken: 0600000d
Module: 001caa78
IsJitted: yes
CodeAddr: 03ef00b8
If IsJitted is "yes," you can run U on the CodeAddr pointer to see a
disassembly of the JITTED code. You can call also DumpClass, DumpMT,
DumpModule on the Class, MethodTable and Module fields above.
\\
COMMAND: token2ee.
Token2EE <module name> <token>
This function allows you to turn a metadata token into a MethodTable or
MethodDesc. Here is an example showing class tokens being resolved:
(lldb) sos Token2EE unittest.exe 02000003
Module: 001caa38
Token: 0x02000003
MethodTable: 0090375c
EEClass: 03ee1ae0
Name: Bank
(lldb) sos Token2EE image00400000 02000004
Module: 001caa38
Token: 0x02000004
MethodTable: 009038ec
EEClass: 03ee1b84
Name: Customer
The module you are "browsing" with Token2EE needs to be loaded in the process.
This function doesn't see much use, especially since a tool like ILDASM can
show the mapping between metadata tokens and types/methods in a friendlier way.
But it could be handy sometimes.
You can pass "*" for <module name> to find what that token maps to in every
loaded managed module. <module name> can also be the debugger's name for a
module, such as mscorlib or image00400000.
\\
COMMAND: dumpmodule.
DumpModule [-mt] <Module address>
You can get a Module address from DumpDomain, DumpAssembly and other
functions. Here is sample output:
(lldb) sos DumpModule 1caa50
Name: /home/user/pub/unittest
Attributes: PEFile
Assembly: 001ca248
LoaderHeap: 001cab3c
TypeDefToMethodTableMap: 03ec0010
TypeRefToMethodTableMap: 03ec0024
MethodDefToDescMap: 03ec0064
FieldDefToDescMap: 03ec00a4
MemberRefToDescMap: 03ec00e8
FileReferencesMap: 03ec0128
AssemblyReferencesMap: 03ec012c
MetaData start address: 00402230 (1888 bytes)
The Maps listed map metadata tokens to CLR data structures. Without going into
too much detail, you can examine memory at those addresses to find the
appropriate structures. For example, the TypeDefToMethodTableMap above can be
examined:
(lldb) dd 3ec0010
03ec0010 00000000 00000000 0090320c 0090375c
03ec0020 009038ec ...
This means TypeDef token 2 maps to a MethodTable with the value 0090320c. You
can run DumpMT to verify that. The MethodDefToDescMap takes a MethodDef token
and maps it to a MethodDesc, which can be passed to dumpmd.
There is a new option "-mt", which will display the types defined in a module,
and the types referenced by the module. For example:
(lldb) sos DumpModule -mt 1aa580
Name: /home/user/pub/unittest
...<etc>...
MetaData start address: 0040220c (1696 bytes)
Types defined in this module
MT TypeDef Name
--------------------------------------------------------------------------
030d115c 0x02000002 Funny
030d1228 0x02000003 Mainy
Types referenced in this module
MT TypeRef Name
--------------------------------------------------------------------------
030b6420 0x01000001 System.ValueType
030b5cb0 0x01000002 System.Object
030fceb4 0x01000003 System.Exception
0334e374 0x0100000c System.Console
03167a50 0x0100000e System.Runtime.InteropServices.GCHandle
0336a048 0x0100000f System.GC
\\
COMMAND: dumpassembly.
DumpAssembly <Assembly address>
Example output:
(lldb) sos DumpAssembly 1ca248
Parent Domain: 0014f000
Name: /home/user/pub/unittest
ClassLoader: 001ca060
Module Name
001caa50 /home/user/pub/unittest
An assembly can consist of multiple modules, and those will be listed. You can
get an Assembly address from the output of DumpDomain.
\\
COMMAND: dumpruntimetypes.
DumpRuntimeTypes
DumpRuntimeTypes finds all System.RuntimeType objects in the gc heap and
prints the type name and MethodTable they refer too. Sample output:
Address Domain MT Type Name
------------------------------------------------------------------------------
a515f4 14a740 5baf8d28 System.TypedReference
a51608 14a740 5bb05764 System.Globalization.BaseInfoTable
a51958 14a740 5bb05b24 System.Globalization.CultureInfo
a51a44 14a740 5bb06298 System.Globalization.GlobalizationAssembly
a51de0 14a740 5bb069c8 System.Globalization.TextInfo
a56b98 14a740 5bb12d28 System.Security.Permissions.HostProtectionResource
a56bbc 14a740 5baf7248 System.Int32
a56bd0 14a740 5baf3fdc System.String
a56cfc 14a740 5baf36a4 System.ValueType
...
This command will print a "?" in the domain column if the type is loaded into multiple
AppDomains. For example:
(lldb) sos DumpRuntimeTypes
Address Domain MT Type Name
------------------------------------------------------------------------------
28435a0 ? 3f6a8c System.TypedReference
28435b4 ? 214d6c System.ValueType
28435c8 ? 216314 System.Enum
28435dc ? 2147cc System.Object
284365c ? 3cd57c System.IntPtr
2843670 ? 3feaac System.Byte
2843684 ? 23a544c System.IEquatable`1[[System.IntPtr, mscorlib]]
2843784 ? 3c999c System.Int32
2843798 ? 3caa04 System.IEquatable`1[[System.Int32, mscorlib]]
\\
COMMAND: dumpsig.
DumpSig <sigaddr> <moduleaddr>
This command dumps the signature of a method or field given by <sigaddr>. This is
useful when you are debugging parts of the runtime which returns a raw PCCOR_SIGNATURE
structure and need to know what its contents are.
Sample output for a method:
0:000> sos DumpSig 0x000007fe`ec20879d 0x000007fe`eabd1000
[DEFAULT] [hasThis] Void (Boolean,String,String)
The first section of the output is the calling convention. This includes, but is not
limited to, "[DEFAULT]", "[C]", "[STDCALL]", "[THISCALL]", and so on. The second
portion of the output is either "[hasThis]" or "[explicit]" for whether the method
is an instance method or a static method respectively. The third portion of the
output is the return value (in this case a "void"). Finally, the method's arguments
are printed as the final portion of the output.
Sample output for a field:
0:000> sos DumpSig 0x000007fe`eb7fd8cd 0x000007fe`eabd1000
[FIELD] ValueClass System.RuntimeTypeHandle
DumpSig will also work with generics. Here is the output for the following
function:
public A Test(IEnumerable<B> n)
0:000> sos DumpSig 00000000`00bc2437 000007ff00043178
[DEFAULT] [hasThis] __Canon (Class System.Collections.Generic.IEnumerable`1<__Canon>)
\\
COMMAND: dumpsigelem.
DumpSigElem <sigaddr> <moduleaddr>
This command dumps a single element of a signature object. For most circumstances,
you should use DumpSig to look at individual signature objects, but if you find a
signature that has been corrupted in some manner you can use DumpSigElem to read out
the valid portions of it.
If we look at a valid signature object for a method we see the following:
0:000> dumpsig 0x000007fe`ec20879d 0x000007fe`eabd1000
[DEFAULT] [hasThis] Void (Boolean,String,String)
We can look at the individual elements of this object by adding the offsets into the
object which correspond to the return value and parameters:
0:000> sos DumpSigElem 0x000007fe`ec20879d+2 0x000007fe`eabd1000
Void
0:000> sos DumpSigElem 0x000007fe`ec20879d+3 0x000007fe`eabd1000
Boolean
0:000> sos DumpSigElem 0x000007fe`ec20879d+4 0x000007fe`eabd1000
String
0:000> sos DumpSigElem 0x000007fe`ec20879d+5 0x000007fe`eabd1000
String
We can do something similar for fields. Here is the full signature of a field:
0:000> dumpsig 0x000007fe`eb7fd8cd 0x000007fe`eabd1000
[FIELD] ValueClass System.RuntimeTypeHandle
Using DumpSigElem we can find the type of the field by adding the offset of it (1) to
the address of the signature:
0:000> sos DumpSigElem 0x000007fe`eb7fd8cd+1 0x000007fe`eabd1000
ValueClass System.RuntimeTypeHandle
DumpSigElem will also work with generics. Let a function be defined as follows:
public A Test(IEnumerable<B> n)
The elements of this signature can be obtained by adding offsets into the signature
when calling DumpSigElem:
0:000> sos DumpSigElem 00000000`00bc2437+2 000007ff00043178
__Canon
0:000> sos DumpSigElem 00000000`00bc2437+4 000007ff00043178
Class System.Collections.Generic.IEnumerable`1<__Canon>
The actual offsets that you should add are determined by the contents of the
signature itself. By trial and error you should be able to find various elements
of the signature.
\\
COMMAND: dumpil.
DumpIL <Managed DynamicMethod object> |
<DynamicMethodDesc pointer> |
<MethodDesc pointer> |
/i <IL pointer>
DumpIL prints the IL code associated with a managed method. We added this
function specifically to debug DynamicMethod code which was constructed on
the fly. Happily it works for non-dynamic code as well.
You can use it in four ways:
1) If you have a System.Reflection.Emit.DynamicMethod object, just pass
the pointer as the first argument.
2) If you have a DynamicMethodDesc pointer you can use that to print the
IL associated with the dynamic method.
3) If you have an ordinary MethodDesc, you can see the IL for that as well,
just pass it as the first argument.
4) If you have a pointer directly to the IL, specify /i followed by the
the IL address. This is useful for writers of profilers that instrument
IL.
Note that dynamic IL is constructed a bit differently. Rather than referring
to metadata tokens, the IL points to objects in a managed object array. Here
is a simple example of the output for a dynamic method:
0:000> sos DumpIL b741dc
This is dynamic IL. Exception info is not reported at this time.
If a token is unresolved, run "sos DumpObj <addr>" on the addr given
in parenthesis. You can also look at the token table yourself, by
running "DumpArray 00b77388".
IL_0000: ldstr 70000002 "Inside invoked method "
IL_0005: call 6000003 System.Console.WriteLine(System.String)
IL_000a: ldc.i4.1
IL_000b: newarr 2000004 "System.Int32"
IL_0010: stloc.0
IL_0011: ldloc.0
IL_0012: ret
\\
COMMAND: verifyheap.
VerifyHeap
VerifyHeap is a diagnostic tool that checks the garbage collected heap for
signs of corruption. It walks objects one by one in a pattern like this:
o = firstobject;
while(o != endobject)
{
o.ValidateAllFields();
o = (Object *) o + o.Size();
}
If an error is found, VerifyHeap will report it. I'll take a perfectly good
object and corrupt it:
(lldb) dumpobj a79d40
Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes
(/home/user/pub/unittest)
Fields:
MT Field Offset Type Attr Value Name
009038ec 4000008 4 CLASS instance 00a79ce4 name
009038ec 4000009 8 CLASS instance 00a79d2c bank
009038ec 400000a c System.Boolean instance 1 valid
(lldb) ed a79d40+4 01 (change the name field to the bogus pointer value 1)
(lldb) sos VerifyHeap
object 01ee60dc: bad member 00000003 at 01EE6168
Last good object: 01EE60C4.
If this gc heap corruption exists, there is a serious bug in your own code or
in the CLR. In user code, an error in constructing PInvoke calls can cause
this problem, and running with Managed Debugging Assistants is advised. If that
possibility is eliminated, consider contacting Microsoft Product Support for
help.
\\
COMMAND: dumplog.
DumpLog [-addr <addressOfStressLog>] [<Filename>]
To aid in diagnosing hard-to-reproduce stress failures, the CLR team added an
in-memory log capability. The idea was to avoid using locks or I/O which could
disturb a fragile repro environment. The DumpLog function allows you to write
that log out to a file. If no Filename is specified, the file "Stresslog.txt"
in the current directory is created.
The optional argument addr allows one to specify a stress log other then the
default one.
(lldb) dumplog
Attempting to dump Stress log to file 'StressLog.txt'
.................
SUCCESS: Stress log dumped
To turn on the stress log, set the following registry keys under
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework:
(DWORD) StressLog = 1
(DWORD) LogFacility = 0xffffffbf (this is a bit mask, almost all logging is on.
This is also the default value if the key
isn't specified)
(DWORD) StressLogSize = 65536 (this is the default value if the key isn't
specified)
(DWORD) LogLevel = 6 (this is the default value if the key isn't
specified. The higher the number the more
detailed logs are generated. The maximum
value is decimal 10)
StressLogSize is the size in bytes of the in-memory log allocated for each
thread in the process. In the case above, each thread gets a 64K log. You
could increase this to get more logging, but more memory will be required for
this log in the process. For example, 20 threads with 524288 bytes per thread
has a memory demand of 10 Megabytes. The stress log is circular so new entries
will replace older ones on threads which have reached their buffer limit.
The log facilities are defined as follows:
GC 0x00000001
GCINFO 0x00000002
STUBS 0x00000004
JIT 0x00000008
LOADER 0x00000010
METADATA 0x00000020
SYNC 0x00000040
EEMEM 0x00000080
GCALLOC 0x00000100
CORDB 0x00000200
CLASSLOADER 0x00000400
CORPROF 0x00000800
REMOTING 0x00001000
DBGALLOC 0x00002000
EH 0x00004000
ENC 0x00008000
ASSERT 0x00010000
VERIFIER 0x00020000
THREADPOOL 0x00040000
GCROOTS 0x00080000
INTEROP 0x00100000
MARSHALER 0x00200000
IJW 0x00400000
ZAP 0x00800000
STARTUP 0x01000000
APPDOMAIN 0x02000000
CODESHARING 0x04000000
STORE 0x08000000
SECURITY 0x10000000
LOCKS 0x20000000
BCL 0x40000000
Here is some sample output:
3560 9.981137099 : `SYNC` RareEnablePremptiveGC: entering.
Thread state = a030
3560 9.981135033 : `GC`GCALLOC`GCROOTS` ========== ENDGC 4194 (gen = 2,
collect_classes = 0) ==========={
3560 9.981125826 : `GC` Segment mem 00C61000 alloc
= 00D071F0 used 00D09254 committed 00D17000
3560 9.981125726 : `GC` Generation 0 [00CED07C, 00000000
] cur = 00000000
3560 9.981125529 : `GC` Generation 1 [00CED070, 00000000
] cur = 00000000
3560 9.981125103 : `GC` Generation 2 [00C61000, 00000000
] cur = 00000000
3560 9.981124963 : `GC` GC Heap 00000000
3560 9.980618994 : `GC`GCROOTS` GcScanHandles (Promotion Phase = 0)
The first column is the OS thread ID for the thread appending to the log,
the second column is the timestamp, the third is the facility category for the
log entry, and the fourth contains the log message. The facility field is
expressed as `facility1`facility2`facility3`. This facilitates the creation of
filters for displaying only specific message categories. To make sense of this
log, you would probably want the Shared Source CLI to find out exactly where
the log comes from.
\\
COMMAND: findappdomain.
FindAppDomain <Object address>
FindAppDomain will attempt to resolve the AppDomain of an object. For example,
using an Object Pointer from the output of DumpStackObjects:
(lldb) sos FindAppDomain 00a79d98
AppDomain: 0014f000
Name: unittest.exe
ID: 1
You can find out more about the AppDomain with the DumpDomain command. Not
every object has enough clues about it's origin to determine the AppDomain.
Objects with Finalizers are the easiest case, as the CLR needs to be able to
call those when an AppDomain shuts down.
\\
COMMAND: histinit.
HistInit
Before running any of the Hist - family commands you need to initialize the SOS
structures from the stress log saved in the debuggee. This is achieved by the
HistInit command.
Sample output:
(lldb) histinit
Attempting to read Stress log
STRESS LOG:
facilitiesToLog = 0xffffffff
levelToLog = 6
MaxLogSizePerThread = 0x10000 (65536)
MaxTotalLogSize = 0x1000000 (16777216)
CurrentTotalLogChunk = 9
ThreadsWithLogs = 3
Clock frequency = 3.392 GHz
Start time 15:26:31
Last message time 15:26:56
Total elapsed time 25.077 sec
.....................................
---------------------------- 2407 total entries -----------------------------
SUCCESS: GCHist structures initialized
\\
COMMAND: histobjfind.
HistObjFind <obj_address>
To examine log entries related to an object whose present address is known one
would use this command. The output of this command contains all entries that
reference the object:
(lldb) histobjfind 028970d4
GCCount Object Message
---------------------------------------------------------
2296 028970d4 Promotion for root 01e411b8 (MT = 5b6c5cd8)
2296 028970d4 Relocation NEWVALUE for root 00223fc4
2296 028970d4 Relocation NEWVALUE for root 01e411b8
...
2295 028970d4 Promotion for root 01e411b8 (MT = 5b6c5cd8)
2295 028970d4 Relocation NEWVALUE for root 00223fc4
2295 028970d4 Relocation NEWVALUE for root 01e411b8
...
\\
COMMAND: histroot.
HistRoot <root>
The root value obtained from !HistObjFind can be used to track the movement of
an object through the GCs.
HistRoot provides information related to both promotions and relocations of the
root specified as the argument.
(lldb) histroot 01e411b8
GCCount Value MT Promoted? Notes
---------------------------------------------------------
2296 028970d4 5b6c5cd8 yes
2295 028970d4 5b6c5cd8 yes
2294 028970d4 5b6c5cd8 yes
2293 028970d4 5b6c5cd8 yes
2292 028970d4 5b6c5cd8 yes
2291 028970d4 5b6c5cd8 yes
2290 028970d4 5b6c5cd8 yes
2289 028970d4 5b6c5cd8 yes
2288 028970d4 5b6c5cd8 yes
2287 028970d4 5b6c5cd8 yes
2286 028970d4 5b6c5cd8 yes
2285 028970d4 5b6c5cd8 yes
322 028970e8 5b6c5cd8 yes Duplicate promote/relocs
...
\\
COMMAND: histobj.
HistObj <obj_address>
This command examines all stress log relocation records and displays the chain
of GC relocations that may have led to the address passed in as an argument.
Conceptually the output is:
GenN obj_address root1, root2, root3,
GenN-1 prev_obj_addr root1, root2,
GenN-2 prev_prev_oa root1, root4,
...
Sample output:
(lldb) histobj 028970d4
GCCount Object Roots
---------------------------------------------------------
2296 028970d4 00223fc4, 01e411b8,
2295 028970d4 00223fc4, 01e411b8,
2294 028970d4 00223fc4, 01e411b8,
2293 028970d4 00223fc4, 01e411b8,
2292 028970d4 00223fc4, 01e411b8,
2291 028970d4 00223fc4, 01e411b8,
2290 028970d4 00223fc4, 01e411b8,
2289 028970d4 00223fc4, 01e411b8,
2288 028970d4 00223fc4, 01e411b8,
2287 028970d4 00223fc4, 01e411b8,
2286 028970d4 00223fc4, 01e411b8,
2285 028970d4 00223fc4, 01e411b8,
322 028970d4 01e411b8,
0 028970d4
\\
COMMAND: histclear.
HistClear
This command releases any resources used by the Hist-family of commands.
Generally there's no need to call this explicitly, as each HistInit will first
cleanup the previous resources.
(lldb) histclear
Completed successfully.
\\
[이 글에 대해서 여러분들과 의견을 공유하고 싶습니다. 틀리거나 미흡한 부분 또는 의문 사항이 있으시면 언제든 댓글 남겨주십시오.]