리눅스 환경의 .NET Core 3/5+ 메모리 덤프를 분석하는 방법 - 두 번째 이야기
예전에 lldb를 이용해 libsos.sos/libsosplugin.so 파일을 직접 로드하는 방법을 소개했습니다.
linux - lldb를 이용한 .NET Core 응용 프로그램의 메모리 덤프 분석 방법
; https://www.sysnet.pe.kr/2/0/12083
그런데, 어느 순간부터 libsos.sos/libsosplugin.so 파일이 안 보입니다. 메모리 덤프 뜨는 방법을 소개했던 (그때 당시에는 실패했었지만) "dotnet-dump"로,
.NET Core 응용 프로그램을 위한 메모리 덤프 방법
; https://www.sysnet.pe.kr/2/0/12078
기능을 통합시켜 버린 듯합니다. 실제로 간단하게 실습해 볼까요? ^^ 우선, dotnet-dump를 설치하고,
// mcr.microsoft.com/dotnet/sdk:3.1 docker 환경에서 설치
# dotnet tool install -g dotnet-dump
Tools directory '/root/.dotnet/tools' is not currently on the PATH environment variable.
If you are using bash, you can add it to your profile by running the following command:
cat << \EOF >> ~/.bash_profile
# Add .NET Core SDK tools
export PATH="$PATH:/root/.dotnet/tools"
EOF
You can add it to the current session by running the following command:
export PATH="$PATH:/root/.dotnet/tools"
You can invoke the tool using the following command: dotnet-dump
Tool 'dotnet-dump' (version '6.0.328102') was successfully installed.
(참고로, dotnet-dump는 .NET 런타임이 아닌 SDK가 설치되어 있어야 합니다.)
이후, 현재 머신에 실행 중인 닷넷 프로세스를 열거하고,
# /root/.dotnet/tools/dotnet-dump ps
1 dotnet /usr/share/dotnet/dotnet dotnet razor31_sample.dll
덤프를 뜬 후,
# /root/.dotnet/tools/dotnet-dump collect --process-id 1
Writing full to /app/core_20221005_221655
Complete
곧장 sos가 로드된 환경의 분석 모드로 들어갈 수 있습니다.
# /root/.dotnet/tools/dotnet-dump analyze /app/core_20221005_221655
Loading core dump: /app/core_20221005_221655 ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.
>
그럼, 저 명령 프롬프트에서 기존의 sos 확장에서 제공하던 명령어들을 입력해 분석할 수 있습니다.
> lm
00007FE692740000 00A1E000 /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.16/System.Private.CoreLib.dll
00007FE6932C0000 0005C000 /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.16/System.Console.dll
...[생략]...
00007FE7088AB000 00335000 /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.16/libclrjit.so
00007FE70C235000 0000A000 /lib/x86_64-linux-gnu/librt-2.31.so
00007FE70C23F000 006F8000 /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.16/libcoreclr.so
00007FE70C977000 0005F000 /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.16/libhostpolicy.so
00007FE70C9D6000 00068000 /usr/share/dotnet/host/fxr/6.0.16/libhostfxr.so
00007FE70CA43000 001D1000 /lib/x86_64-linux-gnu/libc-2.31.so
00007FE70CC18000 0001A000 /lib/x86_64-linux-gnu/libgcc_s.so.1
00007FE70CC32000 00144000 /lib/x86_64-linux-gnu/libm-2.31.so
00007FE70CD76000 001CA000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
00007FE70CF43000 00006000 /lib/x86_64-linux-gnu/libdl-2.31.so
00007FE70CF49000 0001E000 /lib/x86_64-linux-gnu/libpthread-2.31.so
00007FE70CF6E000 00008000 /test/bin/Debug/net6.0/test.dll
Total image size: 29505536
> clrstack
OS Thread Id: 0x1 (0)
Child SP IP Call Site
00007FFF71CB04F0 00007f9a4c41200c [GCFrame: 00007fff71cb04f0]
00007FFF71CB05D0 00007f9a4c41200c [HelperMethodFrame_1OBJ: 00007fff71cb05d0] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007FFF71CB0700 00007F99D30D30D9 System.Threading.Monitor.Wait(System.Object, Int32, Boolean) [/_/src/System.Private.CoreLib/src/System/Threading/Monitor.cs @ 183]
00007FFF71CB0730 00007F99D30D303F System.Threading.Monitor.Wait(System.Object, Int32) [/_/src/System.Private.CoreLib/src/System/Threading/Monitor.cs @ 193]
00007FFF71CB0750 00007F99D30D17EB System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) [/_/src/System.Private.CoreLib/shared/System/Threading/ManualResetEventSlim.cs @ 609]
00007FFF71CB07E0 00007F99D30CF364 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2922]
00007FFF71CB0840 00007F99D30CE732 System.Threading.Tasks.Task.InternalWaitCore(Int32, System.Threading.CancellationToken) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2861]
00007FFF71CB08F0 00007F99D30CDC45 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2820]
00007FFF71CB0920 00007F99D30CDAF4 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs @ 143]
00007FFF71CB0950 00007F99D30CDA46 System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs @ 122]
00007FFF71CB0970 00007F99D4D48A4A System.Runtime.CompilerServices.TaskAwaiter.GetResult() [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs @ 106]
00007FFF71CB0990 00007F99D4D455F2 Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(Microsoft.Extensions.Hosting.IHost) [/_/src/libraries/Microsoft.Extensions.Hosting.Abstractions/src/HostingAbstractionsHostExtensions.cs @ 50]
00007FFF71CB09C0 00007F99D212C3C5 razor31_sample.Program.Main(System.String[]) [/src/razor31_sample/Program.cs @ 17]
00007FFF71CB0CD8 00007f9a4b5252df [GCFrame: 00007fff71cb0cd8]
00007FFF71CB11C0 00007f9a4b5252df [GCFrame: 00007fff71cb11c0]
>
그러니까, dotnet-dump가 windbg + sos 환경을 리눅스에서 그대로 제공하고 있는 것입니다.
여기서 가장 큰 궁금한 점은, core_20221005_221655 덤프 파일을 다른 PC에서도 windbg처럼 분석할 수 있느냐 하는 것입니다. 확인을 해보면 되겠죠? ^^ 해당 파일을 다른 PC, 여기서는 WSL에 복사한 것으로 가정하겠습니다.
위에서 덤프를 뜬 응용 프로그램은 .NET Core 3.1로 만든 것이었는데, 테스트를 위해 WSL Ubuntu에는 .NET 6 SDK를 설치하겠습니다.
// Ubuntu 20.04 환경에서 .NET 6 SDK 설치
$ wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
$ sudo dpkg -i packages-microsoft-prod.deb
$ rm packages-microsoft-prod.deb
$ sudo apt-get update && sudo apt-get install -y dotnet-sdk-6.0
역시 dotnet-dump를 설치하고,
$ dotnet tool install -g dotnet-dump
Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.401
Telemetry
---------
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.
Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
Since you just installed the .NET SDK, you will need to logout or restart your session before running the tool you installed.
You can invoke the tool using the following command: dotnet-dump
Tool 'dotnet-dump' (version '6.0.328102') was successfully installed.
분석을 시도한 결과는?
$ ~/.dotnet/tools/dotnet-dump analyze core_20221005_221655
Loading core dump: core_20221005_221655 ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.
> clrstack
OS Thread Id: 0x1 (0)
Child SP IP Call Site
00007FFF71CB04F0 00007f9a4c41200c [GCFrame: 00007fff71cb04f0]
00007FFF71CB05D0 00007f9a4c41200c [HelperMethodFrame_1OBJ: 00007fff71cb05d0] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007FFF71CB0700 00007F99D30D30D9 System.Threading.Monitor.Wait(System.Object, Int32, Boolean) [/_/src/System.Private.CoreLib/src/System/Threading/Monitor.cs @ 183]
00007FFF71CB0730 00007F99D30D303F System.Threading.Monitor.Wait(System.Object, Int32) [/_/src/System.Private.CoreLib/src/System/Threading/Monitor.cs @ 193]
00007FFF71CB0750 00007F99D30D17EB System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) [/_/src/System.Private.CoreLib/shared/System/Threading/ManualResetEventSlim.cs @ 609]
00007FFF71CB07E0 00007F99D30CF364 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2922]
00007FFF71CB0840 00007F99D30CE732 System.Threading.Tasks.Task.InternalWaitCore(Int32, System.Threading.CancellationToken) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2861]
00007FFF71CB08F0 00007F99D30CDC45 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) [/_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs @ 2820]
00007FFF71CB0920 00007F99D30CDAF4 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs @ 143]
00007FFF71CB0950 00007F99D30CDA46 System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs @ 122]
00007FFF71CB0970 00007F99D4D48A4A System.Runtime.CompilerServices.TaskAwaiter.GetResult() [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs @ 106]
00007FFF71CB0990 00007F99D4D455F2 Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(Microsoft.Extensions.Hosting.IHost) [/_/src/libraries/Microsoft.Extensions.Hosting.Abstractions/src/HostingAbstractionsHostExtensions.cs @ 50]
00007FFF71CB09C0 00007F99D212C3C5 razor31_sample.Program.Main(System.String[])
00007FFF71CB0CD8 00007f9a4b5252df [GCFrame: 00007fff71cb0cd8]
00007FFF71CB11C0 00007f9a4b5252df [GCFrame: 00007fff71cb11c0]
보는 바와 같이 잘 clrstack 명령어 결과가 잘 나옵니다. 오~~~ 멋지지 않나요? 이전에 필요했던 그
복잡한 과정들이 하나도 필요가 없게 되었습니다. ^^
[이 글에 대해서 여러분들과 의견을 공유하고 싶습니다. 틀리거나 미흡한 부분 또는 의문 사항이 있으시면 언제든 댓글 남겨주십시오.]