|

All Topics, MFC / C++ >> System >> Unedited Reader Contributions

API Hooking revealed part 2 - Useful tips
By xryl669

Some useful trick and tips before being able to detect deadlock

| Advanced

C++
Windows
Win32, VS
Dev

| Posted 1 Feb 2005 10:21
| Updated 10 Mar 2005 6:41
| Articles by this author
| 5,092 views

Note: This is an unedited reader contribution

Search:

Articles

Authors

Help! | Articles | Message Boards | StoreFront | Lounge

|
| Toolbox

| Broken Article?
|
| VS.NET 2003 for $899
| Ultimate Toolbox $499
|
| Print version
| Send to a friend
|

Sign in / Sign up

Email |
Password |
Remember me

|

Lost your Password?

3 members have rated this article. Result:

| | | |

Popularity: 1.54. Rating: 3.22 out of 5.

Introduction

This is the second part for building a thread deadlock detector. Please see the first article to understand wait is going on : A (working) implementation of API hooking (Part I)
The next part (with a working deadlock detector) is also here : Thread Deadlock detector

Part II : The tricky part

The API hooked functions for a thread deadlock detector

What should be intercepted in the purpose of detecting deadlock are :
Thread functions (Create[Remote]Thread, [Suspend/Resume]Thread, ExitThread, TerminateThread, OpenThread)
Synchronization functions (WaitFor[Single/Multiple]Object[Ex], SignalObjectAndWait, [Set/Reset/Pulse]Event)
Synchronization objects creation (Create/Open[Mutex/Semaphore/Event], DuplicateHandle)
Synchronization objects deletion (CloseHandle)

To intercept the code, I simply added to the previous code the required functions. Please have a look to the previous article about how to add functions to be hooked.
The main idea is to have a hook structure declared like :

typedef struct
{
char szDLLName[MAX_PATH]; // The DLL name
char szFuncName[MAX_PATH]; // The function name
void * pNewFunc; // The new function pointer
void * pPrevFunc; // The previous function pointer
Flags flags; // The flags (hooked or not, etc...)
} HookStruct;

When hooking the function in all modules, the previous (and true) function pointer is saved in pPrevFunc, while the new function pointer (set to our hooking function) replaces the module IAT.

Then in our function, we can simply call the previous function by converting the pPrevFunc pointer to the correct function's pointer signature. I defined a useful macro for that (the source code will be in part III).

#define CallFunction(X) ((Signature_##X)GetPreviousFunctionAddress(Index_##X))
// with signature defined like this
#define Signature_CreateThread HANDLE (FAR PASCAL *)\
(LPSECURITY_ATTRIBUTES lpThreadAttributes, DWORD dwStackSize, \
LPTHREAD_START_ROUTINE lpStartAddress, LPVOID lpParameter, DWORD \
dwCreationFlags, LPDWORD lpThreadId)
// And index like this
#define Index_CreateThread 3

Okay, we can now call the good function, so let's start the real work.

The communication process

So, how do we inform the server that the "hooked" application is currently performing a monitored action?
The simple answer is to create a structure to send to the server through a WM_COPYDATA message, with the needed informations. The needed informations are application-dependant, and in our case, they are:

The current thread handle (who is calling this function)
The current thread id (the handle is not enough, see below)
The manipulated object id/handle (what are we touching)
An additionnal argument (if any)
The current command (what function are we calling)
The object name, if any (useful to debugging only)
The call stack (required to find from where the call cames from)
The current timestamp (needed to check for deadlocks)
The structure is then defined as :
typedef struct
{
void * lAddress; // The address pointer in the stack
unsigned int lFlags; // Some flags
} StackPointer;

typedef struct
{
HANDLE hObjectID; // The waiting object ID
HANDLE hThread; // The current thread ID
DWORD dwThreadID; // The current thread ID
unsigned int lState; // The current state, count, etc...
HANDLE hFlag; // The stack here
Commands Command; // The current command
char sName[256]; // The object name (if any)
LARGE_INTEGER llTimestamp; // The message timestamp
StackPointer xPointers[10]; // The stack pointers (only 10 pointers are saved)
} CommunicationObject;
// with Commands being an enum like CmdCreateThread = 0, CmdExitThread = 1, etc...

A hooking function will then look like :

// Declare the signature for waiting functions
THREADSPY_API HANDLE WINAPI MyCreateThread(LPSECURITY_ATTRIBUTES
lpThreadAttributes, DWORD dwStackSize, LPTHREAD_START_ROUTINE lpStartAddress,
LPVOID lpParameter, DWORD dwCreationFlags, LPDWORD lpThreadId)
{
DWORD dwID;
HANDLE hHandle = CallFunction(CreateThread)(lpThreadAttributes,
dwStackSize, lpStartAddress, lpParameter, dwCreationFlags, &dwID);
if (hHandle != NULL)
{
CommunicationObject xObj;
memset(&xObj, 0, sizeof(xObj));
// Get the true current thread handle
xObj.hThread = GetTrueCurrentThread();
// Get the manipulated handle (it is a thread id here)
xObj.hObjectID = hHandle;
// And other data
xObj.lState = dwID;
// Save the caller thread ID
xObj.dwThreadID = GetCurrentThreadId();
if (lpThreadId != NULL)
{
*lpThreadId = dwID;
}
// Save the current thread
xObj.Command = CmdCreateThread;
// Then send the structure (but fill it with timestamp and call stack before)
Communicate(&xObj, sizeof(xObj));
// Send another command if the thread is suspended
if (dwCreationFlags & CREATE_SUSPENDED)
{
xObj.lState = 1;
xObj.Command = CmdSuspendThread;
CommunicateWithoutTime(&xObj, sizeof(xObj));
}

// We don't need the handle anymore
if (xObj.hThread != NULL) CallFunction(CloseHandle)(xObj.hThread);
}

return hHandle;
}

The tricky part

Okay, now we "simply" have to fill the structure. Even if it looks easy, it is not because of many problem due to shortage in Win32 API.
For example, if you use the GetCurrentThread function, the Windows API will return a special handle CURRENT_THREAD_HANDLE. This information is , of course, no useful here. This is due to how Windows handles HANDLE. For the same kernel object, one can have multiple handle on it, on different memory space. So with an HANDLE it is not possible to uniquely identify a thread, we need its ID. While it is easy to store the thread handle and ID in a structure in any program, it is not obvious that the debuggee program will have such a mapping.
That's why we need to find how to get the thread ID given its handle.
For example, when a thread call TerminateThread to kill another thread, it only uses the killed thread handle, not its ID. The server will then never be able to match what thread was killed (or this will require a kind of matching algorithm etc, etc...)
Windows Server2k3 provides a function called GetThreadId to get the thread ID, but because it is only in Win2K3, it is not useful.

The solution to this issue is to use the NtQueryInformationThread hidden function from NTDLL.DLL (NTDLL.DLL is mapped in every process memory) like VisualStudio debugger. This function can return a CLIENT_ID structure with the thread ID in it.
For more information, please see the Undocumented NT Internal

Then now that we can identify the object being manipulated, we need to get the stack trace. This can be done by using the StackWalk function from Win32 API. Usually, this function is used by stopping the debuggee thread, retreiving its context, and then resuming the thread. As we don't want to stop the thread (because it will change scheduling order while being debugged), we need to fill up a context structure by ourselve. The trick is to read the EIP register before using StackWalk, and this can be done easily with few asm command like :

CONTEXT c;

_EnterCriticalSection(&mhSection);
// This is a ugly code to get the stack trace
__asm
{
call GetEIP
GetEIP:
pop eax
mov c.Eip, eax
mov c.Ebp, ebp
};
_LeaveCriticalSection(&mhSection);

The last needed trick is to detect when a thread has stopped. Because the injected DLL cannot create a thread without disturbing the program, we are supplying a function called CheckRunningThread that use the same signature as a Thread Start routine, taking a thread handle in client memory space, and returning a DWORD for the thread state. By using the same method for injecting the DLL (thanks to CreateRemoteThread), the server can stop the process being debugged, create a remote thread starting on the CheckRunningThread function, and reading the thread status before resuming the debuggee.
This way, the server can check when a thread finishes and detect for never-released objects

To conclude

In this article there are answers to some "impossible" things (from MSDN) like :

Get a thread ID from its thread handle (google around, and you'll see it is a real issue)
Get the stack trace of a running thread (again it is not a usual practice)
Get a real thread handle instead of the default generic value (using DuplicateHandle)
Spy when a thread in a remote process has finished
Communicate with a server about any action
Hook any Win32 API function

The drawback is that it requires WinNT kernel (like XP, 2k and 2k3), but I'm sure it no more an issue nowadays.

This is for client part. I will provide the source code for both the server and client in the next part (Part III). We will then see how to get the entire log of synchronization function call in a process, and how to map the call stack value to function names.

xryl669

Click here to view xryl669's online profile.

Other popular articles:

Driver Development Part 1: Introduction to Drivers
This article will go into the basics of creating a simple driver.
Driver Development Part 2: Introduction to Implementing IOCTLs
This article will go deeper into the basics of creating a simple driver.
A simple demo for WDM Driver development
WDM Driver programming introduction with three Pseudo Drivers.
API hooking revealed
The article demonstrates how to build a user mode Win32 API spying system

[Top] | Sign in to vote for this article:     PoorExcellent

Premium Sponsor

FAQ

Noise level
Very High

High

Medium

Low

Very low

Search comments

View
Normal (slow)

Preview (slow)

Message View

Topic View

Thread View

Expanded (Supporters only)

Per page
10

25

50 (must logon)

New thread | Msgs 1 to 3 of 3 (Total: 3) (Refresh) | First Prev Next

Subject | Author | Date

| Deadlocks on Timeable Objects? | | Blake Miller | 16:18 1 Feb '05

Can't you catch WAIT_FAILED and WAIT_TIMEOUT on the types of system calls (WaitFor*) discussed in this article?

So, if your thread sees WAIT_TIMEOUT then you might have some kind of 'deadlock'.
Maybe I am just 'different for never using INFINITE in the argument for the wait timeout

I would have thought it most important to catch EnterCriticalSection and LEaveCriticalSection, since you can not specify a timeout for EnterCriticalSection.

I still really like the ideas presented here, however.

[Reply][Email][View Thread][Get Link]

[Modify|Delete]

| Re: Deadlocks on Timeable Objects? | | xryl669 | 4:39 2 Feb '05

Blake Miller wrote:
Can't you catch WAIT_FAILED and WAIT_TIMEOUT on the types of system calls (WaitFor*) discussed in this article?

So, if your thread sees WAIT_TIMEOUT then you might have some kind of 'deadlock'.
Maybe I am just 'different for never using INFINITE in the argument for the wait timeout

In almost all my software, I use a timeout while waiting for an object. However, sometimes it is really difficult to avoid inifite loops (what to do if an object can't be taken x times, for example in a DirectShow filter waiting for more data from the network, as it is not possible to stop the application then). Anyway, using timeout is most of time far from optimal, as it leads to unexpected execution time of a thread. I believe that a good algorithm could never have to timeout a wait.

I would have thought it most important to catch EnterCriticalSection and LeaveCriticalSection, since you can not specify a timeout for EnterCriticalSection.

I still really like the ideas presented here, however.
You can use TryEnterCriticalSection function for Critical sections, without needing a timeout. I think however that it is a very good idea not to have timeout in critical section (because of code complexity needed to handle timeouts). By design, critical section are supposed to be kind of rare and quick.

[Reply][Email][View Thread][Get Link]

[Modify|Delete]

| Re: Deadlocks on Timeable Objects? | | Blake Miller | 10:58 3 Feb '05

I believe that a good algorithm could never have to timeout a wait.
In an ideal world, the user never trips over the Ethernet cable, the phone line does not spontaneously disconnect, the PLC does not lose power or reboot, etc. Mostly, I always have timeouts where communications are involved so that I can recover from such situations where ongoing communication is expected, but perhaps not forthcoming. INFINITE is a long time to wait...
I suppose, then, in such cases, the fact that the thread gets some time after a waitable timeout is 'unexpected' but at least accounted for as part of the design of the system. In our process control software, I have had to rework several designs that previously used INIFNITE and led to a lot of customer dissatisfaction as a result. A thread locked at INIFNITE that will never receive the signal(s) it is waiting for is a hung thread - not deadlocked, just hung

[Reply][Email][View Thread][Get Link]

[Modify|Delete]

Last Visit: 8:07 Monday 21st March, 2005 | First Prev Next

All Topics, MFC / C++ >> System >> Unedited Reader Contributions
Updated: 10 Mar 2005 6:41

Article content copyright xryl669, 2005
everything else Copyright © CodeProject, 1999-2005.
Advertise on The Code Project | Privacy

MSDN Communities | ASPAlliance • Developer Fusion • DevGuru • Programmers Heaven • Planet Source Code • Resource Index • Tek-Tips Forums • What is XML? • VisualBuilder • ZVON • Search Us!