(Memory and Resource) Leak detection for WinCE

Download source and vcproj tarball /wo crtdbg4wince -
14.7 KB

Introduction

(Note: this article deals with alpha 0.06 or higher of http://sourceforge.net/projects/crtdbg4wince/ project)

Resource Leaks are always a pain. But on Windows CE or Windows Mobile devices, with limited resources compared to a desktop system, the pain is much stronger. Not only memory Leaks, which obviously drain free memory from your "always to less" RAM, also other
leaks like Handles itself, open files and so on make faulty applications behave fat and slow.

Finally, the so called "free list fragmentation" makes Windows CE devices sometimes unusable, even if all resources are freed. But "free list fragmentation" is improved with WinCE 6 and anyway beyond the scope of my article

Stop! A question:

Why is a file handle (pointer, 4 bytes) more annoying than a memory leak?

Uhm, you're going like a ball at a gate. Okay, I hope my answer will sharpen your look at the topic: You're right - at the surface, a file handle is just a pointer to an opaque structure, deep inside the File System Driver (FSD). Or the handle is an index of
a table of opaque structures or such. Tell me which leak would be the most painful:


Collapse | Copy
Code

// which leak would be the most painful?
FILE* stream = fopen( "filename.txt", "r" );
HANDLE block = CreateFile( "filename.txt", .... );
HANDLE = CreateEvent( NULL,FALSE,FALSE,NULL );
char *VeryMuchBytes = new char[ 256 ];

Somebody will say: the 256 char block will create the biggest leak. And he will point to:


Collapse | Copy
Code

// winnt.h
typedef void *HANDLE;

sizeof(void*)

is 4 on most win32 platforms, so why focussing on 4 bytes?

This has 3 reasons:

We do not loose the pointer, we loose what it is pointing to
Under the hood, there is most times a structure, allocated by the FSD, owned by the FSD and not released until you closed all handles to this file (reference downcounting, if the file is opened multiple times).
My bet is:

stream

will make the biggest leak, because it points to a FSD structure and a streambuffer, with often
the streambuffer being more than 256 bytes itself.

The same is basically valid for a broad range of resources. Think about Brush Objects, Sockets and so on. Itmay be, that an Event Handle or a Mutex
Handle is really a small thing. But, who knows? And it is not important at all:
size doesn't matter!

Stop again! Why do you state "Size doesn't matter"? My girl happily tells me different ...

She's right, because she (hopefully) thinks about one thing at the same time. But with computer business, we have a "gang" of instances. Imagine, you create a Event Object to sleep on. But because a programming fault, you create this object 10.000 times:


Collapse | Copy
Code

int larger_loop = 10000;
while( larger_loop-- )
{
...
HANDLE WaitMe = CreateEvent( NULL,FALSE,FALSE,NULL );
...
}
WaitForSingleObject(WaitMe,1000);

Do you really believe, the Windows (CE) scheduler will perform with the same speed than it would with only 1 instance of your Event?

Consider: Size doesn't matter, if there are too many. The scheduler holds a list of waitable abjects. Sequencing and reordering this list is highly optimized inside the Scheduler. But nobody can optimize such violently lame programs behavior. Nobody, except
the developer of the faulty program.

Conclusion up to here

leaks with big blocks of memory are bad
large number ob leaked, list-managed resources are bad - even if having small memory footprint
Most modern resources (almost all) are hidden behind void* pointers for good reasons, but this could lead to ignorance about the real size
It's not important which kind of leak we generate. We shall work hard to avoid all of them

Methods and tools ...

Today, you can often access a broad range of leak detection tools. But no tool fits all your needs at once:

static analysis tools
runtime analysis tools
tools integrated with your tool-chain or with a upgraded version of your tool-chain
tools integrated with your operating system
tools integrated or optional with your C-Runtime (CRT)
OS independent tools
OS depending tools
... and so on ...

... and an incomplete Comparisation

static analysis tools, like my preferred "PC-Lint" (not to mix up with free, but less mighty "lint"):

Pro: Are able to find very, very much.

Con: Find too much, you need to decide if it is harmful or just bad style

Con: Can not find runtime-depending leaks like: Internet Server - if you logout, the Username String Buffer will be released, but if you re-login
within same session, it be allocated 2nd time, leaking the 1st.

dynamic (runtime) analysis tools, like CE Application Verifier from MicroSoft:

Pro: sometimes easy to use

Con: sometimes not working (can't resolve symbols and line numbers on CE6 often)

Con: can only find leaks you provoked, so it depends on your test depth an code coverage

dynamic (runtime) analysis tools, like the _CrtDbg from MicroSoft:

Pro: easy to use

Con: not for CE based embedded Windows flavors (until last week)

Con: can only find leaks you provoked, so it depends on your test depth and code coverage

OS independent tools:

Pro: easy to learn, if one already use it for another OS

Con: can only find leaks in C or C++ standard API, not in OS-dependent API

Background

The focus of this article shall be _CrtDbg now, but in a special flavour for Windows CE based platforms (PPC2003, WinCE 4.20, WinCE 5, WiMo 6/6.5, WinCE6).

As a quite experienced developer of Win32 desktop platforms, I learned to love CodeGuard while using Borland tool-chain. But later, I had to switch to VisualStudio. It was a pain, need to work with slightly different API and missing a helper like CodeGuard
has been! But then I learned to use _CrtDbg.h as my new friend. Not as powerful as CG, but still a great help!

Once upon a time, somebody extended my job tasks to write tools for WinCE. Again, painful: no Leak finders avail! Some of my tools and apps have been designed to cross compile on both desktop and mobile terminals. So I was okay to use desktop tools for the
Desktop build, then say 1000 prayers, the WinCE build may behave likewise. PC-Lint provided additional checking, so finally I was less or more able to sleep well. Most times.

At last, I discovered "AppVerify" and established it as part of the Quality Assurance process. Appverify is a bitch, often ranting about singletons, global opened logfiles and so on. Hard to use in a hectic time.

Finally, after switching eVC3 -> eVC4 -> VS2005 -> VS2008 and CE3/4/5/6, Appverify doesn't do the job more often.

I dreamed of _CrtDbg.h and its easy use.
I dreamed of CG/Appverify ability to detect more than only malloc() / new leaks

So I searched the net again and again. Found a lot. But nothing fits my needs perfectly. Until last week. Now, there is my hot candidate: "CrtDbg for WinCE".

This project is hosted on SourceForge since some weeks. It doesn't support eVC3/eVC4 anymore, but helped me fixing some bad leaks. The license of crtdbg4wince is so called "CFU" - "Cheap For Commercial Use, but free for non-profit and educational use". Sound's
not too bad in my ears.

Please read my article and if you like it, consider supporting the developer, so he'll continue this baby and fill in all my needs

"
src="http://www.codeproject.com/script/Forums/Images/smiley_wink.gif" />

This is the link: http://sourceforge.net/projects/crtdbg4wince/ to the
mentioned project. In my eyes, its notable that this project is already now growing into the direction to detect other leaks than only malloc or new.

Using the code

Since it claims to be a Port of _CrtDbg subset, you can use the API almost identical as the original from MicroSoft. Before starting, you should note two or three details:

The header file is currently named "mm_CrtDbg.h", instead of "CrtDbg.h". This is because Modem Man (the developer) told me, he is often using M$ CrtDbg.h simultaneously with its own module, to check the code of
"mm_CrtDbg.h". You can rename his files to "CrtDbg.h", if you like it.
There are some new Flags introduced by Modem Man, which may be bracketed with _WIN32_WCE:


Collapse | Copy
Code

...
int DbgMode = _CRTDBG_LEAK_CHECK_DF;

#ifdef _WIN32_WCE
DbgMode |= _CRTDBG_MM_BOUNDSCHECK;
#endif

_CrtSetDbgFlag( DbgMode );
...

to ensure your code compiles with both Desktop and Mobile Compiler.

since rev. 0.06 you now can keep mm_CrtDbg.h included if you build a Release. It just don't generate any code and don't create a boring #ifdef _DEBUG job for you.

There are also some CrtDbg.h well known functions which are not yet implemented or implemented as dummy. But this didn't harmed me. I never used this kind of calls before. So, why whining?

"
src="http://www.codeproject.com/script/Forums/Images/smiley_wink.gif" />

If YOU want to cross-compile code that used this functions, you can simply define them to be empty macros or set it into #ifdef brackets:


Collapse | Copy
Code

#ifndef _WIN32_WCE
_RPT0(_CRT_WARN,"file a message\n");
#endif

Enough introduction, hands on now!

okay, now let's advance to the "real" work:

Just write a simple WinCE C/C++ program of any flavor (Console or GUI) and include mm_CrtDbg.h:


Collapse | Copy
Code

// the very simplest usage example:
#include <stdio.h>
#include <tchar.h>
#include <windows.h>
#include <new>
#include <mm_CrtDbg.h>

int wmain(int argc, WCHAR* argv[])
{

then set the behavior to report at program end. This is the same way, you use to to with Win32 desktop target platform:


Collapse | Copy
Code

_CrtSetDbgFlag(  _CRTDBG_LEAK_CHECK_DF ); /* Leak check at program exit */

Tell the Leak-finder also to report all informations to the "Output" window of the IDE (debug channel or debug UART if you don't have ActiveSync):


Collapse | Copy
Code

_CrtSetReportMode( _CRT_ASSERT, _CRTDBG_MODE_DEBUG | _CRTDBG_MODE_WNDW );
_CrtSetReportMode( _CRT_WARN  , _CRTDBG_MODE_DEBUG );
_CrtSetReportMode( _CRT_ERROR , _CRTDBG_MODE_DEBUG );

Above, I added

_CRTDBG_MODE_WNDW

to let critical situations arise a message box.

Finally, we need bad code. You can use the below example to start, or start with your own buggy code. I propose to start with the sample, to get a 1st feeling:


Collapse | Copy
Code

// do something very stupid:
TCHAR * lost1 = (TCHAR*) malloc( 10 * sizeof(TCHAR)); //for testing, remove sizeof(TCHAR)
_tcscpy( lost1, _T("looser!") );
TCHAR * lost2 = _tcsdup( lost1 );
free( lost1 );

//
char *alsolost = new char[10];
alsolost = new char[20];
delete [] alsolost;

// the report will come up after executing the return below:
return 0;
}

we will leak

lost2

and

alsolost

,
8 byte and 10 byte.

Lets look on the output inside the IDE's "output" tab:

The output before program termination was:

Collapse | Copy
Code

c:\cpp\crtdbg4wince\sample1.cxx(18) : 'wmain': malloc(0x003986e8,10 byte) registered, ok.
c:\cpp\crtdbg4wince\sample1.cxx(20) : 'wmain': malloc(0x00398790,8 byte) registered, ok.
c:\cpp\crtdbg4wince\sample1.cxx(21) : 'wmain': free for malloc(0x003986e8,10) in c:\cpp\crtdbg4wince\sample1.cxx(18) : 'wmain': , ok
c:\cpp\crtdbg4wince\sample1.cxx(23) : 'wmain': new(0x003986e8,10 byte) registered, ok.
c:\cpp\crtdbg4wince\sample1.cxx(24) : 'wmain': new(0x00398838,20 byte) registered, ok.
c:\cpp\crtdbg4wince\sample1.cxx(24) : 'wmain': delete for new(0x00398838,20) in c:\cpp\crtdbg4wince\sample1.exe(0) : 'unknown_func': , ok

Here we see all activities. We could have it more suppressed it by:


Collapse | Copy
Code

_CrtSetReportMode( _CRT_WARN, 0 );

but this depends on your taste. I personally dislike this "statistical" output to be mapped to _CRT_WARN. Modem Man told me, he's also thinking about introducing a 4th channel _CRT_STAT. I'm looking forward to his solution.

If you have very complex ressource situations, you can make some Perl-parsing with it - as you like it.

But let's advance to the real interesting now.

The output just after program termination:

The header of the termination log always summarizes some global statistics. This is an advance over the original MicroSoft way:


Collapse | Copy
Code

c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': ============================================
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': Leakage Summary at program termination point
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': ============================================
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': peak ever used malloc(): 18 byte, just for information.
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': peak ever used new(): 30 byte, just for information.
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': CreateFile/CloseHandle are okay (or never used).
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': fopen/fclose are okay (or never used).

then we find

error

s listed:


Collapse | Copy
Code

c:\cpp\crtdbg4wince\sample1.exe(0) : error R0001: 'at exit': still 8 byte in use by malloc()!
c:\cpp\crtdbg4wince\sample1.exe(0) : error R0001: 'at exit': still 10 byte in use by new()!

Yes! as predicted by eagle-eye Sarge, we leaked 8 bytes from malloc and 10 bytes from new.

Next lines will tell us, where exactly:


Collapse | Copy
Code

c:\cpp\crtdbg4wince\sample1.cxx(23) : error R0001: 'wmain': delete(0x003986e8,10) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample1.exe(0) : assertion A0001: 'at exit': delete(0x003986e8,10) missing near here. See previous line for new() location.
c:\cpp\crtdbg4wince\sample1.cxx(20) : error R0001: 'wmain': free(0x00398790,8) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample1.exe(0) : assertion A0001: 'at exit': free(0x00398790,8) missing near here. See previous line for malloc() location.

All Lines with

filename(linenumber)
:

can be clicked with the mouse, resulting in the focus jumping immediately to the given source line. In almost all cases, it is able to jump to the resource allocation point, sometimes it is also able to jump to the faulty release line!
Better than MicroSoft! Cool!!!

As you remember, we configured the

assertion

lines to invoke a message box handler. I don't want to bore you with a
screen-shot here. Just imagine it.

Is it all it can do?

No. Here you'll get a more complex sample:

Collapse | Copy
Code

// more complex example:
#include <winsock.h>
#include <mm_CrtDbg.h>

Above we included winsock because I want also to show WSAStartup-leaks. Next, we declare a function and define a dummy class:


Collapse | Copy
Code

void Setup_CrtDbg_Mode( HANDLE CrtFile );

class dummyC
{
public:
dummyC() : x(-1) {OutputDebugString( L"ctor okay\r\n" );}
~dummyC()         {OutputDebugString( L"dtor okay\r\n" );}
private:
int x;
};

It's not often comfortable to only have the output in the debugger tab . Or you have only low bandwidth debug channel or such. So, create a file to collect all the messages immediately on the device. It is not different from MS CrtDbg:


Collapse | Copy
Code

int wmain(int argc, WCHAR* argv[])
{
// open your logfile [optional]
HANDLE CrtFile = CreateFile( "LogFile.txt", GENERIC_READ | GENERIC_WRITE,
0, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL );

Connecting the file with Leak-finder. Details are shown later, inside the function Setup_CrtDbg_Mode():


Collapse | Copy
Code

Setup_CrtDbg_Mode( CrtFile );

again do stupid things, but more complex this time:


Collapse | Copy
Code

// do something very normal (stupidity comes later):
for( int i=3 ; i>0 ; i-- )
{
dummyC C = new dummyC;
WSADATA WSA;
int wsa = WSAStartup( MAKEWORD( 2, 2 ), &WSA );

HANDLE File1 = CreateFile( L"123.txt", GENERIC_READ | GENERIC_WRITE,
0, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL );
HANDLE File2;
DuplicateHandle( GetCurrentProcess(), File1, GetCurrentProcess(),
&File2, 0, FALSE, DUPLICATE_CLOSE_SOURCE | DUPLICATE_SAME_ACCESS );

// do something stupid: forget to free ressources sometimes
if(i!=3) {CloseHandle( File2 );};
if(i!=2) {WSACleanup( &WSA );};
if(i!=1) {delete C;};
_CrtDumpMemoryLeaks();
} // end for 3 loop

do nothing special at program end. The report will come up independently


Collapse | Copy
Code

// a final report will also come up after executing the 'return':
return 0;
}

The Setup_CrtDbg_Mode() helper consist nearly only of calls, you'd also use for Desktop platforms:


Collapse | Copy
Code

// helper function for complex setup of _CrtDbg global settings
void Setup_CrtDbg_Mode( HANDLE CrtFile )
{
int DbgMode;
DbgMode = _CrtSetDbgFlag(  _CRTDBG_LEAK_CHECK_DF   /* Leak check at program exit */
| _CRTDBG_CHECK_ALWAYS_DF /* Check heap every alloc/dealloc */
| _CRTDBG_CHECK_CRT_DF    /* Do Leak check/diff CRT blocks */
);

Above directs the CrtDbg to:

report at exit, as we did before
check on every resource allocation/release
also check CRT internal blocks
return default-preset and the 3 given as

DbgMode

Then switch off the not yet supported "also check CRT internal blocks" and add new buffer overflow/underflow flags. Finally, disable a very chatty alloc/free and set
all this bits together:


Collapse | Copy
Code

DbgMode &= ~_CRTDBG_CHECK_CRT_DF;
DbgMode |= _CRTDBG_MM_BOUNDSCHECK; /* new flag  */
DbgMode &= ~_CRTDBG_MM_CHATTY_ALLOCFREE; /* new flag by maik */
_CrtSetDbgFlag(DbgMode);

As used before, we need to set the 3 "channels" , but this time also to our opened file:


Collapse | Copy
Code

_CrtSetReportMode( _CRT_ASSERT, _CRTDBG_MODE_DEBUG | _CRTDBG_MODE_FILE | _CRTDBG_MODE_WNDW );
_CrtSetReportMode( _CRT_WARN  , _CRTDBG_MODE_DEBUG | _CRTDBG_MODE_FILE );
_CrtSetReportMode( _CRT_ERROR , _CRTDBG_MODE_DEBUG | _CRTDBG_MODE_FILE );

(The above is again 100% identical to Desktop platforms).

Last Sub-Step is to connect the file handle with all wanted "channels". I want to get all written, so I assign the file to all 3:


Collapse | Copy
Code

if( INVALID_HANDLE_VALUE != CrtFile )
{
_CrtSetReportFile( _CRT_ASSERT, CrtFile );
_CrtSetReportFile( _CRT_WARN  , CrtFile );
_CrtSetReportFile( _CRT_ERROR , CrtFile );
}
}}

Let's start the programm and see, what it reports

The output before program termination was again quite helpful:

The most interesting topics are the warning here. I condensed the much messages, it has been very much more in real life:


Collapse | Copy
Code

c:\cpp\crtdbg4wince\sample2.cxx(36) : 'wmain': new(0x003d87a8,16 byte) registered, ok.
ctor okay
c:\cpp\crtdbg4wince\sample2.cxx(38) : 'wmain': WSAStartup(0x00000000,1 refcount) registered, ok.
c:\cpp\crtdbg4wince\sample2.cxx(41) : 'wmain': CreateFile("123.txt", 0x00000fb0,1 F_handle) registered, ok.
c:\cpp\crtdbg4wince\sample2.cxx(44) : 'wmain': CreateFile(0x00000fac,1 F_handle) registered, ok.
c:\cpp\crtdbg4wince\sample2.cxx(44) : 'wmain': CloseHandle for <win32api>("123.txt",0x00000fb0,1) in c:\cpp\crtdbg4wince\sample2.cxx(41) : 'wmain': , ok
c:\cpp\crtdbg4wince\sample2.cxx(38) : 'wmain': WSACleanup for WSAStartup(0x00000000,1) in c:\cpp\crtdbg4wince\sample2.exe(0) : 'unknown_func': , ok
dtor okay
c:\cpp\crtdbg4wince\sample2.cxx(36) : 'wmain': delete for new(0x003d87a8,16) in c:\cpp\crtdbg4wince\sample2.exe(0) : 'unknown_func': , ok
c:\cpp\crtdbg4wince\sample2.cxx(50) : 'wmain': ===[ CrtDumpMemoryLeaks start]===================
...
c:\cpp\crtdbg4wince\sample2.cxx(50) : warning W0001: 'wmain': still 2 F_handle in use by CreateFile()!
c:\cpp\crtdbg4wince\sample2.cxx(44) : warning W0001: 'wmain': CloseHandle(0x00000fac,1) missing for this allocation, or two times allocated to same pointer, or just not closed yet.
c:\cpp\crtdbg4wince\sample2.cxx(50) : warning W0001: 'wmain': CloseHandle(0x00000fac,1) missing near here. See previous line for CreateFile() location.
c:\cpp\crtdbg4wince\sample2.cxx(50) : 'wmain': ===[ CrtDumpMemoryLeaks stopp]===================
...

Again, everything is mouse clickable, so you can immediately jump to the suspicious line. Some explanations:

WSAStartup(0x00000000,1 refcount)

The

WSAStartup

does not create a memory block (0x00000000)

The

WSAStartup

increments by 1 reference count.

CreateFile("123.txt", 0x00000fb0,1 F_handle)

The

CreateFile

opened a file with name "123.txt", which could be a non const runtime value.

The

CreateFile

got handle 0x00000fb0 and did an increment by 1 reference count

special: CreateFile(0x00000fac,1 F_handle) and CloseHandle("123.txt",0x00000fb0,1)

How can

CreateFile

not know the file name?

This is since we see DuplicateHandle() invoked with a file handle here.

And because we said

DUPLICATE_CLOSE_SOURCE

, it then calls CloseHandle() on the former handle of "123.txt".

The output at program termination was:

The most interesting topics are the error and assertion here.
The

assertion

s also
invoked a MessgeBox because


Collapse | Copy
Code

_CrtSetReportMode( _CRT_ASSERT, ..... | _CRTDBG_MODE_WNDW );

but have a look into the clickable (condensed) output. Statistics and well tidy APIs:


Collapse | Copy
Code

..
c:\cpp\crtdbg4wince\sample2.exe(0) : 'at exit': Leakage Summary at program termination point
...
c:\cpp\crtdbg4wince\sample2.exe(0) : 'at exit': peak ever used new(): 16 byte, just for information.
c:\cpp\crtdbg4wince\sample2.exe(0) : 'at exit': peak ever used CreateFile(): 3 F_handle, just for information.
c:\cpp\crtdbg4wince\sample2.exe(0) : 'at exit': malloc/free are okay (or never used).

Then the problem childs:


Collapse | Copy
Code

c:\cpp\crtdbg4wince\sample2.exe(0) : error R0001: 'at exit': still 16 byte in use by new()!
c:\cpp\crtdbg4wince\sample2.exe(0) : error R0001: 'at exit': still 2 F_handle in use by CreateFile()!
c:\cpp\crtdbg4wince\sample2.cxx(36) : error R0001: 'wmain': delete(0x003d88b8,16) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample2.exe(0) : assertion A0001: 'at exit': delete(0x003d88b8,16) missing near here. See previous line for new() location.
c:\cpp\crtdbg4wince\sample2.cxx(38) : error R0001: 'wmain': WSACleanup(0x00000000,1) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample2.exe(0) : assertion A0001: 'at exit': WSACleanup(0x00000000,1) missing near here. See previous line for WSAStartup() location.
c:\cpp\crtdbg4wince\sample2.cxx(44) : error R0001: 'wmain': CloseHandle(0x00000fac,1) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample2.exe(0) : assertion A0001: 'at exit': CloseHandle(0x00000fac,1) missing near here. See previous line for CreateFile() location.

Points of Interest

I feel this project interesting, because it finds all kinds of alloc,new,CreateFile. And it claims to soon find much more.

History

2012-05-26: I fixed some typos (sorry, a dutch-like language is my mothers tongue). In between Modem Man held his promise to fix the DEBUG/RELEASE issue. Modem Man also fixed some problems with altcecrt.h and released 0.06, which is now also compiling with
an unmodified PPC2003 SDK. All his changes are reworked within this article. I added the whole sample code of mine, with VisualStudio project files.

2012-05-25: Tim Corey and Dave Kreskowiak directed me to improve this article. Done. Well?

2012-05-24: My 1st introduction of the project, got some hints from Author of crtdbg4wince and backwards I helped him to fix a bug in his release 0.05.

License

This article, along with any associated source code and files, is licensed under The
Code Project Open License (CPOL)

About the Author

Sergeant
Kolja

Tester / Quality Assurance

Aruba


Did a lot of work in Meduna and Cambria. Mostly bug hunting in the whole little country. Repaired some windows there.