THIS WRITEUP IS VERY VERY LONG AND MOST OF THE PART IS NOT ACTUALLY RELATED TO THE VM. YOU CAN SKIP TO THE “THE MOMENT OF TRUTH” TO ACTUALLY READ THE SOLUTION.

After completing the previous CrackMe based on a very good Anti-Debug technique (which can be further improved as I pointed out in that post by reading a research paper), Elvis, the author of previous challenge gave me another challenge. He already gave me a major hint about the challenge which you usually don’t get in real world scenarios. This challenge is another VM challenge and is an OISC (One Instruction Set Computer)

Running the executable shows a password prompt :

I type anything into it and it keeps watining until unless I exceed certain number of characters, after which it just exits. I don’t know this “certain number” yet but we can find this out by analyzing this in IDA.

Running this in IDA debugger successfully shows this prompt again but pausing the process is something else. We either run this process or stop it, no pausing and resuming execution. This might mean there is anti-debug or something more interesting to discover 🤩

Let’s analyze the start function of this program

aha… ok.. ok.. analysis of start complete 😆

This is spooky. I’ll rename these to mainfn1 and mainfn2 to be able to identify them when I encounter these again, but then again I can also check their XREFs and see when they are called or referenced, but this won’t be able to detect indirect references.

uintptr_t mainfn1() {
    uintptr_t result; // eax
    unsigned int v1; // ecx
    LARGE_INTEGER PerformanceCount; // [esp+0h] [ebp-14h] BYREF
    struct _FILETIME SystemTimeAsFileTime; // [esp+8h] [ebp-Ch] BYREF
    DWORD v4; // [esp+10h] [ebp-4h] BYREF

    if (__security_cookie == -1153374642 || (__security_cookie & 0xFFFF0000) == 0) {
        SystemTimeAsFileTime.dwLowDateTime = 0;
        SystemTimeAsFileTime.dwHighDateTime = 0;
        GetSystemTimeAsFileTime( & SystemTimeAsFileTime);
        v4 = SystemTimeAsFileTime.dwLowDateTime ^ SystemTimeAsFileTime.dwHighDateTime;
        v4 ^= GetCurrentThreadId();
        v4 ^= GetCurrentProcessId();
        v4 ^= GetTickCount64();
        QueryPerformanceCounter( & PerformanceCount);
        result = (uintptr_t) & v4;
        v1 = (unsigned int) & v4 ^ v4 ^ PerformanceCount.LowPart ^ PerformanceCount.HighPart;
        if (v1 == 0xBB40E64E) {
            __security_cookie = 0xBB40E64F;
            dword_B44008 = 0x44BF19B0;
        } else {
            if ((v1 & 0xFFFF0000) == 0) {
                result = (v1 | 0x4711) << 16;
                v1 |= result;
            }
            __security_cookie = v1;
            dword_B44008 = ~v1;
        }
    } else {
        result = ~__security_cookie;
        dword_B44008 = ~__security_cookie;
    }
    return result;
}