GDB anti-debug, Otool/otx anti-disassembly… It’s Challenge number 3 !!!

Today I decided to give a look at Challenge #3 since it promised nasty tricks. Now that looks like a challenge and I love challenges! If you think this is a spoiler then stop reading and come back in a week or so. There is no solution for the challenge; I’m more interested in the “nasty” trick used and why the tools are failing. And I don’t need the Challenge itself to analyse this behavior since I can reproduce it with own code. I have other things to do, so I need to get this out of my mind and maybe someone else can contribute to this analysis.

Let the games begin!

As usual, I started to analyse this challenge by trying to disassemble it with otx. The result was empty, as if __text section can’t be read. Hummm that’s weird. GDB is the next victim… Launch gdb, select binary, run… BANG! Gdb not able to run the program. This is getting interesting! So let’s read the mach-o headers to understand what’s happening here.

$ otool -l Cactus
Load command 0
      cmd LC_SEGMENT
  cmdsize 56
  segname __TEXT
   vmaddr 0x0000b000
   vmsize 0x0000468f
  fileoff 0
 filesize 15789
  maxprot 0x00000007
 initprot 0x00000007
   nsects 0
    flags 0x0
Load command 1
        cmd LC_UNIXTHREAD
    cmdsize 80
     flavor i386_THREAD_STATE
      count i386_THREAD_STATE_COUNT
	    eax 0x00000000 ebx    0x00000000 ecx 0x00000000 edx 0x00000000
	    edi 0x00000000 esi    0x00000000 ebp 0x00000000 esp 0x00000000
	    ss  0x00000000 eflags 0x00000000 eip 0x0000e79c cs  0x00000000
	    ds  0x00000000 es     0x00000000 fs  0x00000000 gs  0x00000000

The binary has only 2 commands and there is no __text section. This is not an usual binary. From other projects I know this is possible and there are at least two articles about this. After searching my bookmarks I found Amit Singh post about this here (have I told you that his book rocks? buy it!). The posted code for tiny.asm doesn’t work in Snow Leopard (I remember it works in Leopard), so here is the working version (we just need to specify nsects and flags – the kernel parser must have been updated in SL).

; tiny.asm for Mac OS X (Mach-O Object File Format)
; nasm -f bin -o tiny tiny.asm
        org   0x1000
        db    0xce, 0xfa, 0xed, 0xfe       ; magic
        dd    7                            ; cputype (CPU_TYPE_X86)
        dd    3                            ; cpusubtype (CPU_SUBTYPE_I386_ALL)
        dd    2                            ; filetype (MH_EXECUTE)
        dd    2                            ; ncmds
        dd    _start - _cmds               ; cmdsize
        dd    0                            ; flags
        dd    1                            ; cmd (LC_SEGMENT)
        dd    56                           ; cmdsize
        db    "__TEXT"                     ; segname
        db    0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ; segname
        dd    0x1000                       ; vmaddr
        dd    0x1000                       ; vmsize
        dd    0                            ; fileoff
        dd    filesize                     ; filesize
        dd    7                            ; maxprot
        dd    5                            ; initprot
        dd    0                            ; nsects
        dd    0                            ; flags
        dd    5                            ; cmd (LC_UNIXTHREAD)
        dd    80                           ; cmdsize
        dd    1                            ; flvaor (i386_THREAD_STATE)
        dd    16                           ; count (i386_THREAD_STATE_COUNT)
        dd    0, 0, 0, 0, 0, 0, 0, 0       ; state
        dd    0, 0, _start, 0, 0, 0, 0, 0  ; state
        xor   eax,eax
        inc   eax
        push  byte 42
        sub   esp, 4
        int   0x80                         ; _exit(42)
filesize equ  $ - $$

Otool isn’t able to disassemble this binary (otx uses otool) but this time gdb is able to run. The difference is that breakpoints aren’t enforced! IDA is able to disassemble the binary directly (else we could point it to the right place taking the EIP as our starting point).

Since I had the idea why otool was failing, I quickly checked its source code to be sure that was the problem (otool is part of cctools package available at Apple Developer Program, if you are interested in checking it). The following can be found at otool/main.c:

                else if(cputype == CPU_TYPE_I386)
                    j = i386_disassemble(sect, size - i, cur_addr, addr,
                                object_byte_sex, relocs, nrelocs, symbols, NULL,
                                nsymbols, sorted_symbols, nsorted_symbols,
                                strings, strings_size, indirect_symbols,
                                nindirect_symbols, cputype, load_commands,
                                ncmds, sizeofcmds, verbose);

The function i386_disassemble is located in otool/i386_disasm.c. Only sections are disassemble and since there is none in the binary then otool can’t disassemble it. Proof is easy since that’s the difference between tiny.asm and nicertiny.asm. I further tested this by manually adding that section to the binary; otool was able to disassemble it without problems (although a 100% correct disassembly is another story). I presume that gdb has the same problem since I think there was/is a common code base for the disassembler and other parts (or similar, cannot recall by memory since last time I messed with gdb source was 1 year ago).

I tried some experiments with gdb and before writing this I had the idea that it could breakpoint after I added the __text section. I can’t reproduce it again so I must have confused with other test binaries I had. Why don’t know why gdb isn’t able to breakpoint these binaries (with or without __text section). I suspect it might have to do with alignment or something. For example, I have tried to compile GNU gdb latest 7.1 version and it can’t even parse these binaries. It crashes like this:

gdb$ file ~/Projects/macserialjunkiechallenge10/Challenge3/tiny
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000010
0x00000001001e10c8 in bfd_mach_o_read_symtab_symbols (abfd=0x1007c2f40) at mach-o.c:1835
1835	  if (sym->symbols)

I have tried to pack some test binaries with UPX and same thing happens (you don’t need to be Sherlock Holmes to see that Challenge #3 is UPX packed). Gdb isn’t able to execute the breakpoints, either software or hardware. A quick Google search hasn’t returned any valid information about this.

While this isn’t a big problem for reversing Challenge #3, this could be an issue with more advanced and complex packers. We can attach gdb to Challenge #3 without any problem after the unpacking and dump whatever we need (hint: there’s an even easier solution for this case ;-)) – don’t forget that vmmap is also messed up in Snow Leopard. But we could miss important stuff in other packers and this could be a good anti-debug trick (maybe IDA debugger is able to do it, I don’t use it). Otool gets confused while disassembling some instructions from Challenge #3 so there is also something else playing in that binary (that could explain why the binary doesn’t even run under gdb).

Now, who can give some hints about this? I’m very much curious about it but I don’t want to spend more time due to other matters that call my attention – gdb source is a pain to read and navigate thru.

Have fun,

Update 1:

psaghelyi left a link in another post to his tool, MachOView. For now it’s a graphical otool -l, that allows you to explore the Mach-O header of binaries. It has potential to be a nice tool as soon as he adds editor capabilities. Help him and contribute to the code 🙂 We need such a tool! Mac is sexy so we need sexy graphical tools!

20 thoughts on “GDB anti-debug, Otool/otx anti-disassembly… It’s Challenge number 3 !!!

  1. the other thing about that challenge is that if you try to unpack it with upx (normally possible) it raises an exception saying it is not packed, but if you try and pack it again it raises an exception saying it is packed

      1. You think gdb code is tough to navigate through, I’ve been wading through UPX source code trying to find out why it won’t -d the target. I think I’m missing something obvious… I’m also sure the answer to the inherent AD trick lies in the UPX source too.


        1. Check your mail 😉
          I checked UPX sources (the loader part is located in src/stub/src) but it doesn’t seem to have any anything special there. If you pack a test binary, the same problem will happen – gdb cannot breakpoint – so I don’t think it’s anti-debugging from UPX but gdb failing.

          1. i have gdb 6.3 and i unpacked it manually. breakpoints work like a charm. don’t know what you gdb does ^^

            the only thing i have is that the unpacked binary runs fine but somehow the appdelegate isn’t connected ?! i’m objective c noob and not familiar with that message handling thing :s

            1. My gdb has some patches but I don’t see them influencing on this problem. I even tested with pristine versions on my Leopard virtual machine.
              Can you mail me the breakpoints you have used so I can try it out?

  2. It’s great to see this causing some head aches 😉
    If you haven’t figured it out by the end of the week I’ll gladly share some of the “nasty tricks” used in this challenge. 🙂

  3. May be the problem related to the __msj segment? Because when I try to place a breakpoint the program crashes with the error you reported caused by the __purr function that is located in this section indeed.
    This segment isn’t evn marked as a CODE one

    1. Maybe ! I have to check it out 🙂
      The interesting thing is that I was talking to someone who is able to breakpoint the crackme and the two asm binaries without any problem.
      At least you can replicate the problem which takes any problem with my OS out of the way.
      Which Xcode version are you using?

  4. Right now i’m testing with the latest version of xcode (3.2.3) . I tryed to debug it with the gdb version shipped with it (6.3.50) and 7.1 but on both of them I can’t run the program and if I try to attach it to the process I can’t set any breakpoint without the program crashing………

    I’m pretty sure this programm uses sysctl to determine if a debugger using ptrace is attached…… and if so it crashes, maybe you should have a look at the main function in the _msj segment beacause I hardly understand what it does

    That’s my first approach on reversing, maybe I should have chosen something easier xD

    1. does it also crash when you breakpoint the entry?

      coz when you break somewhere in the program then the debugger detection crashes the program. for me i could break right on the entry point and modify the jump.

      1. No, when I break on the entry it works in fact I’m trying to understand why it crashes later!
        Strangely neither otool or gdb load the _msj section so I coould’t disassemble it yesterday, but now I’m trying with objdump and i was able dump it. Is there any native tool on osx able to show and/or modify segments and sections ? If not it’ll be nice to create one.
        And between you said that you unpacked it manually! It’ll be nice if you could explain how you were able to do it couse i thought that you had to have some tooll to rebuild the symbols table or something like that! I’m really a beginner in this world so any information is essential for me xD

        1. it crashes because of antidebug trick. thats the syscall wich is right after entrypoint. this trick is also in purr function which is called evertime you do an input in the text field an so on …

          1. yeah, yesterday I was able to modify the program to avoid the antidebug trick :D, in fact I already pointed out that about the use of the syscall in a post before. The only two things that I can’t understand are why I wasn’t able to disassemble the code in the _msj segment with gdb (not debugging only disassembling) and how the author created it.
            Right now I’m not studying the serial generation code, I’ve stopped on half of the _checkPrefixAndFormat function, but I’m trying to unpack manually a sample program couse I think it’s more interesting doing it without the unpacker 😀

            1. Segments with different names can be created without any problems. I see you don’t have the same problem as I have with gdb… I will have to spend some time tracking my problem hehehe

            2. you can easily create this with an with passing attributes to the c or c++ or what ever functions.

              otx only searches for the text segment and display that to you … ida does it’s job well on that 😉

  5. yeah but ida costs such lots of money xD and the version that I have doesn’t want to remote debug my programs….. when I try to launch them it says “unexpected dylib version, will quit” or something like that … Do you guys know why or how to solve this?

    And I’m sorry fG if I made you lose some time thinking that I had the same problem…. really sorry man! Maybe try out with the gdb version from macports and see what happens if it gives you the same problems!

Leave a Reply

Your email address will not be published. Required fields are marked *