How to compile AFL's LLVM mode in OS X

American fuzzy lop aka AFL is one of the easiest and best fuzzers out there and should be part of your development cycle if you care at least one bit about the security of your code.

Its performance in OS X is a bit of a let down because of issues at fork() system call. AFL warns you about this when compiling it:

WARNING: Fuzzing on MacOS X is slow because of the unusually high overhead of fork() on this OS. Consider using Linux or *BSD. You can also use VirtualBox (virtualbox.org) to put AFL inside a Linux or *BSD VM.

Process startup in OS X is a bit expensive because of all the code signature and sandbox checks (this is sort of an educated guess) and since fuzzing is a brute force approach we want to squeeze as many executions per second as possible per CPU core. Back in 2015, AFL introduced a new feature called persistent mode which does in-process fuzzing meaning not so many fork() calls. Instead of launching a new process for each new input this mode sort of rewinds the process back to the beginning and injects the new case there.

While it’s possible to recompile some targets in Linux or other Unix system, there are many targets that use specific OS X features and so they can only be fuzzed within OS X. For those cases the persistent mode is definitely interesting to have working.

Today’s you can find this feature in AFL’s llvm_mode folder. This feature requires clang to be installed, and generates new AFL wrappers called afl-clang-fast and afl-clang-fast++. To get this working in OS X isn’t straightforward because Xcode doesn’t have all the necessary features.

Ben Nagy has a Github repo called osx-afl-llvm with some Makefiles to download the necessary LLVM files and compile everything needed to generate the new wrappers. The problem is that it was last updated two years ago and it is outdated. I used it as my reference guide to get this working with latest AFL (2.45b at the time of writing) and it quickly got me into a heap of trouble. I tried to upgrade Ben’s scripts to latest LLVM (4.0.1 at the time of writing) but things changed at the LLVM side and now CMake is used instead of Makefiles to get LLVM Projects built. Since CMake is not my thing I tried to use a different way to get this compiled.

AFL documentation refers that llvm-config is needed to get this feature compiled. This binary isn’t available with Xcode so we need to download a binary (or source and compile it) package from llvm.org.
Let’s give it a try downloading version 4.0.1 binary package at http://releases.llvm.org/4.0.1/clang+llvm-4.0.1-x86_64-apple-darwin.tar.xz.

First if we try to compile LLVM mode (after compiling main AFL) with Xcode (using Xcode 8.3.2 on Sierra 10.12.5) we get the following error:

$ make
[*] Checking for working 'llvm-config'...
[-] Oops, can't find 'llvm-config'. Install clang or set $LLVM_CONFIG or $PATH beforehand.
    (Sometimes, the binary will be named llvm-config-3.5 or something like that.)
make: *** [test_deps] Error 1

So let’s try to point $LLVM_CONFIG to the 4.0.1 binary version we downloaded:

$ export LLVM_CONFIG=../../clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/bin/llvm-config
$ make
[*] Checking for working 'llvm-config'...
[*] Checking for working 'clang'...
[*] Checking for '../afl-showmap'...
[+] All set and ready to build.
clang -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DBIN_PATH=\"/usr/local/bin\" -DVERSION=\"2.45b\"  afl-clang-fast.c -o ../afl-clang-fast
ln -sf afl-clang-fast ../afl-clang-fast++
clang++ `../../clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/bin/llvm-config --cxxflags` -fno-rtti -fpic -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DVERSION=\"2.45b\" -Wno-variadic-macros -shared afl-llvm-pass.so.cc -o ../afl-llvm-pass.so `../../clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/bin/llvm-config --ldflags`  -Wl,-flat_namespace -Wl,-undefined,suppress
clang -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DBIN_PATH=\"/usr/local/bin\" -DVERSION=\"2.45b\"  -fPIC -c afl-llvm-rt.o.c -o ../afl-llvm-rt.o
[*] Building 32-bit variant of the runtime (-m32)... success!
[*] Building 64-bit variant of the runtime (-m64)... success!
[*] Testing the CC wrapper and instrumentation output...
unset AFL_USE_ASAN AFL_USE_MSAN AFL_INST_RATIO; AFL_QUIET=1 AFL_PATH=. AFL_CC=clang ../afl-clang-fast -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DBIN_PATH=\"/usr/local/bin\" -DVERSION=\"2.45b\"  ../test-instr.c -o test-instr
error: unable to load plugin '../afl-llvm-pass.so': 'dlopen(../afl-llvm-pass.so, 9): Symbol not found:
      __ZN4llvm24DisableABIBreakingChecksE
  Referenced from: ../afl-llvm-pass.so
  Expected in: flat namespace
 in ../afl-llvm-pass.so'
make: *** [test_build] Error 1

We get unknown symbols issues. The reason for this is that we are pointing llvm-config to the downloaded clang version while using Xcode’s clang version to compile this. Both versions were compiled with different flags so some symbols are missing from Xcode’s version.

Instead of setting the $LLVM_CONFIG let’s try to set $PATH as recommended:

$ unset LLVM_CONFIG
$ export PATH=~/Projects/afl/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/bin/:$PATH
$ make
[*] Checking for working 'llvm-config'...
[*] Checking for working 'clang'...
[*] Checking for '../afl-showmap'...
[+] All set and ready to build.
[*] Testing the CC wrapper and instrumentation output...
unset AFL_USE_ASAN AFL_USE_MSAN AFL_INST_RATIO; AFL_QUIET=1 AFL_PATH=. AFL_CC=clang ../afl-clang-fast -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DBIN_PATH=\"/usr/local/bin\" -DVERSION=\"2.45b\"  ../test-instr.c -o test-instr
../test-instr.c:17:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
make: *** [test_build] Error 1

The binary clang version we downloaded misses the system headers so we need to point to them. This can be accomplished using the -isysroot flag. Using this flag and the downloaded clang we need to recompile the whole AFL.
You will need to set up the correct path to Xcode SDK you want to use. In my case I have different Xcode versions installed so I distinguish them by adding the version number to the application name.

Filipe Cabecinhas (@filcab) suggested to use SDKROOT variable instead of -isysroot flag. This is a cleaner option instead of setting the compiler flags as I do below. In this case use

export SDKROOT=`xcrun --show-sdk-path`

instead of the -isysroot parameter to CFLAGS and CXXFLAGS. This will use the SDK for your currently active Xcode version (the -isysroot option allows you to directly set the SDK of another non-active version).

$ cd ..
$ make clean

$ CFLAGS="-isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk" \
CXXFLAGS="-isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk" make
[*] Checking for the ability to compile x86 code...
[+] Everything seems to be working, ready to compile.
cc -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-gcc.c -o afl-gcc
set -e; for i in afl-g++ afl-clang afl-clang++; do ln -sf afl-gcc $i; done
cc -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-fuzz.c -o afl-fuzz
cc -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-showmap.c -o afl-showmap
cc -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-tmin.c -o afl-tmin
cc -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-gotcpu.c -o afl-gotcpu
cc -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-analyze.c -o afl-analyze
cc -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" afl-as.c -o afl-as
ln -sf afl-as as
[*] Testing the CC wrapper and instrumentation output...
unset AFL_USE_ASAN AFL_USE_MSAN; AFL_QUIET=1 AFL_INST_RATIO=100 AFL_PATH=. ./afl-clang -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" test-instr.c -o test-instr
echo 0 | ./afl-showmap -m none -q -o .test-instr0 ./test-instr
echo 1 | ./afl-showmap -m none -q -o .test-instr1 ./test-instr
[+] All right, the instrumentation seems to be working!
[+] LLVM users: see llvm_mode/README.llvm for a faster alternative to afl-gcc.
[+] All done! Be sure to review README - it's pretty short and useful.

WARNING: Fuzzing on MacOS X is slow because of the unusually high overhead of
fork() on this OS. Consider using Linux or *BSD. You can also use VirtualBox
(virtualbox.org) to put AFL inside a Linux or *BSD VM.

NOTE: If you can read this, your terminal probably uses white background.
This will make the UI hard to read. See docs/status_screen.txt for advice.

Now that the main AFL code is compiled we can move to the LLVM mode version.

$ cd llvm_mode

$ CFLAGS="-isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk" CXXFLAGS="-isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk" make
[*] Checking for working 'llvm-config'...
[*] Checking for working 'clang'...
[*] Checking for '../afl-showmap'...
[+] All set and ready to build.
clang -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DBIN_PATH=\"/usr/local/bin\" -DVERSION=\"2.45b\"  afl-clang-fast.c -o ../afl-clang-fast
ln -sf afl-clang-fast ../afl-clang-fast++
clang++ `llvm-config --cxxflags` -fno-rtti -fpic -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DVERSION=\"2.45b\" -Wno-variadic-macros -shared afl-llvm-pass.so.cc -o ../afl-llvm-pass.so `llvm-config --ldflags`  -Wl,-flat_namespace -Wl,-undefined,suppress
clang -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DBIN_PATH=\"/usr/local/bin\" -DVERSION=\"2.45b\"  -fPIC -c afl-llvm-rt.o.c -o ../afl-llvm-rt.o
[*] Building 32-bit variant of the runtime (-m32)... success!
[*] Building 64-bit variant of the runtime (-m64)... success!
[*] Testing the CC wrapper and instrumentation output...
unset AFL_USE_ASAN AFL_USE_MSAN AFL_INST_RATIO; AFL_QUIET=1 AFL_PATH=. AFL_CC=clang ../afl-clang-fast -isysroot /Applications/Xcode8.3.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DBIN_PATH=\"/usr/local/bin\" -DVERSION=\"2.45b\"  ../test-instr.c -o test-instr
echo 0 | ../afl-showmap -m none -q -o .test-instr0 ./test-instr
echo 1 | ../afl-showmap -m none -q -o .test-instr1 ./test-instr
[+] All right, the instrumentation seems to be working!
[+] All done! You can now use '../afl-clang-fast' to compile programs.

And voilá, we have AFL LLVM mode binaries built. There is a problem with the Makefile when we try to install AFL as root, because the compilation will fail. I am not sure why this is happening, need to investigate.

$ sudo make install
[*] Checking for the ability to compile x86 code...
[+] Everything seems to be working, ready to compile.
[*] Testing the CC wrapper and instrumentation output...
unset AFL_USE_ASAN AFL_USE_MSAN; AFL_QUIET=1 AFL_INST_RATIO=100 AFL_PATH=. ./afl-clang -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DDOC_PATH=\"/usr/local/share/doc/afl\" -DBIN_PATH=\"/usr/local/bin\" test-instr.c -o test-instr
echo 0 | ./afl-showmap -m none -q -o .test-instr0 ./test-instr
echo 1 | ./afl-showmap -m none -q -o .test-instr1 ./test-instr

Oops, the instrumentation does not seem to be behaving correctly!

Please ping <lcamtuf@google.com> to troubleshoot the issue.

make: *** [test_build] Error 1

The next step is to integrate AFL into a Xcode project build using xcodebuild from the command line. We don’t need to change anything directly into the Xcode project, just set some environment variables. These are:

Set CC and CXX to afl-clang-fast and afl-clang-fast++.
Set AFL_CC and AFL_CXX to point to our downloaded binary clang and clang++ else Xcode’s clang will be used instead (and fail on symbols).
Set AFL_PATH to the AFL folder we compiled it from (not sure why this is needed - maybe because no install due to the sudo problem?)

$ cd Xcode_Project_Folder
$ export AFL_PATH=~/Projects/afl/afl-2.45b
$ CC=afl-clang-fast AFL_CC=~/Projects/afl/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/bin/clang xcodebuild -configuration Release

This will build the Release version of your project (you don’t want to fuzz unoptimized version) and you can find the binary in the build folder of your project.

Using the persistent_demo.c sample code located at experimental/persistent_demo folder we can compare the performance increase from afl-clang versus afl-clang-fast. On a 6 core 3.5 Ghz TrashCan Mac Pro the fast version gets around 9500 executions per second while the regular version gets around 1500, a 6 times speed increase with the LLVM mode version. This is quite a huge performance gain although the sample code is very simple.

For example fuzzing my own Mach-O parsing library the performance gain is about 2 to 3 times which is quite considerable improvement just from a small compiler change. The AFL LLVM mode README doesn’t expect such dramatic performance gains but its benchmarks were probably based on Linux versions. Mac OS X baseline is much worse because of fork() issues so the performance gains are much more visible.

I am not sure if this is the best way to achieve AFL’s LLVM mode in OS X. It works fine for me although it means you need to keep that clang binary installation and AFL folder somewhere and need to compile projects you want to fuzz via the command line, which isn’t really a big problem considering the benefits we get out of it.

If you have any tips or better ways to achieve this please drop me a mail. I couldn’t find many clues other than Ben’s work on this.

Have fun,
fG!