Almost two years ago (when covid was just starting and we all happily ignored it) I wrote a post about implementing x86 hardware breakpoints in lldb. This critical debugger feature was missing from lldb. Probably because lldb main users are developers and not serious reverse engineers (lol!) dealing with malicious code and/or just reversing/cracking hostile software protections (cracking is the best and most fun RE target practice).
The build process described in that post worked but I wasn’t very happy with it - not easily portable between macOS systems. Some time ago I tried to fix it but I gave up since I wasn’t in the mood to deal with build systems problems.
This week I needed once again to add a feature to lldb. The debug registers aren’t directly available in lldb so we can’t display and modify them. Maybe we could with some Python hacks together with calling system APIs but that’s ugly and not fun.
This time I wanted to solve the portable build problem since I needed to use it in different computers and VMs. Nothing the like right incentive. Because lldb takes a while to compile I wanted to use my Ryzen 3950X for that task. Another incentive to make it portable. Technically only the first compile will take a while since we can use ccache to speed up other builds.
The lldb building documentation isn’t clear at all about creating a self-contained macOS app detached from
Xcode.app structure. My goal was to have an app like Xcode or whatever Xcode Command Line Tools equivalent. The source code contains
cmake files responsible for creating the
Xcode.app. They are located at
A single static binary would be even better but this is not possible because of the split architecture between
lldb (the frontend) and
debugserver (the backend), plus Python scripting support.
After some experiments, lots of frustration, and understanding some of the obstacles I finally built it the way I wanted. So let me guide you on that journey!
The first problem is Python. We definitely want to have Python scripting support otherwise lldbinit doesn’t work and makes lldb really annoying to use. It’s not a real debugger if it doesn’t have Softice looks.
Apple marked scripting languages as deprecated in default installs since Catalina (10.15). Python2 is dead so we definitely want to use Python3. In newer macOS versions python3 is just a stub to install Xcode or Xcode Command Line Tools.
This means that there are filesystem differences between macOS versions. High Sierra and Mojave have a
Python.framework located at
/Library/Frameworks, while in Catalina and Big Sur it can be found at
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/. There is also the problem of different Python versions (3.7 vs 3.8 at least).
Plus, the lldb Caveats documentation has the following note about Python:
To make this possible, LLDB links against the Python shared library. Linking against Python comes with some constraints to be aware of.
It is not possible to build and link LLDB against a Python 3 library and use it from Python 2 and vice versa.
It is not possible to build and link LLDB against one distribution on Python and use it through a interpreter coming from another distribution. For example, on macOS, if you build and link against Python from python.org, you cannot import the lldb module from the Python interpreter installed with Homebrew.
To use third party Python packages from inside LLDB, you need to install them using a utility (such as
pip) from the same Python distribution as the one used to build and link LLDB.
The previous considerations are especially important during development, but apply to binary distributions of LLDB as well.
Apple’s lldb has no problems since it packs its own Python.framework with each Xcode version on newer macOS, or uses system Python in older OSes. I want to have a single distributable app so the Python problem needs to be solved.
If Apple can do it with Xcode.app so we (possibly) can. The question is how. I thought it could be complicated but turned out rather easy.
Python source code has everything needed to build a macOS framework so no Apple proprietary magic involved. We just need to build Python from source code and configure the framework installation into our app bundle. The app is called
lldb-ng (how creative!).
I build a universal ARM64/x86_64 Python since my goal is to build a universal custom lldb (haven’t managed yet). If you don’t need that you can save some space and remove
--enable-universalsdk --with-universal-archs=universal2 from
configure options below.
The build system I used is running macOS Catalina 10.15.7 and Xcode 12.4, on top of a Ryzen 3950X KVM+QEMU VM. The detailed post on how to build this kind of system is still in draft, sorry!
mkdir ~/src cd ~/src curl -L -O https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz tar xfz Python-3.9.6.tgz cd Python-3.9.6 CFLAGS="-mmacosx-version-min=10.13" ./configure --enable-framework=/Applications/lldb-ng.app/Contents/Frameworks/ --enable-universalsdk --with-universal-archs=universal2 --enable-optimizations make -j8 sudo make install
macosx-version-min option to
CFLAGS to build support for older macOS versions other than build machine.
Before compiling and installing things maybe you should verify the package signature. Assuming you have GPG Suite installed:
curl -L -O https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz.asc curl https://keybase.io/bp/pgp_keys.asc | gpg --import gpg --verify Python-3.9.6.tgz.asc
If signature is bad, download and verify again and pray that you and/or python.org are not under attack. Or just go full YOLO and don’t check signature.
After everything is finished the
Python.framework will be installed at
/Applications/lldb-ng.app/Contents/Frameworks. We are directly building the app in
/Applications, which is a bit weird. We could probably use a chroot or something to workaround this. Since I have a VM for this build I don’t care much. We will need to link against this Python version when building lldb so this is maybe the simplest way to do it and avoid further build hacks. I am not an expert in build systems.
The install phase will also install symlinks in
/usr/local/bin. If you are not using a dedicated build system and have a Python install from the official site you might have conflicts. A possible workaround is to check the
Makefile and issue each install step except the one that creates the symlinks.
If everything went well we have a working Python3 copy in the app bundle already.
% /Applications/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/bin/python3 Python 3.9.6 (default, Jul 16 2021, 02:41:04) [Clang 12.0.0 (clang-1220.127.116.11)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>
Before compiling lldb we need to install other dependencies (these are the versions I have used):
I have built and installed all these dependencies from source code. If you are a Homebrew user it should work since they are only used for building not in lldb itself.
After everything is installed we can finally deal with lldb build. We can build out of git repo or from a specific release source package(s). I use the 12.0.1 source package since it’s a faster download and I don’t need git history or newer code other than a stable release. It’s also easier to build out of
llvm-project instead of
lldb source package since parts of
llvm are required to build lldb (all llvm projects were unified into a single git some time ago).
cd ~/src curl -L -O https://github.com/llvm/llvm-project/releases/download/llvmorg-12.0.1/llvm-project-12.0.1.src.tar.xz tar xfz llvm-project-12.0.1.src.tar.xz
To verify the signature (funny enough the key expired two years ago):
curl -L -O https://github.com/llvm/llvm-project/releases/download/llvmorg-12.0.1/llvm-project-12.0.1.src.tar.xz.sig curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/tstellar-gpg-key.asc | gpg --import gpg --verify llvm-project-12.0.1.src.tar.xz.sig
I created a custom cmake configuration that contains all the settings necessary to build into the app bundle. All the build magic is here.
cd ~/src cat <<'EOF' >standalone.cmake set(CMAKE_BUILD_TYPE Release CACHE STRING "") set(CMAKE_EXPORT_COMPILE_COMMANDS ON CACHE BOOL "") set(LLVM_TARGETS_TO_BUILD X86;ARM;AArch64 CACHE STRING "") set(LLVM_ENABLE_PROJECTS clang;lldb CACHE STRING "") set(LLDB_INCLUDE_TESTS OFF CACHE BOOL "") set(LLDB_SKIP_STRIP ON CACHE BOOL "") set(LLDB_NO_INSTALL_DEFAULT_RPATH OFF CACHE BOOL "") set(CMAKE_OSX_DEPLOYMENT_TARGET 10.13 CACHE STRING "") set(LLVM_CCACHE_BUILD ON CACHE BOOL "") set(LLDB_BUILD_FRAMEWORK OFF CACHE BOOL "") set(CMAKE_INSTALL_PREFIX /Applications/lldb-ng.app/Contents/usr CACHE STRING "") set(LLDB_ENABLE_PYTHON ON CACHE BOOL "") set(LLDB_EMBED_PYTHON_HOME OFF CACHE BOOL "") set(Python3_ROOT_DIR "/Applications/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/" CACHE STRING "") set(LLDB_PYTHON_HOME "/Applications/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/bin" CACHE STRING "") set(LLVM_DISTRIBUTION_COMPONENTS lldb liblldb lldb-argdumper darwin-debug debugserver lldb-python-scripts CACHE STRING "") EOF
It is configured to use ccache. If you don’t want to use it then set
OFF or remove that line.
The essential settings are:
LLDB_NO_INSTALL_DEFAULT_RPATH OFF: takes care of setting
rpathso library references are correct.
LLDB_BUILD_FRAMEWORK OFF: don’t build
LLDB.frameworkas in Xcode. Instead a dynamic library
liblldb.dylibis built. I had problems before with the framework and didn’t try to make it work in this build.
CMAKE_INSTALL_PREFIX: the installation path for the binaries. I kept the
usrpart, maybe remove and we end up with
Python3_ROOT_DIR: overrides Python3 search path, otherwise it will not use the version we installed in the app bundle. Without this the binaries would reference the Xcode or system Python and break our portability goal.
LLDB_PYTHON_HOME: path to our Python framework.
LLDB_EMBED_PYTHON_HOME OFF: don’t store
PYTHON_HOMEpath in the binary (this leads to Python startup issues if set).
LLDB_INCLUDE_TESTS is set to off otherwise you need to build
libcxx for tests to work (add it to
LLVM_ENABLE_PROJECTS setting). I’m building out of a stable release so YOLO.
First step is to generate ninja build files. You will need a code signing certificate from Apple otherwise setup a self signing certificate. If you use a self-signing certificate you need to install the certificate in every system where you want to use this version, so hurts portability a bit. The build system expects the certificate to be in the System keychain (weird!) otherwise you might have an error about not finding the code signing identity. Modify
LLDB_CODESIGN_IDENTITY variable below (you just need to use the user id/OU number otherwise the common name can mess up scripts).
CMake Warning at /Users/user/src/llvm-project-12.0.1.src/lldb/tools/debugserver/source/CMakeLists.txt:32 (message): LLDB_CODESIGN_IDENTITY not found: 'XXXXXXXXX' This will cause failures in the test suite.Pass '-DLLDB_USE_SYSTEM_DEBUGSERVER=ON' to use the system one instead.See 'Code Signing on macOS' in the documentation. Call Stack (most recent call first): /Users/user/src/llvm-project-12.0.1.src/lldb/tools/debugserver/source/CMakeLists.txt:95 (get_debugserver_codesign_identity)
debugserver binary needs to be signed to be able to attach to processes (lldb process is just the frontend to the debugger). There is some code signing certificate incompatibility if built on Catalina or Big Sur and then try to use in High Sierra (debugserver will fail to attach because taskgated will not permit it due to code signing error). I have yet to check where is the problem and how to solve it. Mojave or higher it works without problems.
cd ~/src cmake -B lldb-build -G Ninja \ -C standalone.cmake \ -DLLDB_CODESIGN_IDENTITY="INSERT ID HERE" \ llvm-project-12.0.1.src/llvm
And start compiling everything:
ninja -C lldb-build lldb ninja -C lldb-build debugserver ninja -C lldb-build darwin-debug
The lldb target takes the longest to complete. Around 20 mins on a M1 (tested with a native ARM64 build only), and 9 mins on a Ryzen 3950X macOS VM (with all 16 cores/32 threads attributed).
After everything is compiled we just need to install the components.
sudo ninja -C lldb-build install-lldb sudo ninja -C lldb-build install-debugserver sudo ninja -C lldb-build install-liblldb sudo ninja -C lldb-build install-lldb-python-scripts sudo ninja -C lldb-build install-darwin-debug
You might want to create a link in
/usr/local/bin to differentiate from Xcode’s lldb:
sudo ln -s /Applications/lldb-ng.app/Contents/usr/bin/lldb /usr/local/bin/lldb-ng
And finally verify if everything is working as expected.
% lldb-ng /usr/local/bin/lldb-ng [+] Loaded lldbinit version: 2.0.205 (lldbinit) target create "/usr/local/bin/lldb-ng" Current executable set to '/usr/local/bin/lldb-ng' (x86_64). (lldbinit) version lldb version 12.0.1 (lldbinit) script Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D. >>> sys.version '3.9.6 (default, Jul 16 2021, 02:41:04) \n[Clang 12.0.0 (clang-118.104.22.168)]' >>> ^D now exiting InteractiveConsole... (lldbinit) process launch -s (...) Process 32547 stopped * thread #1, stop reason = signal SIGSTOP frame #0: 0x0000000100065000 dyld`_dyld_start Process 32547 launched: '/usr/local/bin/lldb-ng' (x86_64) (lldbinit)
We need to verify if everything is linked and loaded as expected and if it’s using the correct
% vmmap 32546 Process: lldb  Path: /Applications/lldb-ng.app/Contents/usr/bin/lldb Load Address: 0x1018f2000 Identifier: lldb Version: ??? Code Type: X86-64 Platform: macOS Parent Process: zsh  Date/Time: 2021-07-16 05:37:35.173 +0100 Launch Time: 2021-07-16 05:35:08.134 +0100 OS Version: Mac OS X 10.15.7 (19H114) Report Version: 7 Analysis Tool: /Applications/Xcode12.4.app/Contents/Developer/usr/bin/vmmap Analysis Tool Version: Xcode 12.4 (12D4e) ---- Virtual Memory Map of process 32546 (lldb) Output report format: 2.4 -- 64-bit process VM page size: 4096 bytes ==== Non-writable regions for process 32546 REGION TYPE START - END [ VSIZE RSDNT DIRTY SWAP] PRT/MAX SHRMOD PURGE REGION DETAIL __TEXT 1018f2000-101932000 [ 256K 256K 0K 0K] r-x/r-x SM=COW /Applications/lldb-ng.app/Contents/usr/bin/lldb __LINKEDIT 10193a000-101953000 [ 100K 100K 0K 0K] r--/r-- SM=COW /Applications/lldb-ng.app/Contents/usr/bin/lldb __LINKEDIT 101953000-101956000 [ 12K 0K 0K 0K] r--/r-- SM=NUL /Applications/lldb-ng.app/Contents/usr/bin/lldb __TEXT 101956000-101bd6000 [ 2560K 2560K 0K 0K] r-x/rwx SM=COW ...tions/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/Python __DATA_CONST 101bd6000-101bde000 [ 32K 32K 20K 0K] r--/rwx SM=COW ...tions/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/Python __LINKEDIT 101c36000-101d14000 [ 888K 888K 0K 0K] r--/rwx SM=COW ...tions/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/Python __TEXT 103e3d000-107dc5000 [ 63.5M 63.5M 0K 0K] r-x/rwx SM=COW /Applications/lldb-ng.app/Contents/usr/lib/liblldb.12.0.1.dylib __LINKEDIT 1082c5000-108ea4000 [ 11.9M 11.9M 0K 0K] r--/rwx SM=COW /Applications/lldb-ng.app/Contents/usr/lib/liblldb.12.0.1.dylib
We can see that the lldb process is using the correct Python framework and lldb library as we wanted.
% ps aux | grep debugserver user 32548 0.0 0.0 4410532 3648 s008 S 5:36AM 0:00.03 /Applications/lldb-ng.app/Contents/usr/bin/debugserver --fd=7 --native-regs --setsid
debugserver is also our own. Everything is working as expected :-).
We can pack the
lldb-ng.app and move it to another system and use our custom lldb version without much trouble (don’t forget to create the link or fix
The operating system considers the app bundle invalid since there is no main app. We can add the missing pieces to make it valid such as a
Info.plist and a
MacOS folder with some stub app. And then codesign the whole app bundle and make everything by the book.
And that’s it. The whole process is a lot less complicated than I initially thought and ranted about. It required me to dig into the cmake files and understand some build internals. Now you can modify lldb source code and have your own portable version without depending on Xcode releases.
If there are better ways to achieve this or improve the process I am definitely interested to hear about it. Please ping me by email or tweet!
I’ll update the git repo with the cmake and patches later on. Now I have to write the code to manage the debug registers.
It was quite easy to add the debug registers feature I wanted since most of the necessary code was already present.
x86_debug_registers.patch is available in my patches repo. Just apply the patch and build.
(lldbinit) register read -s 3 Debug Registers: dr0 = 0x0000000100011003 dyld`_dyld_start + 3 dr1 = 0x000070000a81b000 dr2 = 0x0000000000000000 dr3 = 0x0000000000000000 dr4 = 0x0000000000000000 dr5 = 0x0000000000000000 dr6 = 0x00000000ffff0ff1 dr7 = 0x0000000000000555 (lldbinit) register write dr0 0 (lldbinit) register read dr0 dr0 = 0x0000000000000000 (lldbinit) register write dr0 0x31337 (lldbinit) register read dr0 dr0 = 0x0000000000031337
To fix the code signing in High Sierra or older we need to fix the
The file is located at
Add the following between the
<key>SecTaskAccess</key> <array> <string>allowed</string> <string>debug</string> </array>
debugserver binary from
lldb-build/bin/ and rebuild
Before this you need to fix Python build because it targets the macOS version you built it on. Pass the
-mmacosx-version-min=10.13 flag to
CFLAGS when configuring Python build. Change the version if you need to compile for even older version than High Sierra.