Almost two years ago (when covid was just starting and we all happily ignored it) I wrote a post about implementing x86 hardware breakpoints in lldb. This critical debugger feature was missing from lldb. Probably because lldb main users are developers and not serious reverse engineers (lol!) dealing with malicious code and/or just reversing/cracking hostile software protections (cracking is the best and most fun RE target practice).
The build process described in that post worked but I wasn’t very happy with it - not easily portable between macOS systems. Some time ago I tried to fix it but I gave up since I wasn’t in the mood to deal with build systems problems.
This week I needed once again to add a feature to lldb. The debug registers aren’t directly available in lldb so we can’t display and modify them. Maybe we could with some Python hacks together with calling system APIs but that’s ugly and not fun.
This time I wanted to solve the portable build problem since I needed to use it in different computers and VMs. Nothing the like right incentive. Because lldb takes a while to compile I wanted to use my Ryzen 3950X for that task. Another incentive to make it portable. Technically only the first compile will take a while since we can use ccache to speed up other builds.
The lldb building documentation isn’t clear at all about creating a self-contained macOS app detached from Xcode.app structure. My goal was to have an app like Xcode or whatever Xcode Command Line Tools equivalent. The source code contains cmake files responsible for creating the Xcode.app. They are located at llvm-project/lldb/cmake/caches/, specifically Apple-lldb-macOS.cmake.
A single static binary would be even better but this is not possible because of the split architecture between lldb (the frontend) and debugserver (the backend), plus Python scripting support.
After some experiments, lots of frustration, and understanding some of the obstacles I finally built it the way I wanted. So let me guide you on that journey!
The first problem is Python. We definitely want to have Python scripting support otherwise lldbinit doesn’t work and makes lldb really annoying to use. It’s not a real debugger if it doesn’t have Softice looks.
Apple marked scripting languages as deprecated in default installs since Catalina (10.15). Python2 is dead so we definitely want to use Python3. In newer macOS versions python3 is just a stub to install Xcode or Xcode Command Line Tools.
This means that there are filesystem differences between macOS versions. High Sierra and Mojave have a Python.framework located at /Library/Frameworks, while in Catalina and Big Sur it can be found at /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/. There is also the problem of different Python versions (3.7 vs 3.8 at least).
Plus, the lldb Caveats documentation has the following note about Python:
To make this possible, LLDB links against the Python shared library. Linking against Python comes with some constraints to be aware of.
It is not possible to build and link LLDB against a Python 3 library and use it from Python 2 and vice versa.
It is not possible to build and link LLDB against one distribution on Python and use it through a interpreter coming from another distribution. For example, on macOS, if you build and link against Python from python.org, you cannot import the lldb module from the Python interpreter installed with Homebrew.
To use third party Python packages from inside LLDB, you need to install them using a utility (such as
pip) from the same Python distribution as the one used to build and link LLDB.The previous considerations are especially important during development, but apply to binary distributions of LLDB as well.
Apple’s lldb has no problems since it packs its own Python.framework with each Xcode version on newer macOS, or uses system Python in older OSes. I want to have a single distributable app so the Python problem needs to be solved.
If Apple can do it with Xcode.app so we (possibly) can. The question is how. I thought it could be complicated but turned out rather easy.
Python source code has everything needed to build a macOS framework so no Apple proprietary magic involved. We just need to build Python from source code and configure the framework installation into our app bundle. The app is called lldb-ng (how creative!).
I build a universal ARM64/x86_64 Python since my goal is to build a universal custom lldb (haven’t managed yet). If you don’t need that you can save some space and remove --enable-universalsdk --with-universal-archs=universal2 from configure options below.
The build system I used is running macOS Catalina 10.15.7 and Xcode 12.4, on top of a Ryzen 3950X KVM+QEMU VM. The detailed post on how to build this kind of system is still in draft, sorry!
mkdir ~/src
cd ~/src
curl -L -O https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz
tar xfz Python-3.9.6.tgz
cd Python-3.9.6
CFLAGS="-mmacosx-version-min=10.13" ./configure --enable-framework=/Applications/lldb-ng.app/Contents/Frameworks/ --enable-universalsdk --with-universal-archs=universal2 --enable-optimizations
make -j8
sudo make install
Add the macosx-version-min option to CFLAGS to build support for older macOS versions other than build machine.
Before compiling and installing things maybe you should verify the package signature. Assuming you have GPG Suite installed:
curl -L -O https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz.asc
curl https://keybase.io/bp/pgp_keys.asc | gpg --import
gpg --verify Python-3.9.6.tgz.asc
If signature is bad, download and verify again and pray that you and/or python.org are not under attack. Or just go full YOLO and don’t check signature.
After everything is finished the Python.framework will be installed at /Applications/lldb-ng.app/Contents/Frameworks. We are directly building the app in /Applications, which is a bit weird. We could probably use a chroot or something to workaround this. Since I have a VM for this build I don’t care much. We will need to link against this Python version when building lldb so this is maybe the simplest way to do it and avoid further build hacks. I am not an expert in build systems.
The install phase will also install symlinks in /usr/local/bin. If you are not using a dedicated build system and have a Python install from the official site you might have conflicts. A possible workaround is to check the Makefile and issue each install step except the one that creates the symlinks.
If everything went well we have a working Python3 copy in the app bundle already.
% /Applications/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/bin/python3
Python 3.9.6 (default, Jul 16 2021, 02:41:04)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Before compiling lldb we need to install other dependencies (these are the versions I have used):
I have built and installed all these dependencies from source code. If you are a Homebrew user it should work since they are only used for building not in lldb itself.
After everything is installed we can finally deal with lldb build. We can build out of git repo or from a specific release source package(s). I use the 12.0.1 source package since it’s a faster download and I don’t need git history or newer code other than a stable release. It’s also easier to build out of llvm-project instead of lldb source package since parts of llvm are required to build lldb (all llvm projects were unified into a single git some time ago).
cd ~/src
curl -L -O https://github.com/llvm/llvm-project/releases/download/llvmorg-12.0.1/llvm-project-12.0.1.src.tar.xz
tar xfz llvm-project-12.0.1.src.tar.xz
To verify the signature (funny enough the key expired two years ago):
curl -L -O https://github.com/llvm/llvm-project/releases/download/llvmorg-12.0.1/llvm-project-12.0.1.src.tar.xz.sig
curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/tstellar-gpg-key.asc | gpg --import
gpg --verify llvm-project-12.0.1.src.tar.xz.sig
I created a custom cmake configuration that contains all the settings necessary to build into the app bundle. All the build magic is here.
cd ~/src
cat <<'EOF' >standalone.cmake
set(CMAKE_BUILD_TYPE Release CACHE STRING "")
set(CMAKE_EXPORT_COMPILE_COMMANDS ON CACHE BOOL "")
set(LLVM_TARGETS_TO_BUILD X86;ARM;AArch64 CACHE STRING "")
set(LLVM_ENABLE_PROJECTS clang;lldb CACHE STRING "")
set(LLDB_INCLUDE_TESTS OFF CACHE BOOL "")
set(LLDB_SKIP_STRIP ON CACHE BOOL "")
set(LLDB_NO_INSTALL_DEFAULT_RPATH OFF CACHE BOOL "")
set(CMAKE_OSX_DEPLOYMENT_TARGET 10.13 CACHE STRING "")
set(LLVM_CCACHE_BUILD ON CACHE BOOL "")
set(LLDB_BUILD_FRAMEWORK OFF CACHE BOOL "")
set(CMAKE_INSTALL_PREFIX /Applications/lldb-ng.app/Contents/usr CACHE STRING "")
set(LLDB_ENABLE_PYTHON ON CACHE BOOL "")
set(LLDB_EMBED_PYTHON_HOME OFF CACHE BOOL "")
set(Python3_ROOT_DIR "/Applications/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/" CACHE STRING "")
set(LLDB_PYTHON_HOME "/Applications/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/bin" CACHE STRING "")
set(LLVM_DISTRIBUTION_COMPONENTS
lldb
liblldb
lldb-argdumper
darwin-debug
debugserver
lldb-python-scripts
CACHE STRING "")
EOF
It is configured to use ccache. If you don’t want to use it then set LLVM_CCACHE_BUILD to OFF or remove that line.
The essential settings are:
LLDB_NO_INSTALL_DEFAULT_RPATH OFF: takes care of settingrpathso library references are correct.LLDB_BUILD_FRAMEWORK OFF: don’t buildLLDB.frameworkas in Xcode. Instead a dynamic libraryliblldb.dylibis built. I had problems before with the framework and didn’t try to make it work in this build.CMAKE_INSTALL_PREFIX: the installation path for the binaries. I kept theusrpart, maybe remove and we end up withbinandlibfolders inContentsroot.Python3_ROOT_DIR: overrides Python3 search path, otherwise it will not use the version we installed in the app bundle. Without this the binaries would reference the Xcode or system Python and break our portability goal.LLDB_PYTHON_HOME: path to our Python framework.LLDB_EMBED_PYTHON_HOME OFF: don’t storePYTHON_HOMEpath in the binary (this leads to Python startup issues if set).
LLDB_INCLUDE_TESTS is set to off otherwise you need to build libcxx for tests to work (add it to LLVM_ENABLE_PROJECTS setting). I’m building out of a stable release so YOLO.
First step is to generate ninja build files. You will need a code signing certificate from Apple otherwise setup a self signing certificate. If you use a self-signing certificate you need to install the certificate in every system where you want to use this version, so hurts portability a bit. The build system expects the certificate to be in the System keychain (weird!) otherwise you might have an error about not finding the code signing identity. Modify LLDB_CODESIGN_IDENTITY variable below (you just need to use the user id/OU number otherwise the common name can mess up scripts).
CMake Warning at /Users/user/src/llvm-project-12.0.1.src/lldb/tools/debugserver/source/CMakeLists.txt:32 (message):
LLDB_CODESIGN_IDENTITY not found: 'XXXXXXXXX' This will cause failures in
the test suite.Pass '-DLLDB_USE_SYSTEM_DEBUGSERVER=ON' to use the system
one instead.See 'Code Signing on macOS' in the documentation.
Call Stack (most recent call first):
/Users/user/src/llvm-project-12.0.1.src/lldb/tools/debugserver/source/CMakeLists.txt:95 (get_debugserver_codesign_identity)
The debugserver binary needs to be signed to be able to attach to processes (lldb process is just the frontend to the debugger). There is some code signing certificate incompatibility if built on Catalina or Big Sur and then try to use in High Sierra (debugserver will fail to attach because taskgated will not permit it due to code signing error). I have yet to check where is the problem and how to solve it. Mojave or higher it works without problems.
cd ~/src
cmake -B lldb-build -G Ninja \
-C standalone.cmake \
-DLLDB_CODESIGN_IDENTITY="INSERT ID HERE" \
llvm-project-12.0.1.src/llvm
And start compiling everything:
ninja -C lldb-build lldb
ninja -C lldb-build debugserver
ninja -C lldb-build darwin-debug
The lldb target takes the longest to complete. Around 20 mins on a M1 (tested with a native ARM64 build only), and 9 mins on a Ryzen 3950X macOS VM (with all 16 cores/32 threads attributed).
After everything is compiled we just need to install the components.
sudo ninja -C lldb-build install-lldb
sudo ninja -C lldb-build install-debugserver
sudo ninja -C lldb-build install-liblldb
sudo ninja -C lldb-build install-lldb-python-scripts
sudo ninja -C lldb-build install-darwin-debug
You might want to create a link in /usr/local/bin to differentiate from Xcode’s lldb:
sudo ln -s /Applications/lldb-ng.app/Contents/usr/bin/lldb /usr/local/bin/lldb-ng
And finally verify if everything is working as expected.
% lldb-ng /usr/local/bin/lldb-ng
[+] Loaded lldbinit version: 2.0.205
(lldbinit) target create "/usr/local/bin/lldb-ng"
Current executable set to '/usr/local/bin/lldb-ng' (x86_64).
(lldbinit) version
lldb version 12.0.1
(lldbinit) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> sys.version
'3.9.6 (default, Jul 16 2021, 02:41:04) \n[Clang 12.0.0 (clang-1200.0.32.29)]'
>>> ^D
now exiting InteractiveConsole...
(lldbinit) process launch -s
(...)
Process 32547 stopped
* thread #1, stop reason = signal SIGSTOP
frame #0: 0x0000000100065000 dyld`_dyld_start
Process 32547 launched: '/usr/local/bin/lldb-ng' (x86_64)
(lldbinit)
We need to verify if everything is linked and loaded as expected and if it’s using the correct debugserver version.
% vmmap 32546
Process: lldb [32546]
Path: /Applications/lldb-ng.app/Contents/usr/bin/lldb
Load Address: 0x1018f2000
Identifier: lldb
Version: ???
Code Type: X86-64
Platform: macOS
Parent Process: zsh [85425]
Date/Time: 2021-07-16 05:37:35.173 +0100
Launch Time: 2021-07-16 05:35:08.134 +0100
OS Version: Mac OS X 10.15.7 (19H114)
Report Version: 7
Analysis Tool: /Applications/Xcode12.4.app/Contents/Developer/usr/bin/vmmap
Analysis Tool Version: Xcode 12.4 (12D4e)
----
Virtual Memory Map of process 32546 (lldb)
Output report format: 2.4 -- 64-bit process
VM page size: 4096 bytes
==== Non-writable regions for process 32546
REGION TYPE START - END [ VSIZE RSDNT DIRTY SWAP] PRT/MAX SHRMOD PURGE REGION DETAIL
__TEXT 1018f2000-101932000 [ 256K 256K 0K 0K] r-x/r-x SM=COW /Applications/lldb-ng.app/Contents/usr/bin/lldb
__LINKEDIT 10193a000-101953000 [ 100K 100K 0K 0K] r--/r-- SM=COW /Applications/lldb-ng.app/Contents/usr/bin/lldb
__LINKEDIT 101953000-101956000 [ 12K 0K 0K 0K] r--/r-- SM=NUL /Applications/lldb-ng.app/Contents/usr/bin/lldb
__TEXT 101956000-101bd6000 [ 2560K 2560K 0K 0K] r-x/rwx SM=COW ...tions/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/Python
__DATA_CONST 101bd6000-101bde000 [ 32K 32K 20K 0K] r--/rwx SM=COW ...tions/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/Python
__LINKEDIT 101c36000-101d14000 [ 888K 888K 0K 0K] r--/rwx SM=COW ...tions/lldb-ng.app/Contents/Frameworks/Python.framework/Versions/3.9/Python
__TEXT 103e3d000-107dc5000 [ 63.5M 63.5M 0K 0K] r-x/rwx SM=COW /Applications/lldb-ng.app/Contents/usr/lib/liblldb.12.0.1.dylib
__LINKEDIT 1082c5000-108ea4000 [ 11.9M 11.9M 0K 0K] r--/rwx SM=COW /Applications/lldb-ng.app/Contents/usr/lib/liblldb.12.0.1.dylib
We can see that the lldb process is using the correct Python framework and lldb library as we wanted.
% ps aux | grep debugserver
user 32548 0.0 0.0 4410532 3648 s008 S 5:36AM 0:00.03 /Applications/lldb-ng.app/Contents/usr/bin/debugserver --fd=7 --native-regs --setsid
And debugserver is also our own. Everything is working as expected :-).
We can pack the lldb-ng.app and move it to another system and use our custom lldb version without much trouble (don’t forget to create the link or fix PATH).
The operating system considers the app bundle invalid since there is no main app. We can add the missing pieces to make it valid such as a Info.plist and a MacOS folder with some stub app. And then codesign the whole app bundle and make everything by the book.
And that’s it. The whole process is a lot less complicated than I initially thought and ranted about. It required me to dig into the cmake files and understand some build internals. Now you can modify lldb source code and have your own portable version without depending on Xcode releases.
If there are better ways to achieve this or improve the process I am definitely interested to hear about it. Please ping me by email or tweet!
I’ll update the git repo with the cmake and patches later on. Now I have to write the code to manage the debug registers.
Have fun,
fG!
Update:
It was quite easy to add the debug registers feature I wanted since most of the necessary code was already present.
The x86_debug_registers.patch is available in my patches repo. Just apply the patch and build.
Sample output:
(lldbinit) register read -s 3
Debug Registers:
dr0 = 0x0000000100011003 dyld`_dyld_start + 3
dr1 = 0x000070000a81b000
dr2 = 0x0000000000000000
dr3 = 0x0000000000000000
dr4 = 0x0000000000000000
dr5 = 0x0000000000000000
dr6 = 0x00000000ffff0ff1
dr7 = 0x0000000000000555
(lldbinit) register write dr0 0
(lldbinit) register read dr0
dr0 = 0x0000000000000000
(lldbinit) register write dr0 0x31337
(lldbinit) register read dr0
dr0 = 0x0000000000031337
Update 2:
To fix the code signing in High Sierra or older we need to fix the Info.plist for debugserver.
The file is located at llvm-project-12.0.1.src/lldb/tools/debugserver/resources/lldb-debugserver-Info.plist
Add the following between the dict entry:
<key>SecTaskAccess</key>
<array>
<string>allowed</string>
<string>debug</string>
</array>
Delete the debugserver binary from lldb-build/bin/ and rebuild debugserver.
Before this you need to fix Python build because it targets the macOS version you built it on. Pass the -mmacosx-version-min=10.13 flag to CFLAGS when configuring Python build. Change the version if you need to compile for even older version than High Sierra.