Rex the Wonder Dog (here and here) is a proof of concept that uses TrustedBSD framework to install kernel level backdoors. Volatility is able to detect these malicious modules with a plugin created by Andrew Case. The plugin works by looking up the TrustedBSD structures and dumping information about the loaded modules.

At SyScan360 I presented a “new” trick to bypass this plugin by creating a shadow structure and leaving the legit one untouched. Volatility looks up the original structure and is unable to detect the malicious modules. The problem of this approach is that modifies kernel code (the references to the structure) so it will raise a flag when verifying the integrity of the running kernel code.

The real Rex is a very cool dog but his only interest in life is food. Fortunately for him the virtual Rex is smarter and eager to learn so let’s teach him something new. This trick exploits a failure in the plugin assumptions to not replicate the exact TrustedBSD plugins call process. It is once again a good example of how assumptions can be problematic, both to the person creating the plugin and its users. The latter are most of the time blind to the underlying assumptions and just want to use the tools. Off-the-shelf tools always have these kind of problems and the more popular they are the more blindly used and trusted. This is not a specific critic to the Volatility project (which I think it’s a great project by the way!) but more directed towards Information Security in general, where this is a frequent problem. I digress, let’s move to the interesting part!

TrustedBSD is one of my favourite features in OS X kernel. It allows to easily extend OS X security or install backdoors into the kernel without modifying kernel code. We just have to load a kernel module configured with the hooks we are interested in listening at. TrustedBSD will do all the dirty work for us and call our functions, where we can control access to resources or do malicious stuff.

The core of TrustedBSD implementation is the mac_policy_list structure, used in a global variable called mac_policy_list. It contains information about all the loaded modules.

/* defined in XNU/security/mac_internal.h */
struct mac_policy_list_element {
        struct mac_policy_conf *mpc;

struct mac_policy_list {
	u_int				numloaded;
	u_int 				max;
	u_int				maxindex;
	u_int				staticmax;
	u_int				chunks;
	u_int				freehint;
	struct mac_policy_list_element	*entries;

typedef struct mac_policy_list mac_policy_list_t;

/* defined in XNU/security/mac_base.c */
mac_policy_list_t mac_policy_list;

The entries array contains the information about the loaded modules and their callbacks in mpc_ops.

  @brief Mac policy configuration

  This structure specifies the configuration information for a
  MAC policy module.  A policy module developer must supply
  a short unique policy name, a more descriptive full name, a list of label
  namespaces and count, a pointer to the registered enty point operations,
  any load time flags, and optionally, a pointer to a label slot identifier.

  The Framework will update the runtime flags (mpc_runtime_flags) to
  indicate that the module has been registered.

  If the label slot identifier (mpc_field_off) is NULL, the Framework
  will not provide label storage for the policy.  Otherwise, the
  Framework will store the label location (slot) in this field.

  The mpc_list field is used by the Framework and should not be
  modified by policies.

struct mac_policy_conf {
  const char  *mpc_name;		/** policy name */
  const char  *mpc_fullname;		/** full name */
  const char  **mpc_labelnames;	/** managed label namespaces */
  unsigned int  mpc_labelname_count;  /** number of managed label namespaces */
  struct mac_policy_ops  *mpc_ops;	/** operation vector */
  int  mpc_loadtime_flags;	/** load time flags */
  int  *mpc_field_off;		/** label slot */
  int  mpc_runtime_flags;	/** run time flags */
  mpc_t  mpc_list;		/** List reference */
  void  *mpc_data;		/** module data */

Iterating over the entries array gives us all the loaded TrustedBSD modules. The contents of mac_policy_list in a clean OS X system are:

gdb$ print (mac_policy_list_t)mac_policy_list
$1 = {
  numloaded = 0x3,
  max = 0x200,
  maxindex = 0x2,
  staticmax = 0x3,
  chunks = 0x1,
  freehint = 0x3,
  entries = 0xffffff800c865000

gdb$ x/10xg 0xffffff800c865000
0xffffff800c865000: 0xffffff7f875dd010 0xffffff7f875f21d0
0xffffff800c865010: 0xffffff7f875fe110 0x0000000000000000
0xffffff800c865020: 0x0000000000000000 0x0000000000000000
0xffffff800c865030: 0x0000000000000000 0x0000000000000000
0xffffff800c865040: 0x0000000000000000 0x0000000000000000

gdb$ print *(struct mac_policy_conf*)0xffffff7f875dd010
$2 = {
  mpc_name = 0xffffff7f875dcfd4 "TMSafetyNet",
  mpc_fullname = 0xffffff7f875dcfe0 "Safety net for Time Machine",
  mpc_labelnames = 0xffffff7f875dd060,
  mpc_labelname_count = 0x1,
  mpc_ops = 0xffffff7f875dd068,
  mpc_loadtime_flags = 0x2,
  mpc_field_off = 0xffffff7f875ddbd8,
  mpc_runtime_flags = 0x1,
  mpc_list = 0x0,
  mpc_data = 0x0

By default three modules are loaded, TMSafetyNet, Sandbox, and Quarantine. If a new TrustedBSD module is loaded the structure is modified to:

gdb$ print (mac_policy_list_t)mac_policy_list
$3 = {
  numloaded = 0x4,
  max = 0x200,
  maxindex = 0x3,
  staticmax = 0x3,
  chunks = 0x1,
  freehint = 0x4,
  entries = 0xffffff800c865000

gdb$ x/10xg 0xffffff800c865000
0xffffff800c865000: 0xffffff7f875dd010 0xffffff7f875f21d0
0xffffff800c865010: 0xffffff7f875fe110 0xffffff7f8860dc40
0xffffff800c865020: 0x0000000000000000 0x0000000000000000
0xffffff800c865030: 0x0000000000000000 0x0000000000000000
0xffffff800c865040: 0x0000000000000000 0x0000000000000000

The number of loaded modules numloaded is increased to four, maxindex and freehint are increased by one, staticmax is unchanged. The meaning of staticmax can be found in XNU_source/security/mac_base.c:

 * mac_policy_list holds the list of policy modules.  Modules with a
 * handle lower than staticmax are considered "static" and cannot be
 * unloaded.  Such policies can be invoked without holding the busy count.
 * Modules with a handle at or above the staticmax high water mark
 * are considered to be "dynamic" policies.  A busy count is maintained

This means that the three “default” modules described above are considered static and cannot be unloaded. If this wasn’t the case, for example, the sandbox module could be unloaded with root access. Just for curiosity, the modules can be configured for unloading by setting the MPC_LOADTIME_FLAG_UNLOADOK flag in mpc_loadtime_flags of struct mac_policy_conf.

How are the modules called aka how TrustedBSD really works? Let’s use the task_for_pid() function to exemplify. Its kernel implementation is:

/* in XNU/bsd/vm/vm_unix.c */
	struct task_for_pid_args *args)
		error = mac_proc_check_get_task(kauth_cred_get(), p);
		if (error) {
			error = KERN_FAILURE;
			goto tfpout;

/* in XNU/security/mac_process.c */
mac_proc_check_get_task(struct ucred *cred, struct proc *p)
	int error;

	MAC_CHECK(proc_check_get_task, cred, p);

	return (error);

In task_for_pid() there is a hook that calls the TrustedBSD function mac_proc_check_get_task(). Inside this function there is a macro MAC_CHECK that iterates the loaded modules and executes the registered callback if the module registered to this operation. There are four of these macros, MAC_CHECK, MAC_GRANT, MAC_BOOLEAN, MAC_PERFORM. Let’s see how MAC_CHECK is implemented to understand the vulnerability:

 * MAC_CHECK performs the designated check by walking the policy
 * module list and checking with each as to how it feels about the
 * request.  Note that it returns its value via 'error' in the scope
 * of the caller.
#define	MAC_CHECK(check, args...) do {					\
	struct mac_policy_conf *mpc;					\
	u_int i;                                               		\
	error = 0;							\
	for (i = 0; i < mac_policy_list.staticmax; i++) {		\
 		mpc = mac_policy_list.entries[i].mpc;              	\
 		if (mpc == NULL)                                	\
 			continue;                               	\
 		if (mpc->mpc_ops->mpo_ ## check != NULL)		\
			error = mac_error_select(      			\
			    mpc->mpc_ops->mpo_ ## check (args),		\
			    error);					\
	}								\
	if (mac_policy_list_conditional_busy() != 0) {			\
		for (; i <= mac_policy_list.maxindex; i++) {		\
 			mpc = mac_policy_list.entries[i].mpc;		\
 			if (mpc == NULL)                                \
 				continue;                               \
 			if (mpc-&gt;mpc_ops-&gt;mpo_ ## check != NULL)	\
				error = mac_error_select(      		\
				    mpc->mpc_ops->mpo_ ## check (args),	\
				    error);				\
		}							\
		mac_policy_list_unbusy();				\
	}								\
} while (0)

The macro first iterates the static modules and continues with the dynamic ones if they exist. Recall that when a new module was loaded the staticmax stayed at three and maxindex increased to three. This means that the new module will hit the second part of the macro.

Now the important question. What happens if staticmax is bigger than maxindex? The dynamic policy will still be called from the staticmax iteration but not from the maxindex iteration.
This means that a rootkit can load a TrustedBSD backdoor and modify the array by moving its pointer ahead and/or modifying staticmax and maxindex values. TrustedBSD will continue to work as usual with these modifications so they are ok.
Where’s the Volatility vulnerability?

This is the vulnerable code snippet from plugin:

list_addr = self.addr_space.profile.get_symbol("_mac_policy_list")

plist = obj.Object("mac_policy_list", offset = list_addr, vm = self.addr_space)
parray = obj.Object('Array', offset = plist.entries, vm = self.addr_space, targetType = 'mac_policy_list_element', 
count = plist.maxindex + 1)

for ent in parray:
    # I don't know how this can happen, but the kernel makes this check all over the place
    # the policy is useful without any ops so a rootkit can't abuse this
    if ent.mpc == None:

    name = ent.mpc.mpc_name.dereference()

    ops = obj.Object("mac_policy_ops", offset = ent.mpc.mpc_ops, vm = self.addr_space)

    # walk each member of the struct
    for check in ops_members:
        ptr = ops.__getattr__(check)

        if ptr != 0:
            good = common.is_known_address(ptr, kernel_symbol_addresses, kmods) 

            yield (good, check, name, ptr)

What happens is that this plugin iterates the modules list using only the maxindex value and doesn’t care about the staticmax. We can easily escape detection by modifying staticmax as previously described. Volatility will not detect the backdoor because it is outside the array it is looking at. You can even position the module far away in the array because there are 0x200 entries pre-allocated. In this case it would be more suspicious but since there is no info printed about the contents of mac_policy_list, that would require someone to pay attention to that detail - also known as experience.

Since the module operations in mpc_ops are just pointers, the zombies rootkit technique can be used to load the malicious backdoor and leave no visible traces of the kernel extension, other than the data and changes in mac_policy_list that Volatility can’t detect yet.

At the same time, this is also a TrustedBSD implementation bug because it assumes the static modules cannot be unloaded, which might not be true at all. If one module loaded as static is unloaded (because it was configured for that, not a common scenario!) a hole will be created in the entries array. The entry will be set to NULL and when you load/reload a module, it will never be called because of the desync that now exists in the staticmax and maxindex fields.

This is a pretty cute way to create a desync between what Volatility is able to see and what is running. It was a very nice bug because it somewhat exploits user trust. I expect most people to trust the plugin output and look no further.
You always need to understand what the tools are doing, and the internals of what you are trying to analyse. This means a lot of extra work and study, and not simple usage of “off-the-shelf” tools.
Keep up the good work Volatility team, be careful with the assumptions!

Have fun,