14. System Calls & Ptrace

Written by Walter Tyree

As alluded to in the introduction to this book, debugging is not entirely about just fixing stuff. Debugging is the process of gaining a better understanding of what’s happening behind the scenes. In this chapter, you’ll explore the foundation of how debugging works, namely, 2 powerful APIs that enable lldb to attach and control a process. They are the mach exception setter APIs and the system call, ptrace.

In addition, you’ll learn some common security tricks developers use with ptrace to prevent a process from attaching to their programs. You’ll also learn some easy workarounds for these developer-imposed restrictions.

System Calls

Wait, wait, wait… ptrace is a system call. What’s a system call?

A system call is an entry point into code handled by the kernel. System calls are the foundation for userland code to do anything of interest, like opening a file, launching a process, consulting the value of a process identifier, etc.

From the userland side, a system call will marshal the appropriate arguments and send them over to the kernel. Userland code is not able to see the implementation details (i.e. the assembly) of what’s happening in the kernel. A userland function takes the arguments and executes a trap, think of it as a arm64 bl or x86_64 call instruction, to a function in the kernel. The kernel takes the arguments, determines if the arguments are well formed and if the process has permission to do the action. The kernel will then carry out that system call or deny accordingly.

For example, getpid, which gets the process identifier for the current process is actually a system call. The userland “source” to this is handwritten assembly found in xnu’s libsyscall/custom/__getpid.s. On the kernel side, the getpid call is picked up and eventually calls getpid(proc_t p, __unused struct getpid_args *uap, int32_t *retval) found in xnu’s bsd/kern/kern_prot.c.

Note: There are many unique system call wrappers that will call into the kernel, but there’s also a generic API to make system calls via the syscall(int, ...) function. One supplies an integer available from the <sys/syscall.h> header (or finds a private syscall number that’s in use) and passes in the expected arguments to that function. For example to mimic the __exit(int status) system call, you’d execute the syscall(SYS_exit, status); where status is the return value you’d pass into __exit.

Finding System Calls

To get a list of system calls, you can peruse the sources of xnu on opensource.apple.com. Alternatively, you can use DTrace to dynamically find them at runtime.

macOS Ventura has about 557 system calls. Open a Terminal window and run the following command to get the number of systems calls available in your system:

sudo dtrace -ln 'syscall:::entry' | wc -l

Note: Remember, you’ll need to disable SIP (See Chapter 1) if you want to use DTrace. In addition, you’ll also need sudo for the DTrace command since DTrace can monitor processes across multiple users, as well as perform some incredibly powerful actions. With great power comes great responsibility — that’s why you need sudo. You’ll learn more about how to bend DTrace to your will in the 5th section of this book. For now you’ll use simple DTrace commands to get system call information out of ptrace.

ptrace

With system calls explained, you’re now going to take a look at the ptrace system call in more depth. The easiest way to describe ptrace is that it enables setting certain debugging related flags for a process that are only accessible from the kernel side. This allows the debugger to catch the debugee if the debugee were to crash. For those interested at exploring the source, look at the ptrace kernel code to see what’s happening and search for references to P_LTRACED.

It’s time to use ptrace for yourself. Open a Terminal console. Before you start, make sure to clear the Terminal console by pressing Command-K. Next execute this DTrace inline script to see how ptrace is called:

sudo dtrace -qn 'syscall::ptrace:entry { printf("%s(%d, %d, %d, %d) from %s\n", probefunc, arg0, arg1, arg2, arg3, execname); }'

Open up the helloptrace application, which you’ll find in the resources folder for this chapter. This is a macOS Terminal command application that does not do much at the moment.

The only thing of interest in this project is a bridging header used to import the ptrace system call API into Swift.

Open main.swift and add the following code to the end of the file:

while true {
  sleep(2)
  print("Hello ptrace!")
}

Next, position Xcode and the DTrace Terminal window you can see them both at the same time. Build and run the application. Once your app has launched observe the output generated by the DTrace script.

Remember from way back in the beginning of the book, that debugserver is the process that LLDB uses to attach to processes. The actual process you’re attaching to is the second parameter of the output, 915 in the screen shot.

Use Open Quickly to read the header for ptrace.h. Press Command-Shift-O and type ptrace.h. Compare the first parameter to ptrace.h header and you’ll see the first parameter, 14, actually stands for PT_ATTACHEXC. What does this PT_ATTACHEXC mean? To get information about this parameter, first, open a Terminal window. Finally, type man ptrace and search for PT_ATTACHEXC.

Note: You can perform case-sensitive searches on man pages by pressing /, followed by your search query. You can search downwards to the next hit by pressing N or upwards to the previous hit by pressing Shift-N.

You’ll find some relevant info about PT_ATTACHEXC with the following output obtained from the ptrace man page:

This request allows a process to gain control of an otherwise unrelated process and begin tracing it. It does not need any cooperation from the to-be-traced process, but the kernel does require the parent process to contain the right privileges. In this case, `pid` specifies the process ID of the to-be-traced process, and the other two arguments are ignored.

With this information, the reason for the first call of ptrace should be clear. This call says “Hey, attach to this process”, and attaches to the process provided in the second parameter.

Onto the next ptrace call from your DTrace output:

ptrace(13, 915, 5635, 0) from debugserver

This one is a bit trickier to understand, since Apple decided to not give any man documentation about this one. This call relates to the internals of a process attaching to another one.

If you look at the ptrace API header, 13 stands for PT_THUPDATE and relates to how the controlling process, in this case, debugserver, handles UNIX signals and Mach messages passed to the controlled process; in this case, helloptrace. The kernel needs to know how to handle signal passing from a process controlled by another process, as in the Signals project from Section 1. The controlling process could say it doesn’t want to send any signals to the controlled process.

This specific ptrace action is an implementation detail of how the Mach kernel handles ptrace internally; there’s no need to dwell on it. Fortunately, there are other documented signals definitely worth exploring through man. One of them is the PT_DENY_ATTACH action, which you’ll learn about now.

Creating Attachment Issues

A process can actually specify it doesn’t want to be attached to by calling ptrace and supplying the PT_DENY_ATTACH argument. This is often used as an anti-debugging mechanism to prevent unwelcome reverse engineers from discovering a program’s internals.

You’ll now experiment with this argument. Open main.swift and add the following line of code before the while loop:

ptrace(PT_DENY_ATTACH, 0, nil, 0)

Build and run, keep on eye on the debugger console and see what happens.

The program will exit and output the following to the debugger console:

Program ended with exit code: 45

Note: You may need to open up the debug console by clicking View ▸ Debug Area ▸ Activate Console or pressing Command-Shift-Y if you’re one of those cool, shortcut devs.

This happened because Xcode launches the helloptrace program by default with LLDB automatically attached. If you execute the ptrace function with PT_DENY_ATTACH, LLDB will exit early and the program will stop executing.

If you were to try and execute the helloptrace program, and tried later to attach to it, LLDB would fail in attaching and the helloptrace program would happily continue execution, oblivious to debugserver’s attachment issues.

There are numerous macOS (and iOS) programs that perform this very action in their production builds. However, it’s rather trivial to circumvent this security precaution. Ninja debug mode activated!

Getting Around PT_DENY_ATTACH

Once a process executes ptrace with the PT_DENY_ATTACH argument, making an attachment greatly escalates in complexity. However, there’s a much easier way of getting around this problem.

Typically a developer will execute ptrace(PT_DENY_ATTACH, 0, 0, 0) somewhere in the main executable’s code — oftentimes, right in the main function.

Since LLDB has the -w argument to wait for the launching of a process, you can use LLDB to “catch” the launch of a process and perform logic to augment or ignore the PT_DENY_ATTACH command before the process has a chance to execute ptrace!

Open a new Terminal window and type the following:

sudo lldb -n "helloptrace" -w

This starts an lldb session and attaches to the helloptrace program, but this time -w tells lldb to wait until a new process with the name helloptrace has started.

You need to use sudo due to an ongoing bug with LLDB and macOS security when you tell LLDB to wait for a Terminal program to launch.

Next, go to the Product menu and select Product ▸ Show Build Folder in Finder.

Next, drag the helloptrace executable into a new Terminal tab. Finally, press Enter to start the executable.

Now, open the previously created Terminal tab, where you had LLDB sit and wait for the helloptrace executable.

If everything went as expected, LLDB will see helloptrace has started and will launch itself, attaching to this newly created helloptrace process.

In LLDB, create the following regex breakpoint to stop on any type of function containing the word ptrace:

(lldb) rb ptrace -s libsystem_kernel.dylib

This will add a breakpoint on the userland gateway to the actual kernel ptrace function. Next, type continue into the Terminal window.

(lldb) continue

You’ll break right before the ptrace function is about to be executed. However, you can simply use LLDB to return early and not execute that function. Do that now like so:

(lldb) thread return 0

Next, simply just continue:

(lldb) continue

Although the program entered the ptrace userland gateway function, you told LLDB to return early and not execute the logic that will execute the kernel ptrace system call.

Note: The “defending” programming side can up the ante and hide the ptrace API by using syscall (i.e. syscall(SYS_ptrace, PT_DENY_ATTACH, 0, nil, 0)). The syscall API is called at a much higher frequency compared to ptrace, which would make breaking on the code of interest more difficult. However, code referencing calls to syscall might also “pop out” to a researcher when dumping the symbol table. One can hide this by dynamically resolving the syscall symbol through dlopen/dlsym, or use less obvious means by resolving syscall. An even better method would be to replicate the required assembly needed to make the ptrace call, completely sidestepping any suspicious references in the symbol table.

Navigate to the Hello ptrace! output tab and verify it’s outputting “helloptrace” over and over. If so, you’ve successfully bypassed PT_DENY_ATTACH and are running lldb while still attached to the helloptrace command!

In a couple chapters, you’ll explore an alternative method to crippling external functions like ptrace by inspecting Mach-O’s __DATA.__la_symbol_ptr section along with the lovely DYLD_INSERT_LIBRARIES environment variable.

Other Anti-Debugging Techniques

Since we’re on the topic of anti-debugging, let’s put iTunes on the spot: for the longest time, iTunes actually used the ptrace’s PT_DENY_ATTACH. However, more recent versions of iTunes has opted for a different technique to prevent debugging: iTunes will now check if it’s being debugged using the powerful sysctl function, then kill itself if true. sysctl is another kernel function (like ptrace) that gets or sets kernel values. iTunes repeatedly calls sysctl while it’s running using a NSTimer to call out to the logic.

Below is a simplified code example in Swift of what iTunes is doing:

let mib = UnsafeMutablePointer<Int32>.allocate(capacity: 4)
mib[0] = CTL_KERN
mib[1] = KERN_PROC
mib[2] = KERN_PROC_PID
mib[3] = getpid()

var size: Int = MemoryLayout<kinfo_proc>.size
var info: kinfo_proc? = nil

sysctl(mib, 4, &info, &size, nil, 0)

if (info.unsafelyUnwrapped.kp_proc.p_flag & P_TRACED) > 0 {
  exit(1)
}

The details of the expected params for sysctl are outside the scope of this chapter that interested readres can explore, but know there’s more than one way to skin a cat.

Key Points

• ptrace is a system call that attaches to other processes.
• Apps can deny ptrace attachments using the PT_DENY_ATTACH argument.

Where to Go From Here?

With the DTrace dumping script you used in this chapter, explore parts of your system and see when ptrace is called.

If you’re feeling cocky, read up on the ptrace man pages and see if you can create a program that will automatically attach itself to another program on your system.

Still have energy? Go man sysctl. That will be some good night-time reading.

Remember, having attachment issues is not always a bad thing!