13. Assembly & the Stack

Written by Derek Selander

In x86_64, when there are more than six parameters passed into a function, the excess parameters are passed through the stack (there’s situations when this is not true, but one thing at a time, young grasshopper). But what does being passed on the stack mean exactly? It’s time to take a deeper dive into what happens when a function is called from an assembly standpoint by exploring some “stack related” registers as well as the contents in the stack.

Understanding how the stack works is useful when you’re reverse engineering programs, since you can help deduce what parameters are being manipulated in a certain function when no debugging symbols are available.

Let’s begin.

The stack, revisited

As discussed previously in Chapter 6, “Thread, Frame & Stepping Around”, when a program executes, the memory is laid out so the stack starts at a “high address” and grows downward, towards a lower address; that is, towards the heap.

Note: In some architectures, the stack grows upwards. But for x64 and ARM for iOS devices, the two you care about, both grow the the stack downwards.

Confused? Here’s an image to help clarify how the stack moves.

The stack starts at a high address. How high, exactly, is determined by the operating system’s kernel. The kernel gives stack space to each running program (well, each thread).

The stack is finite in size and increases by growing downwards in memory address space. As space on the stack is used up, the pointer to the “top” of the stack moves down from the highest address to the lowest address.

Once the stack reaches the finite size given by the kernel, or if it crosses the bounds of the heap, the stack is said to overflow. This is a fatal error, often referred to as a stack overflow. Now you know where your favorite website gets its name from!

Stack pointer & base pointer registers

Two very important registers you’ve yet to learn about are the RSP and RBP. The stack pointer register, RSP, points to the head of the stack for a particular thread. The head of the stack will grow downwards, so the RSP will decrement when items are added to the stack. The RSP will always point to the head of the stack.

Here’s a visual of the stack pointer changing when a function is called.

In the above image, the sequence of the stack pointer follows:

1. The stack pointer currently points to Frame 3.
2. The code pointed to by the instruction pointer register calls a new function. The stack pointer gets updated to point to a new frame, Frame 4, which is potentially responsible for scratchspace and data inside this newly called function from the instruction pointer.
3. Execution is completed in Frame 4 and control resumes back in Frame 3. The stack pointer’s previous reference to Frame 4 gets popped off and resumes pointing to Frame 3

The other important register, the base pointer register (RBP), has multiple uses during a function being executed. Programs use offsets from the RBP to access local variables or function parameters while execution is inside the method/function. This happens because the RBP is set to the value of the RSP register at the beginning of a function in the function prologue.

The interesting thing here is the previous contents of the base pointer are stored on the stack before it’s set to the value of the RSP register. This is the first thing that happens in the function prologue. Since the base pointer is saved onto the stack and set to the current stack pointer, you can traverse the stack just by knowing the value in the base pointer register. A debugger does this when it shows you the stack trace.

Note: Some systems don’t use a base pointer, and it’s possible to compile your application to omit using the base pointer. The logic is it might be beneficial to have an extra register to use. But this means you can’t unwind the stack easily, which makes debugging much harder.

Yeah, an image is definitely needed to help explain.

When a function prologue has finished setting up, the contents of RBP will point to the previous RBP a stack frame lower.

Note: When you jump to a different stack frame by clicking on a frame in Xcode or using LLDB, both the RBP & RSP registers will change values to correspond to the new frame! This is expected because local variables for a function use offsets of RBP to get their values.

If the RBP didn’t change, you’d be unable to print local variables to that function, and the program might even crash. This might result in a source of confusion when exploring the RBP & RSP registers, so always keep this in mind. You can verify this in LLDB by selecting different frames and typing cpx $rbp or cpx $rsp in the LLDB console.

So why are these two registers important to learn about? When a program is compiled with debug information, the debug information references offsets from the base pointer register to get a variable. These offsets are given names, the same names you gave your variables in your source code.

When a program is compiled and optimized for release, the debug information that comes packaged into the binary is removed. Although the names to the references of these variables and parameters are removed, you can still use offsets of the stack pointer and base pointer to find the location of where these references are stored.

Stack related opcodes

So far, you’ve learned about the calling convention and how the memory is laid out, but haven’t really explored what the many opcodes actually do in x64 assembly. It’s time to focus on several stack related opcodes in more detail.

The ‘push’ opcode

When anything such as an int, Objective-C instance, Swift class or a reference needs to be saved onto the stack, the push opcode is used. push decrements the stack pointer (remember, the stack grows downward), then stores the value assigned to the memory address pointed at by the new RSP value.

After a push instruction, the most recently pushed value will be located at the address pointed to by RSP. The previous value would be at RSP plus the size of the most recently pushed value — usually 8 bytes for 64-bit architecture.

To see at a concrete example, consider the following opcode:

push 0x5

This would decrement the RSP, then store the value 5 in the memory address pointed to by RSP. So, in C pseudocode:

RSP = RSP - 0x8 
*RSP = 0x5

The ‘pop’ opcode

The pop opcode is the exact opposite of the push opcode. pop takes the value from the RSP register and stores it to a destination. Next, the RSP is incremented by 0x8 because, again, as the stack gets smaller, it will grow to a higher address.

Below is an example of pop:

pop rdx

This stores the value of the RSP register into the RDX register, then increments the RSP register. Here’s the pseudocode below:

RDX = *RSP
RSP = RSP + 0x8

The ‘call’ opcode

The call opcode is responsible for executing a function. call pushes the address of where to return to after the called function completes; then jumps to the function.

Imagine a function at 0x7fffb34df410 in memory like so:

0x7fffb34de913 <+227>: call   0x7fffb34df410            
0x7fffb34de918 <+232>: mov    edx, eax

When an instruction is executed, first the RIP register is incremented, then the instruction is executed. So, when the call instruction is executed, the RIP register will increment to 0x7fffb34de918, then execute the instruction pointed to by 0x7fffb34de913. Since this is a call instruction, the RIP register is pushed onto the stack (just as if a push had been executed) then the RIP register is set to the value 0x7fffb34df410, the address of the function to be executed.

The pseudocode would look similar to the following:

RIP = 0x7fffb34de918
RSP = RSP - 0x8
*RSP = RIP
RIP = 0x7fffb34df410

From there, execution continues at the location 0x7fffb34df410.

Computers are pretty cool, aren’t they?

The ‘ret’ opcode

The ret opcode is the opposite of the call opcode, in that it pops the top value off the stack (which will be the return address pushed on by the call opcode, provided the assembly’s pushes and pops match) then sets the RIP register to this address. Thus execution goes back to where the function was called from.

Now that you have a basic understanding of these four important opcodes, it’s time to see them in action.

It’s very important to have all push opcodes match your pop opcodes, or else the stack will get out of sync. For example, if there was no corresponding pop for a push, when the ret happened at the end of the function, the wrong value would be popped off. Execution would return to some random place, potentially not even a valid place in the program.

Fortunately, the compiler will take care of synchronizing your push and pop opcodes. You only need to worry about this when you’re writing your own assembly.

Observing RBP & RSP in action

Now that you have an understanding of the RBP and RSP registers, as well as the four opcodes that manipulate the stack, it’s time to see it all in action.

In the Registers application lives a function named StackWalkthrough(int). This C function takes one integer as a parameter and is written in assembly (AT&T assembly, remember to be able to spot the correct location for the source and destination operands) and is located in StackWalkthrough.s. Open this file and have a look around; there’s no need to understand it all just now. You’ll learn how it works in a minute.

This function is made available to Swift through a bridging header Registers-Bridging-Header.h, so you can call this method written in assembly from Swift.

Now to make use of this.

Open ViewController.swift, and add the following below viewDidLoad():

override func awakeFromNib() {
  super.awakeFromNib()
  StackWalkthrough(5)
}

This will call StackWalkThrough with a parameter of 5. The 5 is simply a value used to show how the stack works.

Before exploring RSP and RBP in depth, it’s best to get a quick overview of what is happening in StackWalkthrough. Create a symbolic breakpoint on the StackWalkthrough function.

Once created, build and run.

Xcode will break on StackWalkthrough. Be sure to view the StackWalkthrough function through “source” (even though it’s assembly). Viewing the function through source will showcase the AT&T assembly (because it was written in AT&T ASM).

Xcode will display the following assembly:

push  %rbp       ; Push contents of RBP onto the stack (*RSP = RBP, RSP decreases)

movq  %rsp, %rbp ; RBP = RSP
movq  $0x0, %rdx ; RDX = 0
movq  %rdi, %rdx ; RDX = RDI
push  %rdx       ; Push contents of RDX onto the stack (*RSP = RDX, RSP decreases)

movq  $0x0, %rdx ; RDX = 0
pop   %rdx       ; Pop top of stack into RDX (RDX = *RSP, RSP increases)

pop   %rbp       ; Pop top of stack into RBP (RBP = *RSP, RSP increases)

ret              ; Return from function (RIP = *RSP, RSP increases)

Comments have been added to help understand what’s happening. Read it through and try to understand it if you can. You’re already familiar with the mov instruction, and the rest of the assembly consists of function related opcodes you’ve just learned about.

This function takes the integer parameter passed into it (as you’ll recall, the first parameter is passed in RDI), stores it into the RDX register, and pushes this parameter onto the stack. RDX is then set to 0x0, then the value popped off the stack is stored back into the RDX register.

Make sure you have a good mental understanding of what is happening in this function, as you’ll be exploring the registers in LLDB next.

Back in Xcode, create a breakpoint using Xcode’s GUI on the StackWalkthrough(5) line in the awakeFromNib function of ViewController.swift. Leave the previous StackWalkthrough symbolic breakpoint alive, since you’ll want to stop at the beginning of the StackWalkthrough function when exploring the registers.

Build and run and wait for the GUI breakpoint to trigger.

Now click Debug ▸ Debug Workflow ▸Always Show Disassembly, to show the disassembly. You’ll be greeted with scary looking stuff!

Wow! Look at that! You’ve landed right on a call opcode instruction. Do you wonder what function you’re about to enter?

Note: If you didn’t land right on the call instruction using the Xcode GUI breakpoint, you can either use LLDB’s thread step-inst or more simply, si to single step through assembly instructions. Alternatively, you can create a GUI breakpoint on the memory address that calls the StackWalkthrough function.

From here on out, you’ll step through every assembly instruction while printing out four registers of interest: RBP, RSP, RDI and RDX. To help with this, type the following into LLDB:

(lldb) command alias dumpreg register read rsp rbp rdi rdx

This creates the command dumpreg that will dump the four registers of interest. Execute dumpreg now:

(lldb) dumpreg

You’ll see something similar to the following:

rsp = 0x00007fff5fbfe820
rbp = 0x00007fff5fbfe850
rdi = 0x0000000000000005
rdx = 0x0040000000000000

For this section, the output of dumpreg will be overlaid on each assembly instruction to show exactly what is happening with each of the registers during each instruction. Again, even though the values are provided for you, it’s very important you execute and understand these commands yourself.

Your screen will look similar to the following:

Once you jump into the function call, keep a very close eye on the RSP register, as it’s about to change once RIP jumps to the beginning of StackWalkthrough. As you’ve learned earlier, the RDI register will contain the value for the first parameter, which is 0x5 in this case.

In LLDB, type the following:

(lldb) si

This is an alias for thread step-inst, which tells LLDB to execute the next instruction and then pause the debugger.

You’ve now stepped into StackWalkthrough. Again for each step, dump out the registers using dumpreg.

Take note of the difference in the RSP register. The value pointed at by RSP will now contain the return address to the previous function. For this particular example, RSP, which points to 0x7fff5fbfe758, will contain the value 0x100002455 — the address immediately following the call in awakeFromNib.

Verify this now through LLDB:

(lldb) x/gx $rsp 

The output will match the address immediately following the call opcode in awakeFromNib.

Next, perform an si, then dumpreg for the next instruction.

The value of RBP is pushed onto the stack. This means the following two commands will produce the same output. Execute both of them to verify.

(lldb) x/gx $rsp 

This looks at the memory address pointed at by the stack pointer register.

Note: Wait, I just threw a new command at you with no context. The x command is a shortcut for the memory read command.

The /gx says to format the memory in a giant word (8 bytes, remember that terminology from Chapter 12, “Assembly & Memory”?) in hexadecimal format.

The weird formatting is due to the popularity of this command in gdb, which saw this command syntax ported into lldb to make the transition from debuggers easier.

Now look at the value in the base pointer register.

(lldb) p/x $rbp

Next, step into the next instruction, using si again:

The base pointer is assigned to the value of the stack pointer. Verify both have the same value using dumpreg as well as the following LLDB command:

(lldb) p (BOOL)($rbp == $rsp)

It’s important you put parentheses around the expression, else LLDB won’t parse it correctly.

Execute si and dumpreg again. This time it looks like the following:

RDX is cleared to 0.

Execute si and dumpreg again. This time the output looks the following:

RDX is set to RDI. You can verify both have the same value with dumpreg again.

Execute si and dumpreg. This time it looks the following:

RDX is pushed onto the stack. This means the stack pointer was decremented, and RSP points to a value which will point to the value of 0x5. Confirm that now:

(lldb) p/x $rsp 

This gives the current value pointed at RSP. What does the value here point to?

(lldb) x/gx $rsp 

You’ll get the expected 0x5. Type si again to execute the next instruction:

RDX is set to 0x0. Nothing too exciting here, move along… move along. Type si and dumpreg again:

The top of the stack is popped into RDX, which you know was recently set to 0x5. The RSP is incremented by 0x8. Type si and dumpreg again:

The base pointer is popped off of the stack and reassigned back to the value it originally had when entering this function. The calling convention specifies RBP should remain consistent across function calls. That is, the RBP can’t change to a different value once it leaves a function, so we’re being a good citizen and restoring its value.

Onto the ret opcode. Keep an eye out for the RSP value about to change. Type si and dumpreg again:

The return address was pushed off the stack and set to the RIP register; you know this because you’ve gone back to where the function was called. Control then resumes in awakeFromNib,

Wowza! That was fun! A simple function, but it illustrates how the stack works through call, push, pop and ret instructions.

The stack and 7+ parameters

As described in Chapter 11, the calling convention for x86_64 will use the following registers for function parameters in order: RDI, RSI, RDX, RCX, R8, R9. When a function requires more than six parameters, the stack needs to be used.

Note: The stack may also need to be used when a large struct is passed to a function. Each parameter register can only hold 8 bytes (on 64-bit architecture), so if the struct needs more than 8 bytes, it will need to be passed on the stack as well. There are strict rules defining how this works in the calling convention, which all compilers must adhere to.

Open ViewController.swift and find the function named executeLotsOfArguments(one:two:three:four:five:six:seven:eight:nine:ten:). You used this function in Chapter 11 to explore the registers. You’ll use it again now to see how parameters 7 and beyond get passed to the function.

Add the following code to the end of viewDidLoad:

_ = self.executeLotsOfArguments(one: 1, two: 2, three: 3,
                                four: 4, five: 5, six: 6,
                                seven: 7, eight: 8, nine: 9,
                                ten: 10)

Next, using the Xcode GUI, create a breakpoint on the line you just added. Build and run the app, and wait for this breakpoint to hit. You should see the disassembly view again, but if you don’t, use the Always Show Disassembly option.

As you’ve learned in the Stack Related Opcodes section, call is responsible for the execution of a function. Since there’s only one call opcode between where RIP is right now and the end of viewDidLoad, this means this call must be the one responsible for calling executeLotsOfArguments(one:two:three:four:five:six:seven: eight:nine:ten:).

But what are all the rest of the instructions before call? Let’s find out.

These instructions set up the stack as necessary to pass the additional parameters. You have your usual 6 parameters being put into the appropriate registers, as seen by the instructions before where RIP is now, from mov edx, 0x1 onwards.

But parameters 7 and beyond need to be passed on the stack. This is done with the following instructions:

0x1000013e2 <+178>: mov    qword ptr [rsp], 0x7
0x1000013ea <+186>: mov    qword ptr [rsp + 0x8], 0x8
0x1000013f3 <+195>: mov    qword ptr [rsp + 0x10], 0x9
0x1000013fc <+204>: mov    qword ptr [rsp + 0x18], 0xa

Looks scary, doesn’t it? I’ll explain.

The brackets containing RSP and an optional value indicate a dereference, just like a * would in C programming. The first line above says “put 0x7 into the memory address pointed to by RSP.” The second line says “put 0x8 into the memory address pointed to by RSP plus 0x8.” And so on.

This is placing values onto the stack. But take note the values are not explicitly pushed using the push instruction, which would decrease the RSP register. Why is that?

Well, as you’ve learned, during a call instruction the return address is pushed onto the stack. Then, in the function prologue, the base pointer is pushed onto the stack, and then the base pointer gets set to the stack pointer.

What you haven’t learned yet is the compiler will actually make room on the stack for “scratch space”. That is, the compiler allocates space on the stack for local variables in a function as necessary.

You can easily determine if extra scratch space is allocated for a stack frame by looking for the sub rsp, VALUE instruction in the function prologue. For example, click on the viewDidLoad stack frame and scroll to the top. Observe how much scratch space has been created:

The compiler has been a little bit clever here; instead of doing lots of pushes, it knows it has allocated some space on the stack for itself, and fills in values before the function call passing these extra parameters. Individual push instructions would involve more writes to RSP, which would be less efficient.

Time to look at this scratch space in more depth.

The stack and debugging info

The stack is not only used when calling functions, but it’s also used as a scratch space for a function’s local variables. Speaking of which, how does the debugger know which addresses to reference when printing out the names of variables that belong to that function?

Let’s find out!

Clear all the breakpoints you’ve set and create a new Symbolic breakpoint on executeLotsOfArguments.

Build and run the app, then wait for the breakpoint to hit.

As expected, control should stop at the ever-so-short name of a function: executeLotsOfArguments(one:two:three:four:five:six:seven:eight:nine:ten:), from here on, now referred to as executeLotsOfArguments, because its full name is a bit of a mouthful!

In the lower right corner of Xcode, click on Show the Variables View:

From there, look at the value pointed at by the one variable… it definitely ain’t holding the value of 0x1 at the moment. This value seems to be gibberish!

Why is one referencing a seemingly random value?

The answer is stored by the DWARF Debugging Information embedded into the debug build of the Registers application. You can dump this information to help give you insight into what the one variable is referencing in memory.

In LLDB, type the following:

(lldb) image dump symfile Registers

You’ll get a crazy amount of output. Search for (Cmd + F) the word “one”; include the quotes within your search.

Below is a (very) truncated output that includes the relevant information:

Swift.String, type_uid = 0x300000222
0x7f9b4633a988:     Block{0x300000222}, ranges = [0x1000035e0-0x100003e7f)
0x7f9b48171a20:       Variable{0x30000023f}, name = "one", type = {d50e000003000000} 0x00007f9b4828d2a0 (Swift.Int), scope = parameter, decl = ViewController.swift:39, location =  DW_OP_fbreg(-32)

Based upon the output, the variable named one is of type Swift.Int, found in executeLotsOfArguments, whose location can be found at DW_OP_fbreg(-32). This rather obfuscated code actually means base pointer minus 40, i.e. RBP - 32. Or in hexadecimal, RBP - 0x20.

This is important information. It tells the debugger the variable called one can always be found in this memory address. Well, not always, but always when that variable is valid, i.e. it’s in scope.

You may wonder why it can’t just be RDI, since that’s where the value is passed to the function, and it’s also the first parameter. Well, RDI may need to be reused later on within the function, so using the stack is a safer bet for storage.

The debugger should still be stopped on executeLotsOfArguments. Make sure you’re viewing the Always Show Disassembly output and hunt for the assembly:

mov    qword ptr [rbp - 0x20], rdi

Once you’ve found it in the assembly output of executeLotsOfArguments, create a breakpoint on this line of assembly.

Continue execution so LLDB will stop on this line of assembly.

Try printing out the output of one in LLDB:

(lldb) po one

Gibberish, still. Hmph.

Remember, RDI will contain the first parameter passed into the function. So to make the debugger be able to see the value that one should be, RDI needs to be written to the address where one is stored. In this case, RBP - 0x20.

Now, perform an assembly instruction step in LLDB:

(lldb) si

Print the value of one again.

(lldb) po one

Awwww…. yeah! It’s working! The value one is referencing is correctly holding the value 0x1.

You may be wondering what happens if one changes. Well, RBP - 0x20 needs to change in that case too. This would potentially be another instruction needed to write it there as well as wherever the value is used. This is why debug builds are so much slower than release builds.

Stack exploration takeaways

Don’t worry. This chapter is almost done. But there are some very important takeaways that should be remembered from your stack explorations.

Provided you’re in a function, and the function has finished executing the function prologue, the following items will hold true to x64 assembly:

• RBP will point to the start of the stack frame for this function.
• *RBP will contain the address of the start of the previous stack frame. (Use x/gx $rbp in LLDB to see it).
• *(RBP + 0x8) will point to the return address to the previous function in the stack trace (Use x/gx '$rbp + 0x8' in LLDB to see it).
• *(RBP + 0x10) will point to the 7th parameter (if there is one).
• *(RBP + 0x18) will point to the 8th parameter (if there is one).
• *(RBP + 0x20) will point to the 9th parameter (if there is one).
• *(RBP + 0x28) will point to the 10th parameter (if there is one).
• RBP - X where X is multiples of 0x8, will reference local variables to that function.

Where to go from here?

Now that you’re familiar with the RBP and RSP registers, you’ve got a homework assignment!

Attach LLDB to a program (any program, source or no source) and traverse the stack frame using only the RBP register. Create a breakpoint on an easily triggerable method. One good example is -[NSView hitTest:], if you attach to a macOS application such as Xcode, and click on a view.

It’s important to ensure the breakpoint you choose to add is not a Swift function. You’re going to inspect registers, — and recall you can’t (easily) do this in the Swift context.

Once the breakpoint has been triggered, make sure you’re on frame 0 by typing the following into LLDB:

(lldb) f 0 

The f command is an alias for frame select.

You should see the following two instructions at the top of this function:

push   rbp
mov    rbp, rsp

These instructions form the start of the function prologue and push RBP onto the stack and then set RBP to RSP.

Step over both of these instructions using si.

Now the base pointer is set up for this stack frame, you can traverse the stack frames yourself by inspecting the base pointer.

Execute the following in LLDB:

(lldb) p uintptr_t $Previous_RBP = *(uintptr_t *)$rsp

So now $Previous_RBP equals the old RBP, i.e. the start of the stack frame from the function that called this one.

Recall the first thing on the stack frame is the address to where the function should return. So you can find out where the previous function will return to. This will therefore be where the debugger is stopped in Frame 2.

To find this out and check that you’re right, execute the following in LLDB:

(lldb) x/gx '$Previous_RBP + 0x8'

This will print something like this:

0x7fff5fbfd718: 0x00007fffa83ed11b

Confirm this address equals the return address in Frame 1 with LLDB:

(lldb) f 2

It will look something like this, depending on what you decided to set the initial breakpoint in:

frame #2: 0x00007fffa83ed11b AppKit`-[NSWindow _setFrameCommon:display:stashSize:] + 3234
AppKit`-[NSWindow _setFrameCommon:display:stashSize:]:
    0x7fffa83ed11b <+3234>: xor    ebx, ebx
    0x7fffa83ed11d <+3236>: mov    rsi, qword ptr [rip + 0x1c5a9d8c] ; "_bindingAdaptor"
    0x7fffa83ed124 <+3243>: mov    rdi, r12
    0x7fffa83ed127 <+3246>: call   qword ptr [rip + 0x1c319f53] ; (void *)0x00007fffbee77b40: objc_msgSend

The first address that it spits out should match the output of your earlier x/gx command.

Good luck and may the assembly be with you!