27. SB Examples, Malloc Logging

Written by Walter Tyree

For the final chapter in this section, you’ll go through the same steps I myself took to understand how the MallocStackLogging environment variable is used to get the stack trace when an object is created.

From there, you’ll create a custom LLDB command which gives you the stack trace of when an object was allocated or deallocated in memory — even after the stack trace is long gone from the debugger.

Knowing the stack trace of where an object was created in your program is not only useful for reverse engineering, but also has great use cases in your typical day-to-day debugging. When a process crashes, it’s incredibly helpful to know the history of that memory and any allocation or deallocation events that occurred before your process went off the deep end.

This is another example of a script using stack-related logic, but this chapter will focus on the complete cycle of how to explore, learn, then implement a rather powerful custom command.

Setting Up the Scripts

You have a couple of scripts to use (and implement!) for this chapter. Let’s go through each one of them and how you’ll use them:

• msl.py: This is the command (which is an abbreviation for MallocStackLogging) script you’ll be working on in this chapter. This has a basic skeleton of the logic.
• lookup.py: Wait — you already made this command, right? Yes, but I’ll give you my own version of the lookup command that adds a couple of additional options at the price of uglier code. You’ll use one of the options to filter your searches to specific modules within a process.
• sbt.py: This command takes a backtrace with unsymbolicated symbols, and symbolicate it. You made this in the previous chapter, and you’ll need it at the very end of this chapter. And in case you didn’t work through the previous chapter, it’s included in this chapter’s resources for you to install.

Note: These scripts are also in Appendix C “Helpful Python Scripts” Check it out for some other novel ideas for LLDB scripts. It’s important to note that a lot of scripts in Appendix C have dependencies on other files, so if you try to use only one script then it might not compile until the full set of files are included.

Now for the usual setup. Take all the Python files found in the starter directory for this chapter and copy them into your ~/lldb directory. I am assuming you have the lldbinit.py file already set up, found in Chapter 25, “SB Examples, Improved Lookup.”

Launch an LLDB session in Terminal and go through all the help commands to make sure each script has loaded successfully:

(lldb) help msl
(lldb) help lookup
(lldb) help sbt

MallocStackLogging Explained

In case you’re unfamiliar with the MallocStackLogging environment variable, when the MallocStackLogging environment variable is set to true, it’ll monitor and record allocations and deallocations of memory on the heap. Pretty neat!

Included within the starter directory is the 50 Shades of Ray Xcode project from the last chapter with some additional logic for this chapter. Open the project.

Before you run it, you’ll need to modify the scheme for your purposes. Select the 50 Shades of Ray scheme (make sure there’s no “Stripped” in the name), then press Command-Shift-< to edit the scheme.

1. Select Run.
2. Select Diagnostics.
3. Select Malloc Stack Logging, then All Allocation and Free History.

Once you’ve enabled this environment variable, build the 50 Shades of Ray program and run it.

If the MallocStackLogging environment variable is enabled, you’ll see some output from the LLDB console similar to the following:

ShadesOfRay(12911,0x104e663c0) malloc: stack logs being written into /tmp/stack-logs.12911.10d42a000.ShadesOfRay.gjehFY.index

ShadesOfRay(12911,0x104e663c0) malloc: recording malloc and VM allocation stacks to disk using standard recorder

ShadesOfRay(12911,0x104e663c0) malloc: process 12673 no longer exists, stack logs deleted from /tmp/stack-logs.12673.11b51d000.ShadesOfRay.GVo3li.index

Don’t worry about the details of the output; simply look for the presence of output like this as it indicates the MallocStackLogging is working properly.

While the app is running, click the Generate a Ray button at the bottom.

Once a new Ray is created (that is, you see an instance of Ray Wenderlich’s amazingly innovative & handsome face pop up in the Simulator), perform the following steps:

1. Select the Debug Memory Graph located at the top of the LLDB console in Xcode.
2. Select the Show the Debug navigator in the left panel.
3. At the bottom of the left panel select the Show only content from workspace.
4. Select the memory address that references the RayView.
5. In the right panel of Xcode, make sure the Show the Memory Inspector is selected.

Once you’ve jumped through all those hoops, you’ll have the exact stack trace of where this RayView instance was created through the Backtrace section on the right side of the Xcode window. How cool is that?! The authors of Xcode (and its many modules) have made our lives a bit easier with these memory debugging features!

Plan of Attack

You’ve seen it’s possible to grab a stack trace for an instantiated object, but you’re going to do one better than Apple.

Your command will be able to turn on the MallocStackLogging functionality at will through LLDB, which means you won’t have to rely on an environment variable. This has the additional benefit that you won’t need to restart your process in case you forget to turn it on during a debug session.

So how are you going to figure out how this MallocStackLogging feature works?

When I am absolutely clueless as to where to begin when exploring built-in code, I follow the rather loose process below and alter queries, depending on the scenario or the output:

• I look for chokepoints where I can safely assume some logic of interest will be executed. If I know I can replicate something of interest, I’ll force that action to occur while monitoring it.
• When monitoring the code of interest, I’ll use various tools like LLDB or DTrace (which you’ll learn about in the next section) to find the module holding the code of interest. Again, a module is a dynamic library, framework, NSBundle, or something of that sort.
• Once I find the module of interest, I’ll dump all the code from the module, then filter for what I need using various custom scripts like lookup.py.
• If I find a particular function that looks relevant to my interests, I’ll first try a web search for it. I’ll often find some incredibly useful hints on Apple’s open-source code site that reveal how I can use what I’ve found.
• Searching through Apple’s open-source URLs, I’ll grab as much context as I can about the code of interest. Sometimes there’s code in the C/C++ source file that will give me an idea of how to formulate the parameters into the function, or perhaps I’ll get a description of the code or its purpose in the header file.
• If there’s no documentation to be gained from a web search, I’ll set breakpoints on the code of interest and see if I can trigger that function naturally. Once hit, I’ll explore both the stack frames and registers to see what kind of parameters are being passed in, as well as the context.

You’re going to follow these same steps to see where the code for MallocStackLogging resides, explore the module responsible for handling stack tracing logic, then explore any interesting code of interest within that module.

Let’s get cracking!

Hunting for a Starting Point

As you just saw, Xcode provides a special backtrace for any object that gets allocated when MallocStackLogging is enabled. Go ahead and build and run the app and then tap on Generate a Ray! a few times to create some instances of RayView. Now use the Debug Memory Graph a few times to stop the app. Now inspect some of the RayView instances to look for patterns.

One thing you might notice is that ALL of the malloc stack traces in the Memory inspector have in frame 0 something like this:

_malloc_zone_calloc_instrumented_or_legacy

In the LLDB console, use the lookup command to see if you can find that anywhere:

(lldb) lookup _malloc_zone_calloc_instrumented_or_legacy

From the output, you can see that there is a .dylib that holds that symbol:

****************************************************
1 hits in: libsystem_malloc.dylib
****************************************************
_malloc_zone_calloc_instrumented_or_legacy

Note: your stack trace might show _malloc_zone_calloc and not the _malloc_zone_calloc_instrumented_or_legacy in frame 0.

The module name, libsystem_malloc.dylib fits the bill for something implementing malloc stack logging related logic. Is this it? Maybe. Worth checking out? Totally!

Take a deeper dive into this module and see what it has to offer you.

Using lookup command, explore all the methods implemented by the libsystem_malloc.dylib module that you can execute within your process.

(lldb) lookup . -m libsystem_malloc.dylib

In iOS 16.0, I get 593 hits. I could gloss through all these methods, but I am getting increasingly lazy as a debugger person. Let’s just hunt for everything that pertains to the word “log” (for logging) and see what we get. Type the following in LLDB:

(lldb) lookup [lL]og -m libsystem_malloc.dylib

I get 26 hits from using a case insensitive search for the word log inside the libsystem_malloc.dylib module.

This hit count is bearable enough to weed through. Another way you could have filtered down the 593 hits would be to use the Filter in the bottom right of the LLDB console in Xcode. Just remember to clear the filter when you’re done, or you’ll go crazy wondering why all of your lldb commands aren’t producing output.

Do any of those functions look interesting? Hell yeah! Here are some of the following functions that look interesting to me:

__mach_stack_logging_get_frames

__mach_stack_logging_get_frames_for_stackid

turn_off_stack_logging

turn_on_stack_logging

_malloc_register_stack_logger

Of these, the turn_on_stack_logging and the __mach_stack_logging_get_frames look like they’re worth checking out.

You’ve found the module of interest, as well as some functions worth further exploration. Time to jump out on the Internet and see what’s out there.

Googling JIT Function Candidates

Google for any code pertaining to turn_on_stack_logging. Take a look at this search query:

At the time I wrote this, I got three hits from Google (well, it was actually eight hits with “exclude similar searches” off, but that’s not the point).

These functions are not well-known and are not typically discussed in any circle outside of Apple. In fact, I am rather confident the majority of iOS application developers in Apple don’t know about them either, because when would they use them for writing apps?

This stuff belongs to the low-level C developers of Apple, whom we totally take for granted.

From the Google search, check out the following code from the libmalloc header file at Apple’s open-source site:

typedef enum {
  stack_logging_mode_none = 0,
  stack_logging_mode_all,
  stack_logging_mode_malloc,
  stack_logging_mode_vm,
  stack_logging_mode_lite
} stack_logging_mode_type;

extern boolean_t turn_on_stack_logging(stack_logging_mode_type mode);

This is some really good information to work with. The turn_on_stack_logging function expects one parameter of type int (C enum). The enum stack_logging_mode_type tells you if you want the stack_logging_mode_all option, it will be at value 1.

You’ll run an experiment by turning off the stack logging environment variable, execute the above function via LLDB, and see if Xcode is recording stack traces for any malloc’d object after you’ve called turn_on_stack_logging.

Before you do that, you’ll first explore the other function, __mach_stack_logging_get_frames.

Exploring __mach_stack_logging_get_frames

Fortunately, for your exploration efforts, __mach_stack_logging_get_frames can also be found in the same header file. This function signature looks like the following:

extern kern_return_t __mach_stack_logging_get_frames(
                                        task_t task,   
                          mach_vm_address_t address,
             mach_vm_address_t *stack_frames_buffer,
                          uint32_t max_stack_frames,
                                   uint32_t *count);
    /* Gets the last allocation record (malloc, realloc, or free) about address */

This is a good starting point, but what if there are parameters you’re not 100% sure how to obtain? For example, what’s task_t task all about? This is basically a parameter that specifies the process you want this function to act on. But what if you didn’t know that?

Using Google and searching for any implementation files that contain __mach_stack_logging_get_frames can be a big help when you’re uncertain about things like this.

After a casual Googling, the heap_find.cpp URL provides insight to the first parameter that’s expected within this function.

This file contains the following code:

task_t task = mach_task_self();
/* Omitted code.... */
    stack_entry->address = addr;
    stack_entry->type_flags = stack_logging_type_alloc;
    stack_entry->argument = 0;
    stack_entry->num_frames = 0;
    stack_entry->frames[0] = 0;

    err = __mach_stack_logging_get_frames(task,
                       (mach_vm_address_t)addr,
                           stack_entry->frames,
                                    MAX_FRAMES,
                      &stack_entry->num_frames);

    if (err == 0 && stack_entry->num_frames > 0) {
      // Terminate the frames with zero if there is room
      if (stack_entry->num_frames < MAX_FRAMES)
        stack_entry->frames[stack_entry->num_frames] = 0;
    } else {
      g_malloc_stack_history.clear();
    }
  }
}

The task_t parameter has an easy way to get the task representing the current process through the mach_task_self function located in libsystem_kernel.dylib. You can confirm this yourself with the lookup LLDB command.

Testing the Functions

To prevent you from getting bored to tears, I’ve already implemented the logic for the __mach_stack_logging_get_frames inside the app.

Hopefully, you still have the application running. If not, get the app running with MallocStackLogging still enabled.

It’s always a good idea to build your proof-of-concept JIT code in Xcode first, and once it’s working, then (and only then!) transfer it to your LLDB script. You’re gonna hate your life if you try to write your POC JIT script code straight in LLDB first. Trust me.

In Xcode, navigate to the stack_logger.cpp file. __mach_stack_logging_get_frames was written in C++, so you’ll need to use C++ code to execute it.

The only function in this file is trace_address:

void trace_address(mach_vm_address_t addr) {

  typedef struct LLDBStackAddress {
    mach_vm_address_t *addresses;
    uint32_t count = 0;
  } LLDBStackAddress;   // 1

  LLDBStackAddress stackaddress; // 2
  __unused mach_vm_address_t address = (mach_vm_address_t)addr;
  __unused task_t task = mach_task_self_;  // 3

  stackaddress.addresses = (mach_vm_address_t *)calloc(100,
                                sizeof(mach_vm_address_t)); // 4

  __mach_stack_logging_get_frames(task,
                               address,
                stackaddress.addresses,
                                   100,
                  &stackaddress.count); // 5

  // 6
  for (int i = 0; i < stackaddress.count; i++) {

    printf("[%d] %llu\n", i, stackaddress.addresses[i]);
  }

  free(stackaddress.addresses); // 7
}

Breakdown time!

1. As you know, LLDB only lets you return one object to be evaluated. But, as a creative string-theory version of yourself, can create C structs that contain any types you want to be returned.
2. Declare an instance of said struct for use within the function.
3. Remember mach_task_self that was referenced earlier? The global variable mach_task_self_ is the value returned when calling mach_task_self.
4. Since you’re in a lower level, you don’t have ARC to help you allocate items on the heap. You’re allocating 100 mach_vm_address_t’s, which is more than enough to handle any stack trace.
5. The __mach_stack_logging_get_frames then executes. The addresses array of the LLDBStackAddress struct will be populated with the addresses if there’s any stack trace information available.
6. Print out all the addresses that it found
7. Finally, free the mach_vm_address_t objects you created.

Time to give it a whirl!

LLDB Testing

Make sure the app is running, then tap the Generate a Ray! button a few times. Click the Debug Memory Graph button again to pause the app and bring up the graph.

I have three wondrously magical Ray Wenderlich faces on my simulator, so I get the following output:

Grab any one of those addresses and execute the logic in the trace_address function:

(lldb) po trace_address(0x152d047c0)

You’ll get output that looks like the following truncated snippet:

[0] 4362273848
[1] 4346078816
[2] 4340224704
[3] 4711642824
[4] 4701717412
[5] 4701577380
[6] 4701577128
[7] 4711642824
[8] 4705047740
[9] 4705048576
...

Note: if you didn’t get any output, ensure that you’ve set the Malloc Stack Logging to All Allocations and Free History.

These are the actual addresses of the code where this object is created. Verify the first address is code in memory using image lookup:

(lldb) image lookup -a 4362273848

You’ll get the details about that function:

Address: libsystem_malloc.dylib[0x000000000000f485] (libsystem_malloc.dylib.__TEXT.__text + 56217)
Summary: libsystem_malloc.dylib`calloc + 30

There’s more than one way to skin a memory address. Now use SBAddress to get the information out of this address:

(lldb) script print lldb.SBAddress(4454012240, lldb.target)

You’ll get stack frame 0 in a slightly different format, like so:

libsystem_malloc.dylib`_malloc_zone_calloc_instrumented_or_legacy + 220

Knowing different ways to get the same data can come in handy when you’re writing scripts and are in different contexts with different objects available.

Navigating a C Array With lldb.value

You’ll again use the lldb.value class to parse the return value of this C struct which was generated inline while executing this function.

Set a GUI breakpoint at the end of the trace_address function.

Use LLDB to execute the same function, but honor breakpoints, and remember to replace the address with one of your RayView instances:

(lldb) e -lobjc++ -O -i0 -- trace_address(0x00007fa838414330)

Execution will stop on the final line of trace_address. You know the drill. Grab the reference to the C struct LLDBStackAddress, stackaddress.

(lldb) script print (lldb.frame.FindVariable('stackaddress'))

If successful, you’ll get the synthetic format of the stackaddress variable:

(LLDBStackAddress) stackaddress = {
  addresses = 0x00007fa838515cd0
  count = 30
}

Cast this struct into a lldb.value and call the reference a:

(lldb) script a = lldb.value(lldb.frame.FindVariable('stackaddress'))

Ensure a is valid:

(lldb) script print (a)

You can now easily reference the variables you declared in the LLDBStackAddress struct inside the lldb.value. Type the following into LLDB:

(lldb) script print (a.count)

You’ll get the stack frame count:

(uint32_t) count = 30

What about the addresses array inside the LLDBStackAddress struct?

(lldb) script print (a.addresses[0])

That’s the memory address of the first frame. What about that generateRayViewTapped: method found in frame 2?

(lldb) script print (a.addresses[2])

You’ll get something similar to:

(mach_vm_address_t) [2] = 4454012240

Do you see how this tool is coming together? From finding chokepoints of items of interest, to exploring code in modules, to researching tidbits of useful information in Apple’s open-source site, to implementing proof of concepts in Xcode before jumping to LLDB Python code, there’s a lot of power under the hood.

Don’t slow down — it’s command implementin’ time!

Turning Numbers Into Stack Frames

Included within the starter directory for this chapter is the msl.py script for malloc script logging. You’ve already installed this msl.py script earlier in the “Setting up the scripts” section.

Unfortunately, this script doesn’t do much at the moment, as it doesn’t produce any output. Time to change that.

Open up ~/lldb/msl.py in your favorite editor. Find handle_command and add the following code to it:

command_args = shlex.split(command)
parser = generateOptionParser()
try:
    (options, args) = parser.parse_args(command_args)
except:
    result.SetError(parser.usage)
    return

cleanCommand = args[0]
process = debugger.GetSelectedTarget().GetProcess()
frame = process.GetSelectedThread().GetSelectedFrame()
target = debugger.GetSelectedTarget()

All this logic shouldn’t be new to you, as it’s the “preamble” required to start up the command. The only thing of interest is you opted to omit the posix=False argument that’s sometimes used in the shlex.split(command). There’s no need to provide this parameter, since this command won’t be handling any weird backslash or dash characters. This means the parsing of the output from the options and args variables is much cleaner as well.

Now that you have the basic script going, add the following (meat of this script) right below the code you just wrote:

# 1
script = generateScript(cleanCommand, options)

# 2
sbval = frame.EvaluateExpression(script, generateOptions())

# 3
if sbval.error.fail:
    result.AppendMessage(str(sbval.error))
    return

val = lldb.value(sbval)
addresses = []

# 4
for i in range(val.count.sbvalue.unsigned):
    address = val.addresses[i].sbvalue.unsigned
    sbaddr = target.ResolveLoadAddress(address)
    loadAddr = sbaddr.GetLoadAddress(target)
    addresses.append(loadAddr)

# 5
retString = processStackTraceStringFromAddresses(
                                        addresses,
                                           target)

# 6
freeExpr = 'free('+str(val.addresses.sbvalue.unsigned)+')'
frame.EvaluateExpression(freeExpr, generateOptions())
result.AppendMessage(retString)

Here are the items of interest:

1. Use the generateScript function I supplied, which returns a string containing roughly the same code as in the trace_address function.
2. Execute the code. You know this will return an SBValue.
3. Do a sanity check to see if the EvaluateExpression fails. If it does, dump out the error and exit early.
4. This for-loop enumerates the memory addresses in the val object, which are the output of the script code, and pulls them out into the addresses list.
5. Now that the addresses are pulled out into a list, you pass that list to a predefined function for processing. This returns the stack trace string you’ll spit out.
6. Finally, you manually free memory, as you’re a good memory citizen and always clean up after yourself. Most of these scripts you’ve written leak memory, but now that you’re getting more advanced with this stuff, it’s time to do the right thing and free any allocated memory.

Jump back to the Xcode LLDB console and reload your stuff:

(lldb) reload_script

Provided you have no errors, grab a reference to a RayView from the memory graph. Once you have a reference to a RayView, run your newly created msl command on it, like so:

(lldb) msl 0x00007fa838414330

You’ll get your expected output just like in Xcode!

frame #0 : 0x11197d485 libsystem_malloc.dylib`calloc + 30
frame #1 : 0x10d3cbba1 libobjc.A.dylib`class_createInstance + 85
frame #2 : 0x10d3d5de4 libobjc.A.dylib`_objc_rootAlloc + 42
frame #3 : 0x10cde7550 ShadesOfRay`-[ViewController generateRayViewTapped:] + 64
frame #4 : 0x10e512d22 UIKit`-[UIApplication sendAction:to:from:forEvent:] + 83

Congratulations! You’ve created a script that gives you the stack trace for an object. Now it’s time to level up and give this script some cool options!

Stack Trace From a Swift Object

OK — I know you want me to talk about Swift code. You’ll cover a Swift example as well.

Included in the 50 Shades of Ray app is a Swift module, ingeniously named SomeSwiftModule. Within this module is a class named SomeSwiftCode with a static variable to get your singleton quota going.

The code in SomeSwiftCode.swift is about as simple as you can get:

public final class SomeSwiftCode {
  private init() {}
  static let shared = SomeSwiftCode()
}

You’ll use LLDB to call this singleton and examine the stack trace where this function was created.

First off, you have to import your Swift modules! Enter the following into LLDB:

(lldb) e -lswift -O -- import SomeSwiftModule

You’ll get no result if the above was successful.

In LLDB, access the singleton, like so:

(lldb) e -lswift -O -- SomeSwiftCode.shared

You’ll get the address to this object:

<SomeSwiftCode: 0x600000033640>

Now you’ll pass this address in to the msl command. Use the msl command on this address:

(lldb) msl 0x600000033640

You’ll get your expected stack trace.

Let’s jump to one final topic I want to discuss briefly: how to build these scripts so you “Don’t Repeat Yourself” when creating functionality in your LLDB scripts.

DRY Python Code

Stop the app! In the schemes, select the Stripped 50 Shades of Ray Xcode scheme.

Ensure the MallocStackLogging environment variable is unchecked in the Stripped 50 Shades of Ray scheme.

Good. Ray approves.

Time to try out the turn_on_stack_logging function. Build and run the application.

As you found out in the previous chapter, the “Stripped 50 Shades of Ray” scheme strips the main executable’s contents so there’s no debugging information available. Remember that factoid when you use the msl command.

Once the application is up and running, tap the Generate a Ray! button to create a new instance of the RayView. Since the MallocStackLogging isn’t enabled, let’s see what happens…

Open the Debug Memory Graph again and find one of the RayView instances. Notice in the Memory inspector that there isn’t a backtrace.

See if the msl command works on this address:

(lldb) msl 0x1268051d0

Nothing. That makes sense though, because the environment variable was not supplied to the process. Time to circle back and call turn_on_stack_logging to see what it does. Type the following in LLDB:

(lldb) po turn_on_stack_logging(1)

You’ll get some output similar to the kind you get when you supply your process with the MallocStackLogging environment variable:

Resume execution and create another instance of RayView by tapping the bottom button.

Once you’ve done that, back to the Debug Memory Graph view and inspect all of your RayView instances. Any of them you created since turning on logging should now have a backtrace.

Copy this new address and apply the msl command to it.

(lldb) msl 0x00007f8250f0a170

This will give you the stack trace!

This is awesome! You can enable malloc logging at will to monitor any allocation or deallocation events without having to restart your process.

Wait wait wait. Hold on a second… there’s a symbol that’s stripped.

Ray don’t like no stripped functions.

If you recall in the previous chapter, you created the sbt command which symbolicated a stack trace. In the sbt.py script, you created the processStackTraceStringFromAddresses function which took a list of numbers (representing memory addresses for code) and the SBTarget. This function then returned a potentially symbolicated string for the stack trace.

You’ve already done the hard work to write this function, so why not include this work in the msl.py script to optionally execute it?

Jump to the very top of the msl.py function and add the following import statement:

import sbt

In the handle_command function in msl.py, hunt for the following code:

retString = sbt.processStackTraceStringFromAddresses(
                                            addresses,
                                               target)

Replace that code with the following:

if options.resymbolicate:
    retString = sbt.processStackTraceStringFromAddresses(
                                                addresses,
                                                   target)
else:
    retString = processStackTraceStringFromAddresses(
                                        addresses,
                                           target)

You’re conditionally checking for the options.resymbolicate option (which I’ve already set up for you). If True, then call the logic in the sbt module to see if it can generate a string of resymbolicated functions.

Since you wrote that function to be generic and handle a list of Python numbers, you can easily pass this information from your msl script.

Before you test this out, there’s one final component to implement. You need to make a convenience command to enable the turn_on_stack_logging.

Jump up to the __lldb_init_module function (still in msl.py) and add the following line of code:

debugger.HandleCommand('command alias enable_logging expression -lobjc -O -- extern void turn_on_stack_logging(int); turn_on_stack_logging(1);')

This declares a convenience command to turn on malloc stack logging by calling the method in libsystem_malloc.dylib. Wait what? Remember that the .dylib is always loaded as part of the runtime, and unlike some languages you know, you don’t have to carefully import and define everything in order to use it, just call the function. If something, anything matches the signature, the code runs.

Woot! Done! Jump back to Xcode and reload your script:

(lldb) reload_script

Use the --resymbolicate option on the previous RayView to see the stack in its fully symbolicated form.

(lldb) msl 0x00007f8250f0a170 -r

I am literally crying with happiness in the face of this wholly beautiful stack trace. Snif.

Key Points

• Some Xcode functionalities are just wrappers around LLDB so enhancing them is a good way to practice creating scripts.
• Malloc Stack Logging will log when objects are allocated and deallocated.
• Malloc Stack Logging has an All Allocation and Free History and an All Allocation option. The Free History option records more data.
• Use image lookup -rn or our custom lookup to search for any interesting symbol names you find as a first step in exploring.
• Search https://opensource.apple.com to find header files that often have useful notes and documentation you won’t find anywhere else.
• Write and test code in Xcode for any functions you later want to bring into a Python script as JIT-ed code. The tools for debugging JIT-ed code within your scripts are effectively nonexistent.
• Python scripts can import Python scripts. As you write code, always look for ways to refactor so that you can reuse your best work in multiple places.

Where to Go From Here?

Hopefully, this full circle of idea, research & implementation has proven useful and even inspired you to create your own scripts. There’s a lot of power hidden quietly away in the many frameworks that already exist on your [i|mac|tv|watch]OS device.

All you need to do is find these hidden gems and exploit them for some crazy commercial debugging tools, or even to use in reverse engineering to better understand what’s happening.

Here’s a list of directories you should explore on your actual iOS device:

• /Developer/
• /usr/lib/
• /System/Library/PrivateFrameworks/

Go forth, my little debuggers, and build something that completely blows my mind!