15. Shared Libraries

Written by Walter Tyree

Shared libraries are essential for any program to run. This chapter focuses on the compilation and linking process, highlighting how to write code that uses public and private APIs.

A shared library is a bundle of code loaded into a program at runtime instead of being included at compile time. Shared libraries can’t run by themselves — they need to be loaded in by an executable. Examples of shared libraries in iOS include UIKit and the Foundation frameworks. These are first-party shared libraries provided by Apple that you can link against. You can also create your own frameworks and package them inside your app bundle.

Creating your own shared libraries is an attractive development option. They provide encapsulation of code that can be shared between different projects. It can also lead to a more rigorous testing strategy, where you can test the shared library in isolation and be sure it works as intended.

Shared Libraries 101

Several types of shared libraries can be loaded in at runtime: dynamic libraries, frameworks and plugins.

A dynamic library, or dylib, is a shared executable that only contains executable code.

On the other hand, a framework is more of a directory that can contain executable code — as well as other content, like images and JSON files — almost anything! In fact, a framework doesn’t even need to contain any code for it to be classified as a framework. A framework ends in a *.framework extension and contains the directory for the encapsulated resources.

Finally, there are plugins. These are a bit like frameworks, except they should be self-contained as much as possible. That is, if someone gave you a framework, you’d expect to call those APIs implemented in the framework. A plugin should do as much as possible on its own without having another entity have to call on its APIs.

This chapter’s focus is primarily on dynamic libraries because it’s the simplest option to showcase a shared library and an executable calling code from it.

To appreciate how dynamic libraries, linking and loading work, you’ll build a dynamic library and an executable that references it. You’ll compile all of this using clang without Xcode to appreciate what’s happening. For this example, you’ll create a C executable that calls code from a Swift dynamic library. You’re using Swift with C on purpose — instead of Swift with Swift — as it emphasizes the concept of resolving symbol names. You’ll learn how to import Swift code from a Swift dylib later when you learn about TBDs and module maps.

Building a Swift Dynamic Library

In the following section, you’re encouraged to write the code yourself, but the source is available in the starter directory for copy/pasters.

In Terminal, navigate to the /tmp directory, and use your favorite text editor to add the following code to /tmp/SwiftSharedLibrary.swift:

@_cdecl("swift_function")
public func swift_function() {
   print("hello from \(#function)")
}

public func mangled_function() {
   print("!!!!!! \(#function)")
}

Compile this code into a dylib with the following command:

$ swiftc SwiftSharedLibrary.swift -emit-library -o libSwiftSharedLibrary.dylib

The swiftc command is an integrated front end for the clang compiler. That’s fancy, technical speak for saying it’ll compile Swift code. :]

-emit-library instructs swiftc to create a dynamic library instead of an executable. The -o flag lets you specify the output filename, which will be libSwiftSharedLibrary.dylib. By default, Swift generates this name for you. However, it’s nice to be explicit in case Apple decides to change this in the future.

The SwiftSharedLibrary.swift source code declares three functions (yes, three). The Swift compiler front end creates two mangled functions:

• swift_function, whose symbol mangled name is $s18SwiftSharedLibrary14swift_functionyyF.
• mangled_function, whose mangled name is s18SwiftSharedLibrary16mangled_functionyyF.

In addition, the compiler generates a C-like unmangled function called swift_function, which, in turn, calls the mangled equivalent. This is all done thanks to the @_cdecl("swift_function") attribute in the Swift source.

This is best seen by dumping the symbol table and greping for the word “function”:

$ nm -U libSwiftSharedLibrary.dylib | grep function
0000000000003954 T _$s18SwiftSharedLibrary14swift_functionyyF
0000000000003c3c T _$s18SwiftSharedLibrary16mangled_functionyyF
0000000000003940 T _swift_function

nm‘s -U option filters for local symbols, meaning it displays symbols implemented in libSwiftSharedLibrary.dylib instead of symbols referenced elsewhere. From the output, the capital T indicates that a symbol is public. This lets code in other modules reference this symbol. If you didn’t include the public keyword in the Swift source code, the t would be lowercase, indicating the symbol is private and can’t be referenced from another module — at least not with public APIs like dlsym.

From the output, observe the unmangled _swift_function symbol generated via the @_cdel("swift_function") attribute. Note how the compiler prepends an underscore to (almost) all symbols when compiling code.

You can also get Swift to demangle the names in any output like the one above. You can pipe any output to swift demangle, and you’ll see the demangled names:

$ nm -U libSwiftSharedLibrary.dylib | grep function | swift demangle
0000000000003a48 T SwiftSharedLibrary.swift_function() -> ()
0000000000003cf4 T SwiftSharedLibrary.mangled_function() -> ()
0000000000003a34 T _swift_function

This can be useful if you need to see what each function is in an nm output.

Building a C Executable

You’ll now create the executable to reference the unmangled swift_function implemented in libSwiftSharedLibrary.dylib. Add the following code to /tmp/exe.c:

extern void swift_function(void);

int main() {
  swift_function();
  return 0;
}

The code above externally declares the swift_function() and attempts to execute it. Compile exe.c while linking libSwiftSharedLibrary.dylib:

$ clang exe.c -o exe libSwiftSharedLibrary.dylib

This time, you’re using clang to compile the exe.c file. Remember, swiftc is essentially a wrapper for clang with some additional flag handling baked in for Swift.

You’ve just compiled the executable to exe. clang was smart enough to infer that the libSwiftSharedLibrary.dylib was a dynamic library and automatically linked it into your executable. An alternative way to execute the above command is the following:

$ clang exe.c -o exe -lSwiftSharedLibrary -L./

This format uses clang’s -l option to tell the linker that it needs to link against the SwiftSharedLibrary. When specifying the -l option for a dynamic library, ld will take that name, prepend a lib, and append .dylib to its search query. So, if you passed in -lPumpkinLatte, ld would search for libPumpkinLatte.dylib in its dynamic library search paths.

But how does one specify the search paths to ld? That’s the -L./ flag’s job. This flag can be used multiple times, and it instructs ld of all the possible locations to look for a dynamic library. It’s important you know both ways to link a dynamic library, as developers and developer tools use both methods interchangeably.

Give the exe program a run:

$ ./exe

The expected output from swift_function appears!

hello from swift_function()

Symbols and Dependencies

When linking dynamic frameworks, it’s insightful to be able to inspect dependencies from a particular module. Use otool -L on the exe executable to display the shared libraries exe needs in order to run:

$ otool -L exe

This shows that exe needs the standard system library and your SwiftSharedLibrary.dylib in order to run.

exe:
   libSwiftSharedLibrary.dylib (compatibility version 0.0.0, current version 0.0.0)
   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

The -L option displays framework load commands that are embedded near the beginning of the exe executable, which will be thoroughly discussed in an upcoming chapter. The referenced libSwiftSharedLibrary.dylib is listed as a requirement.

Using the same otool -L option, inspect the libSwiftSharedLibrary.dylib dependencies:

$ otool -L libSwiftSharedLibrary.dylib
libSwiftSharedLibrary.dylib:
  libSwiftSharedLibrary.dylib (compatibility version 0.0.0, current version 0.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
  /usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.8.0)

The library that is important is libswiftCore.dylib, the library that provides the APIs to use print and friends referenced by the Swift source code. Remember the name libswiftCore.dylib, as you’ll use it later when working with static libraries. The final linked library is libSystem.B.dylib, which is imported by pretty much every shared library and isn’t relevant for this example.

At runtime, dyld will take these libraries, resolve exe‘s dependencies, then recursively resolve all of the shared library dependencies. This process repeats to the next layer until there are no more dependencies to load in. You can see this in action by setting the DYLD_PRINT_LIBRARIES environment variable. Here’s a partial excerpt from my computer:

Note: If you were to look at these shared libraries on disk, you’ll notice something peculiar: These libraries don’t exist on disk at the specified location! You may recall from Chapter 7, “Image”, that Apple has combined them into a shared cache. You’ll do a much deeper dive into what’s happening and why by the end of this chapter.

When a dynamic library is linked, it’s typically — but not always! — due to the fact that at least one symbol is needed from that library. Display the external symbols of exe with nm:

$ nm -mu ./exe
  (undefined) external _swift_function (from libSwiftSharedLibrary)

The -m option shows which module a symbol is referenced from. The lowercase -u will only display symbols that are external. Remember, -U for symbols implemented in the module, -u for symbols not implemented in the module.

When compiling, the linker searched for the swift_function symbol in libSwiftSharedLibrary.dylib and found a reference there. You’re not required to replicate this on your end, but if you were to change around the C source code to call a nonexistent function like bad_function(), you’d get the following error because the symbol can’t be resolved:

$ sed  's/swift_function/bad_function/g'  exe.c > bad_exe.c
$ clang exe.c /tmp/libSwiftSharedLibrary.dylib
UUndefined symbols for architecture arm64:
  "_bad_function", referenced from:
      _main in bad_exe-d1e251.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Linking Tricks

The @_cdecl() attribute is a common trick to read Swift symbols in non-Swift code because it’s typically not possible to reference mangled Swift symbols whose names begin with a dollar sign, like $s18SwiftSharedLibrary16mangled_functionyyF.

However, there are some nifty tricks to get around this! You’ll explore several tricks for executing the mangled $s14SwiftSharedLibrary16mangled_functionyyF.

Compile Time Linking

One way to call the mangled Swift function is with an option from ld to make an alias for the desired function.

$ clang exe.c libSwiftSharedLibrary.dylib -Xlinker -alias -Xlinker '_$s18SwiftSharedLibrary16mangled_functionyyF' -Xlinker _swift_function  -o ./exe

The -Xlinker command passes an argument to the linker, ld, one argument at a time. It might seem weird having to pass -Xlinker multiple times, but consider that clang needs to run several executables — in different processes! — and report the results. If you want to see the full list of options ld has, use man ld. Just remember that every argument has to be preceded by -Xlinker if being called from clang. An alternative way of doing this is via the -Wl flag, which expects commas in place of spaces, like in clang exe.c libSwiftSharedLibrary.dylib '-Wl,-alias,_$s18SwiftSharedLibrary16mangled_functionyyF,_swift_function' -o ./exe. This is nice because you only need to type -Wl once. Either method is fine — take your pick.

The -alias option replaces all references of _swift_function with _$s18SwiftSharedLibrary16mangled_functionyyF. Be careful with supplying the correct symbol, as this won’t report any errors on failure. Running the exe now gives the following:

$ ./exe
!!!!!!! hello from mangled_function()

Runtime Linking

Another alternative is to use runtime linking via the dlsym API. You’ll explore this function and dlopen in depth in the next chapter. But for now, you’ll just take a quick look.

Enter the following code snippet into a file called exe2.c:

#include <dlfcn.h>
#include <stdlib.h>

int main() {
  void (*mangled_function)(void) = NULL;
  mangled_function = dlsym(
    RTLD_NEXT, "$s18SwiftSharedLibrary16mangled_functionyyF");

  if (mangled_function) {
    mangled_function();
  }
  return 0;
}

dlsym will attempt to find the symbol at runtime. If successful, the mangled_function function pointer will contain the address.

Be aware that no underscore precedes the symbol when resolving a symbol at runtime.

Then, run the following:

clang exe2.c libSwiftSharedLibrary.dylib -o exe2

Executing it, you see the same thing again:

$ ./exe2
!!!!!!! hello from mangled_function()

Note: The dlsym API only works for public symbols — or, more technically, symbols that contain the N_EXT flag in its corresponding nlist. That means if the source code has a C static or Swift private attribute, the symbol won’t be exported as public. You easily see which symbols are public by observing the uppercase character on the nm output, like the T in 0000000000003894 T _swift_function.

Linking Symbols? Meh!!! Symbols

If you know that a dynamic library would never change (i.e., would never be updated, never recompiled), you can hardcode offsets to that library based on the module’s load address.

For example, if you wanted to call the mangled mangled_function() in libSwiftSharedLibrary.dylib without using the symbol at all, you can find the offset of where the code is located on disk:

$ nm ./libSwiftSharedLibrary.dylib | grep mangled
0000000000003cf4 T _$s18SwiftSharedLibrary16mangled_functionyyF

On my machine, the mangled_function() is located at offset 0000000000003cf4. This is a hexadecimal value and needs a 0x prepended to it.

From there, one can find the starting load address of libSwiftSharedLibrary.dylib and then add the offset to get the function. Here’s the source code for dontdothis.c:

#include <mach-o/dyld.h> // _dyld.* APIs
#include <string.h>      // strcmp
#include <libgen.h>      // basename

int main() {
  uintptr_t base = 0;
  // iterate over loaded images
  for (int i = 0; i < _dyld_image_count(); i++) {
    if (strcmp(basename((char*)_dyld_get_image_name(i)), "libSwiftSharedLibrary.dylib") == 0) {
      // we found the load address for libSwiftSharedLibrary.dylib
      base = (uintptr_t)_dyld_get_image_header(i);
      break;
    }
  }

  // execute mangled_function
  if (base) {
    void (*mangled_function)(void) = (void*)(base + 0x00000000003cf4);
    mangled_function();
  }

  return 0;
}

Compiling and running produces the following on my machine:

clang dontdothis.c libSwiftSharedLibrary.dylib -o ./dontdothis && ./dontdothis
!!!!!!! hello from mangled_function()

This is only here to show you that it can be done. This is a bad idea — don’t do this! Do as I say, not as I do. :]

Defensive Linking

One final trick is to use a weak attribute for a symbol. A weak attribute won’t crash the program if the symbol can’t be resolved.

Add the following to a file called exe3.c:

#include <stdio.h>

__attribute__((weak))
extern void swift_function(void);

__attribute__((weak))
extern void bad_function(void);

int main() {
  swift_function ? swift_function() : printf("swift_function not found!\n");
  bad_function ? bad_function() : printf("bad_function not found!\n");
  return 0;
}

Compile with the -undefined dynamic_lookup option, which tells ld to ignore unresolved symbols at compile time and attempt to link them at runtime:

$ clang exe3.c -o exe3 libSwiftSharedLibrary.dylib -undefined dynamic_lookup && ./exe3
hello from swift_function()
bad_function not found!

Note: Xcode has an annoying warning that complains when using the -undefined flag on iOS. You can get around this by explicitly listing which symbols you want undefined via the -U option, like so: -Wl,-U,_bad_function,-U,_swift_function.

Notice how swift_function was called, but bad_function wasn’t found. Repeat compiling, but compile without linking libSwiftSharedLibrary.dylib and run exe3:

$ clang exe3.c -o exe3 -undefined dynamic_lookup && ./exe3
swift_function not found!
bad_function not found!

Without the __attribute__((weak)), a symbol’s address is assumed to be non-zero, resulting in a runtime crash from dyld.

Static Libraries

Discussing dynamic libraries wouldn’t be complete without mentioning their counterpart — static libraries!

Sometimes, having a separate entity for symbols isn’t the ideal solution. The current dynamic library setup makes libSwiftSharedLibrary.dylib and its path a required dependency for the executable to run. To observe this, change the name of libSwiftSharedLibrary.dylib to something different:

$ mv libSwiftSharedLibrary.dylib libMovedSharedLibrary.dylib
$ ./exe
dyld: Library not loaded: libSwiftSharedLibrary.dylib
  Referenced from: /private/tmp/exe
  Reason: image not found
[1]    6074 abort      exe

If you’re following along with the examples using Terminal, be sure to rename the library back to its original name:

$ mv libMovedSharedLibrary.dylib libSwiftSharedLibrary.dylib

Having this external libSwiftSharedLibrary.dylib dependency requirement might be undesirable — especially if there’s only one consumer using it (which is exe in this example).

Note: There are several ways to resolve finding dependency locations for shared frameworks. A dynamic library can be referenced via an absolute path or a relative path from the calling module. Check out ld‘s rpath option if you’re interested in exploring this.

An alternative to compiling a dynamic library is to compile the SwiftSharedLibrary.swift code as a static library. A static library is a chunk of sharable compiled code that acts a bit like a dynamic library but is packaged inside the calling module.

Recompile SwiftSharedLibrary.swift as a static library:

$ swiftc SwiftSharedLibrary.swift -static -emit-library -o SwiftSharedLibrary.a

Using the -static option along with -emit-library creates a static library at SwiftSharedLibrary.a. Using a *.a is the typical naming convention for static libraries.

Now, compile the exe.c C source file while including the SwiftSharedLibrary.a static library:

$ clang SwiftSharedLibrary.a exe.c -o exe

The compiler will present some errors — OK, lots of errors:

ld: warning: Could not find or use auto-linked library 'swiftSwiftOnoneSupport'
ld: warning: Could not find or use auto-linked library 'swiftCore'
Undefined symbols for architecture arm64:
  "Swift.String.init(stringInterpolation: Swift.DefaultStringInterpolation) -> Swift.String", referenced from:
      SwiftSharedLibrary.swift_function() -> () in SwiftSharedLibrary.a(SwiftSharedLibrary-c49738.o)
      SwiftSharedLibrary.mangled_function() -> () in SwiftSharedLibrary.a(SwiftSharedLibrary-c49738.o)
... snip ...

Uh-oh! You’ve included a static library that has external symbols that you’re not linking against. Remember when you used otool -L on libSwiftSharedLibrary.dylib and saw that it had a dependency, libswiftCore.dylib, to use Swift’s print APIs? Since libSwiftSharedLibrary.a’s code is being compiled directly into exe, exe now needs to link to its dependencies, like /usr/lib/swift/libswiftCore.dylib.

Build again, but now include libSwiftSharedLibrary.a’s dependency of libswiftCore.dylib:

$ clang SwiftSharedLibrary.a exe.c -L/usr/lib/swift -lswiftCore -o exe

Run the executable, then check the compiled local symbols and referenced frameworks included in exe with nm and otool:

$ ./exe

Displays your expected output:

hello from swift_function()

Now, use the -L flag to show the shared libraries with otool:

$ otool -L exe

The symbols that were once separately packaged in libSwiftSharedLibrary.dylib are now implemented directly in exe!

exe:
   /usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 1205.0.24)
   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)

$ nm -mU exe | grep function
0000000100003908 (__TEXT,__text) external _$s14SwiftSharedLibrary14swift_functionyyF
0000000100003bf0 (__TEXT,__text) external _$s14SwiftSharedLibrary16mangled_functionyyF
00000001000038f4 (__TEXT,__text) external _swift_function

That’s because a static library is essentially just a bunch of object files that get added into the final binary at link time. When examining the code with nm, the functions that come from the library appear alongside the rest of the code. The fact that they started in a separate file is no longer obvious.

Be aware: You don’t need a static library if you’re compiling code into only one module. If you’re just doing that, you should add the code directly since, in this case, creating a static library is a superfluous step. A static library is designed to be shareable, multi-architecture compiled code you can hand out to consumers, which is a great option for SDK makers.

Static libraries are great if you’re packaging code that should only be called in one spot. Sometimes, vendors opt for a static library over a dynamic library as it can allow developers more integration flexibility. For example, a consuming codebase can embed the static library directly into the main executable or, instead, integrate the static library into a dynamic library so multiple modules can use it.

Text-Based Dynamic Library Files

In the example above, you created a main executable and a dynamic library. When compiling, the linker had to look into libSwiftSharedLibrary.dylib, parse its symbol table, and ensure that the appropriate symbol was there for the linking to succeed.

When you take a step back and think about this, the linker shouldn’t need to do all that heavy lifting. The linker could be told the same thing just by reading a text file. That’s what a text-based dynamic library stub or *.tbd file does.

Having knowledge of how text-based dynamic library — TBD for short — files work and how to use them is essential for referencing both public and private APIs. If you plan on utilizing and linking against APIs in private frameworks on a remote host, like iOS, using a TBD file is extremely useful! This is because you’re not likely to have physical access to the shared library on your development machine. As of Xcode 7, Apple no longer packages shared libraries for remote platforms, which is understandable given this adds significant bloat to the Xcode bundle size.

Run the find Terminal command:

$ find /Applications/Xcode.app -name "*.tbd"

This dumps all the *.tbd files packaged within Xcode. These files are used in place of shared libraries when referencing symbols. Check out any one of these files that looks interesting to you via an open or a cat Terminal command. Here’s one of my favorites:

$ cat /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/PrivateFrameworks/CoreSymbolication.framework/CoreSymbolication.tbd

TBD Format and TAPI

The text-based dynamic library stubs need to have an agreed-upon format for ld to know how to utilize the TBD file. An open-source implementation called Text-based Application Programming Interface, or TAPI, can generate TBD files from headers or compiled code. tapi has source code found in the LLVM repo and is also part of opensource.apple.com. You can also check out additional documentation of the TBD parameters.

To explore how to work with these, you’ll compile some code and treat this as a private dynamic library you don’t have the source to but do have the header for (via reverse engineering or stumbling upon someone else’s work on GitHub). The starter project directory includes Private.m and Private.h, which contain an Objective-C class called PrivateObjcClass, an NSString constant called SomeStringConstant and a C function called SomeCode.

In Terminal, navigate to where you’ve saved the demo code for this chapter. Using your favorite method, inspect the contents of Private.h and Private.m. Now, compile this code as a shared library, and treat it as a private library.

$ clang -shared -o /tmp/PrivateFramework.dylib Private.m -fmodules -arch arm64e -arch arm64 -arch x86_64

This creates the /tmp/PrivateFramework.dylib shared library for all currently supported macOS hardware. -fmodules is required since the Private.h header imported a module instead of a C #include header (see @import Foundation; in Private.h). Using the version of tapi packaged in Xcode, generate a TBD file for the newly created /tmp/PrivateFramework.dylib.

$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/tapi stubify /tmp/PrivateFramework.dylib -o /tmp/libPrivateFramework.tbd

This creates the TBD file called libPrivateFramework.tbd (remember, the lib part of the name is important for linking!). tapi has several options that can be observed with tapi --help. One of those options is stubify, which extracts symbols from an already compiled module. The -o argument specifies the file’s destination (if you supplied -o -, it will go to standard out).

The TBD file created with the above command has the following contents on my machine:

--- !tapi-tbd
tbd-version:     4
targets:         [ x86_64-macos, arm64-macos, arm64e-macos ]
flags:           [ not_app_extension_safe ]
install-name:    '/tmp/PrivateFramework.dylib'
current-version: 0
compatibility-version: 0
exports:
  - targets:         [ x86_64-macos, arm64-macos, arm64e-macos ]
    symbols:         [ _SomeCode, _SomeStringConstant ]
    objc-classes:    [ PrivateObjcClass ]
...

The TBD above uses a YAML-style way of declaring values. Some of the important values from above:

• targets: The supported executable slices that the shared library includes. You supplied three architectures earlier when compiling the library, which are reflected here.
• install-name: The actual path to the shared library where it’s expected to be. This should match what otool -L would display.
• symbols: A subkey under exports. Declares any “C-like” symbols, including any public code and data.
• objc-classes: A subkey under exports. Declares any Objective classes.

The TBD file above declares _SomeCode and _SomeStringConstant symbols and an Objective-C class called PrivateObjcClass. Along with a corresponding Private.h header, you now have all the components required to compile the executable.

The TBD file will act as a stand-in to the PrivateFramework.dylib compiled module for linking. You’ll create an executable that will reference these symbols and link code via the TBD file.

The starter project includes a file called tbdpoc.m. Compile this file and link it with the TBD file.

$ clang tbdpoc.m -I. -L/tmp/ -lPrivateFramework -fmodules -o /tmp/tbdpoc
$ /tmp/tbdpoc
2023-04-06 14:42:56.710 tbdpoc[9757:587074] much wow, stuff of doing!
2023-04-06 14:42:56.710 tbdpoc[9757:587074] SomeStringConstant is: com.kodeco.tbd.example

The -L/tmp/ -lPrivateFramework combo worked on the TBD file just like a real image should. -I. instructed clang to search the current directory for headers and include files. This is needed so tbdpoc could reference Private.h, found in the same directory.

Instead of the -l/-L flags, you can also specify the TBD file directly, and the compiler will read the file and link to the framework specified in install-name.

This will produce the exact same result:

$ clang tbdpoc.m -I. libPrivateFramework.tbd -fmodules -o /tmp/tbdpoc
ld: warning: text-based stub file libPrivateFramework.tbd and library file libPrivateFramework.tbd are out of sync. Falling back to library file for linking.

It’s worth noting that you can get an annoying little warning because the module’s UUID (which you’ve omitted via the --no-uuids) doesn’t match, making ld complain. Browsing the source code to the linker https://opensource.apple.com/source/ld64/ reveals that the linker will try to match the UUIDs in the TBD file with the ones on disk — see ld64/src/ld/Options.cpp. This can be suppressed with the LD_PREFER_TAPI_FILE environment variable.

Modules and Module Maps

You’ve played with Swift code mangling names and importing them into C, and you’ve created a TBD file importing Objective-C/C code into an Objective-C/C executable. Now, it’s time to take an Objective-C dynamic library and import it into Swift. This is the final piece of the linking puzzle, as Swift requires one additional component to properly import symbols from an external library.

This component is called a module. This term conflicts with the typical meaning of module — compiled code — used throughout this book. The LLVM linker’s version of the module is a “precompiled” grouping of headers that greatly speeds up the compilation process compared to traditional C #include headers. LLVM has a detailed writeup of a module and the parameters one can use.

You’ve executed code before that’s had modules in it with @import SomeModule; in Objective-C code and have told clang to compile modules with the -fmodules argument.

You have several ways to create a usable module in Swift. One way is through the swiftc command utility with the -emit-module flag, which creates a *.swiftmodule that can be linked and referenced in Swift code. Another way is to use a module map, which is a text file that’s understood by clang and serves as the link between C include headers and a module. This method allows you to call private APIs from Swift because you can declare the APIs in a header, include the header in a module, then import that module into Swift code.

First, navigate to the starter directory for the code for this chapter and copy over the Private.h header file to /tmp/:

$ cp Private.h /tmp/

Now, create a module map file using your favorite text editor. Write the following contents to /tmp/module.modulemap:

module YayModule {
  header "Private.h"
  export *
}

The module.modulemap file is important! The compiler will look for this file in the specified include directories. If it exists, clang will automatically pick it up.

This module.modulemap defines a module called YayModule, which exports the C/ObjC symbols declared in Private.h. This will allow your soon-to-be-created Swift source code to use the symbols referenced in Private.h. The poorly chosen YayModule module name is to highlight that you can choose any arbitrary name so long as you import the same module name in the Swift source code.

The export * declaration indicates YayModule should export all of its imported declarations. This is a bit cryptic, so an example will better describe this: YayModule imports Objective-C headers in Private.h in order to declare the Objective-C class. If you didn’t export these declarations, you’d have to manually import an Objective-C module in addition to the YayModule module. You should default to always including an export * statement unless you know what you’re doing.

You’re almost there. Create a Swift source file named /tmp/mmpoc.swift with the following code:

import YayModule

print("calling external: \(SomeStringConstant)")
SomeCode();
let c = PrivateObjcClass()
c.doStuff()

As a reminder, you should now have the following files in the /tmp directory, which will be required to compile mmpoc.swift:

• mmpoc.swift: The file that you’ll compile.
• libPrivateFramework.tbd: The TBD file used by ld to resolve symbols referenced by mmpoc.swift.
• PrivateFramework.dylib: The actual compiled framework that’s loaded at runtime whose symbols will be referenced. Also used by libPrivateFramework.tbd to indicate who implements the APIs.
• module.modulemap: The Swift header file equivalent used to import the symbols.
• Private.h: The header file referenced by module.modulemap and also for declaring code in Private.m.

After ensuring the above files are present, give the compilation a go. If successful, run mmpoc:

$ swiftc mmpoc.swift -I. -L/tmp -lPrivateFramework -o /tmp/mmpoc && /tmp/mmpoc
calling external: com.kodeco.tbd.example
2023-04-06 15:22:45.109 mmpoc[10164:605856] SomeStringConstant is: com.kodeco.tbd.example
2023-04-06 15:22:45.110 mmpoc[10164:605856] much wow, stuff of doing!

Excellent! You can now call, link to and execute “private” APIs in Swift or Objective-C.

Xcode Equivalent

You jumped down to the command line to do all this work. It’s worth going back up to Xcode to see how to achieve the same thing.

If you want to link to a library — -lDynamicLibrary or -framework FrameworkName — select your desired Xcode target, and then click the plus button in Xcode’s Framework and Libraries under General. Alternatively, you can specify a library to link with under Build Phases and then Link Binary With Libraries.

After compiling the module, otool -L shows that your library has been added to the executable:

Hopefully, all the *.a/*.tbd/*.framework/*.dylibs now make a little bit more sense when linking in a library.

dyld Shared Cache

If you were to inspect any executable’s linked frameworks, you’d notice that libSystem.B.dylib is included in pretty much everything.

For example, building a Swift file with no source code will still link to libSystem.B.dylib:

$ touch /tmp/anexample.swift && swiftc /tmp/anexample.swift -o /tmp/anexample && otool -L /tmp/anexample
/tmp/anexample:
   /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)

But if you were to try and inspect this file…

$ file /usr/lib/libSystem.B.dylib
/usr/lib/libSystem.B.dylib: cannot open '/usr/lib/libSystem.B.dylib' (No such file or directory)

You’ll notice that this file doesn’t exist on disk. That’s because Apple aggressively caches frequently used libraries into a “mega library bundle” known as the dyld shared cache. This cache provides a significant speed boost, as referencing resident memory is so much faster than querying the disk hundreds of times to load libraries.

The internals of dyld and the cache are complex and outside the scope of this book (see http://newosxbook.com/index.php for excellent writing on this), but you do need to at least be able to list what modules are packed into the shared cache to be able to link to and explore their symbols in memory.

You’ll use your insights gained in this chapter by compiling a Swift executable that lists all the modules packed into the dyld shared cache. You’ll achieve this via private C APIs that are referenced in the source code of dyld under dyld_priv.h. A header with the same dyld_priv.h name has been provided in the starter project and includes a watered-down selection of these APIs. You’ll use the same module.modulemap to import this dyld_priv.h header and reference these APIs via Swift.

In Terminal, open the module.modulemap in the starter directory, and add header "dyld_priv.h" between header "Private.h" and export *. The module.modulemap file now looks like this:

module YayModule {
  header "Private.h"
  header "dyld_priv.h"
  export *
}

The source code has already been written in dyldlist.swift, available in the starter directory.

import YayModule  

let cache_uuid =
  UnsafeMutablePointer<UInt8>.allocate(capacity: 16)
let manager = FileManager.default

if _dyld_get_shared_cache_uuid(cache_uuid) {
    let cachePath = String(
      cString: dyld_shared_cache_file_path())
    print("Inspecting dyld cache at \"\(cachePath)\"")

    dyld_shared_cache_iterate_text(cache_uuid) { info in
        if let module = info?.pointee {
            let uuid = UUID(uuid: module.dylibUuid).uuidString
            let path = String(cString: module.path)
            let exists = manager.fileExists(atPath: path)

            print("\(exists ? "*" : " ") \(uuid) - \(path)")
        }
    }
}

The details of these APIs will be left as an exercise for you to explore on your own. What matters is the compilation.

In Terminal, while in the starter directory, compile dyldlist.swift. If you get errors, make sure that the version of the module.modulemap you updated is the one in the starter directory and not the one in tmp.

$ swiftc -I. -o /tmp/dyldlist dyldlist.swift

You’re referencing APIs, but you didn’t have to explicitly link to a library when compiling via clang. Why is that? Do you remember libSystem and how it’s implicitly included in every process? libSystem will reexport these symbols, which belong to /usr/lib/system/libdyld.dylib. Since libSystem automatically imports libdyld.dylib, it tells the linker, “Don’t worry, I’ve got this”.

Consulting Xcode’s libSystem TBD file, you can verify all the *_dyld* symbols it handles:

$ cat $(xcrun --show-sdk-platform-path)/Developer/SDKs/MacOSX.sdk/usr/lib/libSystem.tbd | grep _dyld

Give the dyldlist executable a run. You’ll see a significant number of libraries held in the dyld shared cache:

$ /tmp/dyldlist  | wc -l
  2500

The source code adds an asterisk to any file it can find in the cache and also on disk. grep for an asterisk at the beginning of the output:

$ /tmp/dyldlist  | grep -E "^\*" | wc -l
4
$ /tmp/dyldlist  | grep -E "^\*"
* AED7DD2C-0325-3172-83E7-3BE31F6D4069 - /usr/lib/dyld
* 222F8841-2BFD-3804-AA0C-F6D80A73FBDF - /usr/lib/system/libsystem_kernel.dylib
* 7AF7B500-9A6E-3121-A66A-397C209B5C83 - /usr/lib/system/libsystem_pthread.dylib
* 61D6CE46-BF8C-34EA-B81A-879743AD4063 - /usr/lib/system/libsystem_platform.dylib

Key Points

• Shared libraries are code external to your code that are loaded at runtime or compile time.
• Unless you’re planning to actually share the code with multiple clients, making a library is often not worth the overhead.
• The nm and otool commands let you inspect shared libraries to find function names.
• The swift demangle command converts mangled function names into the form you can use in your code to call them.
• The linker will resolve dependencies for dynamic frameworks automatically. You need to do the resolution yourself for static libraries.
• The linker has a weak attribute you can use when linking so that a program won’t crash if a symbol can’t be resolved.
• A text-based dynamic library, or TBD, file with the .tbd extension can stand in for a shared library when compiling code.
• A module map file serves as a bridge between Swift and Objective-C code.

Where to Go From Here?

Does your head hurt? In this chapter, you learned more than you ever wanted to know about dynamic frameworks. You explored the compiling and linking process while realizing it’s not about the language, but more about the linker needing to resolve symbols. You’ve learned how to generate \*.tbd files to link to a library you don’t physically have on your computer’s disk. You’ve learned how to use clang‘s modules to import private code to use in Swift. Finally, you’ve learned about the dyld shared cache and how to list its libraries in Swift.

Don’t forget to refer back to Chapter 7, “Image”, for strategies to dump things out of the shared cache. Now, you can inspect them and link to them as you explore.

Although mentioned earlier, you really should check out the man pages for ld. Some fascinating options can be performed with the linker that could save you hours of headaches and looking at half-baked answers on Stack Overflow.