Modern, Efficient Core Data

Core Data is one of the venerable Apple Frameworks that’s been around for a long time. Since the release of NSPersistentContainer in iOS 10, Apple has shown Core Data a lot of love. The most recent Core Data additions step it up another huge notch. There are now batch insert requests, persistent history and derived attributes which can definitely make Core Data usage more efficient.

In this tutorial, you’ll improve an app by making the data store much more efficient. You’ll learn how to:

• Create a batch insert request.
• Query the persistent store’s transaction history.
• Control how and when the UI updates in response to new data.

And you just might save the human race in the process!

Getting Started

Fireballs! They’re everywhere! Is anyone paying attention? Fireballs could be the first sign of an alien invasion or a portent of a coming Armageddon. Someone has to keep watch. This is your mission. You’ve made an app that downloads fireball sightings from the NASA Jet Propulsion Laboratory (JPL) so you can group them and report on suspicious fireball activity.

Download the starter project using the Download Materials button at the top or bottom of this tutorial. Open the starter project. Look at what you have so far.

Exploring Fireball Watch

Build and run the app, so you can get a feel for how it works. The app downloads the latest fireball data from the JPL, creates records for each fireball sighting and stores them in a Core Data stack. You can also create groups and add fireballs to groups for reporting purposes.

When it launches, the list will be empty, so tap the refresh button at the top right of the Fireballs list. Pretty soon, the list should fill up. You can tap it again to see it doesn’t add duplicate records for the same data. If you swipe left on some fireball cells and delete a few, then tap refresh again, you’ll see those fireballs recreated after downloading the data.

If you tap the Groups tab, you can add a group. Make some groups and then go back to the Fireballs tab and tap a fireball in the list. Then, tap the in-tray button at top right to select one or more groups in which to include that fireball. When you tap a group listed in the Groups tab, it’ll show you a map with all of the fireballs in that group.

Examining the Core Data Stack

Now take a look at how the app’s Core Data stack is set up.

Open Persistence.swift. You’ll see a class called PersistenceController. This class handles all your Core Data setup and data importing. It uses an NSPersistentContainer which creates a standard SQLite store or, optionally, an in-memory store that’s used for SwiftUI previews.

The persistent container’s viewContext is the managed object context the app uses for the fetch requests that produce the data for the lists. This is a typical setup. You have two entities in your model: Fireball and FireballGroup.

PersistenceController has fetchFireballs() which downloads the fireball data and calls the private importFetchedFireballs(_:) to import the resulting array of FireballData structs as Fireball managed objects. It does this as a background task, using the persistent container’s performBackgroundTask(_:).

importFetchedFireballs(_:) loops through the FireballData array, creates a managed object and saves the managed object context. Because the persistent container’s viewContext has automaticallyMergesChangesFromParent set to true, this might stall the UI while the app saves all the objects. This is a problem which can make an app feel quite clunky and is the target of your first improvement.

Making a Batch Insert Request

The list of reported fireballs will only grow larger, and what if there’s a sudden fireball swarm? A fireball swarm could indicate a possible alien landing site that could herald a new invasion attempt!

You want the initial download to be as snappy as possible. Your app needs to quickly get you up to speed with the most current data. Any pauses, delays or hangs are unacceptable — lives depend on it.

Batch inserting comes to the rescue! A batch insert request is a special persistent store request that allows you to import large amounts of data directly into the persistent store. You’ll need a method that creates a batch insert request for this operation. Open Persistence.swift and add the following method to PersistenceController:

private func newBatchInsertRequest(with fireballs: [FireballData])
  -> NSBatchInsertRequest {
  // 1
  var index = 0
  let total = fireballs.count

  // 2
  let batchInsert = NSBatchInsertRequest(
    entity: Fireball.entity()) { (managedObject: NSManagedObject) -> Bool in
    // 3
    guard index < total else { return true }

    if let fireball = managedObject as? Fireball {
      // 4
      let data = fireballs[index]
      fireball.dateTimeStamp = data.dateTimeStamp
      fireball.radiatedEnergy = data.radiatedEnergy
      fireball.impactEnergy = data.impactEnergy
      fireball.latitude = data.latitude
      fireball.longitude = data.longitude
      fireball.altitude = data.altitude
      fireball.velocity = data.velocity
    }

    // 5
    index += 1
    return false
  }
  return batchInsert
}

This method takes the array of FireballData objects and creates an NSBatchInsertRequest to insert them all. Here's how:

1. You first create local variables to hold the current loop index and the total fireball count.
2. Create a batch insert request using NSBatchInsertRequest(entity:managedObjectHandler:). This method requires an NSEntity and a closure executed for every insert you want to perform — one for each fireball. The closure must return true if it's your last insert.
3. Inside the closure, you first check that you've reached the end of the fireballs array and if so return true, completing the request.
4. This is where you insert new data. The closure is called with an NSManagedObject instance. This is a new object, and after checking that its type is Fireball (it always will be, but you should always be safe), you set the object's properties to match the fetched fireball data.
5. Finally, you increment the index and return false, indicating that the insert request should call the closure again.

Batch Inserting Fireballs

So that's the request creation done. Now, how do you use it? Add the following method to PersistenceController:

private func batchInsertFireballs(_ fireballs: [FireballData]) {
  // 1
  guard !fireballs.isEmpty else { return }

  // 2
  container.performBackgroundTask { context in
    // 3
    let batchInsert = self.newBatchInsertRequest(with: fireballs)
    do {
      try context.execute(batchInsert)
    } catch {
      // log any errors
    }
  }
}

Here's what that does:

1. You first check that there's actually work to do, making sure the array is not empty.
2. Then ask the PersistentContainer to execute a background task using performBackgroundTask(_:).
3. Create the batch insert request and then execute it, catching any errors it might throw. The batch request inserts all of your data into the persistent store in a single transaction. Because your Core Data model has a unique constraint defined, it'll only create new records if they do not exist and update existing records if required.

One final change: Go to fetchFireballs() and, instead of calling self?.importFetchedFireballs($0), change it to:

self?.batchInsertFireballs($0)

You may also comment or delete importFetchedFireballs(_:), because it's no longer needed.

All that's left to do is build and run!

But you may notice that something is wrong. If you delete a fireball and then tap the refresh button again, the list doesn't update. That's because the batch insert request inserts data into the persistent store, but the view context is not updated, so it has no idea that anything has changed. You can confirm this by restarting the app, after which you'll see that all the new data now appears in the list.

Previously, you were creating objects in the background queue context and saving the context, which pushed the changes to the persistent store coordinator. It was automatically updated from the persistent store coordinator after saving the background context because you have automaticallyMergeChangesFromParent set to true on the view context.

Part of the efficiency of persistent store requests is that they operate directly on the persistent store and avoid loading data into memory, or generating context save notifications. So while the app is running, you'll need a new strategy for updating the view context.

Enabling Notifications

Of course, updating the store in the background is not an uncommon situation. For example, you might have an app extension which updates the persistent store, or your app supports iCloud and your app's store updates from another device's changes. Happily, iOS offers a notification — NSPersistentStoreRemoteChange — which is delivered whenever a store update occurs.

Open Persistence.swift again and jump to init(inMemory:). Just before the line that calls loadPersistentStores(completionHandler:) on PersistentContainer, add this line:

persistentStoreDescription?.setOption(
  true as NSNumber,
  forKey: NSPersistentStoreRemoteChangeNotificationPostOptionKey)

Adding that one line causes your store to generate notifications whenever it updates.

Now, you need to use this notification somehow. First, add an empty method to PersistenceController, that'll be a placeholder for all of your update processing logic:

func processRemoteStoreChange(_ notification: Notification) {
  print(notification)
}

Your placeholder method simply prints the notification to the Xcode console.

Next, subscribe to the notification using the NotificationCenter publisher by adding this to the end of init(inMemory:):

NotificationCenter.default
  .publisher(for: .NSPersistentStoreRemoteChange)
  .sink {
    self.processRemoteStoreChange($0)
  }
  .store(in: &subscriptions)

Whenever your app receives the notification, it will call your new processRemoteStoreChange(_:).

Build and run, and you'll see a notification arrive in the Xcode console for each update. Try refreshing the fireball list, adding groups, deleting fireballs and so on. All updates to the store will generate a notification.

So how does this notification help you? If you want to keep it simple, you can simply refresh the view context whenever you receive the notification. But there's a smarter, and much more efficient, way. And this is where you dive into persistent history tracking.

Enabling Persistent History Tracking

If you enable persistent history tracking, Core Data retains the transaction history of everything going on in your persistent store. This enables you to query the history to see exactly what objects were updated or created and merge only those changes into your view context.

To enable persistent history tracking, add this line in init(inMemory:) just before the line that calls loadPersistentStores(completionHandler:) on PersistentContainer:

persistentStoreDescription?.setOption(
  true as NSNumber, 
  forKey: NSPersistentHistoryTrackingKey)

That's it! Now, the app will save the transaction history of every change to your persistent store and you can query that history with a fetch request.

Making a History Request

When your app receives the store's remote change notification, it can now query the store's history to discover what's changed. Because store updates can come from multiple sources, you'll want to use a serial queue to perform the work. That way, you'll avoid conflicts or race conditions processing multiple sets of changes if they happen simultaneously.

Add the queue property to your class just before init(inMemory:):

private lazy var historyRequestQueue = DispatchQueue(label: "history")

Now, you can return to processRemoteStoreChange(_:), remove the print() statement and add the following code that'll perform a history request:

// 1
historyRequestQueue.async {
  // 2
  let backgroundContext = self.container.newBackgroundContext()
  backgroundContext.performAndWait {
    // 3
    let request = NSPersistentHistoryChangeRequest
      .fetchHistory(after: .distantPast)

    do {
      // 4
      let result = try backgroundContext.execute(request) as? 
        NSPersistentHistoryResult
      guard 
        let transactions = result?.result as? [NSPersistentHistoryTransaction],
        !transactions.isEmpty 
      else {
        return
      }
       
      // 5
      print(transactions)
    } catch {
      // log any errors
    }
  }
}

Here what's going on in the code above:

1. You run this code as a block on your history queue to handle each notification in a serial manner.
2. To perform the work, you create a new background context and use performAndWait(_:) to run some code in that new context.
3. You use NSPersistentHistoryChangeRequest.fetchHistory(after:) to return a NSPersistentHistoryChangeRequest, a subclass of NSPersistentStoreRequest, that you can execute to fetch history transaction data.
4. You execute the request and coerce the results into an array of NSPersistentHistoryTransaction objects. The default result type of a history request is just such an array of objects. The objects also contain NSPersistentHistoryChange objects that are all the changes related to the transactions returned.
5. This is where you'll process the changes. For now, you just print the returned transactions to the console.

Build and run and do the usual testing dance: Tap the refresh button, delete a few fireballs, refresh again and so on. You'll find the notifications arrive and an array of transaction objects prints to your Xcode console.

Revealing a Conundrum: Big Notifications

This reveals a conundrum, and if you've already noticed it, well done!

Any change to the persistent store triggers a notification, even if your user adds or deletes a managed object from a user interaction. That's not all: Notice that your history fetch request also returns all changes from the beginning of the transaction log.

Your notifications are too big!

Your intention is to avoid doing any unnecessary work for the view context, taking control of when to refresh the view context. No problem at all, you have it covered. To make the whole process clear, you'll do this in a few easy-to-follow steps.

Step 1: Setting a Query Generation

The first step — a small one toward taking control of the view context — is to set a query generation. In Persistence.swift, add this to init(inMemory:) before the NotificationCenter publisher:

if !inMemory {
  do {
    try viewContext.setQueryGenerationFrom(.current)
  } catch {
    // log any errors  
  }
}

You are pinning the view context to the most recent transaction in the persistent store with the call to setQueryGenerationFrom(_:). However, because setting query generation is only compatible with an SQLite store, you do so only if inMemory is false.

Step 2: Saving the History Token

Your history request uses a date to limit the results, but there's a better way.

An NSPersistentHistoryToken is an opaque object that marks a place in the persistent store's transaction history. Each transaction object returned from a history request has a token. You're able to store it so you know where to start when you query persistent history.

You'll need a property in which to store the token for use while the app is running, a method to save the token as a file on disk and a method to load it from the saved file.

Add the following property to PersistenceController just after historyRequestQueue:

private var lastHistoryToken: NSPersistentHistoryToken?

That'll store the token in memory and, of course, you need a place to store it on disk. Next, add this property:

private lazy var tokenFileURL: URL = {
  let url = NSPersistentContainer.defaultDirectoryURL()
    .appendingPathComponent("FireballWatch", isDirectory: true)
  do {
    try FileManager.default
      .createDirectory(
        at: url, 
        withIntermediateDirectories: true, 
        attributes: nil)
  } catch {
    // log any errors
  }
  return url.appendingPathComponent("token.data", isDirectory: false)
}()

tokenFileURL will attempt to create the storage directory the first time you access the property.

Next, add a method to save the history token as a file to disk:

private func storeHistoryToken(_ token: NSPersistentHistoryToken) {
  do {
    let data = try NSKeyedArchiver
      .archivedData(withRootObject: token, requiringSecureCoding: true)
    try data.write(to: tokenFileURL)
    lastHistoryToken = token
  } catch {
    // log any errors
  }
}

This method archives the token data to a file on disk and also updates lastHistoryToken.

Return to processRemoteStoreChange(_:) and find the following code:

let request = NSPersistentHistoryChangeRequest
  .fetchHistory(after: .distantPast)

And replace it with this:

let request = NSPersistentHistoryChangeRequest
  .fetchHistory(after: self.lastHistoryToken)

This simply changes from requesting the whole history to requesting the history since the last time the token was updated.

Next, you can grab the history token from the last transaction in your returned transaction array and store it. Under the print() statement, add:

if let newToken = transactions.last?.token {
  self.storeHistoryToken(newToken)
}

Build and run, watch the Xcode console, and tap the refresh button. The first time you should see all the transactions from the beginning. The second time you should see far fewer and perhaps none. Now that you've downloaded all the fireballs and stored the last transaction history token, there are probably no newer transactions.

Unless there's a new fireball sighting!

Step 3: Loading the History Token

When your app starts, you'll also want it to load the last saved history token if it exists, so add this method to PersistenceController:

private func loadHistoryToken() {
  do {
    let tokenData = try Data(contentsOf: tokenFileURL)
    lastHistoryToken = try NSKeyedUnarchiver
      .unarchivedObject(ofClass: NSPersistentHistoryToken.self, from: tokenData)
  } catch {
    // log any errors
  }
}

This method unarchives the token data on disk, if it exists, and sets the lastHistoryToken property.

Call this method by adding it to the end of init(inMemory:):

loadHistoryToken()

Build and run and watch the console again. There should be no new transactions. In this way your app will be ready to query the history log, right off the bat!

Step 4: Setting a Transaction Author

You can refine your history processing even further. Every Core Data managed object context can set a transaction author. The transaction author is stored in the history and becomes a way to identify the source of each change. It's a way you can tell changes made by your user directly from changes made by background import processes.

First, at the top of PersistenceController, add the following static properties:

private static let authorName = "FireballWatch"
private static let remoteDataImportAuthorName = "Fireball Data Import"

These are the two static strings that you'll use as author names.

Next, add the following to line to init(inMemory:), right below the call to set viewContext.automaticallyMergesChangesFromParent:

viewContext.transactionAuthor = PersistenceController.authorName

This sets the transaction author of the view context using the static property you just created.

Next, scroll down to batchInsertFireballs(_:) and, within the closure you pass to performBackgroundTask(_:), add this line at the beginning:

context.transactionAuthor = PersistenceController.remoteDataImportAuthorName

This sets the transaction author of the background context used for importing data to the other static property. So now the history that's recorded from changes to your contexts will have an identifiable source, and importantly, different from the transaction author for UI-updates like deleting by swiping a row.

Step 5: Creating a History Request Predicate

To filter out any transactions caused by the user, you'll need to add a fetch request with a predicate.

Find processRemoteStoreChange(_:) and add the following right before do:

if let historyFetchRequest = NSPersistentHistoryTransaction.fetchRequest {
  historyFetchRequest.predicate = 
    NSPredicate(format: "%K != %@", "author", PersistenceController.authorName)
  request.fetchRequest = historyFetchRequest
}

First, you create an NSFetchRequest using the class property NSPersistentHistoryTransaction.fetchRequest and set its predicate. The predicate test will return true if the transaction author is anything other than the string you created to identify the transactions made by the user. Then, you set the fetchRequest property of the NSPersistentHistoryChangeRequest with this predicated fetch request.

Build and run, and watch the console. You'll see the result of all this work. Delete a fireball and you'll see no transactions printed to the console because you're filtering out the transactions generated by the user directly. However, if you then tap the refresh button, you'll see a new transaction appear, because that's a new record added by the batch import. Success!

Phew! That was a long stretch — how are you doing? In these trying times, it's always good to remember your app's core mission: to save humanity from alien invasion. It's all worth it!

Step 6: Merging Important Changes

OK, you've added all of the optimizations you need to make sure that your view context processes changes from only the most relevant transactions. All that's left to do is merge those changes into the view context to update the UI. And that is relatively simple.

Add the following method to your PersistenceController:

private func mergeChanges(from transactions: [NSPersistentHistoryTransaction]) {
  let context = viewContext
  // 1
  context.perform {
    // 2
    transactions.forEach { transaction in
      // 3
      guard let userInfo = transaction.objectIDNotification().userInfo else {
        return
      }

      // 4
      NSManagedObjectContext
        .mergeChanges(fromRemoteContextSave: userInfo, into: [context])
    }
  }
}

Here what's going on in the code above:

1. You make sure to do the work on the view context's queue, using perform(_:).
2. You loop over each transaction passed to this method.
3. Each transaction contains all the details of each change, but you need it in a form you can pass to mergeChanges(fromRemoteContextSave:into:): a userInfo dictionary. objectIDNotification().userInfo has just the dictionary you need.
4. Passing that to mergeChanges(fromRemoteContextSave:into:) will bring the view context up to date with the transaction changes.

Remember the query generation you set previously? One of the effects of the mergeChanges(fromRemoteContextSave:into:) method is to update the context's query generation. Handy!

All that remains is to call your new method. Add the following line to processRemoteStoreChange(_:) just before the call to print(_:) (you can also remove that call to print(_:) if you'd like!):

self.mergeChanges(from: transactions)

The process changes method now filters the transactions and passes only the most relevant ones to the mergeChanges(from:) method.

Build and run!

Forget the console, check out your app. Refresh twice and you should see nothing happen the second time because no work is required. Then, delete a fireball and then tap the refresh button. You'll see it appear again!

Adding Derived Attributes

You're able to add fireballs to groups, so it'd be nice to show the fireball count in the group list as well.

Derived attributes are a recent addition to Core Data that allow you to create an entity attribute that's computed from child entity data each time the context is saved and stored in the persistent store. This makes it efficient, because you don't need to recompute it each time it's read.

You create a derived attribute in your managed object model. Open FireballWatch.xcdatamodeld and select the FireballGroup entity. Find the Attributes section and click the plus button to add a new attribute. Call it fireballCount and set the type to Integer 64.

In the Data Model inspector on the right, check the Derived checkbox, which reveals the Derivation field. In this field, type the following:

fireballs.@count

This uses the predicate aggregate function @count and acts on the existing fireballs relationship to return the count of how many fireballs are child entities of this group.

Remember to save your managed object model.

All that's left to do is display the count.

Open FireballGroupList.swift, located in the View group, and find the following line:

Text("\(group.name ?? "Untitled")")

Replace it with the following:

HStack {
  Text("\(group.name ?? "Untitled")")
  Spacer()
  Image(systemName: "sun.max.fill")
  Text("\(group.fireballCount)")
}

This simply adds an icon and the fireball count to each row. Build and run to see how it displays:

Perfect!

Where to Go From Here?

Well done! Take a step back and admire your work. You should feel proud knowing the world is a little safer from alien invasion because of your app.

You can download the completed project using the Download Materials button at the top or bottom of this tutorial.

If you're looking for a challenge, try adding code to delete the unnecessary transaction history after it's already been processed, to save the history from growing indefinitely. There's a handy method for the job: NSPersistentHistoryChangeRequest.deleteHistoryBefore(_:).

If you're looking to learn even more about Core Data, I recommend:

• Making Apps with Core Data from WWDC2019
• Using Core Data With CloudKit from WWDC 2019
• Core Data: Sundries and maxims from WWDC 2020

I hope you enjoyed this Core Data tutorial. If you have any questions or comments, please join the forum discussion below.

Stay vigilant!