8. Transforming Operators in Practice

Written by Marin Todorov

In the previous chapter, you learned about the real workhorses behind reactive programming with RxSwift: the map and flatMap dynamic duo. Of course, those aren’t the only two operators you can use to transform observables, but a program can rarely do without using those two at least few times. The more experience you gain with these two, the better (and shorter) your code will be.

You already got to play around with transforming operators in the safety of a Swift playground, so hopefully you’re ready to take on a real-life project. Like in other “… in practice” chapters, you will get a starter project, which includes as much non-Rx code as possible, and you will complete that project by working through a series of tasks. In the process, you will learn more about map and flatMap, and in which situations you should use them in your code.

Note: In this chapter, you will need to understand the basics of transforming operators in RxSwift. If you haven’t worked through Chapter 7, “Transforming Operators”, do that first and then come back to this chapter.

Without further ado, it’s time to get this show started!

Getting started with GitFeed

I wonder what the latest activity is on the RxSwift repository? In this chapter, you’ll build a project to tell you this exact thing.

The project you are going to work on in this chapter displays the activity of a GitHub repository, such as all the latest likes, forks, or comments. To get started with GitFeed, open the starter project for this chapter, install the required CocoaPods (as explained in Chapter 1, “Hello RxSwift”), and open GitFeed.xcworkspace.

The app is a simple navigation controller project and features a single table view controller in which you will display the latest activity fetched from GitHub’s JSON API.

Note: The starter project is set to display the activity of https://github.com/ReactiveX/RxSwift, but if you’d like to change it to any other repository of your choice, feel free.

Run the app and you will see the empty default screen:

There’s nothing too complex going on right now, but you’ll soon have this whole setup ablaze!

The project will feature two distinct storylines:

• The main plot is about reaching out to GitHub’s JSON API, receiving the JSON response, and ultimately converting it into a collection of objects.
• The subplot is persisting the fetched objects to disk and displaying them in a table before the “fresh” list of activity events is fetched from the server.

You will see that these two complement each other perfectly — and there are plenty of opportunities to use both map and flatMap to build what’s required.

Fetching data from the web

Hopefully you’ve used the URLSession API before and have a general idea of its workflow. In summary: you create a URLRequest containing a web URL and parameters, then send it off to the Internet. After a bit, you receive the server response.

With your current knowledge of RxSwift, it won’t be difficult to add a reactive extension to the URLSession class. Although you will specifically look as adding a proper reactive extension to URLSession in Chapter 17, “Creating a Custom Reactive Extension,” in this chapter you will simply use a solution boxed with RxCocoa — RxSwift’s companion library.

If you peek into GitFeed’s Podfile, you will notice that you import two different CocoaPods: RxSwift and RxCocoa. What gives?

RxCocoa is a library based on RxSwift, which implements many helpful APIs to aid with developing against RxSwift on Apple’s platforms. In an effort to keep RxSwift itself as close as possible to the common Rx API shared between all implementations such as RxJS, RxJava, and RxPython, all “extra functionality” is separated into RxCocoa. You will learn about it in more detail in Chapters 12 and 13.

You will use the default RxCocoa URLSession extension to quickly fetch JSON from GitHub’s API in this chapter.

Using map to build a request

The first task you will undertake is to build a URLRequest you will send off to GitHub’s server. You will follow a reactive approach that might not make sense immediately, but don’t worry — when you re-visit that part of the project later on, you will appreciate it!

Open ActivityController.swift and peek inside. You configure the view controller’s UI in viewDidLoad(), and when you’re finished, you call refresh(). refresh() in turn calls fetchEvents(repo:) and hands over to it the repo name "ReactiveX/RxSwift".

It is in fetchEvents(repo:) where you will add most of your code in this section. To get started, add the following:

let response = Observable.from([repo])

To start building the web request, you begin with a simple string, which is the repository’s full name. The idea to start with a string instead of directly building a URLRequest is to be flexible with the observable’s input. This means you won’t have a lot of issues if you decide to change which repo you work with — which is what you will do in the Challenges section.

Next, take the address string and create the fully qualified URL of the activity API endpoint:

.map { urlString -> URL in
  return URL(string: "https://api.github.com/repos/\(urlString)/events")!
}

You use a couple of shortcuts to create the full URL by using a hard-coded string and force unwrapping the result. You end up with the URL to access the latest events’ JSON. Have you noticed that you specified the closure’s output type? Did you really have to do that? The obvious answer is no; usually you don’t need to explicitly spell out closure input and output types. You can usually leave it to the compiler to figure those out.

However, especially in code where you have several map and/or flatMap operators chained together, you might need to help the compiler out. It will sometimes get lost in figuring out the proper types, but you can aid it by at least spelling out the output types. If you see an error about mismatched or missing types, you can add more type information to your closures and it’ll probably fix the problem.

But enough about compiler woes — back to coding!

Now that you have a URL, you can move on to transforming it into a complete request. Chain to the last operator:

.map { url -> URLRequest in
  return URLRequest(url: url)
}

Easy enough: you use map to transform a URL to a URLRequest by using the provided web address.

Nice work! You’ve chained a couple of map operators to create a more complex transformation:

Now it’s time to bring flatMap into play and fetch some JSON.

Using flatMap to wait for a web response

In the previous chapter, you learned that flatMap flattens out observable sequences. One of the common applications of flatMap is to add some asynchronicity to a transformation chain. Let’s see how that works.

When you chain several transformations, that work happens synchronously. That is to say, all transformation operators immediately process each other’s output:

When you insert a flatMap in between, you can achieve different effects:

• You can flatten observables that instantly emit elements and complete, such as the Observable instances you create out of arrays of strings or numbers.
• You can flatten observables that perform some asynchronous work and effectively “wait” for the observable to complete, and only then let the rest of the chain continue working.

What you need to do in your GitFeed code is something like this:

To do that, append the following code to the operator chain that you have so far:

.flatMap { request -> Observable<(response: HTTPURLResponse, data: Data)> in
  return URLSession.shared.rx.response(request: request)
}

You use the RxCocoa response(request:) method on the shared URLSession object. That method returns an Observable<(response: HTTPURLResponse, data: Data)>, which completes whenever your app receives the full response from the web server. You will learn more about the RxCocoa rx extensions and how to extend Foundation and UIKit classes yourself later on in the book.

Note: Since response(request:) can error out if there’s no connectivity or the URL is malformed, you should catch any errors inside the flatMap body. You will see how to do that in Chapter 14.

In the code you just wrote, flatMap allows you to send the web request and receive a response without the need of protocols and delegates. How cool is that? Freely mixing map and flatMap transformations (as above) enables the kind of linear yet asynchronous code you hopefully are starting to appreciate more and more in this book.

Finally, to allow more subscriptions to the result of the web request, chain one last operator. You will use share(replay:, scope:) to share the observable and keep in a buffer the last emitted event:

.share(replay: 1)

Unlike in Chapter 6, “Filtering Operators in Practice”, this time you use share(replay:, scope:). Let’s shortly have a look why.

share() vs. share(replay: 1)

URLSession.rx.response(request:) sends your request to the server, and upon receiving the response, emits a .next event just once with the returned data, and then completes.

In this situation, if the observable completes and then you subscribe to it again, that will create a new subscription and will fire another identical request to the server.

To prevent situations like this, you use share(replay:scope:). This operator keeps a buffer of the last replay elements emitted and feeds them to any newly subscribed observers. Therefore, if your request has completed and a new observer subscribes to the shared sequence (via share(replay:scope:)), it will immediately receive the buffered response from the previously-executed network request.

There are two scopes available to choose from: .whileConnected and .forever. The former will buffer elements up to the point where it has no subscribers, and the latter will keep the buffered elements forever. That sounds nice, but consider the implications on how much memory is used by the app.

Let’s see how the app would behave when using either scope:

• .forever: the buffered network response is kept forever. New subscribers get the buffered response.
• .whileConnected: the buffered network response is kept until there are no more subscribers, and is then discarded. New subscribers get a fresh network response.

The rule of thumb for using share(replay:scope:) is to use it on any sequences you expect to complete, or ones that cause a heavy workload and are subscribed to multiple times; this way you prevent the observable from being re-created for any additional subscriptions.

You can also use this if you’d like new observers to automatically receive the last n emitted events.

Transforming the response

It will probably not come as a surprise that along with all map transformations you did before sending the web request, you will need to do some more after you receive its response.

If you think about it, the URLSession class gives you back a Data object, and this is not an object you can work with right away. You need to transform it to an array of native objects you can safely use in your code.

You’ll now create a subscription to the response observable that converts the response data into objects. Just after that last piece of code you wrote, add the following code on a new line:

response
  .filter { response, _ in
    return 200..<300 ~= response.statusCode
  }

With the filter operator above, you easily discard all error response codes. Your filter will only let through responses having a status code between 200 and 300, which is all the success status codes.

Note: Interested in the HTTP response codes list? Check out this article on Wikipedia: https://bit.ly/1fFATGL.

What’s with that pesky, built-in ~= operator? It’s one of the lesser-known Swift operators, and when used with a range on its left side, checks if the range includes the value on its right side.

Also note you’re going to ignore the non-successful status codes, instead of having your observable send an error event. This is a stylistic choice meant to keep the code simple for now, but you’ll see in later chapters how easy error propagation with Rx can be.

The data you receive will generally be a JSON-encoded server response containing a list of event objects. You will use a Decodable-conforming struct to try and decode the data response you received.

Open Event.swift from the starter project and you will see an Event struct, already conforming to the Codable protocol.

Back in ActivityController.swift, you will add another operator after the freshly inserted filter. This time around, you’d like to transform the Data you receive from the API response into a list of Events.

In case the response data cannot be decoded into events, you will stop processing the response altogether.

You could achieve the above with a map and a following filter, but just as with Swift’s collection types, you can use a shorthand for this called compactMap.

In ActivityController.swift, append a compactMap operator immediately after the last filter:

.compactMap { _, data -> [Event]? in
  return try? JSONDecoder().decode([Event].self, from: data)
}

compactMap “lets through” any non-nil values, and filters any nils.

Let’s deconstruct this piece of code:

• You discard the response object and take only the response data.
• You create a JSONDecoder and attempt to decode the response data as an array of Events.
• You use a try? to return a nil value in case the decoder throws an error while decoding the JSON data.

It’s really cool how RxSwift forces you to encapsulate these discrete pieces of work by using operators. And as an added benefit, you are always guaranteed to have the input and output types checked at compile time.

The compactMap operator will effectively discard any error responses or any responses that do not contain new events since you last checked. You’ll implement fetching only new events later in the chapter, but you can account for this now and help out your future self.

Finally, it’s time to wrap up this seemingly endless chain of transformations and get to updating the UI. To simplify the code, you will write the UI code in a separate method. For now, simply append this code to the final operator chain:

.subscribe(onNext: { [weak self] newEvents in
  self?.processEvents(newEvents)
})
.disposed(by: bag)

Processing the response

Yes, it’s finally time to perform some side effects. You started with a simple string, built a web request, sent it off to GitHub, and received an answer back. You transformed the response to JSON and then to native Swift objects. Now it’s time to show the user what you’ve been cooking up behind the scenes all this time.

ActivityController already includes a placeholder method called processEvents(_:) ready to be fleshed out. In this method, you’ll grab the last 50 events from the repository’s event list and store the list into the subject property events on your view controller. You’ll do that manually for now, since you haven’t yet learned how to directly bind sequences to subjects.

Insert into processEvents():

var updatedEvents = newEvents + events.value
if updatedEvents.count > 50 {
  updatedEvents = [Event](updatedEvents.prefix(upTo: 50))
}

events.accept(updatedEvents)

You append the newly fetched events to the list by using events.accept(_:). Additionally, you cap the list to 50 objects. This way you will show only the latest activity in the table view.

Finally, you set the value of events and are ready to update the UI. Since the data source code is already included in ActivityController, you simply reload the table view to display the new data. To the end of processEvents(_:), add the following line:

tableView.reloadData()

Run the app, and you should see the latest activity from GitHub. Yours will be different, depending on the current state of the repo in GitHub.

However, at some point, the app should crash pretty heavily and you will notice Xcode showing you the issue in the code editor:

Unfortunately, you still haven’t looked into managing threads with RxSwift, so even though that’s not the recommended way to do things, let’s just use GCD to switch to the main thread and update the table. Wrap the call to reloadData() like so:

DispatchQueue.main.async {
  self.tableView.reloadData()
}

Since the code that came with the starter project in viewDidLoad() sets up a table refresh control, you can try to pull down the table.

As soon as you pull far enough, the refresh control calls refresh() and reloads the events.

If someone forked or liked the repo since the last time you fetched the repo’s events, you will see new cells appear on top.

There is a little issue when you pull down the table view: the refresh control never disappears, even if your app has finished fetching data from the API.

To hide it when you’ve finished fetching events, add the following code just below self.tableView.reloadData():

self.refreshControl?.endRefreshing()

endRefreshing() will hide the refresh control and reset the table view to its default state.

So far, you should have a good grasp of how and when to use map and flatMap. Throughout the rest of the chapter, you are going to tie off a few loose ends of the GitFeed project to make it more complete.

In the challenges, you will again work through some tasks requiring smart observable sequence transformations.

Persisting objects to disk

In this section, you are going to work on the subplot as described in the introduction, where you will persist objects to disk, so when the user opens the app they will instantly see the events you last fetched.

In this example, you are about to persist the events to a .plist file. The amount of objects you are about to store is small, so a .plist file will suffice for now.

First, add a new property to the ActivityController class:

private let eventsFileURL = cachedFileURL("events.json")

eventsFileURL is the file URL where you will store the events file on your device’s disk. It’s time to implement the cachedFileURL function to grab a URL to where you can read and write files. Add this outside the definition of the view controller class:

func cachedFileURL(_ fileName: String) -> URL {
  return FileManager.default
    .urls(for: .cachesDirectory, in: .allDomainsMask)
    .first!
    .appendingPathComponent(fileName)
}

Now, scroll down to processEvents(_:) and append this to the bottom:

let encoder = JSONEncoder()
if let eventsData = try? encoder.encode(updatedEvents) {
  try? eventsData.write(to: eventsFileURL, options: .atomicWrite)
}

In this code, you try to encode updatedEvents as a Data object. Next, you call write(to:options:) with the resulting piece of data and provide it the URL of the file where you want to create the file or overwrite an existing one.

Cool! processEvents(_:) is the place to perform side effects, so writing the events to disk in that place feels right. But where can you add the code to read the saved events from disk?

Since you need to read the objects back from the file just once, you can do that in viewDidLoad(). This is where you will check if there’s a file with stored events, and if so, load its contents into events.

Scroll up to viewDidLoad() and add this just above the call to refresh():

let decoder = JSONDecoder()
if let eventsData = try? Data(contentsOf: eventsFileURL),
   let persistedEvents = try? decoder.decode([Event].self, from: eventsData) {
  events.accept(persistedEvents)
}

This code works similarly to the one you used to save the objects to disk —  but in reverse.

You first read the store Data from disk; then, you create a JSONDecoder and attempt to decode the data back into an array of Events. You add the array of events into the events relay using its accept method, or an empty array on error; since you persisted the events to disk, they all should be valid, but hey — safety first!

That should do it. Delete the app from the Simulator, or from your device if you’re working there. Then run the app, wait until it displays the list of events, and then stop it from Xcode. Run the project a second time, and observe how the table view instantly displays the older data while the app fetches the latest events from the web.

Add a last-modified header to the request

To exercise flatMap and map one more time (yes, they simply are that important), you will optimize the current GitFeed code to request only events it hasn’t fetched before. This way, if nobody has forked or liked the repo you’re tracking, you will receive an empty response from the server and save on network traffic and processing power.

First, add a new property to ActivityController to store the file name of the file in question:

private let modifiedFileURL = cachedFileURL("modified.txt")

This time you don’t need a .plist file, since you essentially need to store a single string like Mon, 30 May 2017 04:30:00 GMT. This is the value of a header named Last-Modified that the server sends alongside the JSON response. You need to send the same header back to the server with your next request. This way, you leave it to the server to figure out which events you last fetched and if there are any new ones since then.

As you did previously for the events list, you will use a subject to keep track of the Last-Modified header. Add the following new property to ActivityController:

private let lastModified = BehaviorRelay<String?>(value: nil)

Scroll to viewDidLoad() and add this code above the call to refresh():

if let lastModifiedString = try? String(contentsOf: modifiedFileURL, encoding: .utf8) {
  lastModified.accept(lastModifiedString)
}

If you’ve previously stored the value of a Last-Modified header to a file, you will fetch it back by using Data(contentsOf:). This data is then used to optionally create a String which you then pass to the lastModified relay.

Start with filtering out the error responses. Move to fetchEvents() and create a second subscription to the response observable by appending the following code to the bottom of the method:

response
  .filter { response, _ in
    return 200..<400 ~= response.statusCode
  }

Next you need to:

• Filter all responses that do not include a Last-Modified header.
• Grab the value of the header.
• Filter the sequence once more, taking the header value into consideration.

It does sound like a lot of work, and you might be planning on using a filter, map, another filter, or more. In this section, you will use a single flatMap to easily filter the sequence.

You can use flatMap to filter the responses that don’t feature a Last-Modified header.

Append this to the operator chain from above:

.flatMap { response, _ -> Observable<String> in
  guard let value = response.allHeaderFields["Last-Modified"] as? String else {
    return Observable.empty()
  }
  return Observable.just(value)
}

You use guard to check if the response contains an HTTP header by the name of Last-Modified, whose value can be cast to a String.

If you can make the cast, you return an Observable<String> with a single element; otherwise, you return an Observable, which never emits any elements:

Now that you have the final value of the desired header, you can proceed to update the lastModified property and store the value to the disk. Add the following:

.subscribe(onNext: { [weak self] modifiedHeader in
  guard let self = self else { return }

  self.lastModified.accept(modifiedHeader)
  try? modifiedHeader.write(to: self.modifiedFileURL, atomically: true, encoding: .utf8)
})
.disposed(by: bag)

In your subscription’s onNext closure, you add the latest date to the lastModified relay using its accept(_) method and then call modifiedHeader.write(to:atomically:encoding:) to save to disk. In the end, you add the subscription to the view controller’s dispose bag.

To finish working through this part of the app, you need to use the stored header value in your request to GitHub’s API. Scroll toward the top of fetchEvents(repo:) and find the particular map below where you create a URLRequest:

.map { url -> URLRequest in
  return URLRequest(url: url)
}

Replace the above code with this:

.map { [weak self] url -> URLRequest in
  var request = URLRequest(url: url)
  if let modifiedHeader = self?.lastModified.value {
    request.addValue(modifiedHeader,
      forHTTPHeaderField: "Last-Modified")
  }
  return request
}

In this new piece of code, you create a URLRequest just as you did before, but you add an extra condition: if lastModified contains a value, no matter whether it’s loaded from a file or stored after fetching JSON, add that value as a Last-Modified header to the request.

This extra header tells GitHub that you aren’t interested in any events older than the header date. This will not only save you traffic, but responses which don’t return any data won’t count towards your GitHub API usage limit. Everybody wins!

In this chapter, you learned about different real-life use cases for map and flatMap — and built a cool project along the way, even though you still need to handle the results on the main thread (like the smart programmer you are).

But you can still do better! In the challenges section, you will work on adding a threading strategy to the project so that you can do transformations on a background thread and switch to the main thread to do UI updates. This will keep your app snappy and responsive.

In a further challenge, you will see how you can easily extend the project by throwing even more maps and flatMaps into the mix.

Once you work through the challenges, you can move on to the next chapter, where you will finally learn about combining operators to greatly simplify more complex subscriptions.

Challenge

Challenge: Fetch top repos and spice up the feed

In this challenge, you will go through one more map/flatMap exercise. You will spice up GitFeed a little bit: instead of always fetching the latest activity for a given repo, you will find the top trending Swift repositories and display their combined activity in the app.

At first sight, this might look like a lot of work, but in the end you’ll find it’s only about a dozen lines of code.

To get started, replace let response = Observable.from([repo]) in fetchEvents(repo:) with:

let response = Observable.from(["https://api.github.com/search/repositories?q=language:swift&per_page=5"])

This API endpoint will return a list of the top five popular Swift repositories. Since you don’t specify an order parameter in that API call, GitHub will order the returned results by their “score”, which is a secret magic GitHub computed property that has to do with each item’s relevance to the search terms.

Note: The GitHub JSON API is a great tool to play with. You can grab a bunch of very interesting data such as trending repositories, public activity, and more. If you are interested to learn more, visit the API homepage at https://developer.github.com/v3/.

Now proceed in exactly the same manner as you did in the chapter to transform that string into a URL and transform that in turn into a URLRequest. There’s no need to include a Last-Modified header.

Since you don’t need the response headers, you can use URLSession.shared.rx.json(request:), which is a method which directly returns the transformed JSON instead of raw data.

As the last step, you will need to get the JSON response as a [String: Any] dictionary and try grabbing its items key. items should contain a list of [String: Any] dictionaries, which represent each of the trending repos. You need the full_name of each of these.

This is the repo name that includes the user name and the repo name, such as icanzilb/EasyAnimation, realm/realm-cocoa, ReactiveX/RxSwift, and so on.

Use flatMap, and in case any of those assumptions fail, return Observable.empty() just as you did previously. If everything goes according to plan, return an Observable<String> created out of the list of the trending repos’ full names.

Now you can chain the existing code to that flatMap like so:

let response = Observable.from(["https://api.github.com/search/repositories?q=language:swift&per_page=5"])

[map to convert to to URLRequest]

[flatMap to fetch JSON back]

[flatMap to convert JSON to list of repo names,
  and create Observable from that list]

[existing code follows below]

.map { urlString -> URL in
  return URL(string: "https://api.github.com/repos/\(urlString)/events?per_page=5")!
}
.map { [weak self] url -> URLRequest in
  var request = URLRequest(url: url)
  ...
}

Now, each time you start the app or pull down the table to refresh, the app will get the list of top five Swift repositories and then fire off five different requests to GitHub to fetch the events for each repo.

If you end up seeing too many events from the same repository, you can cap the server response by adding a per_page=5 query parameter to the URL. Then it will store the events locally and update the table with the latest data:

If you’d like to play around some more, you can sort the combined list of events by date and other interesting ways. What other types of sorting or filtering can you come up with?

If you wrapped up this challenge successfully, you can consider yourself a transformation pro! Oh… if you could only use a map in real life to turn lead into gold, that would really be something! But data transformation with RxSwift comes a close second — and that’s great, too.