14. In Practice: Project "News"

Written by Marin Todorov

In the past few chapters, you learned about few practical applications of the Combine integration in Foundation types. You learned how to use URLSession‘s data task publisher to make network calls, you saw how to observe KVO-compatible objects with Combine and more.

In this chapter, you will combine your solid knowledge about operators with some of the Foundation integrations you just discovered and will work through a series of tasks like in the previous “In Practice” chapter. This time around, you will work on building a Hacker News API client.

“Hacker News,” whose API you are going to be using in this chapter, is a social news website focused on computers and entrepreneurship. If you haven‘t already, you can check them out at: https://news.ycombinator.com.

In this chapter, you will work in an Xcode playground focusing only on the API client itself.

In Chapter 15, “In Practice: Combine & SwiftUI,” you will take the completed API and use it to build a real Hacker News reader app by plugging the network layer into a SwiftUI-based user interface. Along the way, you will learn the basics of SwiftUI and how to make your Combine code work with the new declarative Apple framework for building amazing, reactive app UIs.

Without further ado, let‘s get started!

Getting Started With the Hacker News API

Open the included starter playground API.playground in projects/starter and peek inside. You will find some simple starter code included to help you hit the ground running and let you focus on Combine code only:

Inside the API type, you will find two nested helper types:

• An enum called Error which features two custom errors your API will throw in case it cannot reach the server or it cannot decode the server response.
• A second enum called EndPoint which contains the URLs of the two API endpoints your type is going to be connecting to.

Further down, you will find the maxStories property. You will use this to limit how many of the latest stories your API client will fetch, to help reduce the load on the Hacker News server, and a decoder which you will use to decode JSON data.

Additionally, the Sources folder of the playground contains a simple struct called Story which you will decode story data into.

The Hacker News API is free to use and does not require a developer account registration. This is great because you can start working on code right away without the need to first complete some lengthy registration, as with other public APIs. The Hacker News team wins a ton of karma points!

Getting a Single Story

Your first task is to add a method to API which will contact the server using the EndPoint type to get the correct endpoint URL and will fetch the data about a single story. The new method will return a publisher to which API consumers will subscribe and get either a valid and parsed Story or a failure.

Scroll down the playground source code and find the comment saying // Add your API code here. Just below that line, insert a new method declaration:

func story(id: Int) -> AnyPublisher<Story, Error> {
  return Empty().eraseToAnyPublisher()
}

To avoid compilation errors in your playground, you return an Empty publisher which completes immediately. As you‘ll finish building the method body, you‘ll remove the expression and return your new subscription, instead.

As mentioned, this publisher‘s output is a Story and its failure is the custom API.Error type. As you will see later on, in case there are network errors or other mishaps, you will need to convert those into one of the API.Error cases to match the expected return type.

Start modeling the subscription by creating a network request to the single-story endpoint of the Hacker News API. Inside the new method, above the return statement, insert:

URLSession.shared
  .dataTaskPublisher(for: EndPoint.story(id).url)

You start by making a request to Endpoint.story(id).url. The url property of the endpoint contains the complete HTTP URL to request. The single story URL looks like this (with a matching ID): https://hacker-news.firebaseio.com/v0/item/12345.json (Visit https://bit.ly/2nL2ojS if you’d like to preview the API response.)

Next, to parse JSON on a background thread and keep the rest of the app responsive, let‘s create a new custom dispatch queue. Add a new property to API above the story(id:) method like so:

private let apiQueue = DispatchQueue(
  label: "API",
  qos: .default,
  attributes: .concurrent
)

You will use this queue to process JSON responses and, therefore, you need to switch your network subscription to that queue. Back in story(id:), add the line below calling dataTaskPublisher(for:):

.receive(on: apiQueue)

Once you‘ve switched to the background queue, you need to fetch the JSON data out of the response. The dataTaskPublisher(for:) publisher returns an output of type (Data, URLResponse) as a tuple but for your subscription, you need only the data.

Add another line to the method to map the current output to only the data from the resulting tuple:

.map(\.data)

The output type of this operator is Data, which you can feed to a decode operator and try converting the response to a Story.

Append to the subscription:

.decode(type: Story.self, decoder: decoder)

In case it receives anything but a valid story JSON, decode(...) will throw an error and the publisher will complete with a failure.

You will learn about error handling in detail in Chapter 16, “Error Handling.” In the current chapter, you will use few operators and get a taste of a few different ways to handle errors but you will not go into the nitty-gritty of how things work.

For the current story(id:) method, you will return an empty publisher in case things go south for any reason. This is easy to do by using the catch operator. Add to the subscription:

.catch { _ in Empty<Story, Error>() }

You ignore the thrown error and return Empty(). This, as you hopefully still remember, is a publisher that completes immediately without emitting any output values like so:

Handling the upstream errors this way via catch(_) allows you to:

• Emit the value and complete if you get back a Story.
• Return an Empty publisher which completes successfully without emitting any values, in case of a failure.

Next, to wrap up the method code and return your neatly designed publisher, you need to replace the current subscription at the end. Add:

.eraseToAnyPublisher()

You can now remove the temporary Empty you‘ve added before. Find the following line and remove it:

return Empty().eraseToAnyPublisher()

Your code should now compile with no issues, but just to make sure you haven‘t missed a step in all the excitement, review your progress so far and make sure your completed code looks like this:

func story(id: Int) -> AnyPublisher<Story, Error> {
  URLSession.shared
    .dataTaskPublisher(for: EndPoint.story(id).url)
    .receive(on: apiQueue)
    .map(\.data)
    .decode(type: Story.self, decoder: decoder)
    .catch { _ in Empty<Story, Error>() }
    .eraseToAnyPublisher()
}

Even though your code compiles, this method still won‘t produce any output just yet. You‘re about to take care of that next.

Now you can instantiate API and try calling into the Hacker News server.

Scroll down just a bit and find the comment line // Call the API here. This is a good spot to make a test API call. Insert the following code:

let api = API()
var subscriptions = [AnyCancellable]()

Fetch a story by providing a random ID to test with by adding:

api.story(id: 1000)
   .sink(receiveCompletion: { print($0) },
         receiveValue: { print($0) })
   .store(in: &subscriptions)

You create a new publisher by calling api.story(id: 1000) and subscribe to it via sink(...) which prints any output values or completion event. To keep the subscription alive until the request has completed you store it in subscriptions.

As soon as the playground runs again, it will make a network call to hacker-news.firebaseio.com and print the result in the console:

The returned JSON data from the server is a rather simple structure like this:

{
  "by":"python_kiss",
  "descendants":0,
  "id":1000,
  "score":4,
  "time":1172394646,
  "title":"How Important is the .com TLD?",
  "type":"story",
  "url":"http://www.netbusinessblog.com/2007/02/19/how-important-is-the-dot-com/"
}

The Codable conformance of Story parses and stores the values of the following properties: by, id, time, title and url.

Once the request completes successfully, you‘ll see the following output, or a similar output in case you changed the 1000 value in the request, in the console:

How Important is the .com TLD?
by python_kiss
http://www.netbusinessblog.com/2007/02/19/how-important-is-the-dot-com/
-----
finished

The Story type conforms to CustomDebugStringConvertible and it has a custom debugDescription that returns the title, author name and story URL neatly ordered, like above.

The output ends with a finished completion event. To try what happens in case of an error, replace the id 1000 with -5 and check the output in the console. You will only see finished printed because you caught the error and returned Empty().

Nice work! The first method of the API type is complete and you exercised some of the concepts you covered in previous chapters like calling the network and decoding JSON. Additionally you got a gentle introduction to basic dispatch queue switching and some easy error handling. You will cover these in more detail in future chapters.

Despite what an incredibly nice exercise this task was, you‘re probably hungry for more. So, in the next section, you will dig deeper and lay some serious code down.

Multiple Stories via Merging Publishers

Getting a single story out of the API server was a relatively straight forward task. Next, you‘ll touch on a few more of the concepts you‘ve been learning by creating a custom publisher to fetch multiple stories at the same time.

The new method mergedStories(ids:) will get a story publisher for each of the given story ids and merge them all together. Add this new method declaration to the API type after the story(id:) method you implemented earlier:

func mergedStories(ids storyIDs: [Int]) -> AnyPublisher<Story, Error> {

}

What this method will essentially do is call story(id:) for each of the given ids and then flatten the result into a single stream of output values.

First of all, to reduce the number of network calls during development, you will fetch only the first maxStories ids from the provided list. Start the new method by inserting the following code:

let storyIDs = Array(storyIDs.prefix(maxStories))

To get started, create the first publisher:

precondition(!storyIDs.isEmpty)

let initialPublisher = story(id: storyIDs[0])
let remainder = Array(storyIDs.dropFirst())

By using story(id:), you create the initialPublisher publisher that fetches the story with the first id in the list.

Next, you will use reduce(_:_:) from the Swift standard library on the remaining story ids to merge each next story publisher into the initial publisher like so:

To reduce the rest of the stories into the initial publisher add:

return remainder.reduce(initialPublisher) { combined, id in

}

reduce(_:_:) will start with the initial publisher and provide each of the ids in the remainder array to the closure to process. Insert this code to create a new publisher for the given story id in the empty closure, and merge it to the current combined result:

return combined
  .merge(with: story(id: id))
  .eraseToAnyPublisher()

The final result is a publisher which emits each successfully fetched story and ignores any errors that each of the single-story publishers might encounter.

Note: Congratulations, you just created a custom implementation of the MergeMany publisher. Working through the code yourself was not in vain though. You learned about operator composition and how to apply operators like merge and reduce in a real-world use case.

With the new API method completed, scroll down to this code and comment or delete it to speed up the execution of the playground while testing your newer code:

api.story(id: -5)
   .sink(receiveCompletion: { print($0) },
         receiveValue: { print($0) })
   .store(in: &subscriptions)

In place of the just deleted code, insert:

api.mergedStories(ids: [1000, 1001, 1002])
   .sink(receiveCompletion: { print($0) },
         receiveValue: { print($0) })
   .store(in: &subscriptions)

Let the playground run one more time with your latest code. This time, you should see in the console these three story summaries:

How Important is the .com TLD?
by python_kiss
http://www.netbusinessblog.com/2007/02/19/how-important-is-the-dot-com/
-----

Wireless: India's Hot, China's Not
by python_kiss
http://www.redherring.com/Article.aspx?a=21355
-----

The Battle for Mobile Search
by python_kiss
http://www.businessweek.com/technology/content/feb2007/tc20070220_828216.htm?campaign_id=rss_daily
-----
finished

Another success along your path of learning Combine! In this section, you wrote a method that combines any number of publishers and reduces them to a single one. That‘s very helpful code to have around, as the built-in merge operator can merge only up to 8 publishers. Sometimes, however, you just don‘t know how many publishers you‘ll need in advance!

Getting the Latest Stories

In this final chapter section, you will work on creating an API method that fetches the list of latest Hacker News stories.

Do you see a pattern in this chapter? First, you reused your single story method to fetch multiple stories. Now, you are going to reuse the multiple stories method to fetch the list of latest stories.

Add the new empty method declaration to the API type as follows:

func stories() -> AnyPublisher<[Story], Error> {
  return Empty().eraseToAnyPublisher()
}

Like before, you return an Empty object to prevent any compilation errors while you construct your method body and publisher.

Unlike before, though, this time your returned publisher‘s output is a list of stories. You will design the publisher to fetch multiple stories and accumulate them in an array, emitting each intermediary state as the responses come in from the server.

This behavior will allow you to, in the next chapter, bind this new publisher directly to a List UI control that will automatically animate the stories live on-screen as they come in from the server.

Begin, as you did previously, by firing off a network request to the Hacker News API. Insert the following in your new method, above the return statement:

URLSession.shared
  .dataTaskPublisher(for: EndPoint.stories.url)

The stories endpoint lets you hit the following URL to get the latest story ids: https://hacker-news.firebaseio.com/v0/newstories.json.

Again, you need to grab the data component of the emitted result. So, map the output by adding:

.map(\.data)

The JSON response you will get from the server is a plain list like this:

[1000, 1001, 1002, 1003]

You need to parse the list as an array of integer numbers and, if that succeeds, you can use the ids to fetch the matching stories.

Append to the subscription:

.decode(type: [Int].self, decoder: decoder)

This will map the current subscription output to an [Int] and you will use it to fetch the corresponding stories one-by-one from the server.

Now is the moment, however, to go back to the topic of error handling for a moment. When fetching a single story, you just ignore any errors. But, in stories(), let‘s see how you can do a little more than that.

API.Error is the error type to which you will constrain the errors thrown from stories(). You have two errors defined as enumeration cases:

• invalidResponse: for when you cannot decode the server response into the expected type.
• addressUnreachable(URL): for when you cannot reach the endpoint URL.

Currently, your subscription code in stories() can throw two types of errors:

• dataTaskPublisher(for:) could throw different variations of a URLError when a network problem occurs.
• decode(type:decoder:) could throw a decoding error when the JSON doesn‘t match the expected type.

Your next task is to handle those various errors in a way that would map them to the single API.Error type to match the expected failure of the returned publisher.

You will jump the gun yet another time and get a “soft” introduction to another error handling operator. Append this code to your current subscription, after decode:

.mapError { error -> API.Error in
  switch error {
  case is URLError:
    return Error.addressUnreachable(EndPoint.stories.url)
  default:
    return Error.invalidResponse
  }
}

mapError handles any errors occurring upstream and allows you to map them into a single error type — similar to how you use map to change the type of the output.

In the code above, you switch over any errors and:

• In case error is of type URLError and therefore occurred while trying to reach the stories server endpoint, you return .addressUnreachable(_).
• Otherwise, you return .invalidResponse as the only other place where an error could occur. Once successfully fetched, the network response is decoding the JSON data.

With that, you matched the expected failure type in stories() and can leave it to the API consumers to handle errors downstream. You will use stories() in the next chapter. So, you will do a little more with error handling before you get to Chapter 16, “Error Handling,” and dive into the details.

So far, the current subscription fetches a list of ids from the JSON API but doesn‘t do much on top of that. Next, you will use a few operators to filter unwanted content and map the id list to the actual stories.

First, filter empty results — in case the API goes bonkers and returns an empty list for its latest stories. Append:

.filter { !$0.isEmpty }

This will guarantee that downstream operators receive a list of story ids with at least one element. This is very handy because, as you remember, mergedStories(ids:) has a precondition ensuring that its input parameter is not empty.

To use mergedStories(ids:) and fetch the story details, you will flatten all the story publishers by appending a flatMap operator:

.flatMap { storyIDs in
  return self.mergedStories(ids: storyIDs)
}

Merging all the publishers into a single downstream will produce a continuous stream of Story values. The publisher emits these downstream as soon as they are fetched from the network:

You could leave the current subscription as is right now but you‘d like to design the API to be easily bindable to a list UI control. This will allow the consumers to simply subscribe stories() and assign the result to an [Story] property in their view controller or SwiftUI view.

To achieve that, you will need to aggregate the emitted stories and map the subscription to return an ever-growing array — instead of single Story values.

It‘s time for some serious magic! Remember the scan operator from Chapter 3, “Transforming Operators” I know that was some time ago, but, this is the operator that will help you achieve your current task. So, if needed, jump back to that chapter and come back here when refreshed on scan.

Append to your current subscription:

.scan([]) { stories, story -> [Story] in
  return stories + [story]
}

You let scan(...) start emitting with an empty array. Each time a new story is being emitted, you append it to the current aggregated result via stories + [story].

This addition to the subscription code changes its behavior so that you get the — sort of — buffered contents each time you receive a new story from the batch you are working on:

Finally, it can‘t hurt to sort the stories before emitting output. Story conforms to Comparable so you don‘t need to implement any custom sorting. You just need to call sorted() on the result. Append:

.map { $0.sorted() }

Wrap up the current, rather long, subscription by type erasing the returned publisher. Append one last operator:

.eraseToAnyPublisher()

At this point, you can find the following temporary return statement, and remove it:

return Empty().eraseToAnyPublisher()

Your playground should now finally compile with no errors. However, it still shows the test data from the previous chapter section. Find and comment out:

api.mergedStories(ids: [1000, 1001, 1002])
   .sink(receiveCompletion: { print($0) },
         receiveValue: { print($0) })
   .store(in: &subscriptions)

In its place, insert:

api.stories()
   .sink(receiveCompletion: { print($0) },
         receiveValue: { print($0) })
   .store(in: &subscriptions)

This code subscribes to api.stories() and prints any returned output and completion events.

Once you let the playground run one more time, you should see a dump of the latest Hacker News stories in the console. You print the list iteratively. Initially, you will see the story fetched first on its own:

[
More than 70% of America’s packaged food supply is ultra-processed
by xbeta
https://news.northwestern.edu/stories/2019/07/us-packaged-food-supply-is-ultra-processed/
-----]

Then, the same one accompanied by a second story:

[
More than 70% of America’s packaged food supply is ultra-processed
by xbeta
https://news.northwestern.edu/stories/2019/07/us-packaged-food-supply-is-ultra-processed/
-----, 
New AI project expects to map all the word’s reefs by end of next year
by Biba89
https://www.independent.co.uk/news/science/coral-bleaching-ai-reef-paul-allen-climate-a9022876.html
-----]

Then, a list of the same stories plus a third one and so on:

[
More than 70% of America’s packaged food supply is ultra-processed
by xbeta
https://news.northwestern.edu/stories/2019/07/us-packaged-food-supply-is-ultra-processed/
-----, 
New AI project expects to map all the word’s reefs by end of next year
by Biba89
https://www.independent.co.uk/news/science/coral-bleaching-ai-reef-paul-allen-climate-a9022876.html
-----, 
People forged judges’ signatures to trick Google into changing results
by lnguyen
https://arstechnica.com/tech-policy/2019/07/people-forged-judges-signatures-to-trick-google-into-changing-results/
-----]

Please note, since you‘re fetching live data from the Hacker News website the stories, what you see in your console will be different as more and more stories are added every few minutes. To see that you are indeed fetching live data, wait a few minutes and re-run the playground. You should see some new stories show up alongside the ones you already saw.

Nice effort working through this somewhat longer section of the chapter! You‘ve completed the development of the Hacker News API client and are ready to move on to the next chapter. There, you will use SwiftUI to build a proper Hacker News reader app.

Challenges

There is nothing to add per se to the API client but you can still play around a little if you‘d like to put some more work into this chapter‘s project.

Challenge 1: Integrating the API Client With UIKit

As already mentioned, in the next chapter, you will learn about SwiftUI and how to integrate it with your Combine code.

In this challenge, try to build an iOS app that uses your completed API client to display the latest stories in a table view. You can develop as many details as you want and add some styling or fun features but the main point to exercise in this challenge is subscribing the API.stories() and binding the result to a table view — much like the bindings you worked on in Chapter 8, “In Practice: Project ‘Collage’.”

In case you’re not interested in working with UIKit - no worries, this challenge is just an excercise you can also skip and dive head first into Chapter 15, “In Practice: Combine & SwiftUI“.

If you successfully work through the challenge as described, you should see the latest stories “pour in” when you launch the app in the simulator, or on your device:

Key Points

• Foundation includes several publishers that mirror counterpart methods in the Swift standard library and you can even use them interchangeably as you did with reduce in this chapter.
• Many of the pre-existing APIs, such as Decodable, have also integrated Combine support. This lets you use one standard approach across all of your code.
• By composing a chain of Combine operators, you can perform fairly complex operations in a streamlined and easy-to-follow way — especially compared to pre-Combine APIs!

Where to Go From Here?

Congratulations on completing the “Combine in Action” section! What a ride this was, wasn‘t it?

You‘ve learned most of what Combine‘s foundations has to offer, so it‘s now times to pull out the big guns in an entire section dedicated to advanced topics in the Combine framework, starting with building an app that uses both SwiftUI and Combine.