Vapor and Job Queues: Getting Started

In this Vapor tutorial, you’ll learn how to dispatch a job from your Vapor app to a queue hosted on Redis. You’ll also learn how to schedule jobs that you want to run in the future.

If you’ve ever gone to the supermarket or driven on the highway, chances are you’ve been in a real-life queue. With software, it’s not all that different. The program schedules a series of data to process, one after another.

Usually, this processing pattern follows a first-in-first-out (FIFO) principle.

In this tutorial, you’ll learn how to:

• Configure and run a queue
• Dispatch a job to your queue
• Create a scheduled job

Job queues in particular serve to coordinate asynchronous service-to-service communication. Generally, one process sends while another receives. Each interacts with the queue independent of the other. They likely aren’t even aware of each other’s existence!

Queues are especially used in serverless and microservice architectures. But even monolithic applications can see benefits from job queues, such as offloading CPU or disk-writing intensive tasks.

Here are some examples of tasks that queues could help with:

• Sending emails and text messages, as part of two-factor or email authentication, for example
• Performing complex or long-running database operations, such as importing and processing thousands of items into your database
• Speeding up response time by delaying non-critical processing
• Buffering or batching work when you operate on large data sets
• Smoothing spiky workloads during peak hours by offloading the whole request
• Ensuring increased job integrity and resilience by persisting until the jobs are finished

Getting Started

Download the starter project by clicking the Download Materials button at the top or bottom of this tutorial. Then, navigate to the starter folder.

Looking at the Vapor Project

Open the Vapor app in Xcode by double-clicking the Package.swift.

While you wait for the Swift Package Manager (SwiftPM) to resolve dependencies, check out the existing project in /Sources/App:

File explorer for the starter project with highlight of key folders and files.

You should pay attention to a few things:

1. Controllers: You’ll see one controller — NewsletterRecipientAPIController.swift. It contains the method to sign up for your newsletter.
2. Migrations: You’ll find the migrations for Newsletter and NewsletterRecipient here. You’ll also find a third migration — SeedNewsletter. It creates a new newsletter in your database.
3. Models: This contains the two database models, Newsletter and NewsletterRecipient.
4. configure.swift: Here’s where you’ll find everything required to get started with the tutorial. This includes Databases, Migrations and Routes. This also is where you’ll register your Queue and Jobs later.
5. routes.swift: You’ll need only one route for the project, which is to sign up for a newsletter. It’s registered under /api/newsletter/sign-up.

Now, you’re ready to begin the project!

Running the Vapor Project

After SwiftPM finishes resolving the project’s dependencies, set the Working Directory to your project folder.

In Xcode, go to the Newslettering run scheme.

Under Options, turn on Working Directory and select your project folder.

Set your build folder in Xcode before running the app.

Close the Scheme Editor window and verify that your selected platform is My Mac. Then, build and run.

Vapor Queue Build and Run

Now, switch to Paw (or your preferred REST Client) and prepare the following request:

POST http://localhost:8080/api/newsletter/sign-up
Content-Type: application/json

{
   "email": "test1@newslettering.com",
   "name": "Test User 1"
}

Next, send the request:

Newsletter sign-up request and response.

You should get a 201 Created response from the server. If you check the response body, you’ll see the email and name you sent in the request.

Getting Started with Vapor Queues

You’ll use the Vapor Queues package in the rest of the tutorial. It’s a job queue system that allows you to run jobs on request as well as schedule them to run in the future.

To take advantage of the Queues package, you’ll need a job storage driver. For this project, you’ll use Vapor’s QueuesRedisDriver.

Configuring Your QueuesRedisDriver

Open configure.swift and, as part of the list of packages to import, add the following:

import Queues
import QueuesRedisDriver

Here, you expose the Queues and QueuesRedisDriver APIs so you can configure them.

Next, configure your queue to run on Redis.

Inside configure(_:), after try Application.autoMigrate().wait(), add the following:

try app.queues.use(.redis(url: "redis://127.0.0.1:6379"))

This is where you register that the queue will be running on Redis with the predefined URL.

Now, let’s get your Redis server up and running.

Start Your Redis Server

Open your terminal and navigate to the project folder.

Start the Redis container by running:

$ docker-compose run -p 6379:6379 cache

This starts a Docker Redis container that is listening on port :6379 and exposes the port to your local machine.

Running Your Queue and Scheduled Jobs

There are two “modes” you can run your scheduled jobs as:

• Separate process
• In-app process

To run the queue as a separate process, run the following command in your terminal:

$ swift run Run queues

This command starts a separate worker in the background that listens for any dispatched jobs.

To run scheduled jobs in a separate process, pass the --scheduled flag to the same command above.

The second option is to run the queue as an in-app process, which is what you’ll do for the rest of the tutorial.

Inside configure.swift, right under where you configured the queue driver, add the following:

// 1
try app.queues.startInProcessJobs()
// 2
try app.queues.startScheduledJobs()

With this code, you:

1. Start the queue as an in-app process. You’ll run your jobs on the default queue, so you won’t need a queue name as an argument.
2. Start the scheduled jobs as an in-app process. Your app will then check for any scheduled jobs. If none exist, it’ll exit the worker again.

Now, build and run your app. You’ll see a warning in the debug editor before the server starts:

[ WARNING ] No scheduled jobs exist, exiting scheduled jobs worker.

Console output after you have registered the scheduled job

You’ve learned how to run jobs. Now it’s time to dispatch them.

Dispatching Your First Job

When a new reader signs up to your newsletter, you want to send them a welcome email to say, “Thank you”.

This is a typical example of a process that you would dispatch to a job because you don’t want the responsiveness of your webpage to depend on a third-party email API to completing.

Creating RecipientWelcomeEmailJob

In Xcode, add a new Jobs folder to your app and add a RecipientWelcomeEmailJob.swift file.

Open the file and add the following:

// 1
import Queues
import Vapor

// 2
struct WelcomeEmail: Codable {
 let to: String
 let name: String
}

// 3
struct RecipientWelcomeEmailJob: Job {
  // 4
  typealias Payload = WelcomeEmail
  // 5
  func dequeue(
    _ context: QueueContext,
    _ payload: WelcomeEmail
  ) -> EventLoopFuture<Void> {
    print("Send welcome email to \(payload.to). Greet the user as \(payload.name).")
    return context.eventLoop.future()
  }
  // 6
  func error(
    _ context: QueueContext,
    _ error: Error,
    _ payload: WelcomeEmail
  ) -> EventLoopFuture<Void> {
    return context.eventLoop.future()
  }
}

Here’s what this file does:

1. Imports Queues and Vapor, exposing their APIs in the file
2. Creates a struct — WelcomeEmail — to hold the information you need to send your welcome email
3. Creates a struct — RecipientWelcomeEmailJob — that conforms to the Job protocol
4. Defines WelcomeEmail as your job payload
5. Adds a dequeue(_:_:) method that processes the job on dequeue. In this tutorial, it prints a greeting to the logs and returns EventLoopFuture.
6. Adds an error(_:_:_:) method, which handles any errors that might occur. This tutorial ignores error handling.

Now, open configure.swift and, between configuring and starting the queue, add the following:

let recipientJob = RecipientWelcomeEmailJob()
app.queues.add(recipientJob)

Here, you instantiate the RecipientWelcomeEmailJob and register it in the queues namespace.

Finally, open NewsletterRecipientAPIController.swift and replace the // TODO on line 54 with:

.flatMap { recipient in
  req.queue.dispatch(
    RecipientWelcomeEmailJob.self,
    WelcomeEmail(to: recipient.email, name: recipient.name)
  )
  .transform(to: recipient)
}

This closure dispatches RecipientWelcomeEmailJob to the queue. It has a payload containing the email and name of the created recipient.

Finally, you transform the response back to the recipient, and the method can finish.

Now, build and run.

Console output after you have registered the scheduled job

In Paw, post a new recipient. You should see that the response doesn’t look different from before. This is expected.

Navigate back to Xcode, and you’ll see four new lines in your debug editor:

[ INFO ] POST /api/newsletter/sign-up [request-id: EBC14B25-4F8A-4AC9-87C8-0001859FD203]
[ INFO ] Dispatched queue job [job_id: 3A4B0EBE-09AA-4E7B-A9E0-8D715F0C75A9, job_name: RecipientWelcomeEmailJob, queue: default, request-id: EBC14B25-4F8A-4AC9-87C8-0001859FD203]
[ INFO ] Dequeuing job [job_id: 3A4B0EBE-09AA-4E7B-A9E0-8D715F0C75A9, job_name: RecipientWelcomeEmailJob, queue: default]
Send welcome email to test2@newslettering.com. Greet the user as Test User 2.

Here’s a breakdown of what the logs tell you:

1. Which endpoint was called and what the unique request-id is
2. Your app dispatched a RecipientWelcomeEmailJob to the default queue with the provided job_id
3. A job from the default queue was dequeued
4. The print message you created earlier

It’s now time to fill in some gaps regarding the job.

Options When Dispatching to the Queue

When you dispatch a job, you must provide the job’s type and the required payload.

In the example above, the job type was RecipientWelcomeEmailJob and the payload was WelcomeEmail.

When you dispatch a job, you also can provide two other options:

1. maxRetryCount, which takes an Int. It is zero by default.
2. delayUntil, which takes an optional Date. It is nil by default.

maxRetryCount is the number of times to retry the job on failure. This is especially important if you are calling an external API and you want to make sure it goes through.

By setting delayUntil, you delay the process of the job until after the date you provided. If the driver dequeues the job too early, it’ll make sure to re-queue it until after the delay time.

When delayUntil is nil or some date in the past, the driver processes the job the first time it is dequeued.

Those are helpful. But what if you want jobs to repeat at specific days or times?

Scheduling Jobs

If you want to run a task at a certain time, such as:

• Sending a newsletter of the first of every month;
• Greeting all your friends and family with a Christmas email every December;
• Or reminding your kid that you are the coolest parent every hour.

You can arrange it as a scheduled job in your Vapor app.

Creating SendNewsletterJob

In Xcode, create a new file, SendNewsletterJob.swift, inside the Jobs folder.

Now, insert:

// 1
import Fluent
import Queues
import Vapor

// 2
struct SendNewsletterJob: ScheduledJob {
  // 3
  func run(context: QueueContext) -> EventLoopFuture<Void> {
    return getNewsletter(on: context.application.db)
      .and(self.getRecipients(on: context.application.db))
      .map { newsletter, recipients in
        // 4
        let message: String = recipients
          .map(\.name)
          .map { "Send newsletter with title: \(newsletter.title) to: \($0)" }
          .joined(separator: "\n")
        print(message)
    }
  }

  // 5
  private func getNewsletter(on db: Database) -> EventLoopFuture<Newsletter> {
    let today = Calendar.current.startOfDay(for: Date())
    return Newsletter.query(on: db)
      .filter(\.$sendAt, .equal, today)
      .first()
      .unwrap(or: Abort(.notFound))
  }

  // 6
  private func getRecipients(
    on db: Database
  ) -> EventLoopFuture<[NewsletterRecipient]> {
    return NewsletterRecipient.query(on: db).all()
  }
}

Here, you:

1. Import Fluent, Queues and Vapor and expose their APIs in the file
2. Create a SendNewsletterJob struct and make it conform to ScheduledJob
3. Create the required run(context:) method. It fetches the newsletter that should be sent out (#5) and all the newsletter recipients (#6).
4. Map the list of recipients to a list of messages that informs you who received the email. Then print the message.
5. Create private helper method that fetches the current newsletter. If the newsletter doesn’t exist, the method returns a failed EventLoopFuture with an error.
6. Create private helper method to fetch all the recipients

Next, open configure.swift and, under the recipientJob, add the following:

let sendNewsletterJob = SendNewsletterJob()
app.queues.schedule(sendNewsletterJob).minutely().at(5)

Here, you instantiate the SendNewsletterJob and register it to run on the fifth second of every minute.

The queues package comes with a handful of convenient helpers to schedule your job:

• at(_:) takes a specific date for a job that should run only once.
• yearly() identifies a yearly occurrence. It can be further configured with the month it should run.
• monthly() sets a monthly schedule, and can be further configured with the day it should run.
• weekly() specifies that the job should occur weekly, and you can further specify on which day of the week it should run.
• daily() schedules the job to execute daily. You can further specify the time it should run.
• hourly() sets the job to an hourly schedule, and can be further configured with the minutes it should run.
• minutely() configures the job to run every minute. You can further specify which seconds it should run.
• everySecond() schedules the job for every second and has no further configuration.

Ready to take this out for a spin?

Running Your Scheduled Job

Build and run your app.

Vapor Queue Build And Run No Empty Queue Message

The first thing you notice is that the warning in the console is gone. That means your Vapor app registered your scheduled job and the worker is running.

Next, create a few extra recipients in your API client and wait for the job to be executed. You’ll see something like below:

Send newsletter with title: Mock Newsletter to: Test User 1
Send newsletter with title: Mock Newsletter to: Test User 2
Send newsletter with title: Mock Newsletter to: Test User 3
Send newsletter with title: Mock Newsletter to: Test User 4

It prints a line for each recipient in your database, letting you know that the newsletter was sent to them.

Congratulations! You now understand the fundamentals of queues in Vapor and how to schedule jobs.

Where to Go From Here?

You can download the completed project files by clicking the Download Materials button at the top or bottom of this tutorial.

In this article, you learned what a job queue is, how to use it to run parts of your code in a background thread and how to dispatch specific jobs for sending a welcome email to your new recipient.

You also learned how to schedule jobs that you want to run at some point in the future or at recurring times.

If you’re looking for a challenge beyond this tutorial, here are a few things you could try:

• Use the Mailgun package to parse your WelcomeEmail payload into a real email and send it with Mailgun.
• Try using one of the other drivers that the Vapor community has created, such as QueuesFluentDriver or QueuesMongoDriver.
• Set up a second microservice and have it process your job queue instead of doing it in-process.

If you want to learn more about queues and the Vapor Queues package, you can find the official documentation here.

We hope you enjoyed this tutorial! If you have any questions or comments, please join the forum discussion below!