Vapor and Job Queues: Getting Started
Using Vapor’s Redis and Queues libraries, learn how to configure, dispatch, and process various jobs in a queue. By Heidi Hermann.
Sign up/Sign in
With a free Kodeco account you can download source code, track your progress, bookmark, personalise your learner profile and more!
Create accountAlready a member of Kodeco? Sign in
Sign up/Sign in
With a free Kodeco account you can download source code, track your progress, bookmark, personalise your learner profile and more!
Create accountAlready a member of Kodeco? Sign in
Sign up/Sign in
With a free Kodeco account you can download source code, track your progress, bookmark, personalise your learner profile and more!
Create accountAlready a member of Kodeco? Sign in
Contents
Vapor and Job Queues: Getting Started
20 mins
- Getting Started
- Looking at the Vapor Project
- Running the Vapor Project
- Getting Started with Vapor Queues
- Configuring Your QueuesRedisDriver
- Start Your Redis Server
- Running Your Queue and Scheduled Jobs
- Dispatching Your First Job
- Creating RecipientWelcomeEmailJob
- Options When Dispatching to the Queue
- Scheduling Jobs
- Creating SendNewsletterJob
- Running Your Scheduled Job
- Where to Go From Here?
In this Vapor tutorial, you’ll learn how to dispatch a job from your Vapor app to a queue hosted on Redis. You’ll also learn how to schedule jobs that you want to run in the future.
If you’ve ever gone to the supermarket or driven on the highway, chances are you’ve been in a real-life queue. With software, it’s not all that different. The program schedules a series of data to process, one after another.
Usually, this processing pattern follows a first-in-first-out (FIFO) principle.
In this tutorial, you’ll learn how to:
- Configure and run a queue
- Dispatch a job to your queue
- Create a scheduled job
Job queues in particular serve to coordinate asynchronous service-to-service communication. Generally, one process sends while another receives. Each interacts with the queue independent of the other. They likely aren’t even aware of each other’s existence!
Queues are especially used in serverless and microservice architectures. But even monolithic applications can see benefits from job queues, such as offloading CPU or disk-writing intensive tasks.
Here are some examples of tasks that queues could help with:
- Sending emails and text messages, as part of two-factor or email authentication, for example
- Performing complex or long-running database operations, such as importing and processing thousands of items into your database
- Speeding up response time by delaying non-critical processing
- Buffering or batching work when you operate on large data sets
- Smoothing spiky workloads during peak hours by offloading the whole request
- Ensuring increased job integrity and resilience by persisting until the jobs are finished
Getting Started
Download the starter project by clicking the Download Materials button at the top or bottom of this tutorial. Then, navigate to the starter folder.
- Xcode 11 and Swift 5.2 (or newer versions)
- Docker: If you don’t have Docker yet, visit Docker install for Mac.
- A REST client to run requests against your back end. This tutorial uses Paw, but Postman, RESTed, and even cURL work fine.
Looking at the Vapor Project
Open the Vapor app in Xcode by double-clicking the Package.swift.
While you wait for the Swift Package Manager (SwiftPM) to resolve dependencies, check out the existing project in /Sources/App:
You should pay attention to a few things:
- Controllers: You’ll see one controller — NewsletterRecipientAPIController.swift. It contains the method to sign up for your newsletter.
-
Migrations: You’ll find the migrations for
Newsletter
andNewsletterRecipient
here. You’ll also find a third migration —SeedNewsletter
. It creates a new newsletter in your database. -
Models: This contains the two database models,
Newsletter
andNewsletterRecipient
. - configure.swift: Here’s where you’ll find everything required to get started with the tutorial. This includes Databases, Migrations and Routes. This also is where you’ll register your Queue and Jobs later.
- routes.swift: You’ll need only one route for the project, which is to sign up for a newsletter. It’s registered under /api/newsletter/sign-up.
Now, you’re ready to begin the project!
Running the Vapor Project
After SwiftPM finishes resolving the project’s dependencies, set the Working Directory to your project folder.
In Xcode, go to the Newslettering run scheme.
Under Options, turn on Working Directory and select your project folder.
Close the Scheme Editor window and verify that your selected platform is My Mac. Then, build and run.
Now, switch to Paw (or your preferred REST Client) and prepare the following request:
POST http://localhost:8080/api/newsletter/sign-up
Content-Type: application/json
{
"email": "test1@newslettering.com",
"name": "Test User 1"
}
Next, send the request:
You should get a 201 Created
response from the server. If you check the response body, you’ll see the email and name you sent in the request.
Getting Started with Vapor Queues
You’ll use the Vapor Queues package in the rest of the tutorial. It’s a job queue system that allows you to run jobs on request as well as schedule them to run in the future.
To take advantage of the Queues package, you’ll need a job storage driver. For this project, you’ll use Vapor’s QueuesRedisDriver.
Configuring Your QueuesRedisDriver
Open configure.swift and, as part of the list of packages to import, add the following:
import Queues
import QueuesRedisDriver
Here, you expose the Queues and QueuesRedisDriver APIs so you can configure them.
Next, configure your queue to run on Redis.
Inside configure(_:)
, after try Application.autoMigrate().wait()
, add the following:
try app.queues.use(.redis(url: "redis://127.0.0.1:6379"))
This is where you register that the queue will be running on Redis with the predefined URL.
Now, let’s get your Redis server up and running.
Start Your Redis Server
Open your terminal and navigate to the project folder.
Start the Redis container by running:
$ docker-compose run -p 6379:6379 cache
This starts a Docker Redis container that is listening on port :6379
and exposes the port to your local machine.
Running Your Queue and Scheduled Jobs
There are two “modes” you can run your scheduled jobs as:
- Separate process
- In-app process
To run the queue as a separate process, run the following command in your terminal:
$ swift run Run queues
This command starts a separate worker in the background that listens for any dispatched jobs.
To run scheduled jobs in a separate process, pass the --scheduled
flag to the same command above.
The second option is to run the queue as an in-app process, which is what you’ll do for the rest of the tutorial.
Inside configure.swift, right under where you configured the queue driver, add the following:
// 1
try app.queues.startInProcessJobs()
// 2
try app.queues.startScheduledJobs()
With this code, you:
- Start the queue as an in-app process. You’ll run your jobs on the default queue, so you won’t need a queue name as an argument.
- Start the scheduled jobs as an in-app process. Your app will then check for any scheduled jobs. If none exist, it’ll exit the worker again.
Now, build and run your app. You’ll see a warning in the debug editor before the server starts:
[ WARNING ] No scheduled jobs exist, exiting scheduled jobs worker.
You’ve learned how to run jobs. Now it’s time to dispatch them.
Dispatching Your First Job
When a new reader signs up to your newsletter, you want to send them a welcome email to say, “Thank you”.
This is a typical example of a process that you would dispatch to a job because you don’t want the responsiveness of your webpage to depend on a third-party email API to completing.
Creating RecipientWelcomeEmailJob
In Xcode, add a new Jobs folder to your app and add a RecipientWelcomeEmailJob.swift file.
Open the file and add the following:
// 1
import Queues
import Vapor
// 2
struct WelcomeEmail: Codable {
let to: String
let name: String
}
// 3
struct RecipientWelcomeEmailJob: Job {
// 4
typealias Payload = WelcomeEmail
// 5
func dequeue(
_ context: QueueContext,
_ payload: WelcomeEmail
) -> EventLoopFuture<Void> {
print("Send welcome email to \(payload.to). Greet the user as \(payload.name).")
return context.eventLoop.future()
}
// 6
func error(
_ context: QueueContext,
_ error: Error,
_ payload: WelcomeEmail
) -> EventLoopFuture<Void> {
return context.eventLoop.future()
}
}
Here’s what this file does:
- Imports
Queues
andVapor
, exposing their APIs in the file - Creates a struct —
WelcomeEmail
— to hold the information you need to send your welcome email - Creates a struct —
RecipientWelcomeEmailJob
— that conforms to theJob
protocol - Defines
WelcomeEmail
as your job payload - Adds a
dequeue(_:_:)
method that processes the job on dequeue. In this tutorial, it prints a greeting to the logs and returnsEventLoopFuture
. - Adds an
error(_:_:_:)
method, which handles any errors that might occur. This tutorial ignores error handling.
Now, open configure.swift and, between configuring and starting the queue, add the following:
let recipientJob = RecipientWelcomeEmailJob()
app.queues.add(recipientJob)
Here, you instantiate the RecipientWelcomeEmailJob
and register it in the queues
namespace.
Finally, open NewsletterRecipientAPIController.swift and replace the // TODO
on line 54 with:
.flatMap { recipient in
req.queue.dispatch(
RecipientWelcomeEmailJob.self,
WelcomeEmail(to: recipient.email, name: recipient.name)
)
.transform(to: recipient)
}
This closure dispatches RecipientWelcomeEmailJob
to the queue. It has a payload containing the email and name of the created recipient.
Finally, you transform the response back to the recipient
, and the method can finish.
Now, build and run.
In Paw, post a new recipient. You should see that the response doesn’t look different from before. This is expected.
Navigate back to Xcode, and you’ll see four new lines in your debug editor:
[ INFO ] POST /api/newsletter/sign-up [request-id: EBC14B25-4F8A-4AC9-87C8-0001859FD203]
[ INFO ] Dispatched queue job [job_id: 3A4B0EBE-09AA-4E7B-A9E0-8D715F0C75A9, job_name: RecipientWelcomeEmailJob, queue: default, request-id: EBC14B25-4F8A-4AC9-87C8-0001859FD203]
[ INFO ] Dequeuing job [job_id: 3A4B0EBE-09AA-4E7B-A9E0-8D715F0C75A9, job_name: RecipientWelcomeEmailJob, queue: default]
Send welcome email to test2@newslettering.com. Greet the user as Test User 2.
Here’s a breakdown of what the logs tell you:
- Which endpoint was called and what the unique
request-id
is - Your app dispatched a
RecipientWelcomeEmailJob
to the default queue with the providedjob_id
- A job from the default queue was dequeued
- The print message you created earlier
It’s now time to fill in some gaps regarding the job.
Options When Dispatching to the Queue
When you dispatch a job, you must provide the job’s type and the required payload.
In the example above, the job type was RecipientWelcomeEmailJob
and the payload was WelcomeEmail
.
When you dispatch a job, you also can provide two other options:
-
maxRetryCount
, which takes anInt
. It is zero by default. -
delayUntil
, which takes an optionalDate
. It isnil
by default.
maxRetryCount
is the number of times to retry the job on failure. This is especially important if you are calling an external API and you want to make sure it goes through.
By setting delayUntil
, you delay the process of the job until after the date you provided. If the driver dequeues the job too early, it’ll make sure to re-queue it until after the delay time.
When delayUntil
is nil
or some date in the past, the driver processes the job the first time it is dequeued.
Those are helpful. But what if you want jobs to repeat at specific days or times?
Scheduling Jobs
If you want to run a task at a certain time, such as:
- Sending a newsletter of the first of every month;
- Greeting all your friends and family with a Christmas email every December;
- Or reminding your kid that you are the coolest parent every hour.
You can arrange it as a scheduled job in your Vapor app.
Creating SendNewsletterJob
In Xcode, create a new file, SendNewsletterJob.swift, inside the Jobs folder.
Now, insert:
// 1
import Fluent
import Queues
import Vapor
// 2
struct SendNewsletterJob: ScheduledJob {
// 3
func run(context: QueueContext) -> EventLoopFuture<Void> {
return getNewsletter(on: context.application.db)
.and(self.getRecipients(on: context.application.db))
.map { newsletter, recipients in
// 4
let message: String = recipients
.map(\.name)
.map { "Send newsletter with title: \(newsletter.title) to: \($0)" }
.joined(separator: "\n")
print(message)
}
}
// 5
private func getNewsletter(on db: Database) -> EventLoopFuture<Newsletter> {
let today = Calendar.current.startOfDay(for: Date())
return Newsletter.query(on: db)
.filter(\.$sendAt, .equal, today)
.first()
.unwrap(or: Abort(.notFound))
}
// 6
private func getRecipients(
on db: Database
) -> EventLoopFuture<[NewsletterRecipient]> {
return NewsletterRecipient.query(on: db).all()
}
}
Here, you:
- Import
Fluent
,Queues
andVapor
and expose their APIs in the file - Create a
SendNewsletterJob
struct and make it conform toScheduledJob
- Create the required
run(context:)
method. It fetches the newsletter that should be sent out (#5) and all the newsletter recipients (#6). - Map the list of recipients to a list of messages that informs you who received the email. Then print the message.
- Create private helper method that fetches the current newsletter. If the newsletter doesn’t exist, the method returns a failed
EventLoopFuture
with an error. - Create private helper method to fetch all the recipients
Next, open configure.swift and, under the recipientJob
, add the following:
let sendNewsletterJob = SendNewsletterJob()
app.queues.schedule(sendNewsletterJob).minutely().at(5)
Here, you instantiate the SendNewsletterJob
and register it to run on the fifth second of every minute.
The queues package comes with a handful of convenient helpers to schedule your job:
-
at(_:)
takes a specific date for a job that should run only once. -
yearly()
identifies a yearly occurrence. It can be further configured with the month it should run. -
monthly()
sets a monthly schedule, and can be further configured with the day it should run. -
weekly()
specifies that the job should occur weekly, and you can further specify on which day of the week it should run. -
daily()
schedules the job to execute daily. You can further specify the time it should run. -
hourly()
sets the job to an hourly schedule, and can be further configured with the minutes it should run. -
minutely()
configures the job to run every minute. You can further specify which seconds it should run. -
everySecond()
schedules the job for every second and has no further configuration.
Ready to take this out for a spin?
Running Your Scheduled Job
Build and run your app.
The first thing you notice is that the warning in the console is gone. That means your Vapor app registered your scheduled job and the worker is running.
Next, create a few extra recipients in your API client and wait for the job to be executed. You’ll see something like below:
Send newsletter with title: Mock Newsletter to: Test User 1
Send newsletter with title: Mock Newsletter to: Test User 2
Send newsletter with title: Mock Newsletter to: Test User 3
Send newsletter with title: Mock Newsletter to: Test User 4
It prints a line for each recipient in your database, letting you know that the newsletter was sent to them.
Congratulations! You now understand the fundamentals of queues in Vapor and how to schedule jobs.
Where to Go From Here?
You can download the completed project files by clicking the Download Materials button at the top or bottom of this tutorial.
In this article, you learned what a job queue is, how to use it to run parts of your code in a background thread and how to dispatch specific jobs for sending a welcome email to your new recipient.
You also learned how to schedule jobs that you want to run at some point in the future or at recurring times.
If you’re looking for a challenge beyond this tutorial, here are a few things you could try:
- Use the Mailgun package to parse your
WelcomeEmail
payload into a real email and send it with Mailgun. - Try using one of the other drivers that the Vapor community has created, such as QueuesFluentDriver or QueuesMongoDriver.
- Set up a second microservice and have it process your job queue instead of doing it in-process.
If you want to learn more about queues and the Vapor Queues package, you can find the official documentation here.
We hope you enjoyed this tutorial! If you have any questions or comments, please join the forum discussion below!