Work queues allow you to perform tasks that are slow or error-prone outside of the current user’s request to improve their experience with your site. Read on to learn how you can use work queues in Laravel for your projects.
Why You Should Be Using Work Queues
Two or three jobs ago (depending on how you look at it), I managed a help desk and the software we used to record tickets. We were using open source software that allowed us to email all of the participants of a ticket. The problem we continued to have is that every time we performed an action that sent an email, the application would open a connection to our email server, send one email, and then close the connection. Every save action emailed at least one person and sometimes it would email four or five. This caused the save action to take one to five seconds. Every so often the connection would fail and a portion of the emails would go out so we would have to save again and it would email out duplicates. This became such a problem that we eventually abandoned the software completely.
Two or one jobs ago (again depending on how you look at it), I helped write an application that had thousands of users per customer. At least once a year our customers would provide us with a CSV file with all their current employees in the file and we would add new users, remove old users, and make changes based on what the file contained. This process took an extremely long time and we kept having time out and memory usage issues with some of the larger customers.
Both of these examples illustrate times when processes take a long time or are error-prone and cause a poor user experience. Work queues allow us to tell our application that we need to do a unit of work but allows us to handle that unit of work outside the normal application process. This gives the users of our application a potentially huge speed improvement.
Queues, Jobs, and Workers
There are three concepts we need to quickly define so it’s easier to understand the rest of this tutorial.
The first concept is a work queue. A queue is simply a list of items where you put items into the queue and remove them in the same order (First In/First Out or FIFO). Work queues provide a central location for us to store the units of work we want to be done.
The second concept is a job. In Laravel jobs are the classes that are performing the unit for work we want to be done.
The third concept is a worker. A worker is a process generally on the same server as the main application (it’s not a requirement which is great for scaling) that performs the unit of work. Laravel provides a simple interface for how to get a worker to run our jobs.
For work queues, a queue driver is what we’re using to store the jobs we need done. Out of the box, Laravel provides support for two different types of queue drivers. The easiest solution is using your existing database as the queue (which is the solution we’ll be using in this article). The other option is using a Redis server to store the data. Redis is a NoSQL solution that can handle queue like data exceptionally well. We highly recommend that as your application’s usage goes up you look into using Redis as the database can quickly become a bottleneck.
Creating Your First Job
The first thing we need to do is create our first job. We’ve been developing an example application in this series that has
Projects that have
Tasks. The next feature we need to add is the ability to add a task to every project that is not completed.
To create our job we’re going to use “artisan’s”
This will create a new class in “app/Jobs/AddTaskToNotCompletedProjects.php” that will look like the following:
The important part of this class is the
handle() function. The
handle() function is what does the actual work in our job. To get us started we’re going to create a simple function that loops through all the open
Projects (based on our logic that a null “end_date” indicates not completed) and adds a new
Task to the
Then to test this, we’re going to short circuit the whole queue complexity and add the following in our “routes/web.php” file.
The call to
dispatchNow() causes the job to run inside the current request. We don’t recommend doing this in a production environment but it’s great for testing.
Now if we access our application the job will run and we can check the results by looking into the “tasks” table.
See how easy that was?
Specifying the “Name”
The downside to this is that all of our tasks have a name of “Scheduled Task”. We could create one job for every type of task we want to add but that would be a maintenance nightmare. Thankfully, that empty
__construct() function can be used to set parameters for our job.
We’re going to create a parameter named
$name which will allow us to specify the task name at runtime.
Now when we call
dispatchNow() we can pass it a
string which will be passed directly to
Now if we check our “tasks” table we can see that our name is being set correctly.
Getting Started With A Database Queue
We have our job defined now but we still haven’t handled the whole problem of it running in the background.
The first thing we need to do is create the tables that Laravel will use to store the job information. This is done by running
php artisan queue:table and
php artisan queue:failed-table to generate the migrations that will be used.
And then we’re going to run
php artisan migrate to get the tables created.
Next, we need to open our “.env” file and set “QUEUE_CONNECTION” to “database” so Laravel will know to save the jobs to the database tables we just created.
To get our job into the queue we’re going to run the following command. Please note we changed
Now we can look into the “jobs” table and see that our job is there (the “payload” column contains so much data we removed the contents of the column from this article).
Managing Your Workers
Workers are the final component in this process we need to discuss. There are two ways of running your workers.
The first way is to use
php artisan queue:listen.
This method works well in cases where you’re changing the source code files between jobs a lot but don’t have a lot of jobs running. This is because in this mode the jobs are run so it looks for changes in files but the downside is that it’s less performant. We recommend running your workers in this mode when you’re developing your application.
The second way is to use
php artisan queue:work.
This has better performance in comparison to
queue:listen but doesn’t notice changes to files when it’s running. To get around this you’ll need to use
php artisan queue:restart to force the workers to reset themselves whenever we deploy a new version of our application.
Manually Managing Your Failed Jobs
One of the benefits of all this extra work was that it was supposed to be easier for us to handle failures more gracefully. Let’s discuss what happens when a job fails and we need to intervene.
Let’s say we accidentally changed ‘name’ to ‘name2’ in our
handle() function so it fails to write to the database.
Now when our worker tries to run through the jobs it has it’s going to fail.
All of these failed jobs get moved into the “failed_jobs” table. We recommend adding something to your application that surfaces these errors so you’re aware of them. We generally display them on a “systems” dashboard so we can intervene.
To manually check the failed jobs you could do a “select” query through your MySQL client of choice or you can use the
queue:failed command to see them.
The ID column is important here because it’s what we’re going to use to interact with these jobs. It’s also important to notice that the ID here is different than the number in brackets above when the job fails. That is because the first number is the id column from the “jobs” table and the second is the id column from the “failed_jobs” table.
Let’s say we determine job #1 isn’t important because job 2 did the same work so repeating it would be wasteful. We can tell Laravel to delete (or forget) the job by using the
queue:forget command which removes the job from the “failed_jobs” table.
We could also determine we want to have the workers retry the jobs using the
queue:retry command. In this case, we’ll retry #2 but we could have given it a range like 2-6 to retry more than 1 or
queue:retry all to retry all of them.
Finally, if we determine none of the jobs are worth keeping we can
queue:flush the jobs out.
The huge downside to the process above is that it’s a manual process. Laravel provides two levers that allow us to tweak how it automatically retries failed jobs. The first is the
$tries variable which determines how many times it should attempt to do the job. The second is the
$retryAfter variable which determines how much time between each of the tries. We recommend setting a
$retryAfter on all of your jobs that attempt to do something with an external service (send an email, connect to an API, etc.) because usually if the service is having a problem it needs a few minutes to fix itself.
For example, let’s say we’ve determined we want to try our
AddTaskToNotCompletedProjects task 3 times and wait 10 seconds between the attempts. We would set the following variables inside our
Then when the job fails (maybe when we still haven’t fixed that ‘name2’ issue) we can see that it tries three times with 10 seconds between the attempts before it fails.
Another thing that we haven’t covered is that right now all of the jobs are being handled by a single queue. While this does work, there are benefits to defining multiple queues. We generally recommend that you define one queue per workload area so that you can more easily determine if you need to spin up extra workers for that type of job.
To start we generally create:
- One queue for every external service
- One queue for scheduled tasks
- One queue to handle reports
The default queue is named “default” (shocker I know), to place a job on a different queue you need to call the
onQueue() function after the
If we look at the “queue” column in the “jobs” table we can see that it’s now getting a different name than “default”.
Now if we run our worker as we have above the jobs on the “tasks” queue won’t be run. This is because by default the worker only looks for jobs in the “default” queue. To fix this we need to specify the name of the queue(s) we want to have it run jobs from using the
--queue command line flag.
Again, we recommend at least one worker per work queue but sometimes you can get away with running a single worker for your test environment. To get this to work you’ll need to specify all the queues it should be pulling from.
To make this example a little easier to follow we changed our
handle() function to look like the following.
Now we’re going to specify the queues that we want this worker to process in a comma-separated list.
There are two things to notice in this output. The first is that the worker processed all the “tasks” queue jobs before it processed the “default” queue jobs. This is because of the order we passed the queue names to the artisan command determines in what order the worker checks each queue for jobs. The second is that the queue name is blank for the “default” queue.
Work queues are excellent solutions to perform work that is error-prone or time-intensive. Laravel provides an excellent interface for creating queues, jobs, and workers for your application.
In our next post, we’ll discuss how to schedule our
AddTaskToNotCompletedProjects so it runs every week.
Hopefully, this has been as helpful to you as it has to me. Check back soon for more artisan commands.
Scott is the Director of Technology at WeCare Connect where he strives to provide solutions for his customers needs. He's the father of two and can be found most weekends working on projects around his house with his loving partner.