Features and Changes Coming to Laravel 8's Queue System

Laravel 8 ships with cool new features as well as some changes to the queue system. In this post we'll look into these features and changes.

Changes

Backoff

The retryAfter() method and the retryAfter property of queued jobs, mailers, notifications, and listeners has been renamed to backoff.

The --delay option of the php artisan queue:work command has been renamed to --backoff as well.

You should also know that you can now pass an array to the backoff property to instruct the worker to implement an exponential backoff:

public $backoff = [30, 60];

Or return an array from the backoff() method:

public function backoff ()
{
    return [30, 60]; 
}

Using an exponential backoff here, we instructed the worker to delay retrying the job 30 seconds after the first failure, then 60 seconds for each failure after that.

You can also use an exponential backoff on the queue:work command:

php artisan queue:Work --backoff=30,60

Job Expiration

The timeoutAt property of queued jobs, notifications, and listeners has been renamed to retryUntil.

Using $retryUntil instructs the worker to keep retrying the job until a certain time in the future.

You can add retryUntil as a public property on the job class or a retryUntil method:

public function retryUntil()
{
    return now()->addDay();
}

New Features

Queued Closures

You can now chain a catch() method while dispatching a queued closure:

dispatch(function () {
    // Job logic...
})->catch(function (Throwable $e) {
    // Handle Failure...
});

The closure provided to the catch() method will be invoked if the job fails.

Database Driver Reliability

When releasing a job back to the queue while using the database queue driver, Laravel will perform the operation inside a transaction now. That means the job will not be deleted from the queue unless the new—released—instance is added. This decreases the chance of losing jobs significantly.

Redis Driver Efficiency

When dispatching a group of jobs in bulk while using the redis queue driver, Laravel will perform the operation by sending a single command to redis. Previously, Laravel used to send multiple rpush command to redis; one for each job.

Worker Graceful Termination

Starting Laravel 8, workers will exit gracefully and call any terminating callbacks registered by App::terminating().

Worker Self Termination

In order to avoid memory leaks, it's a common practice to terminate your workers every now and then and let your process monitoring tool start new ones. This was usually done by adding a CRON job that runs the queue:restart artisan command.

In Laravel 8, you can instruct workers to exit after processing a certain number of jobs or after running for a specific number of seconds:

php artisan queue:work --max-jobs=1000 --max-time=3600

Naming Workers

You can now add a --name option to the queue:work command:

php artisan queue:work --name=notifications

This feature was mainly added to allow people to customize how the worker picks which queue to process jobs from at runtime:

Worker::popUsing('notifications', function ($pop) {
    $queues = time()->atNight() 
        ? ['mail', 'webhooks'] 
        : ['push-notifications', 'sms', 'mail', 'webhooks'];

    foreach ($queues as $queue) {
        if (! is_null($job = $pop($queue))) {
            return $job;
        }
    }
});

Job Batching

Laravel's job batching allows you to dispatch a number of jobs to be processed in parallel by your workers. You can perform an action after all jobs in the batch are processed or if any of the batch jobs fail:

Bus::batch([
    new ProcessFile(1),
    new ProcessFile(2),
    new ProcessFile(3),
])->dispatch();

You can find more information on Job Batching in the official documentation.

Job Chaining

You can now dispatch a chain of jobs using the Bus facade:

Bus::chain([
    new ExtractReports,
    new GenerateReport,
    new SendResults,
])->dispatch();

You can also add a catch() callback that'll be invoked if any of the jobs in the chain fails:

Bus::chain([
    new ExtractReports,
    new GenerateReport,
    new SendResults,
])->catch(function(){
    // Handle the chain failure.
})
->dispatch();

Horizon Balancing Rate

Two new configuration options were added to Horizon; balanceMaxShift and balanceCooldown.

 'environments' => [
    'environment' => [
        'supervisor-1' => [
            'balanceMaxShift' => 5,
        ],
    ],
],

balanceMaxShift sets the maximum number of worker processes to add or remove each time Horizon scales the workers pool. In previous versions on Horizon, only a single worker was added or removed, now you can control that number.

As for balanceCooldown, it sets the number of seconds to wait between each scaling action. This was hard-coded to 3 seconds in previous versions of Horizon.

 'environments' => [
    'environment' => [
        'supervisor-1' => [
            'balanceCooldown' => 1,
        ],
    ],
],