Illuminate Queues Everywhere (Laravel 4 Queues Component)

Published on by Safeer

In this blog I’ll discuss how to use the Illuminate Queues component in a non Laravel project. There are a lot of reasons why you might be interested in using a component from a framework without the full framework. Regardless of the reason I’m a strong believer in leveraging and building on the shoulders of giants.

Installation

To begin with we’ll include the following package as a requirement in out composer.json file:

"illuminate/queue": "4.1.*",
"pda/pheanstalk": "~2.1"

If you don’t use Composer, now’ll be a great time to start (click here to learn more). We’ll also want to register our namespace for this project by adding:

"autoload": {
    "psr-0": {
        "MyQueue": "./src/"
    }
}

Now in the root directory of this project create a folder named src and then create another directory called MyQueue inside it.

Then run composer update, this will pull in the required components.

Container

The Queues component requires a Illuminate IoC container, so we’ll use one. IoC containers allow us to decouple the creation of objects that our code depends on. All of this provides more flexiability if we ever decide to change the queues implementation.

As this is a simple and contrived example we’ll start by creating a simple loader.php file which will be used to setup the shared pieces of code for the Queue publisher and consumer. So let’s create a new file called loader.php and place the following code inside it:

<?php

include_once './vendor/autoload.php';

$app = new \Illuminate\Container\Container;

This starts by including the composer autoloader and then creates a new IoC Container object.

Queues Object

Now we’ll create the Queues object, for now we’ll stick with beanstalkd queues, however Illuminate Queues also supports Amazon SQS and Iron MQ. You’ll need to make sure you have beanstalkd installed and running, the great news is that Homestead comes with it setup – you can find out more on Homestead by visiting the official website of Laravel

To setup the Queues Object, add the following chunk of code to the end of your loader.php file:

$app->bindIf('queue', function($app) {
    $queue = new \Illuminate\Queue\Capsule\Manager($app);

    $queue->addConnection(array(
        'driver' => 'beanstalkd',
        'host' => 'localhost',
        'queue' => 'default',
    ), 'default');

    $queue->setAsGlobal();

    return $queue;
}, true);

This binds a closure to ‘queue’ in the IoC Container. The parameter after the closure is whether this should be a shared object or not. As we only need this object to be created once we’ll enable this.

It’s worth noting that the Queue Object will only be created once. It will be created when the we request queue from the IoC Container.

Queue Publisher

Okay so now we have the implementation of how to create the queue object, so we can now start pushing jobs onto our Queue. To do this let’s create a new file, I’m calling mine push.php and placing it in the root directory of this project.

<?php

include_once './loader.php';

$app['queue']->push('\MyQueue\DummyHandler', array());

This uses the loader.php file to setup the system to use the Queue object we defined. Then it adds a new Queue item to the Queue. Before we go any further lets break up what’s happening in the following line:

$app['queue']->push('\MyQueue\DummyHandler', array());

The Queue object push method has two parameters which are the:

  1. Job Handler – This is the class that will process the Queue item. Unless you explictly specify a method it’ll expect a method called fire. To use a different method adjust this parameter to use the following template: 'Class@Method'
  2. Payload – This is the data for this Queue Job item

Queue Job Handler

Create a new file in the src/MyQueue directory called DummyHandler.php and place the following code inside it:

<?php namespace MyQueue;

class DummyHandler {

    public function fire($job, $data)
    {
        echo $job->getJobId() . " - Attempt: " . $job->attempts() . "\n";
        throw new \Exception("Error Processing Request", 1);
    }

}

Okay, so you’ll have figured out that this is a very basic example which prints the Job ID and the number of attempts this Job has had. Once done it throws an Exception, in a real example we would call $job->delete once done. However for this example we just want to keep pushing the job back until it exhausts it’s attempts.

Queue Worker

We’ll use the Illuminate Worker to handle running of our Queue Job Items as it handles some important things such as:

  1. Sleeping between jobs
  2. Deleting queue items
  3. Retrying queue items
  4. If defined firing events and handling failed jobs

To use the worker we’ll want to define it in our IoC Container, we can do this by modifying our loader.php file and adding the following to the bottom of it:

$app->bindIf('queue.worker', function($app) {
    return new \Illuminate\Queue\Worker($app['queue']->getQueueManager(), null, null);
}, true);

This binds a shared refernce to how we want our Queue worker to be setup. The Worker accepts upto three parameters and they are:

  1. The Queue object
  2. Optional: An implementation of the FailedJobProviderInterface. This isn’t covered in this article however passing an implementation of this allows you to decouple how you want to handle failed jobs. The Illuminate Container component comes with a Illuminate Database implemention of this.
  3. Optional: A Events dispatcher object, this will allow you to listen to events fired such as illuminate.queue.failed

Queue Consumer

We now have a Queue Publisher and the worker setup. Let’s create something to Consume these Queue Job Items. Start by creating a new file called pull.php in the root directory of this project and we want to add the following code to it:

<?php

include_once './loader.php';

while (true)
{
    try {
        $app['queue.worker']->pop('default', 'default', 3, 64, 30, 3);
    } catch (\Exception $e) {
        // Log or do something here
    }
}

This code chunk starts by setting up the system to use the Queue. It then creates a endless loop that uses the queue worker we setup to process the new job items. Here’s an explanation of what $app['queue.worker']->pop… parameters mean:

  1. Connection to use, you can define multiple Queue Connections and then reference them by the Connection name you set. You can set the name of the Connection by using the second parameter in addConnection when defining the Queue closure.
  2. The Queue to use, this is the value you pass as queue when adding a connection
  3. Number of seconds to delay a job for if it fails
  4. Maximum memory, at time of writing this doesn’t do anything.
  5. Time in seconds to sleep when no Queue job is returned
  6. Maximum number of times to retry the Job item

Conclusion / Wrapping up

As you can see using the Queue component is simple even without the full Laravel Framework. Here’s things that are not covered in the article which you’ll want to consider:

How you’ll handle failed Queue Job Items?

You could have a look into implementing a FailedJobProviderInterface class and then passing it through to the queue.worker

How you’ll keep your queue processor running?

You’ll want to look into a process control system such as supervisord