Yii2 Queue with Redis on Docker Swarm is a game changer

I don't know why I haven't utilized this before, but the yiisoft/yii2-queue is a fantastic way to quickly and easily split some of your application processing out to asynchronous workers.

Getting it set up with a redis storage engine on a docker stack is super simple -- we just add a persistent redis service to the docker stack, configure the application to include the queue component, and move our slow process(es) into a queue job.

Since we're using a Docker Swarm stack we don't need to worry about setting up a cron or systemd service, we set up the queue listener as it's own service in the stack, which also gives us the ability to scale the queue workers dynamically as the need arises.

Add the yiisoft/yii2-queue package

To begin, you need to add the yiisoft/yii2-queue package to your composer.json file. You can add it manually, or use the command line to add it:

composer require --prefer-dist yiisoft/yii2-queue

Add the queue component to your application config

You will need to modify both the web.php and the console.php (assuming that both are pushing to the queue and/or the console is going to be running listener - you can always have a different application processing the queue - the concept remains the same, just add the queue configuration to that application console instead.)

We're using the redis queue class, because it's very easy to toss a redis instance into our swarm stack and it requires no migrations to work out of the box (plus it's super speedy and a great use of redis).

'bootstrap' => [
        ...
        'queue',
    ],
    'components' => [
        ...
        'redis' => [
            'class'=>'yii\redis\Connection',
            'hostname' => 'myredis', // the swarm service name
            'port' => 6379,
            'database' => 0,
        ],
        'queue' => [
            'class'   => \yii\queue\redis\Queue::class,
            'redis'   => 'redis',
            'channel' => 'queue',
        ],
    ],

Note: the redis config in the application does not use the swarm identifier prepended to the service name - the swarm handles those negotiations. Just identify it exactly as you have it listed in your services definition block (see below).

Configure your swarm services

In your docker-compose.yml file for the swarm, you will need to create a new service for running the queue, based on the php-cli of your choice.

It should look something along the lines of:

services:
    runner:
      image: php:7.3-cli
      deploy: 
        replicas: 1
        restart_policy: 
          condition: on-failure
      networks:
        - mynetwork
      volumes:
        - ./:/var/www/html
      command: ['bash','-c',"/var/www/html/yii queue/listen --verbose"]

Where ./ is the folder that has your codebase for the worker to implement. Note: You should used named volumes rather than direct mounts like this, unless you're just doing local testing.

If you don't have redis in the stack yet, it should look something like, in your services: section

myredis: 
    image: redis:5.0
    volumes:
      - redis:/data
    deploy:
      constraints: [node.role == manager]
    networks:
      - mynetwork

Now, when your stack is deployed, you will have one listener set up to run the queue. If you need to scale it you can do so simply, either by updating the replica count in the docker-compose.yml file and re-deploying the stack, or by using the docker service scale command:

docker service scale mystack_runner=5

Verify your worker is listening

Once the stack has been deployed you can verify that the worker is listening by running the docker service logs command

docker service logs mystack_runner -f

Which should output something like:

mystack_runner.1.w6ww64d5lgxy@docker-desktop | 2019-05-30 15:38:38 [pid: 1] - Worker is started

You now have everything you need to start implementing queued items within your application!

Implementing the jobs is very well explained in the Yii2-Queue Manual, so I will not dive into it in depth here.

Once you've implemented your jobs, you can monitor them progressing through the queue by using the same docker service logs command we used before, which should look something like:

mystack_runner.1.w6ww64d5lgxy@docker-desktop | 
     2019-05-30 16:39:59 [18] app\components\queues\MyJob (attempt: 1, pid: 1) - Started 
mystack_runner.1.w6ww64d5lgxy@docker-desktop | 
     2019-05-30 16:40:00 [18] app\components\queues\MyJob (attempt: 1, pid: 1) - Done (0.310 s) 

You can also query for the queue info from your swarm service directly:

docker exec -it $(docker ps -lq -f name=mystack_runner) /var/www/html/yii queue/info

(Note the handy $(docker ps -lq -f) trick: that will give you the latest single container id that matches the requested filter)

Which will output something along the lines of:

Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 2

Happy optimizing!!

---

Caveats:

• This example uses the base php:7.3-cli image. You probably need a slightly customized image depending on what php extensions your runner needs access to.
• This example assumes you're running your stack as 'mystack' - please sub in whatever the name of your stack is anywhere you see 'mystack' in this example.
• If you're adding a redis instance for this you want to make sure you set the data to be persistent and set up a healthcheck for your services before running it in production.