dreaming of an extension for Gearman

Translation of the original in Catalan. Read it in: CatalàCastellano

I must say that I am not very tidy when presenting new technologies since first of all

I ran a benchmark on Gearman

before making an introduction to it; well then, since in this article I want to talk about some possible extensions I want to work on with Gearman, first of all I will give a short introduction to the project.

Introduction

Gearman

is what we commonly call a task server, that is, when our code has to request a task, feature, job, or something similar it is very interesting with regard to:

  • scalability: we can have as many servers and/or processes consuming tasks as we are interested in.
  • parallelism: tasks can be consumed in parallel.
  • load balancing: we can do map/reduce over the tasks and send them to the servers we are interested in to distribute the load.
  • independence between languages: the code that requests the task and the one that consumes the task can be totally different; the libraries Gearman has are: PHP, Pearl, Ruby, C, Python, etc.
  • HTTP interface: besides, it has an HTTP client interface that will allow us to inject tasks from languages not supported by the Gearman libraries.

using a server of this kind, since besides allowing us to request synchronous tasks, we can also request asynchronous tasks from it. That is, not only do we not know who is doing the job for us, limiting ourselves to receiving the result, but we can also ask that this job be done whenever possible.

As if all this were not enough there are still more advantages:

  • Open Source
  • Programat en C
  • Petit i molt ràpid
  • It supports several backends: RAM, SQLite, Memcached, Tokyo Cabinet, etc.
gearman stack
gearman stack

The people who started implementing Gearman were those of

Danga Interactive

famous for

LiveJournal

i

SixApart

.

My ideas

After this introduction, now I can talk about the things I would like Gearman to do but that it does not do. First of all I have to talk about the advantages it would have if it could have a backend against Redis. What I am after when connecting Redis with Gearman is to achieve:

  • persistence of tasks even if Gearman is restarted
  • persistence of tasks on disk even if Redis is restarted, thanks to:
    • asynchronous writing to disk
    • bgrewriteaof: prevents data from being lost when Redis is abruptly restarted because of the asynchronous writing of information
  • publishing to a Redis PubSub channel the changes made to a task that has been sent to 'background'

Integration with Redis

It is about doing the same as was done to integrate the tokyo cabinet backend:

queue_libtokyocabinet.c

the problem of using tokyo cabinet against disk is the brutal loss of performance compared to using it against RAM, since writes are done synchronously.

At code level the greatest similarities are with:

queue_libmemcached.c

, although the problem this implementation has is that every time we restart memcached we have no persistence of the information that had been stored in memcached, it is as if the keys that had been introduced in the previous session had been erased. Besides, memcached does not support persistence in its data either.

So, what has to be done is to take the best of both integrations and make the module with Redis.

Subscribing to the updates of a task via Redis

When a task is sent in the background to Gearman it returns a 'Handler' so we can ask about the state of the task; the problem is that if we want to know how the task is progressing or have it tell us when it has finished there is no way to know it other than by 'pooling'. On the other hand, the 'worker' keeps updating the task whenever it sees fit so that Gearman can know what its state is.

My idea is that when using the Redis backend, at the same moment the state of the task is updated it is also published (publish) to a channel

PubSub de Redis

so that the code that sent the task can subscribe (subscribe) to this channel and follow the state of the task in real time and with a very low resource cost. This would save us the need for Gearman to be able to call a callback method to inform us of its state, since there are some languages where doing this is not so simple.

In the diagram I attach below we can see a sketch I have made about this:

sketch of ideas for Gearman with Redis
sketch of ideas for Gearman with Redis
  1. our code sends a task in 'background' to Gearman and it returns a 'Handler' to identify the task.

  2. the task is stored in Redis (set)

  3. our code subscribes to the PubSub channel of the task

  4. a worker requests the task

  5. the state of the task is published

  6. the state of the task keeps being updated

  7. points (5) and (6) are repeated until the task is finished

Feedback

As always, all kinds of criticism and ideas about my 'paranoia' are welcome.