Python Celery Asynchronous Task Decorator

A while ago I was working on a project in Django that had many tasks that needed to be executed asynchronously such as sending emails, generating reports, and checking some database related issues.

Executing asynchronous tasks in Python can be made in multiple ways, one of them is using celery, a distributed task queue written in Python which has proved itself to be a great solid task queue system.

And if you’re using Django then there is a Django app to make your life easier, django-celery.

Now as always I like to make my own life easier also by implementing generic solutions for my needs, so I have written this simple piece of code to create a function decorator that declares the function as an asynchronous task, which means that by just adding this decorator to the function it becomes asynchronous.

So let’s get started!

Description

The following code will allow you to decorate Python functions with a decorator that will allow them to be executed asynchronously in a seamless easy way. You don’t have to change any thing in your code, just add the asynchronous_task decorator to the functions you want them to be executed asynchronously and you’re done.
This code can be used with Django also, just uncomment the Django block in it and it’ll work.

I wrote this code in a file called celery_tasks.py which probably the place you’ll have your all celery tasks, it adds a task called execute that is responsible for executing the passed function and parameters.

The code looks like a mess I know, but simply what it does is that you pass this task the following parameters:

  1. Python module name to get the function from
  2. Method name to be executed
  3. Positional and named parameters

The second part is defining the decorator, lets assume that this file is called celery_decorator.py:
Note that I’m assuming that celery_decorator.py is in the same folder of your celery_tasks.py file.

This decorator simply wraps the function in a call to the execute task on celery, so that it’ll be executed asynchronously, it accept an optional parameter that defines whether the decorator should check if celery is running and if it’s not then execute the function synchronously, it is set to False by default because it takes some time to check celery, so it makes the performance poor a little bit.

Now we can us it on any function inside our code:

Now in your code you just simply don’t change anything, all the calls to these two functions will be asynchronous calls.

Download

The source code is licensed under the GNU Public License (GPL).

You can download a copy of celery_tasks.py and celery_decorators.py from this Gist: https://gist.github.com/3874979

Hope this helps somebody to perform faster 🙂

One thought on “Python Celery Asynchronous Task Decorator

  1. Seems usefull, but maybe someone with an eye for security could give this some thought?

    I suspect there being a significant risk in havind a celery worker execute anything thrown at it. I like the idea of the necessary common codebase for the workers, which gives you some control over what workers will do. With the approach shown here, more security emphasis is on the the messaging middleware.

Leave a Reply