Async tasks queuing with celery — A conceptual approach

you can find dozens of “introductions” and “how to” articles on the internet about different matters, but the real game changer for you is a quick review to give you the needed insight about the topic you are searching for.

here I’ve tried to present my own experience about celery with my words, gained trough reading documentation and watching hours of content on YouTube, combined with real-world examples at work.

For fast access you can use headings

Based on celery docs, Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

It’s a task queue with focus on real-time processing, while also supporting task scheduling.

What are we going to improve with celery

With distributing long-running tasks among servers called “Worker” we reduce overhead on the main server of our web application, which in real world applications means :

• Fast user experience with main server only processing user requests and lets the workers handle the heavy computational tasks
• Lowering resource usage by optimizing usage for the respective tasks they are gonna execute resulting in green IT
• Reducing resource cost for the company

Important terminologies

In celery world, whether reading the docs or other articles you are gonna encounter some new words, so let’s see what are those:

read them all several times and let the connection between these terms shape in your mind

Broker : The middle-man between worker and task producer

Backend : Also called backend-result, a database to store success/failure of tasks

Tasks : A function that needs to be executed

Task Queue : Queue of tasks (LOL)

Producer : In short it’s client, which means the entity that enqueues tasks into the broker

Beat : Celery built-in command to enqueue tasks, producers can be other python applications or even functions as well
Worker : A daemon that executes the function (task) that was enqueued on the broker

You will get to know these terms better as you work with celery

Life story of a stalk

Imagine you have a web application with a dozens of users and each one of them wants to do something data-intensive inside your application like:

• Generating bulk emails — sending them
• Creating thumbnails for uploaded pic
• Converting video
• Generating pdf
• Recomputing ratings
• Spam filtering
• Factoring a large number
• Searching a directory structure

and so on …

If you code your application in sync mode, you will have hard time responding to users and the slow response time of your application will not satisfy users

Guess who is sad now ? YOU ARE, because users will find an alternative to your services and get the job done, So use celery for you own good not users

Here comes celery as an async task queuing system, but remember “celery Is not a native queuing system by itself, rather it makes the process of creating a queue system easier for you.”

Like the diagram below, when client requests for a specific service we enqueue that to a queue inside broker as a task, on the other hand we had set one worker (a server or a process in a server) to listen for that queue and as soon as it sees the new task, worker will start running that, returning the result to broker and then client.

Lets imagine several clients doing this at the same time with different tasks

Here is the scenario :

Client №1, whom I am (Sina), asks the application to generate a large pdf

Client №2, whom you are, asks the application to convert a large video

Client №3, whom he is a teenage hacker, asks the application for a search query (he is trying to bring our web application down — you fool ! )

Our main server, Producer, puts all these tasks in the dedicated queue inside broker and is free to process any other requests from clients, no more overheads.

At the same time we had set each worker to listen to a specific queue and run the tasks inside if any.

Here if we have many clients each client asking for many services concurrently, your application will work like an Atomic Clock while all the heavy loads are being processed in the background on dedicated servers and the result will get back to clients when done.

What is it like to be a celery producer

Like being the king, you just order your servant (broker) around and create chores for the workers to do, the result of their work will get to you by the servant (broker) when it’s done.

We can use Django, Flask, and other web frameworks as producer, or use a python code and celery built-in producer, beat, to create tasks

What is it like to be a celery broker

Middle-man, servant or the leaderboard where tasks are listed in different sections (queues) both teams, the producer and workers, interacts with it.

If you search message queuing you are gonna find some technologies like redis (in memory DB) or RabbitMQ (message-broker) that are being used as brokers with celery — check celery docs

What is it like to be celery worker

Boring, you sit there looking at a whiteboard like thing called broker and when your name pops up, you get your shovel and start working on the job broker tells you to

Do not see workers as some alien technology its the same server as before just running a specific task that we told it to, here is some example of what workers can do:

• make changes in DB
• Update UI via web-hooks or callbacks
• Add items to cache
• Process file, send email
• Queue future tasks and more

*One worker can listen to several queue and run tasks on both

The queue ensures that each worker only gets one task at a time and that each task is only being processed by one worker, while a worker is running a task it’s not available so incoming tasks are stored in the queue, When a worker becomes available, it takes the first task from the front of the queue and begins processing it, what about many workers ? They take turns in order.

You have your insight right now, Let’s dive deeper into stuff and after that we are gonna run a simple celery application.

Best practice celery

• Make tasks idempotent when you can
• It’s difficult troubleshooting any issues that arise when tasks are state based given that they are executing async
• Avoid having one task depend on other tasks
• It’s already in the background and not blocking the user, but in my experience I had to do this because of reducing run-time for dependent tasks.
• Tasks still use resources so code efficient
• Tasks are not fire and forget, then fail
• When completion of a is crucial, make sure to code error scenarios and any retry mechanism
• Pass object ids instead of objects themselves
• If you pass an object’s instance instead of it’s id it ay have become stale by the time task has started running, it’s best to re-fetch via the object if from DB
• Try to avoid database brokers, they don’t work as well as brokers specially designed for message brokers
• Be sure to test with CELERY_ALWAYS_EAGER=True, this forces the task to be dispatched and completed in a async call, which lets you know right away if there are issues with your task
• Be sure to test with CELERY_ALWAYS_EAGER=False, to make sure your code works and errors are not hidden by them being in the background

Avoid queue mechanism for

• Processing payment.
• Address validation
• Replacing cron
• If you don’t use advance feature of queue there is no difference between cron and task queue
• As a multi-threading mechanism
• Execute a million workers for data sort it’s better to use multi-thread libraries
• Tasks that aren’t that slow or frequent
• Don’t complicate it for something simple
• Logging : you want logging to be in order not in a async form without order
• If you fire off a task and stall for it to finish

Final words

that was all from my point of view, I’ve tried to simplify the concept behind the matter with story telling and not going deep into coding, if you have any questions about celery or other topics, I’m here to help and gain some new friends.
you can find me on social media via @haririsina
on medium via @hariri_sina
have a nice one :)