Why use Celery instead of RabbitMQ?

PythonMessage QueueRabbitmqCelery

Python Problem Overview


From my understanding, Celery is a distributed task queue, which means the only thing that it should do is dispatching tasks/jobs to others servers and get the result back. RabbitMQ is a message queue, and nothing more. However, a worker could just listen to the MQ and execute the task when a message is received. This achieves exactly what Celery offers, so why need Celery at all?

Python Solutions


Solution 1 - Python

You are right, you don't need Celery at all. When you are designing a distributed system there are a lot of options and there is no right way to do things that fits all situations.

Many people find that it is more flexible to have pools of message consumers waiting for a message to appear on their queue, doing some work, and sending a message when the work is finished.

Celery is a framework that wraps up a whole lot of things in a package but if you don't really need the whole package, then it is better to set up RabbitMQ and implement just what you need without all the complexity. In addition, RabbitMQ can be used in many more scenarios besides the task queue scenario that Celery implements.

But if you do choose Celery, then think twice about RabbitMQ. Celery's message queueing model is simplistic and it is really a better fit for something like Redis than for RabbitMQ. Rabbit has a rich set of options that Celery basically ignores.

Solution 2 - Python

Celery basically provides a nice interface to doing just what you said, and deals with all the configuration for you. Yes you could do it by hand, but you'd just be rewriting celery.

Solution 3 - Python

In my opinion, it's easy to integrate celery with flower and other monitoring packages than RabbitMQ.

It all depends on the use case anyways...

If you don't need other functionalities celery provides, RabbitMQ would be an easy way out. Weighing out your options wouldn't be a bad idea...

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionKarView Question on Stackoverflow
Solution 1 - PythonMichael DillonView Answer on Stackoverflow
Solution 2 - PythonmjtamlynView Answer on Stackoverflow
Solution 3 - PythonYusuf GaniyuView Answer on Stackoverflow