Python Problem Overview
My rough understanding is that Redis is better if you need the in-memory key-value store feature, however I am not sure how that has anything to do with distributing tasks?
Does that mean we should use Redis as a message broker IF we are already using it for something else?
Solution 1 - Python
I've used both recently (2017-2018), and they are both super stable with Celery 4. So your choice can be based on the details of your hosting setup.
- If you must use Celery version 2 or version 3, go with RabbitMQ. Otherwise...
- If you are using Redis for any other reason, go with Redis
- If you are hosting at AWS, go with Redis so that you can use a managed Redis as service
- If you hate complicated installs, go with Redis
- If you already have RabbitMQ installed, stay with RabbitMQ
In the past, I would have recommended RabbitMQ because it was more stable and easier to setup with Celery than Redis, but I don't believe that's true any more.
AWS now has a managed service that is equivalent to RabbitMQ called Amazon MQ, which could reduce the headache of running this as a service in production. Please comment below if you have any experience with this and celery.