Why do we need to use rabbitmq
PythonDjangoTwistedRabbitmqPython Problem Overview
Why do we need RabbitMQ when we have a more powerful network framework in Python called Twisted. I am trying to understand the reason why someone would want to use RabbitMQ.
Could you please provide a scenario or an example using RabbitMQ?
Also, where can I find a tutorial on how to use RabbitMQ?
Python Solutions
Solution 1 - Python
Let me tell you a few reasons that makes using MOM (Message Oriented Middleware) probably the best choice.
Decoupling:
It can decouple/separate the core components of the application. There is no need to bring all the benefits of the decoupled architecture here. I just want to point it out that this is one of the main requirement of writing a quality and maintainable software.
Flexibility:
It is actually very easy to connect two totally different applications written on different languages together by using AMQP protocol. These application will talk to each other by the help of a "translator" which is MOM.
Scalability:
By using MOM we can scale the system horizontally. One message producer can transmit to unlimited number of message consumers a task, a command or a message for processing and for scaling this system all we need to do is just create new message consumers. Lets say we are getting 1000 pictures per second and we must resize them. Solving this problem with traditional methods could be a headache. With MOM we can transmit images to the message consumers which can do their job asynchronously and make sure data integrity is intact.
They are other benefits of using MOM as well but these 3 are the most significant in my opinion.
Solution 2 - Python
Twisted is not a queue implementation. Apart from that RabbitMQ offers enterprise-level queuing features and implements the AMQP protocol which is often needed in an enterprise world.
Solution 3 - Python
Twisted is a networking library that implements a number of network protocols as well as allowing you to create your own. One of the protocols that you can use with Twisted is AMQP https://launchpad.net/txamqp
RabbitMQ is an AMQP broker, i.e. a services that runs outside of your application, probably on a separate cluster of servers. AMQP is merely the protocol that is used to communicate with a message queueing broker like RabbitMQ. You get a lot of things from RabbitMQ. You can send messages persistently with guaranteed delivery so they will arrive even if your app crashes, and even if the RabbitMQ broker ends up being restarted. You get load balancing between message consumers if you have multiple consumers on the same queue. You get interoperability with apps in other languages as long as you use a reasonably open serialization format for your message bodies. AMQP allows you to break up a monolithic app into many loosely coupled parts that can run on different servers. This is a big win for long term maintenance of an application.
Solution 4 - Python
RabbitMQ is a bit more than mere messaging... It's a common platform that has ability to inter-connect applications. Using RabbitMQ a Java application can speak to a Linux server and/or a .NET app, to a Ruby & rails + almost anything that finds its place in the corporate web development. And most importantly it implements the "fire and forget" model proposed by AMQP. Its just a perfect replacement for JMS or ESB, especially if you are dealing with cross platform architecture, with a guaranty of reliability. There is even a special feature called RPC (Remote procedure call) that adds to the ease of development in the distributed arch.
Apart from all these, in the world financial services like Stock-exchange or share-market where a lot of reliable and efficient routing is required (suppose you don't know the actual number of people subscribed to your services, but want to ensure that who ever does so, receives your pings whether they are connected in this moment, or will connect later), RabbitMQ rules because it's based on ERLANG & the Open-telecom platform that assures high performance while using minimum resources. For the most convenient introduction to RabbitMQ, see rabbitmq.com/getstarted.html for your native development language.
Solution 5 - Python
RabbitMQ is an implementation of AMQP, which defines an interoperable protocol for message oriented middleware. As such, it defines semantics for message creation, publication, routing and consumption that can be implemented on any platform.
Conceptually, it could be considered as a specialization of a networking engine like Twisted, but based on an industry accepted standard.
Here is a blog from Ross Mason that discusses the interest of interoperable publish-subscribe with AMQP: http://blogs.mulesoft.org/inter-operable-publishsubscribe-with-amqp/
Solution 6 - Python
I use RabbitMQ as message broker for Celery.
Also, I have worked with Twisted. It is different.
See here for more on AMQP: http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol
Solution 7 - Python
RabbitMQ works on message queueing technologies like AMQP which helps keep things clean and latency-free.
And the best scenario to make use of RabbitMQ is for background processing of data which can take more time to be processed and cannot be served over HTTP. For example, if you want to download a report from your web app. And that report generation takes like 15-20 mins to be processed and get downloaded. Then in that case you should be pushing the download request to the RabbitMQ queue and then you should be expecting that report to be delivered to you via email or notification.
To know about exactly how RabbitMQ works or how it solves such use cases you should check out this YouTube video - https://youtu.be/vvxvlF6CJkg and https://youtu.be/0dXwJa7veI8