What problem do MOM (Message Oriented Middleware) solve? Scalability? Integration?

In which domain are they typically used and in which domains are they typically *not* used?

For example, say, is Google using such solution for it&#39;s main search engine or to power GMail?

What about big websites like Walmart, eBay, FedEx (pretty much a Java shop) and buy.com (pretty much an MS shop)? Does MOM solve a need there?

Does it make any sense when you&#39;re writing a Webapp where you control the server-side and have an homogenous environment (say tens of Amazon EC2 instances all running Linux + Java JVMs) there and where the clients are, well, Web browsers?

Does it make sense for desktop apps that need to communicate with a server?

Or is it &#39;only&#39; for big enterprise stuff where you typically have a happy mix of countless of different systems that needs to communicate in a way or another?  

I&#39;m a bit confused as to what they&#39;re useful for and I think that with example of where they&#39;re appropriate and where they&#39;re not appropriate I could better understand their use.


In which domains are message oriented middleware like AMQP useful?

I&#39;m learning the messaging system and got confused by those terminology. 

All the messaging system below provides loose coupling between services with different sets of features.

`queue` - FIFO, pulling mechanism, 1 consumer each queue but any number of producers?

`message bus` - pub/sub model, any number of consumers with any number of producers processing messages? Is `Azure Service Bus` an implementation of `message bus`? 

`event bus` - pub/sub model, any number of consumers with any number of producers processing events? 

Do people use `message bus` and `event bus` interchangeably as far as terminology goes?

What are the difference between events and messages? Are those just synonyms in this context? 

`event hub` - pub/sub model, partition, replay, consumers can store events in external storage or close to real-time data analysis. What exactly is event hub? 

`event grid` - it can be used as a downstream service of event hub. What does it exactly do that `event hub` doesn&#39;t do? 

Can someone provide some historical context as how each technology evolve to another each tied with some practical use cases?

I&#39;ve found [message bus vs. message queue][1] helpful


  [1]: https://stackoverflow.com/questions/57345003/message-bus-and-message-queue-understanding

Message bus vs. Service bus vs. Event hub vs Event grid

I&#39;ve been evaluating messaging technologies for my company but I&#39;ve become very confused by the conceptual differences between a few terms:

**Pub/Sub** vs **Multicast** vs **Fan Out**
I am working with the following definitions:

 - **Pub/Sub** has publishers delivering a separate copy of each message to
   each subscriber which means that the opportunity to guarantee delivery exists
 - **Fan Out** has a single queue pushing to all listening
   clients. 
 - **Multicast** just spams out data and if someone is listening
   then fine, if not, it doesn&#39;t matter.  No possibility to guarantee a client definitely gets a message.

Are these definitions right?  Or is Pub/Sub the pattern and multicast, direct, fanout etc. ways to acheive the pattern?

I&#39;m trying to work the out-of-the-box RabbitMQ definitions into our architecture but I&#39;m just going around in circles at the moment trying to write the specs for our app.

Please could someone advise me whether I am right?

Messaging Confusion: Pub/Sub vs Multicast vs Fan Out

Under what conditions would one favor apps talking via a message queue instead of via web services (I just mean XML or JSON or YAML or whatever over HTTP here, not any particular type)?

I have to talk between two apps on a local network. One will be a web app and have to request commands on another app (running on different hardware). The requests are things like creating users, moving files around, and creating directories. Under what conditions would I prefer XML Web Services (or straight TCP or something) to using a Message queue? 

The web app is Ruby on Rails, but I think the question is broader than that.

Message Queue vs. Web Services?

I am new to message brokers like [RabbitMQ](https://www.rabbitmq.com/) which we can use to create tasks / message queues for a scheduling system like [Celery](http://www.celeryproject.org/).

Now, here is the question:

- I can create a table in [PostgreSQL](https://www.postgresql.org/) which can be appended with new tasks and consumed by the consumer program like Celery.

- Why on earth would I want to setup a whole new tech for this like RabbitMQ?

Now, I believe scaling cannot be the answer since our database like PostgreSQL can work in a distributed environment.

I googled for what problems does the database poses for the particular problem, and I found:

- polling keeps the database busy and low performing
- locking of the table -&gt; again low performing
- millions of rows of tasks -&gt; again, polling is low performing

Now, how does RabbitMQ or any other message broker like that solves these problems?

Also, I found out that `AMQP` protocol is what it follows. What&#39;s great in that?

Can [Redis](https://redis.io/) also be used as a message broker? I find it more analogous to Memcached than RabbitMQ.

Please shed some light on this!

Why do we need message brokers like RabbitMQ over a database like PostgreSQL?

I was just reading abit about JMS and Apache ActiveMQ.
And was wondering what real world use have people here used JMS or similar message queue technologies for ?


Real world use of JMS/message queues?

I could use some pseudo-code, or better, Python.  I am trying to implement a rate-limiting queue for a Python IRC bot, and it partially works, but if someone triggers less messages than the limit (e.g., rate limit is 5 messages per 8 seconds, and the person triggers only 4), and the next trigger is over the 8 seconds (e.g., 16 seconds later), the bot sends the message, but the queue becomes full and the bot waits 8 seconds, even though it&#39;s not needed since the 8 second period has lapsed.

What&#39;s a good rate limiting algorithm?

I am trying to understand what JMS and how it is connected to AMQP terminology.
I know JMS is an API and AMQP is a protocol. 

Here are my assumptions (and questions as well)

   - RabbitMQ uses AMQP protocol (rather implements AMQP protocol)
   - Java clients need to use AMQP protocol client libraries to connect / use RabbitMQ
   - Where does JMS API come into play here? JMS API should use AMQP client libraries to connect to RabbitMQ?
   - Usually we use JMS to connect Message brokers like RabbitMQ, ActiveMQ, etc. Then what is the default protocol used here instead of AMQP?

Some of the above may be dumb. :-) But trying to wrap my head around it.

JMS and AMQP - RabbitMQ

We&#39;d be interested to hear any experiences with the pros and cons of ActiveMQ vs RabbitMQ vs ZeroMQ. Information about any other interesting message queues is also welcome.

ActiveMQ or RabbitMQ or ZeroMQ or

And are there any? To me, MB knows both subscribers and publishers and acts as a mediator, notifying subscribers on new messages (effectively a &quot;push&quot; model). MQ, on the other hand, is more of a &quot;pull&quot; model, where consumers pull messages off a queue.

Am I completely off track here?

Message Queue vs Message Bus -- what are the differences?

Looking to use a message queue in a small web app I&#39;m building with node.js. I looked at resque but not sure that&#39;s appropriate. The goal is to push notifications to clients based on backend and other client actions with socketio. I could do this with just socketio but I thought maybe a proper message queue would make this cleaner and I wouldn&#39;t have to reinvent the wheel. 

What are the options out there?

What are good message queue options for nodejs?

All of the [examples](https://www.google.com/#q=kafka+producer+key+value) of [Kafka](https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html) | [producers](https://www.tutorialspoint.com/apache_kafka/apache_kafka_simple_producer_example.htm) show the `ProducerRecord`&#39;s key/value pair as not only being the same type (all examples show `&lt;String,String&gt;`), but the same **value**. For example:

    producer.send(new ProducerRecord&lt;String, String&gt;(&quot;someTopic&quot;, Integer.toString(i), Integer.toString(i)));

But in the Kafka docs, I can&#39;t seem to find where the key/value concept (and its underlying purpose/utility) is explained. In traditional messaging (ActiveMQ, RabbitMQ, etc.) I&#39;ve always fired a message at a particular topic/queue/exchange. But Kafka is the first broker that seems to require key/value pairs instead of just a regulare &#39;ole string message.

So I ask: **What is the purpose/usefulness of requiring producers to send KV pairs?**

What&#39;s the purpose of Kafka&#39;s key/value pair-based messaging?

From my understanding, Celery is a distributed task queue, which means the only thing that it should do is dispatching tasks/jobs to others servers and get the result back. RabbitMQ is a message queue, and nothing more. However, a worker could just listen to the MQ and execute the task when a message is received. This achieves exactly what Celery offers, so why need Celery at all?

Why use Celery instead of RabbitMQ?

I have checked the official Android documentation/guide for `Looper`, `Handler` and `MessageQueue` . But I couldn&#39;t get it. I am new to android, and got very confused with these concepts. 

What is the relationship between Looper, Handler and MessageQueue in Android?

1. What are the allowed types of messages (strings, bytes, integers, etc.)?
2. What is the maximum size of a message?
3. What is the maximum number of queues and exchanges?

What are the limits of messages, queues and exchanges?

I&#39;ve been getting more into the internals of the Node.js architecture, and a term I see coming up a lot is &quot;tick&quot; as in &quot;next tick of the event loop&quot; or the function [nextTick()][1].

What I haven&#39;t seen is a solid definition of what exactly a &quot;tick&quot; is. Based on various articles ([such as this one][2]), I&#39;ve been able to piece a concept together in my head, but I&#39;m not sure how accurate it is.

Can I get a precise and detailed description of a Node.js event loop tick?

  [1]: http://nodejs.org/docs/latest/api/process.html#process_process_nexttick_callback
  [2]: http://howtonode.org/understanding-process-next-tick

What exactly is a Node.js event loop tick?

I&#39;ve been asked to evaluate RabbitMQ instead of Kafka but found it hard to find a situation where a message queue is more suitable than Kafka. Does anyone know use cases where a message queue fits better in terms of  throughput, durability, latency, or ease-of-use?

When to use RabbitMQ over Kafka?

I need a high performance message bus for my application so I am evaluating performance of `ZeroMQ`, `RabbitMQ` and `Apache Qpid`. To measure the performance, I am running a test program that publishes say 10,000 messages using one of the message queue implementations and running another process in the same machine to consume these 10,000 messages. Then I record time difference between the first message published and the last message received. 

Following are the settings I used for the comparison.

1. `RabbitMQ`: I used a &quot;fanout&quot; type exchange and a queue with default configuration. I used the RabbitMQ C client library.
2. `ZeroMQ`: My publisher publises to `tcp://localhost:port1` with `ZMQ_PUSH` socket, My broker listens on `tcp://localhost:port1` and resends the message to tcp://localhost:port2 and my consumer listens on `tcp://localhost:port2` using `ZMQ_PULL` socket. I am using a broker instead of peer to to peer communication in `ZeroMQ` to to make the performance comparison fair to other message queue implementation that uses brokers.
3. `Qpid` C++ message broker: I used a &quot;fanout&quot; type exchange and a queue with default configuration. I used the Qpid C++ client library.

Following is the performance result:

1. `RabbitMQ`: it takes about 1 second to receive 10,000 messages.
2. `ZeroMQ`: It takes about 15 milli seconds to receive 10,000 messages.
3. `Qpid`:  It takes about 4 seconds to receive 10,000 messages.

**Questions:**

1. Have anyone run similar performance comparison between the message queues? Then I like to compare my results with yours.
2. Is there any way I could tune `RabbitMQ` or `Qpid` to make it performance better? 

**Note:**

The tests were done on a virtual machine with two allocated processor. The result may vary for different hardware, however I am mainly interested in relative performance of the MQ products.

Performance comparison between ZeroMQ, RabbitMQ and Apache Qpid

I need to choose a new Queue broker for my new project.  

This time I need a scalable queue that supports pub/sub, and keeping message ordering is a must.  

I read Alexis comment: He writes:

&gt; &quot;Indeed, we think RabbitMQ provides stronger ordering than Kafka&quot;

I read the message ordering section in rabbitmq docs:

    

&gt; &quot;Messages can be returned to the queue using AMQP methods that feature
&gt; a requeue
&gt;      parameter (basic.recover, basic.reject and basic.nack), or due to a channel 
&gt;      closing while holding unacknowledged messages...With release 2.7.0 and later 
&gt;      it is still possible for individual consumers to observe messages out of 
&gt;      order if the queue has multiple subscribers. This is due to the actions of 
&gt;      other subscribers who may requeue messages. From the perspective of the queue 
&gt;      the messages are always held in the publication order.&quot;

If I need to handle messages by their order, I can only use rabbitMQ with an exclusive queue to each consumer?  

Is RabbitMQ still considered a good solution for ordered message queuing?




RabbitMQ - Message order of delivery

I am trying to learn messaging system. I have found that RabbitMq and NServiceBus are using together in few places. My questions are

 1. If I am using the RabbitMQ then why do i need NServiceBus? and vice versa
 2. What NServiceBus can do but RabbitMQ or Kafka cannot?
 3. Can I use NServiceBus and kafka together? Or Apache-Kafka does not require NServiceBus 



NServiceBus and Rabbit MQ or Kafka

I&#39;ve been playing with message queues (System V, but POSIX should be ok too) in Linux recently and they seem perfect for my application, but after reading The Art of Unix Programming I&#39;m not sure if they are really a good choice.

[http://www.faqs.org/docs/artu/ch07s02.html#id2922148][1]
&gt; The upper, message-passing layer of System V IPC has largely fallen out of use. The lower layer, which consists of shared memory and semaphores, still has significant applications under circumstances in which one needs to do mutual-exclusion locking and some global data sharing among processes running on the same machine. These System V shared memory facilities evolved into the POSIX shared-memory API, supported under Linux, the BSDs, MacOS X and Windows, but not classic MacOS.

[http://www.faqs.org/docs/artu/ch07s03.html#id2923376][2]
&gt;The System V IPC facilities are present in Linux and other modern Unixes. However, as they are a legacy feature, they are not exercised very often. The Linux version is still known to have bugs as of mid-2003. Nobody seems to care enough to fix them.

Are the System V message queues still buggy in more recent Linux versions? I&#39;m not sure if the author means that POSIX message queues should be ok?

It seems that sockets are the preferred IPC for almost anything(?), but I cannot see how it would be very simple to implement message queues with sockets or something else. Or am I thinking too complexly?

I don&#39;t know if it&#39;s relevant that I&#39;m working with embedded Linux?

  [1]: http://www.faqs.org/docs/artu/ch07s02.html#id2922148
  [2]: http://www.faqs.org/docs/artu/ch07s03.html#id2923376

Are message queues obsolete in linux?

I don&#39;t have any previous experience with *MQs and I&#39;m looking to build knowledge on JMS and message queues in general. That way, I wonder whether I should start with ActiveMQ or just &quot;ignore&quot; it altogether and start by teaching myself Apollo. Is Apollo as feature-complete as ActiveMQ? Does it implement JMS 2.0 (I see that ActiveMQ got stuck with 1.1)? Will I be missing something really important?

Also, how does Kafka compare to these two solutions?

ActiveMQ vs Apollo vs Kafka

At work, we need to build a jobs server for things like sending emails, building PDFs, crunching some data, etc.  Obviously, we&#39;d like to build on some sort of generic queueing system. I&#39;m familiar with Gearman, and that this is exact problem that it tries to solve: putting jobs on a queue where workers come to pick them up.  However, I&#39;m seeing lots of mentions of Rabbitmq and am unclear how it&#39;s used in this scenario. 

Is Rabbitmq a good framework to build a distributed jobs system on top of?  

Rabbitmq or Gearman - choosing a jobs queue

We are implementing a REST API, which will kick off multiple long running backend tasks. I have been reading the RESTful Web Services Cookbook and the recommendation is to return HTTP 202 / Accepted with a Content-Location header pointing to the task being processed. (e.g. http://www.example.org/orders/tasks/1234), and have the client poll this URI for an update on the long running task. 

The idea is to have the REST API immediately post a message to a queue, with a background worker role picking up the message from the queue and spinning up multiple backend tasks, also using queues. The problem I see with this approach is how to assign a unique ID to the task and subsequently let the client request a status of the task by issuing a GET to the Content-Location URI. 

If the REST API immediately posts to a queue, then it could generate a GUID and attach that as an attribute on the message being added to the queue, but fetching the status of the request becomes awkward. 

Another option would be to have the REST API immediately add an entry to the database (let&#39;s say an order, with a new order id), with an initial status and then subsequently put a message on the queue to kick off the back ground tasks, which would then subsequently update that database record. The API would return this new order ID in the URI of the Content-Location header, for the client to use when checking the status of the task. 

Somehow adding the database entry first, then adding the message to the queue seems backwards, but only adding the request to the queue makes it hard to track progress. 

What would be the recommended approach? 

Thanks a lot for your insights. 



Long running REST API with queues

Can I get the comparison between RabbitMQ and MSMQ. It will be helpful performance information on different factors are available.

Comparison between RabbitMQ and MSMQ

What&#39;s the maximum number of queues that RabbitMQ can handle on a single server?

Does it depend on RAM? Does it depends on erlang processes?

RabbitMQ - How many queues can RabbitMQ handle on a single server?

Kafka splits incoming messages up into partitions, according to the partition assigned by the producer. Messages from partitions then get consumed by consumers in different consumer groups. 

This architecture makes me wary of using Kafka as a work/task queue, because I have to specify the partition at time of production, which indirectly limits which consumers can work on it because a partition is sent to only one consumer in a consumer group. I would rather not specify the partition ahead of time, so that whichever consumer is available to take that task can do so. Is there a way to structure partitions/producers in a Kafka architecture where tasks can be pulled by the next available consumer, without having to split up work ahead of time by choosing a partition when the work is produced?

Using only one partition for this topic would put all the tasks in the same queue, but then the number of consumers is limited to 1 per consumer group, so each consumer would have to be in a different group. Then all of the task get distributed to each consumer group, though, which is not the kind of work queue I&#39;m looking for.

Is Apache Kafka appropriate for use as a task queue?

Is Apache Kafka appropriate for use as an unordered task queue?

This two images are from http://zguide.zeromq.org/page:all.

What is the difference between this two pattern if we ignore sink in push-pull pattern ?
Is there a difference in how a message gets transfer, if yes what is the difference ?

![](https://github.com/imatix/zguide/raw/master/images/fig4.png)
![](https://github.com/imatix/zguide/raw/master/images/fig5.png)


 

difference between pub-sub and push-pull pattern in zeroMq

I have been studying apache kafka for a month now. I am however, stuck at a point now. My use case is, I have two or more consumer processes running on different machines. I ran a few tests in which I published 10,000 messages in kafka server. Then while processing these messages I killed one of the consumer processes and restarted it. Consumers were writing processed messages in a file. So after consumption finished, file was showing more than 10k messages. So some messages were duplicated. 

In consumer process I have disabled auto commit. Consumers manually commit offsets batch wise. So for e.g if 100 messages are written to file, consumer commits offsets. When single consumer process is running and it crashes and recovers duplication is avoided in this manner. But when more than one consumers are running and one of them crashes and recovers, it writes duplicate messages to file.

Is there any effective strategy to avoid these duplicate messages? 

Effective strategy to avoid duplicate messages in apache kafka consumer

I&#39;m a newbie with Rabbitmq(and programming) so sorry in advance if this is obvious.  I am creating a pool to share between threads that are working on a queue but I&#39;m not sure if I should use connections or channels in the pool.  

I know I need channels to do the actual work but is there a performance benefit of having one channel per connection(in terms of more throughput from the queue)? or am I better off just using a single connection per application and pool many channels?


note: because I&#39;m pooling the resources the initial cost is not a factor, as I know connections are more expensive than channels.  I&#39;m more interested in throughput.

Is there a performance difference between pooling connections or channels in rabbitmq?

Does anyone know if there&#39;s a way to check the number of messages in a RabbitMQ queue from a client application?

I&#39;m using the .NET client library.

Check RabbitMQ queue size from client

We&#39;ve got an application which will be using RabbitMQ and have several different queues for passing messages between tiers.

Initially, I was planning to use multiple direct exchanges, with one for each message type, but it looks like having a single topic exchange with queues using different routing key bindings will achieve the same thing.

Having a single exchange also seems like it would be a bit easier to maintain, but I was wondering if there is any benefit (if any) of doing it one way over the other?

Option 1, using multiple direct exchanges:

    ExchangeA (type: direct)
    -QueueA

    ExchangeB (type: direct)
    -QueueB

    ExchangeC (type: direct)
    -QueueC

Option 2, using single topic exchange:

    Exchange (type: topic)
    -QueueA  (receives messages from exchange with routing key of &quot;TypeA&quot;)
    -QueueB  (receives messages from exchange with routing key of &quot;TypeB&quot;)
    -QueueC  (receives messages from exchange with routing key of &quot;TypeC&quot;)

All Message Queue Solutions on Gang of Coders

Total of 32 Message Queue Solutions