I am running in my locahost both Zookeeper and Kafka (1 instance each).

I create succesfully a topic from kafka:

    ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --replication-factor 1 --partitions 1 --topic Hello-Nicola

    Created topic &quot;Hello-Nicola&quot;.

Kafka logs show:

    [2017-12-06 16:00:17,753] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
    [2017-12-06 16:03:19,347] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Hello-Nicola-0 (kafka.server.ReplicaFetcherManager)
    [2017-12-06 16:03:19,393] INFO Loading producer state from offset 0 for partition Hello-Nicola-0 with message format version 2 (kafka.log.Log)
    [2017-12-06 16:03:19,406] INFO Completed load of log Hello-Nicola-0 with 1 log segments, log start offset 0 and log end offset 0 in 35 ms (kafka.log.Log)
    [2017-12-06 16:03:19,408] INFO Created log for partition [Hello-Nicola,0] in /tmp/kafka-logs with properties {compression.type -&gt; producer, message.format.version -&gt; 1.0-IV0, file.delete.delay.ms -&gt; 60000, max.message.bytes -&gt; 1000012, min.compaction.lag.ms -&gt; 0, message.timestamp.type -&gt; CreateTime, min.insync.replicas -&gt; 1, segment.jitter.ms -&gt; 0, preallocate -&gt; false, min.cleanable.dirty.ratio -&gt; 0.5, index.interval.bytes -&gt; 4096, unclean.leader.election.enable -&gt; false, retention.bytes -&gt; -1, delete.retention.ms -&gt; 86400000, cleanup.policy -&gt; [delete], flush.ms -&gt; 9223372036854775807, segment.ms -&gt; 604800000, segment.bytes -&gt; 1073741824, retention.ms -&gt; 604800000, message.timestamp.difference.max.ms -&gt; 9223372036854775807, segment.index.bytes -&gt; 10485760, flush.messages -&gt; 9223372036854775807}. (kafka.log.LogManager)
    [2017-12-06 16:03:19,409] INFO [Partition Hello-Nicola-0 broker=0] No checkpointed highwatermark is found for partition Hello-Nicola-0 (kafka.cluster.Partition)
    [2017-12-06 16:03:19,411] INFO Replica loaded for partition Hello-Nicola-0 with initial high watermark 0 (kafka.cluster.Replica)
    [2017-12-06 16:03:19,413] INFO [Partition Hello-Nicola-0 broker=0] Hello-Nicola-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)


But Zookeeper logs show:

    2017-12-06 16:03:19,299 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x1000177fb3d0001 type:create cxid:0x43 zxid:0x26 txntype:-1 reqpath:n/a Error Path:/brokers/topics/Hello-Nicola/partitions/0 Error:KeeperErrorCode = NoNode for /brokers/topics/Hello-Nicola/partitions/0
    2017-12-06 16:03:19,302 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x1000177fb3d0001 type:create cxid:0x44 zxid:0x27 txntype:-1 reqpath:n/a Error Path:/brokers/topics/Hello-Nicola/partitions Error:KeeperErrorCode = NoNode for /brokers/topics/Hello-Nicola/partitions


If I try to produce messages:

    ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello-Nicola
    &gt;ciao
    [2017-12-06 16:04:21,897] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
    [2017-12-06 16:04:22,000] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)


server.properties (in kafka) is:

    broker.id=0
    listeners=PLAINTEXT://mylocal-0:9092
    num.network.threads=3
    num.io.threads=8
    socket.send.buffer.bytes=102400
    socket.receive.buffer.bytes=102400
    socket.request.max.bytes=104857600
    num.partitions=1
    num.recovery.threads.per.data.dir=1
    offsets.topic.replication.factor=1
    transaction.state.log.replication.factor=1
    transaction.state.log.min.isr=1
    log.retention.hours=168
    log.segment.bytes=1073741824
    log.retention.check.interval.ms=300000
    zookeeper.connect=localhost:2181
    zookeeper.connection.timeout.ms=6000
    group.initial.rebalance.delay.ms=0


It seems that Zookeeper didn&#39;t registrer any broker.

Any suggestion, please?





Kafka + Zookeeper: Connection to node -1 could not be established. Broker may not be available

I&#39;m learning how to use Kafka on this website [link][1](except I&#39;m using port 2182 in zookeeper), but it shows:

&gt; zookeeper is not a recognized option

after executing:

&gt; sudo ./bin/kafka-console-consumer.sh --topic test --zookeeper
&gt; localhost:2182

How to fix it?

Env:

    kafka_2.11-2.1.0
    zookeeper-3.4.10


  [1]: https://data-artisans.com/blog/kafka-flink-a-practical-how-to
  [2]: https://stackoverflow.com/questions/53341838/kafka-consumer-zookeeper-is-not-a-recognized-option

zookeeper is not a recognized option when executing kafka-console-consumer.sh

I have installed zookeeper in 3 different aws servers. The following is the configuration in all the servers

    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/var/zookeeper
    clientPort=2181
    server.1=x.x.x.x:2888:3888
    server.2=x.x.x.x:2888:3888
    server.3=x.x.x.x:2888:3888

All the three instance have a `myid` file at `var/zookeeper` with appropriate id in it. All the three servers have all ports open from the aws console. But when I run the zookeeper server, I get the following error in all the instances.

    2015-06-19 12:09:22,989 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] 
      - Cannot open channel to 2 at election address /x.x.x.x:3888
    java.net.ConnectException: Connection refused
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:579)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
      at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
      at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
    2015-06-19 12:09:23,170 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382]
       - Cannot open channel to 3 at election address /x.x.x.x:3888
    java.net.ConnectException: Connection refused
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:579)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
      at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
      at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
      at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
    2015-06-19 12:09:23,170 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 25600

Zookeeper error: Cannot open channel to X at election address

I was trying to run Kafka on Windows machine and when I try to start the zookeeper I am facing this weird error:

&gt;classpath is empty. please build the project first e.g. by running &#39;gradlew jarall&#39;

classpath is empty. please build the project first

I have written a kafka consumer and producer that worked fine until today.
This morning, when I started zookeeper and kafka, my consumer was not able to read messages, and I found this in the zookeeper logs:

    INFO Got user-level KeeperException when processing sessionid:0x151c41e62e10000 
         type:create cxid:0x2a zxid:0x1e txntype:-1 reqpath:n/a 
         Error Path:/brokers/ids 
         Error:KeeperErrorCode = NodeExists for /brokers/ids   
         (org.apache.zookeeper.server.PrepRequestProcessor)



Zookeeper &amp; Kafka error KeeperErrorCode=NodeExists

One of the first things I think about when using a new service (such as a non-RDBMS data store or a message queue) is: &quot;How should I structure my data?&quot;.

I&#39;ve read and watched some introductory materials. In particular, take, for example, [Kafka: a Distributed Messaging System for Log Processing][sw], which writes:

* &quot;a Topic is the container with which messages are associated&quot;
* &quot;the smallest unit of parallelism is the partition of a topic. This implies that all messages that ... belong to a particular partition of a topic will be consumed by a consumer in a consumer group.&quot;

Knowing this, what would be a good example that illustrates how to use topics and partitions? When should something be a topic? When should something be a partition?

[sw]: http://www.systemswemake.com/papers/kafka

As an example, let&#39;s say my (Clojure) data looks like:

    {:user-id 101 :viewed &quot;/page1.html&quot; :at #inst &quot;2013-04-12T23:20:50.22Z&quot;}
    {:user-id 102 :viewed &quot;/page2.html&quot; :at #inst &quot;2013-04-12T23:20:55.50Z&quot;}

Should the topic be based on `user-id`? `viewed`? `at`? What about the partition?

How do I decide?

Data Modeling with Kafka? Topics and Partitions

In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of data from the broker). Given this, I do not want the overhead of using Zookeeper; Can I not just use the broker only? Why is a Zookeeper must?

Is Zookeeper a must for Kafka?

I&#39;ve been looking at [Zookeeper][1] recently and wondered whether anybody was using it currently and what they were specifically using it for storing.

The most common use case is for configuration information, but what kind of data and how much data are you storing?

  [1]: http://wiki.apache.org/hadoop/ZooKeeper

Real World Use of Zookeeper

I&#39;ve generally implemented **sequence number generation** using database sequences in the past.

e.g. Using Postgres SERIAL type [http://www.neilconway.org/docs/sequences/][1]

I&#39;m curious though as how to generate sequence numbers for large distributed systems where there is no database.  Does anybody have any experience or suggestions of a best practice for achieving sequence number generation in a **thread safe** manner for multiple clients?


  [1]: http://www.neilconway.org/docs/sequences/

Distributed sequence number generation?

I&#39;m using ***kafka 0.10*** without *zookeeper*. I want to get kafka topics list. This command is not working since we&#39;re not using zookeeper: 
`bin/kafka-topics.sh --list --zookeeper localhost:2181`. How can I get the same output without zookeeper?

List all Kafka 0.10 topics using - -zookeeper flag without access to Zookeeper

Is there a way to delete all the data from a topic or delete the topic before every run?

Can I modify the KafkaConfig.scala file to change the `logRetentionHours` property?  Is there a way the messages gets deleted as soon as the consumer reads it?


I am using producers to fetch the data from somewhere and sending the data to a particular topic where a consumer consumes, can I delete all the data from that topic on every run?  I want only new data every time in the topic.  Is there a way to reinitialize the topic somehow?

Is there a way to delete all the data from a topic or delete the topic before every run?

I exploring a few options to setup kafka and I knew that the Zookeeper has to be up and running to initiate a kafka.


I would like to know how can I find the below.


1) hostname and port for my zookeeper instance---I checked the zoo.cfg and I could only find the ClientPort not the hostname, will hostname be the hostname of my box??


2) To check if ZooKeeper is up and running---I tried to do a `ps -ef | grep &quot;zoo&quot;` I could not find anything. May be I am using a wrong key word to search??

Any help would be really appreciated?

How to check if ZooKeeper is running or up from command prompt?

I am trying to understand ZooKeeper, how it works and what it does. Is there any application which is comparable to ZooKeeper? 

If you know, then how would you describe ZooKeeper to a layman?

I have tried apache wiki, zookeeper sourceforge...but I am still not able to relate to it.

I just read thru http://zookeeper.sourceforge.net/index.sf.shtml, so aren&#39;t there more services like this? Is it as simple as just replicating a server service?

Explaining Apache ZooKeeper

[ZooKeeper][1] is a highly available coordination service for data centers. It originated in the Hadoop project. One can implement locking, fail over, leader election, group membership and other coordination issues on top of it.
Are there any alternatives to ZooKeeper? (free software of course)

  [1]: http://zookeeper.apache.org

ZooKeeper alternatives? (cluster coordination service)

I installed the stable version of kafka (0.8.1 with 2.9.2 Scala) as per their website and am running it with a 3 node zookeeper ensemble (3.4.6). I tried to create a test topic but keep seeing that there is no leader assigned to the partition of the topic:

    [kafka_2.9.2-0.8.1]$ ./bin/kafka-topics.sh --zookeeper &lt;zookeeper_ensemble&gt; --describe --topic test-1
    Topic:test-1	PartitionCount:1	ReplicationFactor:3	Configs:
    	Topic: test-1	Partition: 0	**Leader: none**	Replicas: 0,1,2	**Isr:** 

I tried to write to the topic anyway using the console producer but ran into the LeaderNotAvailableException exception:

    [kafka_2.9.2-0.8.1]$ ./kafka-console-producer.sh --broker-list &lt;broker_list&gt; --topic test-1
    
    hello world
    
    [2014-04-22 11:58:48,297] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -&gt; 
    No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
    
    [2014-04-22 11:58:48,321] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -&gt; 
    No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
    
    [2014-04-22 11:58:48,322] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)
    
    [2014-04-22 11:58:48,445] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -&gt; 
    No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
    
    [2014-04-22 11:58:48,467] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -&gt; 
    No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
    
    [2014-04-22 11:58:48,467] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)
    
    [2014-04-22 11:58:48,590] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -&gt; 
    No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
    
    [2014-04-22 11:58:48,612] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -&gt; 
    No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
    
    [2014-04-22 11:58:48,612] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)
    
    [2014-04-22 11:58:48,731] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -&gt; 
    No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
    
    [2014-04-22 11:58:48,753] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -&gt; 
    No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
    
    [2014-04-22 11:58:48,754] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)
    
    [2014-04-22 11:58:48,876] WARN Error while fetching metadata [{TopicMetadata for topic test-1 -&gt; 
    No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
    
    [2014-04-22 11:58:48,877] ERROR Failed to send requests for topics test-1 with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
    
    [2014-04-22 11:58:48,878] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread)
    kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
    	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
    	at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
    	at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
    	at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
    	at scala.collection.immutable.Stream.foreach(Stream.scala:547)
    	at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
    	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)

I should also state that this was working initially for a few days and then suddenly any topic that was created had this missing leader problem.

Running into LeaderNotAvailableException when using Kafka 0.8.1 with Zookeeper 3.4.6

One of the problems, I face with supervisord is that when I have a command which in turn spawns another process, supervisord is not able to kill it. 

For example I have a java process which when runs normally is like

     $ zkServer.sh start-foreground
     $ ps -eaf | grep zk
     user 30404 28280  0 09:21 pts/2    00:00:00 bash zkServer.sh start-foreground
     user 30413 30404 76 09:21 pts/2    00:00:10 java -Dzookeeper.something..something

The supervisord config file looks like:

    [program:zookeeper]
    command=zkServer.sh start-foreground
    autorestart=true
    stopsignal=KILL

These kind of processes which have multiple childs are not well handled by supervisord when it comes to stopping them from `supervisorctl`. So when I run this from the supervisord and try to stop it from supervisorctl, only the top level process gets killed but not the actual java process.


All Apache Zookeeper Solutions on Gang of Coders

Total of 16 Apache Zookeeper Solutions