This page describes how you setup Apache Kafka to function with 4S Microservices on your local machine. The instructions below have been verified to work on OSX 10.12.1 with Docker 1.12.1 (build: 12133).
Run Kafka on your local machine by following these steps.
Get the scripts and docker definitions from 4S bitbucket: https://bitbucket.org/4s/kafka
You must setup a docker network by issuing the command:
docker network create <network-name>
This network is shared by all microservices. Network setup must only be carried out once.
Verify that everything went well by issuing the command;
docker network ls
In order to build a Kafka docker image and start all necessary services (e.g. Zookeeper, kafka-manager and kafka) issue the command:
docker-compose -f docker-compose.yml -f docker-compose.opentele3.dev.yml up -d --force-recreate
For more information about docker-compose, visit the docker website: https://docs.docker.com/compose/reference/overview/
Since Kafka needs to know its own “host name” when it starts up, problems can arise if accessing kafka from various environments (local and global). Therefore, two aliases have been created for kafka and zookeeper in this setup. The aliases are 'kafka' and 'zookeeper', respectively.
Properties props = new Properties(); props.put("bootstrap.servers", "kafka:9092"); KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
You need to add kafka to your
/etc/hosts file to be able to reach kafka on your local machine. Add the following lines to your
127.0.0.1 kafka 127.0.0.1 zookeeper
Create a Kafka producer:
kafkacat -P -b kafka:9092 -t important
Create a Kafka consumer:
kafkacat -C -b kafka:9092 -t important
If this is the first time you are running this on your machine starting the consumer should yield:
% Reached end of topic important  at offset 0
Now try typing some text - like
foo and hitting enter in the terminal where you are running the producer. This should yield the following in the terminal where you are running the consumer:
foo % Reached end of topic important  at offset 1
Sometimes you may experience, that even though you have setup up everything according to the book no communication is going through between producers and consumers. This can be because Docker creates an anonymous data container for Kafka. This is not removed and recreated when using –force-recreate with docker-compose. This may cause problems in some situations, like after upgrading Docker. To completely start over with Docker containers execute the following commands:
docker-compose -f docker-compose.yml -f docker-compose.opentele3.dev.yml stop docker-compose -f docker-compose.yml -f docker-compose.opentele3.dev.yml rm docker-compose -f docker-compose.yml -f docker-compose.opentele3.dev.yml up -d