Site Tools


4smicroservices:kafka

Kafka through Docker

This page describes how you setup Apache Kafka to function with 4S Microservices on your local machine. The instructions below have been verified to work on OSX 10.12.1 with Docker 1.12.1 (build: 12133).

Install and run locally

Run Kafka on your local machine by following these steps.

Get the scripts

Get the scripts and docker definitions from 4S bitbucket: https://bitbucket.org/4s/kafka

Network setup

You must setup a docker network by issuing the command:

docker network create <network-name>

This network is shared by all microservices. Network setup must only be carried out once.

Verify that everything went well by issuing the command;

docker network ls

Start kafka services

In order to build a Kafka docker image and start all necessary services (e.g. Zookeeper, kafka-manager and kafka) issue the command:

docker-compose -f docker-compose.yml -f docker-compose.opentele3.dev.yml up -d --force-recreate

For more information about docker-compose, visit the docker website: https://docs.docker.com/compose/reference/overview/

Aliases for kafka and zookeeper

Since Kafka needs to know its own “host name” when it starts up, problems can arise if accessing kafka from various environments (local and global). Therefore, two aliases have been created for kafka and zookeeper in this setup. The aliases are 'kafka' and 'zookeeper', respectively.

Example of Java client
Properties props = new Properties();
props.put("bootstrap.servers", "kafka:9092");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
Kafka on localhost

You need to add kafka to your /etc/hosts file to be able to reach kafka on your local machine. Add the following lines to your /etc/hosts file:

127.0.0.1 kafka
127.0.0.1 zookeeper

Verifying that it works

To verify that your Kafka setup works you may for instance use the command-line utility kafkacat (see link for installation instructions).

Create a Kafka producer:

kafkacat -P -b kafka:9092 -t important

Create a Kafka consumer:

kafkacat -C -b kafka:9092 -t important

If this is the first time you are running this on your machine starting the consumer should yield:

% Reached end of topic important [0] at offset 0

Now try typing some text - like foo and hitting enter in the terminal where you are running the producer. This should yield the following in the terminal where you are running the consumer:

foo
% Reached end of topic important [0] at offset 1

Troubleshooting

No connection between producers and consumers

Sometimes you may experience, that even though you have setup up everything according to the book no communication is going through between producers and consumers. This can be because Docker creates an anonymous data container for Kafka. This is not removed and recreated when using –force-recreate with docker-compose. This may cause problems in some situations, like after upgrading Docker. To completely start over with Docker containers execute the following commands:

docker-compose -f docker-compose.yml -f docker-compose.opentele3.dev.yml stop
docker-compose -f docker-compose.yml -f docker-compose.opentele3.dev.yml rm
docker-compose -f docker-compose.yml -f docker-compose.opentele3.dev.yml up -d
4smicroservices/kafka.txt · Last modified: 2018/12/12 13:27 (external edit)