Local Binary

Check Out

Check out the following repositories to your local working directory. I am using networknt in my home directory.

cd ~/networknt
git clone https://github.com/networknt/light-kafka.git
git clone https://github.com/networknt/light-example-4j.git

Confluent Local

Before running the tutorial, we need to have confluent platform started locally. If you are using Linux, you can install confluent locally and start it from the command line.

cd ~/tool/confluent-6.1.0/bin
confluent local services start

You can also start the confluent locally with a docker-compose.

cd ~/networknt/light-kafka
docker-compose up

Once the confluent platform is started, go to http://localhost:9021 to access the control center. Create a topic named test6 without key schema and value schema.

Start Backend

Let’s first start the ReactiveConsumer backend service within the IDE from the following folder. We will start it from the command line later.

~/networknt/light-example-4j/kafka/sidecar-backend

Start Sidecar

Start the Kafka sidecar within the IDE from the following folder.

~/networknt/kafka-sidecar

set the JVM Option to

-Dlight-4j-config-dir=config/local

Binary Value

update the config/local/values.yml to the following.

# producer format configuration
kafka-producer.keyFormat: string
kafka-producer.valueFormat: binary

# consumer format configuration
kafka-consumer.topic: test6
kafka-consumer.keyFormat: string
kafka-consumer.valueFormat: binary

# Service Startup and Shutdown Hooks
service.com.networknt.server.StartupHookProvider:
  - com.networknt.mesh.kafka.ProducerStartupHook
  # - com.networknt.mesh.kafka.ActiveConsumerStartupHook
  # - com.networknt.mesh.kafka.KsqldbConsumerStartupHook
  - com.networknt.mesh.kafka.ReactiveConsumerStartupHook
  # - com.networknt.mesh.kafka.AdminClientStartupHook
service.com.networknt.server.ShutdownHookProvider:
  - com.networknt.mesh.kafka.ProducerShutdownHook
  # - com.networknt.mesh.kafka.ActiveConsumerShutdownHook
  # - com.networknt.mesh.kafka.KsqldbConsumerShutdownHook
  - com.networknt.mesh.kafka.ReactiveConsumerShutdownHook
  # - com.networknt.mesh.kafka.AdminClientShutdownHook

Curl command line.

curl -k --location --request POST 'https://localhost:8443/producers/test6' \
--header 'Content-Type: application/json' \
--data-raw '{"records":[{"key":"alice","value":"YWJj"},{"key":"john","value":"ZGVm"},{"key":"alex","value":"eHl6"}]}'

Binary Key

Update the config/local/values.yml to the following.

# producer format configuration
kafka-producer.keyFormat: binary
kafka-producer.valueFormat: string

# consumer format configuration
kafka-consumer.topic: test6
kafka-consumer.keyFormat: binary
kafka-consumer.valueFormat: string

# Service Startup and Shutdown Hooks
service.com.networknt.server.StartupHookProvider:
  - com.networknt.mesh.kafka.ProducerStartupHook
  # - com.networknt.mesh.kafka.ActiveConsumerStartupHook
  # - com.networknt.mesh.kafka.KsqldbConsumerStartupHook
  - com.networknt.mesh.kafka.ReactiveConsumerStartupHook
  # - com.networknt.mesh.kafka.AdminClientStartupHook
service.com.networknt.server.ShutdownHookProvider:
  - com.networknt.mesh.kafka.ProducerShutdownHook
  # - com.networknt.mesh.kafka.ActiveConsumerShutdownHook
  # - com.networknt.mesh.kafka.KsqldbConsumerShutdownHook
  - com.networknt.mesh.kafka.ReactiveConsumerShutdownHook
  # - com.networknt.mesh.kafka.AdminClientShutdownHook

Command Line

We are going to start both Kafka Sidecar and the sidecar-backend with the command line and run the curl command from the command line to test the sidecar.

Docker Compose

The easy way to get the Kafka-sidecar and backend API started together is the docker-compose-demo.yml in the Kafka-sidecar repo.

Checkout kafka-sidecar and start the confluent platform with docker-compose.

cd ~/networknt
git clone https://github.com/networknt/kafka-sidecar.git
cd ~/networknt/kafka-sidecar
docker-compose up -d

Start the kafka sidecar and backend API with another docker-compose with host network.

cd ~/networknt/kafka-sidecar
docker-compose -f docker-compose-demo.yml up

Create a topic test6 and dead letter topic test6.dlq from the control center and send a test request with curl.

curl -k --location --request POST 'https://localhost:8443/producers/test6' \
--header 'Content-Type: application/json' \
--data-raw '{"records":[{"key":"YWxpY2U=","value":"abc"},{"key":"am9obg==","value":"def"},{"key":"YWxleA==","value":"xyz"}]}'

You should see the logs in the docker-compose-demo console for the new messages.

The following is a video tutorial, and please ignore the DOCKER_HOST_IP set up in the demo as it is not necessary with the host network.

Kubernetes