Following on from my previous post on setting up Rancher/K8S on RancherOS in a VM in Windows for local development, a common task will be setting up container services within the cluster but then accessing those services from your local Windows machine (e.g. while developing in Visual Studio). In a lot of cases this is probably straightforward, either exposing ports directly using a service or using Ingress to route host headers to the correct internal service. However in the case of Kafka it's a bit more complex due to the way in which the brokers address themselves when the initial connection is received and the broker list is sent back. In a nutshell, the default Kafka setup from the Catalog Apps in Rancher binds the brokers to their POD IP, when the broker list is sent to Windows it cannot address these IPs (unless you want to set up some kind of natting). After some Googling and help from the following posts: I came up with the following instructions: STEP 1 (install Kafka in cluster): Install Kafka from the Rancher catalogue
  1. your-dev-cluster > default > Catalog Apps > Launch
  2. find and select "Kafka"
  3. switch off the "Topics UI Layer 7 Loadbalancer" (near the bottom) - don't need it in dev.
  4. click "Launch"
  5. .. Wait until all the kafka services are running ..
  6. You can now verify that the Landoop UI is running and seeing brokers by visiting the endpoint is has produced, e.g. http://rancherdev.yourdomain:30188 <-- random port, check what it says!!
Kafka is now available in the cluster, but not from Windows. Continue with step 2 --> STEP 2 (expose Kafka externally): Change the Kafka startup command for multiport listening
  1. your-dev-cluster > default > workloads > kafka-kafka
  2. Three dots, click "Edit"
  3. Click "show advanced options"
  4. Under Command > Entrypoint - paste the following:
  5. Click "Upgrade"
Add service discovery for the new ports
  1. your-dev-cluster > default > Service Discovery
  2. Click "View/Edit YAML" on kafka-kafka..
  3. Use the following lines for section "spec > ports" (assuming you have 3 instances of Kafka)
      - name: broker
        port: 9092
        protocol: TCP
        targetPort: 9092
      - name: broker-ext0
        port: 9093
        protocol: TCP
        targetPort: 9093
      - name: broker-ext1
        port: 9094
        protocol: TCP
        targetPort: 9094
      - name: broker-ext2
        port: 9095
        protocol: TCP
        targetPort: 9095
Configure nginx to use TCP ConfigMap
  1. your-dev-cluster > system > workloads > nginx-ingress-controller
  2. Three dots > edit
  3. Environment variables:
  4. "Add from Source" > "Config Map" > "tcp-services"
  5. Click "Upgrade"
Expose the port using Ingress TCP ConfigMap
  1. your-dev-cluster > system > resources > config maps > ns: ingress-nginx > tcp-services
  2. Three dots, click "Edit"
  3. Add the following entries:
            - key = 9093
            - value = kafka/kafka-kafka:9093
            - key = 9094
            - value = kafka/kafka-kafka:9094
            - key = 9095
            - value = kafka/kafka-kafka:9095
Reboot the kafka services
  1. your-dev-cluster > default > workloads > tick all and click 'redeploy'
Now from Windows try telnet to 9093/9094/9095 or even better from WSL bash, install kafkacat and run: kafkacat -b -L