geo-no-gzk.md, this is copy from sijie's gist

geo-no-gzk.md

This doc demonstrates how to do geo-replication across multiple pulsar clusters without a global configuration store (zookeeper).

This demo is using docker-compose to start 3 pulsar clusters. Each pulsar cluster has 1 zk, 1 bk, and 1 broker. The docker compose file can be found at https://gist.github.com/jiazhai/72aef9bfbfdd3fe421084c2973247a64

The information of all the three clusters is listed in the following table:

	zk	configuration store	broker
beijing	zk-beijing	zk-beijing	broker-beijing
shanghai	zk-shanghai	zk-shanghai	broker-shanghai
guangzhou	zk-guangzhou	zk-guangzhou	broker-guangzhou

0. prepare the cluster

Prepare the docker image

This demon is using pulsar's test image apachepulsar/pulsar-test-latest-version. You can build the docker image locally or pull the image from docker hub.

1. build it from source code:

mvn install -DskipTests -Pdocker   

2. or use docker pull

docker pull apachepulsar/pulsar-test-latest-version 

start all nodes：

Download the docker compose file from https://gist.github.com/sijie/63737459112471a82957ae20bd78adb5.

In the directory that contains the downloaded docker compose file, run following command to start 3 clusters.

docker-compose up

Since it will start 9 docker containers locally, please make sure you allocate enough memory for your docker daemon.

1. Start additional containers as client for each cluster.

Start a client container for cluster beijing

Open a new terminal:

a. start a docker instance

docker run --name client-beijing  -it --rm  --network cluster_pulsar apachepulsar/pulsar-test-latest-version:latest /bin/bash

b. in this docker instance run this command to replace the broker address in client.conf

$ sed -i "s/localhost/broker-beijing/g" conf/client.conf

Start a client container for cluster shanghai

Open a new terminal:

a. start a docker instance

docker run --name client-shanghai  -it --rm  --network cluster_pulsar apachepulsar/pulsar-test-latest-version:latest /bin/bash

b. in this docker instance run this command to replace the broker address in client.conf

$ sed -i "s/localhost/broker-shanghai/g" conf/client.conf

Start a client container for cluster guangzhou

Open a new terminal:

a. start a docker instance

docker run --name client-guangzhou  -it --rm  --network cluster_pulsar apachepulsar/pulsar-test-latest-version:latest /bin/bash

b. in this docker instance run this command to replace the broker address in client.conf

$ sed -i "s/localhost/broker-guangzhou/g" conf/client.conf

2. Create clusters

NOTE: this is the extra step required for settinig up geo-replication when there is no global configuration store.

Since there is no global configuration store for all three clusters, they don't know each other at this moment. You need to create clusters on each cluster to tell a cluster how it can access the other two clusters.

a. Create shanghai and guangzhou clusters on beijing cluster

On client-beijing container, run following commands to create cluster shanghai and guangzhou.

bin/pulsar-admin clusters create --url http://broker-shanghai:8080 --broker-url pulsar://broker-shanghai:6650 shanghai
bin/pulsar-admin clusters create --url http://broker-guangzhou:8080 --broker-url pulsar://broker-guangzhou:6650 guangzhou

These commands basically tells the brokers in beijing cluster how they can access cluster shanghai and guangzhou.

b. Create beijing and guangzhou clusters on shanghai cluster

On client-shanghai container, run following commands to create cluster beijing and guangzhou.

bin/pulsar-admin clusters create --url http://broker-beijing:8080 --broker-url pulsar://broker-beijing:6650 beijing
bin/pulsar-admin clusters create --url http://broker-guangzhou:8080 --broker-url pulsar://broker-guangzhou:6650 guangzhou

c. Create beijing and shanghai clusters on guangzhou cluster

On client-guangzhou container, run following commands to create cluster beijing and shanghai.

bin/pulsar-admin clusters create --url http://broker-beijing:8080 --broker-url pulsar://broker-beijing:6650 beijing
bin/pulsar-admin clusters create --url http://broker-shanghai:8080 --broker-url pulsar://broker-shanghai:6650 shanghai

After executing the above commands, each cluster knows how to connect to the other two clusters now. Let's move to create tenants and namespaces.

2. Create tenants and namespace

Since we don't setup a global configuration store for three clusters, these three clusters don't share tenants and namespace policies. So you have to create the tenants and namespaces on all three clusters.

on the client containers (client-beijing / client-shanghai / client-guangzhou) run following commands to create a tenant and a namespace.

bin/pulsar-admin tenants create my-tenant  --allowed-clusters beijing,shanghai,guangzhou
bin/pulsar-admin namespaces create my-tenant/my-namespace --clusters beijing,shanghai,guangzhou

3. verify messages send and receive.

a. in both client-shanghai and client-guangzhou, create a subscription for the test topic, they will wait receive messages from client-beijing

in client-shanghai

bin/pulsar-client consume -s "sub-shanghai" my-tenant/my-namespace/t1 -n 0

in client-guangzhou

bin/pulsar-client consume -s "sub-guangzhou" my-tenant/my-namespace/t1 -n 0

b. in client-beijing, produce message to the topic

bin/pulsar-client produce  my-tenant/my-namespace/t1  --messages "hello-from-beijing-1" -n 10

You will see both client-shanghai and client-guangzhou will receive all the messages produced by client-beijing.