ActiveMQ on Zookeeper Setup Guide

guide-amq_zk.md

Image grabbed from ElasticCloudApps.com

Ingredients

• 3 Linux machines with Installed JDK 1.7+
  • Default setup is 3 Replicas
• Apache ActiveMQ 5.13.4 (stable)
  • Extract to /opt/activemq/apache-activemq-5.13.4 (We call this ACTIVEMQ_HOME from now on)
• Apache Zookeeper 3.4.8 (stable)
  • Extract to /opt/zookeeper/zookeeper-3.4.8 (We call this ZOOKEEPER_HOME from now on)

Setup Hostnames

• Update the value of the HOSTNAME field from /etc/sysconfig/network. Name must be unique in an ensemble.
• Add hostnames of the brokers in the ensemble at /etc/hosts
• Remember to restart each machine after updating hostnames and hosts file.

Setup Zookeeper and ActiveMQ for each Machine

• ⚠️ Config files should be the same for each machine.
• Make sure that the zookeeper jar file versions are similar in the server (ZOOKEEPER_HOME) and client (ACTIVEMQ_HOME/lib/optional)
• It is most likey that the jar file in ACTIVEMQ_HOME has a lower version number than what is in ZOOKEEPER_HOME.

cd ACTIVEMQ_HOME/lib/optional
rm zookeeper-3.4.6.jar
ln -s ZOOKEEPER_HOME/zookeeper-3.4.8.jar

Setup Zookeeper

• Go to ZOOKEEPER_HOME
• Create file conf/zoo.cfg with following settings

tickTime=2000                # REQUIRED - This will be the basis of timing events (milliseconds)
initLimit=5                  # REQUIRED - Amount of ticks in initialization
syncLimit=2                  # REQUIRED - Amount of ticks to determine if node is still connected
dataDir=/var/lib/zookeeper   # REQUIRED - Where to store zookeeper sync data
clientPort=2181              # REQUIRED - Port which clients connect to
server.1=hostname1:2888:3888 # or "0.0.0.0:2888:3888" if server.1 is the current node
server.2=hostname2:2888:3888 # or "0.0.0.0:2888:3888" if server.2 is the current node
# and so on...
server.N=hostname3:2888:3888 # or "0.0.0.0:2888:3888" if server.N is the current node

Where hostname{1,2,3} are the hostnames declared in your /etc/hosts

• Save the ID of this node to myid file in your dataDir.
  • Example: In server.1, the content of myid is 1, 2 for server.2, 3 for server.3, and so on...

Setup ActiveMQ

• Go to ZOOKEEPER_HOME
• Replace persistenceAdapter from kahaDB to replicatedLevelDB

<broker ... brokerName="broker" ...>
	...
	<persistenceAdapter>
		<replicatedLevelDB
			directory="${activemq.data}/leveldb"
			replicas="3"
			bind="tcp://0.0.0.0:0"
			zkAddress="hostname1:2181,hostname2:2181,hostname3:2181"
			hostname="hostnameX"
		/>
	</persistenceAdapter>
	...
</broker>

Where:

• hostname{1,2,3} are the hostnames declared in your /etc/hosts
• hostnameX in 'hostname' is the hostname declared in your /etc/sysconfig/network
• All the broker nodes that are part of the same replication set should have matching brokerName XML attributes.
• The the port number used (2181) is the clientPort from zookeeper's configuration file.

Start Zookeeper and ActiveMQ

sudo ZOOKEEPER_HOME/bin/zkServer.sh start
sudo ACTIVEMQ_HOME/bin/activemq start

Connecting to HA ActiveMQ

• Given the setup above are performed without any errors, you can configure the ActiveMQ client to connect to this broker URL:

failover:(tcp://hostname1:61616,tcp://hostname2:61616,tcp://hostname3:61616)

Sample Zookeeper and ActiveMQ Configuration

https://gist.github.com/kana0011/3ad1c960b9aa24b8af8d7b623c4cb2b3

Further Readings

• https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_A-MQ/6.1/html/Configuring_Broker_Persistence/RepLevelDBConfig.html
• http://activemq.apache.org/clustering.html
• http://activemq.apache.org/replicated-leveldb-store.html
• http://elasticcloudapps.com/page0/files/c1f6bea32e025aa68542a95f9d664ea9-12.html
• http://stackoverflow.com/a/30993130/3091701
• http://stackoverflow.com/a/26710215/3091701
• http://zookeeper.apache.org/doc/r3.4.8//zookeeperStarted.html

Thanks a lot for that great guide!
I was able to successfully setup an ActiveMQ/Zookeeper cluster following your guide without significant problems.

The only thing that I would like to propose you as a suggestion is adding to the guide information on the binding address in the activemq.xml configuration file. In particular, the difference between tcp://0.0.0.0:0 and tcp://0.0.0.0:SOME_PORT_HERE

As I found out the value tcp://0.0.0.0:0 means that the Zookeeper could use a dynamically selected port, which might differ from run to run. With that setup, the ActiveMQ log was containing records like INFO | Master started: tcp://host1:37649.
In my case with ActiveMQ/Zookeeper cluster setup on 2 CentOS Linux machines that was an issue since it was needed to open corresponding port in the firewall. But it was hard to do that because of that dynamic binding port selection.
So I changed the binding value from tcp://0.0.0.0:0 to tcp://0.0.0.0:61619, opened port 61619 in the firewall and it started working successfully.