Cassandra Cluster Setup

Adding Nodes

If for some reason the cluster needs to be expanded on the same host, this can be accomplished launching new Cassandra container and make it point to the seed container. Make sure to provide a valid seed container so the new node can bootstrap properly.

docker run -d -e CASSANDRA_SEEDS=mwmanager-cassandra -d \
              --link mwmanager-cassandra  \
              brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-mm-7-tech-preview/middleware-manager-datastore:latest

Note

if the cluster needs to run on a separate host, the option -e CASSANDRA_BROADCAST_ADDRESS=<public-ip> must be specified on each node that is launched, where <public-ip> is the IP of the machine on the network, is not possible to combine the approach of having multiple nodes on the same host and nodes on a remote host for one cluster.

Verify the container is launched using docker ps, then verify the status of the node, this can be done running the nodetool command inside the container.

#  docker exec -it mwmanager-cassandra /opt/apache-cassandra/bin/nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.2  490.35 KB  256          50.2%             ead3a0ee-b040-4873-8b62-aa700d02b0c1  rack1
UN  172.17.0.4  312.28 KB  256          49.8%             8e4a73eb-5545-48e1-9464-ef2728f8852e  rack1

Once all nodes appear UP the cleanup process needs to run in each other nodes of the cluster. This operation removes all unnecessary keys that don’t belong to the node itself. The cleanup operation can be done using nodetool inside each container.

docker exec -it <container_id> /opt/apache-cassandra/bin/nodetool cleanup

The cleanup process needs temporary disk space (proportional to the amount of data stored) and is an I/O intensive operation, so it should be postponed to lower usage hours.

This process would be repeated each time we want to add a new node to the cluster.

Remove a node.

For remove a node from the cluster those are the steps:

Select a node for be removed form the cluster, the list of nodes can be obtained running docker ps
Once the node is selected check whether the node is up or down using nodetool status
Then a decommission process needs to run inside the container,

This process will move data to the other nodes, and replicate the appropriate data.

  docker exec -it my_container_id /usr/bin/nodetool decommission

Note	If the node is running on another machine, you need to execute the above command on that machine.

The progress of the process can be monitored using the command:

docker exec -it <container_id> /opt/apache-cassandra/bin/nodetool netstats

Mode: DECOMMISSIONED
Not sending any streams.
Read Repair Statistics:

Once the process finish the container can be safely deleted.

docker stop my_container_id
docker rm my_container_id

docker rm --force my_container_id

Replace a node

If something goes wrong with one node, we want to replace it with a new one. In order to accomplish that, we need to follow the following steps:

Get the status of the cluster using nodetool status

docker exec -it mwmanager-cassandra /opt/apache-cassandra/bin/nodetool status

=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.3  179.25 KiB  256          67.6%             3dc29aa0-5cb7-4352-a45d-72600f87ee48  rack1
DN  172.17.0.2  166.47 KiB  256          63.4%             b73c89e5-e2e1-470e-874b-f926b7243b49  rack1
UN  172.17.0.4  83.71 KiB  256          69.0%             fe1e6949-1fc7-496a-a572-a3416b47d16f  rack1

See if some node is dead, if there is a dead node get the IP address of that node.
Start a new node container using the following command:

docker run -d -e JVM_OPTS="-Dcassandra.replace_address=<node_replaced_ip>" \
           -e CASSANDRA_SEEDS=mwmanager-cassandra\
           -e CASSANDRA_START_RPC=true \
           --link mwmanager-cassandra \
           brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-mm-7-tech-preview/middleware-manager-datastore:latest

Note	If the node is running on another machine, you need to execute the above command on that host and must be specify the option -e CASSANDRA_BROADCAST_ADDRESS=<public-ip>

Where node_replaced_ip is the IP of the node we want to replace. (see step 2)

run nodetool status again to see if the node was replaced.
once the new node finish the bootstrap process the old node can be removed.