https://gist.github.com/diegopacheco/2c36ba8bca1c3a2e44c5f422357a060e
1. bin/nodetool status (Copy the IP_ADDRESS of the node down(DN) that you want to replace.
2. Edit conf/cassandra.yaml change rpc_address and listen_address(to new node ip .i.e: $(hostname -i)
3. (OPTIONAL - If you DONT have DNS) Assuming you ARE using DNS for seeds there is no need to change the seeds ips(otherwise you need change the ip there)
4. vim conf/cassandra-env.sh -> JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=IP_DEAD_NODE_REPLACE_HERE_STEP_1"
5. Start cassandra with bin/cassandra and WAIT
6. (IF you are using DNS) DNS should be the last thing to be update. After cassandra-env.sh and replace process finish. So do it now.... and them wait 60s to have dns updated.
7. STOP cassandra remove line conf/cassandra-env.sh -> JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=IP_DEAD_NODE_REPLACE_HERE_STEP_1"
8. START CASSANDRA bin/cassandra
9. DONE1. Never use nodetool removenode - this is only for decomission / will shufle data over network
2. When you start cass on replace node will take sometime.
3. NOTE that you cant run nodetool repair with deadnodes (cass 4 will allow that)
4. After running this process when running nodetool status you should see all nodes in UN and ZERO nodes with DN.
5. Assuming you have 3 node cluster and replication factory 3. Owns (effective) should be 100% for all nodes on nodetool status
6. The new node need come DOWN - if he started before the process you need stop it and delete whole data/* dir.
7. Update the Seed nodes as last thing(consider a seed node died) otherwise replace dont work. If is not a seed node is fine you dont need to touch it. If you are using DNS update dns as last thing. Dont update DNS before replace process.