Remove Elasticsearch Node

Example assumes the node IP Address is 172.21.1.21 and that you have jq installed. If you do not have jq remove | jq . from the commands.

Expand the number of shards that can be reallocated

Choose settings appropriate for your hardware and network resources. There’s no magic bullet here. You’ll have to do some testing to find out these numbers.

curl -s -H 'Content-Type: application/json' -XPUT localhost:9200/_cluster/settings -d '{
  "transient":{
    "cluster.routing.allocation.node_concurrent_recoveries":20,
    "indices.recovery.max_bytes_per_sec": "200mb",
    "cluster.routing.allocation.cluster_concurrent_rebalance":5
  }
}' | jq .


Result:

{
  "acknowledged": true,
  "persistent": {},
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "cluster_concurrent_rebalance": "5",
          "node_concurrent_recoveries": "20"
        }
      }
    },
    "indices": {
      "recovery": {
        "max_bytes_per_sec": "200mb"
      }
    }
  }
}

Begin shard relocation.

curl -H 'Content-Type: application/json' -XPUT localhost:9200/_cluster/settings -d '{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "172.21.1.21"
  }
}' | jq .

Result:
{
  "acknowledged": true,
  "persistent": {},
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "exclude": {
            "_ip": "172.21.1.21"
          }
        }
      }
    }
  }
}

Monitor relocation.

curl -s localhost:9200/_cat/shards | grep RELOCATING

or to just get the number left to relocate. When the number is zero, the process is finished.

watch -n 60 -d "curl -s localhost:9200/_cat/shards | grep RELOCATING | wc -l"

Stop the elasticsearch process or shutdown node.

systemctl stop elasticsearch.service

or

service elasticsearch stop

Clean-up settings

Setting value to null returns value to default.

curl -H 'Content-Type: application/json' -XPUT localhost:9200/_cluster/settings -d '{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : null,
    "cluster.routing.allocation.node_concurrent_recoveries": null, 
    "indices.recovery.max_bytes_per_sec": null, 
    "cluster.routing.allocation.cluster_concurrent_rebalance": null
  }
}' | jq .