Elasticsearch High Availability Production Cluster Setup
This article provides steps to setup your own Elasticsearch Cluster in Single and Multiple server Environment that is accessible from external sources using Using URL or IP address in Production Configuration where failover scenario is covered in case of single master failure.
I wanted to setup my own ELK Cluster where I can monitor System, Apache, JMX, APM and Logs from different applications.
Elasticsearch
is one the widely used document databases and is also part of ELK stack which
is used for Log processing.
Elasticsearch can be operated in different modes, below are few.
- Master node
- Data node
- Coordinating node
I faced
many issues while trying to setup a Production Cluster which could be accessed from other
server for which I have to bind it to network interface. On doing to I have to
assign Physical IP address to my Elasticsearch as the localhost is not
accessible from outside and it makes it Production Configuration. To use
production mode Bootstrap check which caused the issues.
Below steps would help every one who are trying to setup their own cluster.
Current Configuration is done using Elasticseach 7.4.0 version.
Download Elasticsearch 7.4.0
To Setup a Production cluster we require minimum of two or more Elasticsearch master instances configured in Production mode(Configured with Internal\External IP address). More number of instance of master are required to avert the situation of breakdown due to failure of a master node in cluster.
Note: If there are less than two instances of master nodes running Elasticsearch will fail to elect master node as it would not meet its minimum requirements.
Enough of the talk, let dive into the configuration of High Availability Production Cluster Configuration with Basic\Opensource License.
Under "config" folder you will find "elasticsearch.yml" which is where configuration is done to create a Production cluster.
1. Cluster Name: This attribute should be same for all the nodes that would be configured to join the same cluster.
cluster.name: prod
2. Node name: This should be unique for all the nodes in the cluster. Node name could also Server host name that could be resolved through DNS. Name of node should be changed as per the mode it is configured to.
node.name: master-node-1
3. Network Host: This is the IP address or Server Domain name with DNS resolution
network.host: 192.168.0.14
4. HTTP Port: This is port on which Elastic search will be using for http traffic, this should be unique for the instances running on the server. Default port 9200
http.port: 9202
5. Transport Port: This is port where Elasticsearch instances in cluster communicate between each other, this port should be unique per instance running on the server. Default port 9300
transport.port: 9302
6. Master Discovery Configuration: To enable Elasticsearch for discovering master eligible nodes in the cluster below configuration is to be made. (IPAddress:transportport). In the below i have 3 master eligible nodes on the same server.
discovery.seed_hosts: ["192.168.0.14:9300", "192.168.0.14:9301", "192.168.0.14:9302"]
cluster.initial_master_nodes: ["192.168.0.14:9300", "192.168.0.14:9301", "192.168.0.14:9302"]
7. Create Dedicated Eligible Nodes: Each Elasticsearch node can perform multiple functions, but it is better to dedicated nodes for each. Below are few important nodes are required.
- Dedicated Master Node
node.master: true
node.data: false
node.ingest: false
node.voting_only: false
node.ml: false
xpack.ml.enabled: false
cluster.remote.connect: false
- Dedicated Data Node
node.master: false
node.data: true
node.ingest: false
node.voting_only: false
node.ml: false
xpack.ml.enabled: false
cluster.remote.connect: false
- Dedicated Coordinating Node: Coordinating nodes act as smart load balancer all the traffic to Custer should be routed through these node which could be Put or Get
node.master: false
node.data: false
node.ingest: false
node.voting_only: false
node.ml: false
xpack.ml.enabled: false
cluster.remote.connect: false
8. Communication between nodes in Cluster: To enable communication between nodes in Cluster we need add the below configuration, port in configuration should be http.port.
xpack.monitoring.exporters.my_remote:
type: http
host: ["192.168.0.14:9200", "192.168.0.14:9201", "192.168.0.14:9202"]
Below snapshot shows all three master nodes (node1, node2, node3), data nodes (data1, data2), coordinating node (client1) in cluster
As the one of the master is down other master node got elected as master.
Once the master node is restarted it rejoined the cluster and could be elected as master once other master node is down
All the configuration files are available at GitHub.