top of page

Setup Multi-Region Elasticsearch cluster with AWS EC2


In your daily work, you might received the request of setting up multi-region Elasticsearch cluster that serves content globally, the cluster have to meet these requirements:

  • Data Consistency.

  • High Availability.

  • Privacy & Security.

  • Single Endpoint & Easy to use.


Please check below recommended solution.

Figure 1: Implementation Diagram

TL;DR:

  • Setup VPC & VPC Peering

  • Setup AWS EC2 & Elasticsearch Installation

  • Implement ALB for every regions

  • Configure Route53, Health Check and Geo Location



 

Step 1: Setup VPC & VPC Peering

Setup VPC

First of all, we need to setup VPC with non-overlapping CIDR as on diagram:

  • Region us-east-1: 172.16.1.0/24

  • Region eu-central-1: 172.17.1.0/24

  • Region ap-southeast-1: 172.18.1.0/24

CIDR have to be non-overlap due to set up VPC Peering and Routing Table

Initiate VPC Peering between the regions, which make mesh network.

IMPORTANT: The Elasticsearch cluster need mesh connections between nodes due to failover & synchronization seemlessly.

Setup VPC Peering

Follow this instruction from official AWS.

There are 3 VPC Peering Connections we have to setup is:

  • Connection 1: between us-east-1 and eu-central-1

  • Connection 2: between us-east-1 and ap-southeast-1

  • Connection 3: between eu-central-1 and ap-southeast-1

If you implement Elasticsearch cluster on 4 regions, you must create 6 VPC Peering Connections. Such as: 4 regions contain us-east-1, eu-central-1, ap-southeast-1, sa-east-1

  • Connection 1: between us-east-1 and eu-central-1

  • Connection 2: between us-east-1 and ap-southeast-1

  • Connection 3: between eu-central-1 and ap-southeast-1

  • Connection 4: between us-east-1 and sa-east-1

  • Connection 5: between eu-central-1 and sa-east-1

  • Connection 6: between ap-souteast-1 and sa-east-1

The fomular is:

Number of VPC Peering Connection Needed = <number_of_regions> Combinations of 2


After this step, please make sure all VPC Peering work properly 100%.


Step 2: Setup AWS EC2 & Elasticsearch Installation

Setup AWS EC2

Follow this instruction from official AWS.

Create these rules in your security group that allow Elasticsearch services running within EC2 can communicate each others for data transfer & synchronization.

  • Allow from us-east-1 VPC CIDR 172.16.1.0/24 access to port 9200 and 9300 on EC2.

  • Allow from eu-central-1 VPC CIDR 172.17.1.0/24 acess to port 9200 and 9300 on EC2.

  • Allow from ap-southeast-1 VPC CIDR 172.18.1.0/24 acess to port 9200 and 9300 on EC2.

  • Allow from any IP from local subnet of launched EC2 for Application Load Balancer to access to port 9200 on EC2.

Elasticsearch Installation

Before setting up Elasticsearch cluster, we should calculate how many master eligible nodes for cluster, the common fomular is: (N/2)+1

Elasticsearch cluster election process based on quorum-based, so we need +1 to create enough votes due to quick decision in election.


Let see the sample elasticsearch.yml configuration file:


[...]
cluster.name: sample_cluster
network.host: <node_ip>
node.roles: [master, data, voting_only] # it's up to you
[...]
discovery.seed_hosts: ["172.16.1.5:9300", "172.16.1.6:9300", "172.17.1.4:9300", "172.17.1.5:9300", "172.18.1.4:9300", "172.18.1.5:9300"]
cluster.initial_master_nodes: ["172.16.1.5:9300", "172.16.1.6:9300", "172.17.1.4:9300", "172.17.1.5:9300", "172.18.1.4:9300", "172.18.1.5:9300"]


Step 3: Implement ALB for every regions

Follow this instruction from official AWS.


Create AWS ALB as above instruction with target is Elasticsearch nodes for every region, you will get 3 dns names, like this:

  • us-east-1: ALB-US-ES-1056884759.us-east-1.elb.amazonaws.com

  • us-central-1: ALB-EU-ES-1056884759.us-east-1.elb.amazonaws.com

  • ap-southeast-1: ALB-AP-ES-1056884759.us-east-1.elb.amazonaws.com

Step 4: Configure Route53, heath check & geolocation

Configure Route53 records

Configure health check for Route53, follow this instruction with these steps:

  • Create an Cloudwatch alarm that fires when healthy targets of ALB below two.

  • Configure Route53 health check based on Cloudwatch Alarm.

Now configure Route53 for your hosted zone with following configuration:

Now you have 3 endpoints in 3 regions. After having 3 endpoints, configure Route53 geolocation with health check, we recommend as below:

You can modify DNS records anytime you want.




437 views0 comments

Komentar

Dinilai 0 dari 5 bintang.
Belum ada penilaian

Tambahkan penilaian
Stationary photo

Be the first to know

Subscribe to our newsletter to receive news and updates.

Thanks for submitting!

Follow us
bottom of page