A Guide to Deploy Elasticsearch Cluster on Google Kubernetes Engine

This is a simple guide that helps you to deploy Elasticsearch cluster on Google Kubernetes Engine. This guide should be relevant on any Kubernetes cluster with a simple tweak on the persistent volume part.

The overall step-by-step is like the following:

1. Enable the persistent volume via Storage Classes.
2. Enable the Elasticsearch node discovery via Headless Service.
3. Deploy the Elasticsearch Cluster via Stateful Sets.

1. Persistent Volume

The first step is to create a Storage Class on your cluster. Create new file called storage.yaml with the following content:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ssd
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
  zone: asia-southeast1-a

Note that the Storage Class provisioner that we use is GCE ( kubernetes.io/gce-pd ) with the following parameters:

1. type: pd-ssd to enable the SSD as persistent disk. You may update the parameters type with type: pd-standard to enable the standard disk as persistent disk
2. zone: asia-southeast1-a for the compute zone. You may update this to your compute zone. You can also use zones parameter for multiple zones like the following: zones: asia-southeast1-a, asia-southeast1-b .

Then enable the storage class using the following command:

kubectl apply -f storage.yaml

You should be able to see the Storage Class on the dashboard:

If you deploy the Elasticsearch Cluster on other than Google Kubernetes Engine, please update the provisioner and the parameters. You can see the available provisioners and parameters in here. The rest of this guide will applicable to any Kubernetes Cluster.

2. Elasticsearch Node Discovery

The second step is to enable elasticsearch node discovery via headless service. Create new file called service.yaml with the following content:

apiVersion: v1
kind: Service
metadata:
  name: es
  labels:
    service: elasticsearch
spec:
  clusterIP: None
  ports:
  - port: 9200
    name: serving
  - port: 9300
    name: node-to-node
  selector:
    service: elasticsearch

Then enable the headless service using the following command:

kubectl apply -f service.yaml

You should be able to see the service on the dashboard:

Now, every pod that have label service: elasticsearch should be accessible via $PODNAME.es.default.cluster.local inside a kubernetes cluster. This will helps our elasticsearch nodes to discover each other and form a cluster.

3. Elasticsearch Cluster

The last step is to deploy the elasticsearch cluster using the StatefulSet. Create new file called elasticsearch.yaml with the following content:

to deploy the Elasticsearch cluster, run the following command:

kubectl apply -f elasticsearch.yaml

It may takes time before the cluster is ready, it depends on the size of the cluster. You can see if the cluster is ready or not by accessing the workloads dashboard.

If your cluster is ready, you can check if cluster is created or not by accessing one of the elasticsearch node via port-forward:

kubectl port-forward elasticsearch-0 9200:9200

this will forward all request to http://localhost:9200 to the elasticsearch-0 node. Then:

curl http://localhost:9200/_cluster/state?pretty

It should show you that the cluster is formed by 5 elasticsearch nodes.

That’s it! now you can enjoy my favorite gif: