Setup Multi-Cluster ServiceMesh with Istio on EKS

#Kubernetes #Terraform #AWS #Istio

1. Provisioning
2. Istio Setup with Helm chart
3. Cross network gateway validation

Hey! In this post, We will be exploring a technology called ServiceMesh powered by Istio.

Short Intro on Non-Istio users

Istio is an open source service mesh that layers transparently onto existing distributed applications. Istio’s powerful features provide a uniform and more efficient way to secure, connect, and monitor services. Its powerful control plane brings vital features, including:

Secure service-to-service communication in a cluster with TLS encryption, strong identity-based authentication and authorization

Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic

Fine-grained control of traffic behavior with rich routing rules, retries, failovers, and fault injection

A pluggable policy layer and configuration API supporting access controls, rate limits and quotas

Automatic metrics, logs, and traces for all traffic within a cluster, including cluster ingress and egress

Service Mesh

Most of you are already familiar with Istio. Since, Kubernetes federation is currently not yet available and the latest version is on a Beta version and you want to distribute your traffic across different clusters with production grade deployment.

So, Let’s quickly go through the Step-by-Step procedure to implement Multi Cluster Deployment with Istio.

This tutorial is highly based on AWS and Terraform and also Helm Charts.

1. Provisioning

Required resources

- 2 EKS Clusters
- Security Group / Rule
- S3 Bucket for terraform states
- IAM Roles
- IAM Permissions
- AWS ALB (Application LoadBalancer)

IAM Role

Create your IAM Role for your EKS Cluster.

iam.tf

Security Group

AWS Security Group is required in able for your cluster to communicate.

securitygroup.tf

EKS Clusters

In this section, you will provision the 2 EKS Cluster.

eks.tf

After creating necessary tf configuration. It’s now time to apply it.

First, Create a tf workspace.

terraform workspace new istio-service-mesh

Next, Verify if your tf configuration is smooth.

terraform init
terraform workspace select istio-service-mesh
terraform fmt
terraform validate
terraform plan -out='plan.out'

Then, Apply it.

terraform apply 'plan.out'

It is now provisioning:

Provisioning

After 20 minutes or more, Your cluster’s is ready!

Cluster on AWS Console

2. Istio Setup with Helm chart

It’s now time to install Istio on both clusters.

Required Charts

1. Istio Base helm chart
2. Istiod helm chart
3. Istio ingress gateway helm chart

First, Add the helm istio repository via:

helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

Cluster1

Installing Istio Base helm chart via:

helm upgrade --install istio-base istio/base \
    --create-namespace -n istio-system \
    --version 1.13.2 --wait

Now, Istio based is now installed. Next one is the Istio Control Plane.

Note: You must specify the meshID, clusterName and network to uniquely identify your clusters when installing Istio control Plane.

helm upgrade --install istiod istio/istiod -n istio-system --create-namespace \
    --wait --version 1.13.2 \
    --set global.meshID="cluster1" \
    --set global.multiCluster.clusterName="cluster1" \
    --set global.network="cluster1"

Now, It’s time to expose the cluster with ingress or what so called edge router by install istio ingressgateway. In my case, I prepare to use ALB instead of prepared loadbalancer by Istio 😆.

kubectl create namespace istio-ingress
kubectl label namespace istio-ingress istio-injection=enabled

helm upgrade --install istio-ingressgateway istio/gateway \
     -n istio-ingress --create-namespace \
     --version 1.13.2 --set service.type="NodePort"

Finally, create an ingress resource then associate the ingress to istio-ingressgateway NodePort service.

ingress.yaml

Cluster2

Same steps applied to cluster2. But you must change the meshID, clusterName and network values on Istio Control plane chart.

Installing Istio base chart via:

helm upgrade --install istio-base istio/base \
    --create-namespace -n istio-system \
    --version 1.13.2 --wait

Installing Istio Control Plane:

helm upgrade --install istiod istio/istiod \
    -n istio-system --create-namespace \
    --wait --version 1.13.2 \
    --set global.meshID="cluster2" \
    --set global.multiCluster.clusterName="cluster2" \
    --set global.network="cluster2"

On cluster2, We don’t have to setup additional edge ingressgateway. Since, the connection will be started from cluster1. But, How can we distribute the traffic from cluster1 to cluster2 ?

Answer: By exposing cluster services 💡

On cluster1, Create additional Loadbalancer by installing additional istio-ingressgateway.

helm upgrade --install istio-crossnetworkgateway istio/gateway \
     -n istio-system --create-namespace --version 1.13.2

For cluster2:

helm upgrade --install istio-crossnetworkgateway istio/gateway \
     -n istio-system --create-namespace --version 1.13.2

Exposing services for both cluster. istio-exposeservice.yaml

Now, Services are now exposed. But how does istio identify or discover resources from the other cluster? We need to enable Endpoint Discovery.

On cluster1, I assume that your kubeconfig file is pointed to cluster1 context. This way, We can create istio secret file that can give access on both clusters in able for them to discover resources.

Create an istio secret for cluster2. This command should be done on cluster1 context:

istioctl x create-remote-secret --name=cluster1 > cluster2-secret.yaml

On cluster2, Create an istio secret for cluster1. This command should be done on cluster2 context:

istioctl x create-remote-secret --name=cluster2 > cluster1-secret.yaml

If you view the file, It’s just a kubeconfig context from both cluster contexts enabling API Access. Next, We should apply the secret to both clusters.

Cluster1:

kubectl apply -f cluster1-secret.yaml

Cluster2:

kubectl apply -f cluster2-secret.yaml

Last, But now least. Verify if your clusters has already a trust configuration.

diff \
   <(export KUBECONFIG=$(pwd)/kubeconfig_cluster1.yaml && kubectl -n istio-system get secret cacerts -ojsonpath='{.data.root-cert\.pem}') \
   <(export KUBECONFIG=$(pwd)/kubeconfig_cluster2.yaml && kubectl -n istio-system get secret cacerts -ojsonpath='{.data.root-cert\.pem}')

If there’s no certificate found on both clusters. You can generate a self-signed root CA certificate.

Kindly, visit for more info: https://github.com/redopsbay/Istio-Multi-Cluster/tree/master/istio-tool

Generate Certificates

Istio provides basic security by default in able for the services not being accidentally exposed publicly. Istio will automatically drop client connection if the TLS handshake doesn’t meet the requirements.

Because Istio verifies service-to-service communication by using Trust Configurations.

Creating a root-ca certificate.

cd istio-tool
mkdir -p certs
pushd certs
make -f ../Makefile.selfsigned.mk root-ca

Generate a cluster1 certificate.

make -f ../Makefile.selfsigned.mk cluster1-cacerts

Generate a cluster2 certificate.

make -f ../Makefile.selfsigned.mk cluster2-cacerts

Now, apply both the certificates on both cluster.

For Cluster1:

kubectl create secret generic cacerts -n istio-system \
      --from-file=cluster1/ca-cert.pem \
      --from-file=cluster1/ca-key.pem \
      --from-file=cluster1/root-cert.pem \
      --from-file=cluster1/cert-chain.pem

For Cluster2:

kubectl create secret generic cacerts -n istio-system \
      --from-file=cluster2/ca-cert.pem \
      --from-file=cluster2/ca-key.pem \
      --from-file=cluster2/root-cert.pem \
      --from-file=cluster2/cert-chain.pem

After applying all the necessary steps. The cluster1 and cluster2 should now be able to distribute traffic on both clusters.

3. Cross network gateway validation

After applying the necessary steps. Of course, you need to verify if it’s actually working. I’ve created a basic application called MetaPod that allows you to extract the pod information or metadata through the web. So you can determine if your traffic is actually being forwarded to the 2nd cluster.

MetaPod sample deployment manifest.

For Cluster1, Try to deploy the test deployment.

Note: You must change the hosts values to make it work on your end.

For Cluster2:

After a few seconds, Try to visit the registered gateway on your end. In my case, https://metapod.example.com and it should look like this:

Cluster1

As you can see under the CLUSTER NAME. Your traffic is forwarded to Cluster1. If you constantly refresh your browser page. You’ll notice that your traffic is being forwarded also to Cluster2. See below:

Cluster2

Alright! That’s it. You may encounter a lot of problems during your journey. But it’s worth to try. You can message me directly here or on my twitter account if you need help. I will try my best to help you out to fix it. 😅😅😅

Hope you like it. Cheers!!!🍻🍻 Thanks!!!

References

GitHub - redopsbay/Istio-Multi-Cluster: Istio Multi Cluster Service Mesh

GitHub - redopsbay/MetaPod: Display Pod MetaData when using Kubernetes Multi Cluster Management

Documentation

Plug in CA Certificates

Install Multi-Primary on different networks

Troubleshooting Multicluster