Comparing 2 approaches of managing cloud resources for applications: Terraform vs Kubernetes

Overview

Terraform

Terraform is an open-source tool for managing infrastructure as code. It provides HCL language and CLI tools to provision and manage infrastructure resources. Terraform also has many cloud and API providers such as AWS, GCP, Azure, Kubernetes, Docker, etc. You can see a full list here. To track changes of resources and discover what will be changed if you apply your code, Terraform stores information about managed resources in state. State is a JSON representation of managed resources, and information about them. With Terraform you can store your state in different formats such as local file, AWS S3, GCP Storage Bucket, Azure Blobstorage, etc.

Pros:

• Cloud agnostic. Flexibility to choose which cloud provider to work with.
• Extensible. To allow Terraform to work with your custom API, you can create a custom provider and write a logic on how terraform should interact with your resources.
• Multi-platform. You can run Terraform CLI tool in Linux, Mac OS, FreeBSD, OpenBSD and Solaris platforms.
• Predictable. Provides a way to check infrastructure changes before applying the changes.
• Reusable. Provides a way to publish and reuse your code via modules. One module can have dependencies on another module.
• Has dependency graph. Allows you to specify which resources are dependent on each other and which are not, and thus can be executed in parallel.
• Big community. At the moment of writing Terraform has 31k stars on Github.
• Agentless. You don’t need to run any agent or anything.

Cons:

• No rollbacks. There is no auto rollback feature.
• New language. You need to learn a new language syntax.
• Implicit dependencies don’t work well. Sometimes errors occur when you haven’t specified explicit dependencies.

Kubernetes / Helm

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It provides an API that allows you to create various resources that can be configured via the YAML language. Configurations of resources are stored in etcd storage. You can use default resources to manage containers and communication between them, or you can define your own custom resource definitions and manage them via operators and controllers.

To apply changes in Kubernetes you can use raw manifest files, in which you can specify all necessary properties and deploy them into the Kubernetes cluster. The problem is when you want to reuse and version these resources in the manifest. This is where Helm can help you. Helm is a simple template tool for Kubernetes manifests which has an ability to package your configuration for specific components into artifacts, called Charts. With Charts you can set a version for the Chart, store it in a repository called Chartmuseum, and reuse Charts for deploying a component with values. Simply, it provides templating and package management for your Kubernetes manifests with some additional features. To track managed resources it uses the state approach which is used by Terraform, but for each deployment it creates another state which is also called Release. Having one Release for each deployment allows Helm to do rollbacks in case something goes wrong.

Pros:

• Templating. It allows parameterization of raw manifest files.
• Reusable. Provides package management functionality via Chartmuseum and packaging helm charts. One helm chart can be dependent on another.
• Provides stages of deployment. Has a lifecycle hooks feature that allows you to specify what resources should be created before or after installation/upgrade.
• Extensible. CLI tool is extensible via plugins and you can support your custom resources via extending Kuberetes API through CRDs and operators.
• Has rollbacks. Provides a way to rollback your application to the previous state in case a problem occurs.
• Agentless. You don’t need to run any agent or anything starting from v3.
• Predictable. Provides a way to check what Helm will do when you deploy changes.
• Big community. At the moment of writing Kubernetes has 21.1k stars on Github.

Cons:

• Weak tracking of actual state. Tracks actual state of resources through last deployed release instead of API calls.
• No dependency management. Resources can be installed only all at once. There is no dependency graph. So you can’t specify an order for deployed resources.

Describing environment

Our application environment will consists of these components:

• Wordpress application.
• AWS RDS.

Hands on

To create Kubernetes cluster (EKS) in both cases we will use terraform. You can find the Source code here: …

Terraform approach

First, let’s configure our input variables in a variable.tf file. In our case we need to set region for AWS and EKS remote state to configure Kubernetes and Helm providers:

variable.tf

    variable "region" {
    type    = string
    default = "eu-central-1"
    }


    variable "eks_remote_state_bucket" {
    type = string
    }


    variable "eks_remote_state_key" {
    type = string
    }
                    

Then let’s create a vars folder in which we will set variables and backend configuration:

vars/default.tfvars

    eks_remote_state_bucket = "artcher-infrastructure"
    eks_remote_state_key = "eks/terraform.tfstate"
                    

vars/default.tfbackend

    bucket = "artcher-infrastructure"
    key    = "application/terraform.tfstate"
    region = "eu-central-1"
                    

In the next step we need to define providers in main.tf and data resources for remote state in data.tf:

data.tf

    data "terraform_remote_state" "eks" {
        backend = "s3"

        config = {
            bucket = var.eks_remote_state_bucket
            key    = var.eks_remote_state_key
            region = var.region
            }
    }

    data "aws_eks_cluster" "cluster" {
        name = data.terraform_remote_state.eks.outputs.cluster_id
    }

    data "aws_eks_cluster_auth" "cluster" {
        name = data.terraform_remote_state.eks.outputs.cluster_id
    }

                    

main.tf

    terraform {
        required_providers {
            aws = {
            source  = "hashicorp/aws"
            version = "~> 3.0"
            }
            kubernetes = {
            source = "hashicorp/kubernetes"
            version = "2.8.0"
            }
            helm = {
            source = "hashicorp/helm"
            version = "2.4.1"
            }
        }


        backend "s3" {
        }
    }

    provider "aws" {
        region = var.region
    }

    provider "kubernetes" {
        host = data.aws_eks_cluster.cluster.endpoint
        token = data.aws_eks_cluster_auth.cluster.token
        insecure = true
    }

    provider "helm" {
        kubernetes {
            host = data.aws_eks_cluster.cluster.endpoint
            token = data.aws_eks_cluster_auth.cluster.token
            insecure = true
        }
    }

    resource "kubernetes_namespace" "main" {
        metadata {
            name = "dev"
        }
    }
                    

Let’s define application namespace in main.tf file:

main.tf

    resource "kubernetes_namespace" "main" {
        metadata {
            name = "dev"
        }
    }
                    

Next step is defining an RDS instance for our Wordpress installation in db.tf file. Good practice is to allow Terraform to generate a password for future usage, so we will use random_password resource to generate password for our database. Also, because VPC creation is usually done by Terraform, we can simply refer to VPC id, VPC cidr block and subnet ids via terraform_remote_state resource.

db.tf

    locals {
        db_username = "wordpress"
        db_password = random_password.db_password.result
        db_name = "wordpress"
    }


    resource "aws_db_subnet_group" "main" {
        name       = "wordpress"
        subnet_ids = data.terraform_remote_state.eks.outputs.vpc_subnet_ids
    }


    resource "aws_security_group" "allow_mysql" {
        name        = "allow_mysql"
        description = "Allow MySQL inbound traffic"
        vpc_id      = data.terraform_remote_state.eks.outputs.vpc_id


        ingress {
            description      = "MYSQL from VPC"
            from_port        = 3306
            to_port          = 3306
            protocol         = "tcp"
            cidr_blocks      = [data.terraform_remote_state.eks.outputs.vpc_cidr_block]
        }


        egress {
            from_port        = 0
            to_port          = 0
            protocol         = "-1"
            cidr_blocks      = ["0.0.0.0/0"]
            ipv6_cidr_blocks = ["::/0"]
        }
    }


    resource "aws_db_instance" "main" {
        allocated_storage    = 10
        engine               = "mysql"
        engine_version       = "5.7"
        instance_class       = "db.t3.micro"
        name                 = local.db_name
        username             = local.db_username
        password             = local.db_password
        skip_final_snapshot  = true
        db_subnet_group_name = aws_db_subnet_group.main.name
        vpc_security_group_ids = [aws_security_group.allow_mysql.id]
    }


    resource "random_password" "db_password" {
        length           = 16
        special          = false
    }
                    

Deployment AWS Elasticache via Terraform is straightforward. Let’s define it in cache.tf file:

cache.tf

    resource "aws_elasticache_cluster" "main" {
        cluster_id           = "wordpress"
        engine               = "memcached"
        node_type            = "cache.t3.micro"
        num_cache_nodes      = 1
        parameter_group_name = "default.memcached1.6"
        port                 = 11211
        subnet_group_name    = aws_elasticache_subnet_group.main.name
    }

    resource "aws_elasticache_subnet_group" "main" {
        name       = "wordpress"
        subnet_ids = data.terraform_remote_state.eks.outputs.vpc_subnet_ids
    }
                    

And finally, let’s define Wordpress deployment in wordpress.tf file. Address of the site will be wordpress.<yourdomain>. All parameters that depend on other Terraform resources we can take from properties of these resources. Wordpress admin password will be generated via Terraform as well.

wordpress.tf

    locals {
        wordpress_values = <<EOF
        wordpressUsername: "admin"
        wordpressPassword: "${random_password.wordpress_password.result}"
        ingress:
        enabled: true
        hostname: wordpress.${data.terraform_remote_state.eks.outputs.dns_domain}
        annotations:
            kubernetes.io/ingress.class: "nginx"
        service:
        type: ClusterIP
        mariadb:
        enabled: false
        externalDatabase:
        host: ${aws_db_instance.main.address}
        database: ${aws_db_instance.main.name}
        user: ${local.db_username}
        password: ${local.db_password}
        EOF
    }


    resource "random_password" "wordpress_password" {
        length           = 16
        special          = true
        override_special = "_%@"
    }


    resource "helm_release" "wordpress" {
        name       = "wordpress"
        namespace  = kubernetes_namespace.main.metadata[0].name
        chart      = "wordpress"
        repository = "https://charts.bitnami.com/bitnami"


        values = [
            local.wordpress_values
        ]


        wait = true
    }
                    

Let’s define outputs:

outputs.tf

    output "wordpress_password" {
        value = random_password.wordpress_password.result
    }
                    

Next step will be initialization of terraform script and applying changes:

    $ terraform init -backend-config ./vars/default.tfbackend
    $ terraform apply -var-file ./vars/default.tfvars
    > yes
                    

Wait until script is complete and find a password of a wordpress in outputs array of this command:

    $ terraform show -json
                    

Check result via browser.

Kubernetes / Helm approach

If we want to use Kubernetes to manage the database of Wordpress, we need to install AWS RDS controller in our EKS cluster. Let’s download the chart of rds-controller and put it in infra layer of terraform EKS provisioning:

    $ helm pull oci://public.ecr.aws/aws-controllers-k8s/rds-chart --version v0.0.17
    $ tar -xf rds-chart-v0.0.17.tgz
    $ mv rds-chart terraform/src/infra/charts/rds-chart
                    

First, we need to add IAM permissions to add permissions for cluster to manage RDS instances. For educational purposes, let’s simply modify and deploy policy in terraform/src/eks/iam.tf:

    resource "aws_iam_policy" "nodes_dns_policy" {
        name = "additional.nodes.eks"

        policy = <<EOF
        {
        "Version": "2012-10-17",
        "Statement": [
        {
            "Effect": "Allow",
            "Action": [
            "route53:ChangeResourceRecordSets"
            ],
            "Resource": [
            "arn:aws:route53:::hostedzone/${aws_route53_zone.primary.zone_id}"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
            "route53:ListHostedZones",
            "route53:ListResourceRecordSets"
            ],
            "Resource": [
            "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "rds:*",
            "Resource": "*"
        }
        ]
        }
        EOF
    }

    resource "aws_iam_role_policy_attachment" "nodes_dns_policy" {
        for_each   = module.eks.eks_managed_node_groups
        policy_arn = aws_iam_policy.nodes_dns_policy.arn
        role       = each.value.iam_role_name
    }
                    

After that let’s add file controllers.tf in infra layer and deploy configuration of rds-chart deployment:

locals {
  rds_values = <<EOF
aws:
  region: eu-central-1
EOF
}

resource "kubernetes_namespace" "controllers" {
  metadata {
    name = "controllers"
  }
}

resource "helm_release" "rds_controller" {
  name       = "rds-controller"
  namespace  = kubernetes_namespace.controllers.metadata[0].name
  chart      = "./charts/rds-chart"
  values     = [local.rds_values]

  wait = true
}
                    

After terraform script is deployed, let’s check that controller is up and running:

    $ aws eks update-kubeconfig –-name eks # Fetch kubeconfig
    $ kubectl get po -n controllers
    NAME                                        READY   STATUS    RESTARTS   AGE
    rds-controller-rds-chart-6965c55d8f-cv596   1/1     Running   0          3h3
                    

When EKS cluster is ready, we can start writing a helm chart for our RDS instance.

    $ helm create db
    $ cd db
    $ find ./templates ! -name '_helpers.tpl' -type f -exec rm -f {} +
    $ echo > values.yaml
                    

We need to define DBInstance resource in templates/dbinstance.yaml:

templates/dbinstance.yaml

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
  name: {{ include "db.fullname" . }}
spec:
  allocatedStorage: {{ .Values.db.allocatedStorage}}
  engine: {{ .Values.db.engine }}
  engineVersion: {{ .Values.db.engineVersion | quote }}
  dbInstanceClass: {{ .Values.db.instanceClass }}
  dbInstanceIdentifier: {{ .Values.db.instanceIdentifier }}
  dbName: {{ .Values.db.name }}
  masterUsername: {{ .Values.db.masterUsername }}
  masterUserPassword:
    name: {{ include "db.fullname" . }}-creds
    key: password
    namespace: {{ .Release.Namespace }}
  dbSubnetGroupName: {{ .Values.dbSubnetGroup.name }}
  vpcSecurityGroupIDs:
    {{- range $.Values.db.vpcSecurityGroupIDs }}
    - {{ . | quote }}
    {{- end }}
                    

Also, we need a DBSubnetGroup resource for DBInstance. It will be in templates/dbsubnetgroup.yaml:

templates/dbsubnetgroup.yaml

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBSubnetGroup
metadata:
  name: {{ include "db.fullname" . }}
spec:
  name: {{ .Values.dbSubnetGroup.name }}
  description: {{ .Values.dbSubnetGroup.description }}
  subnetIDs:
    {{- range $.Values.dbSubnetGroup.subnetIDs }}
    - {{ . | quote }}
    {{- end }}
                    

Then, let’s create a secret for our RDS instance password in templates/secret-db-creds.yaml. We will need to specify this password explicitly, because Helm doesn’t have a way to generate passwords and keep them persistent between upgrades:

templates/secret-db-creds.yaml

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: {{ include "db.fullname" . }}-creds
data:
  password: {{ .Values.db.masterUserPassword | b64enc | quote }}
                    

And finally, we need to create values.yaml file for default values:

values.yaml

db:
  name: ""
  allocatedStorage: 20
  engine: mysql
  engineVersion: "5.7"
  instanceClass: ""
  instanceIdentifier: ""
  masterUsername: ""
  masterUserPassword: ""
  vpcSecurityGroupIDs: []


dbSubnetGroup:
  name: ""
  description: ""
  subnetIDs: []
                    

Unfortunately, we cannot deploy db and wordpress charts simultaneously, because we will know about DBInstance endpoint only after it’s created and ready, we don’t have a way in Helm and Kubernetes to use status properties of existing resources as input for new resources. So we need to deploy the db helm chart first, wait until the AWS RDS instance is created, fetch the endpoint from resource status properties and then use it to deploy the wordpress helm chart. Let’s create values for db chart:

db-values.yaml

db:
  name: wordpress
  instanceClass: db.t3.micro
  instanceIdentifier: wordpress
  masterUsername: wordpress
  masterUserPassword: mMZTJqGJRcfQvM5K
  vpcSecurityGroupIDs:
    - sg-0e9cfcd062b3ca0d5


dbSubnetGroup:
  name: wordpress
  description: "Wordpress Subnet Group"
  subnetIDs:
    - subnet-099d349fc9781289c
    - subnet-08dc04b820f394868
                    

Deploy helm chart:

    $ kubectl create ns dev
    $ helm upgrade --install --namespace dev -f db-values.yaml --wait rds ./charts/db
                    

When deployment is finished you can see in status properties of dbinstance resource values like these:

    ...
    Endpoint:
        Address:                            wordpress.cscrfrhxk6c6.eu-central-1.rds.amazonaws.com
        Hosted Zone ID:                     Z1RLNUO7B9Q6NB
        Port:                               3306
    ...
                    

Copy endpoint address and use it in your values for wordpress chart:

wordpress-values.yaml

externalDatabase:
  database: wordpress
  host: wordpress.cscrfrhxk6c6.eu-central-1.rds.amazonaws.com
  password: mMZTJqGJRcfQvM5K
  user: wordpress
service:
  type: ClusterIP
ingress:
  annotations:
    kubernetes.io/ingress.class: nginx
  enabled: true
  hostname: wordpress.ds-pet.xyz
mariadb:
  enabled: false
wordpressPassword: nHLHm%yQWGRYK2x1
wordpressUsername: admin
                    

Deploy wp chart:

    helm upgrade --install --n amespace dev -f wordpress-values.yaml --wait wordpress bitnami/wordpress --version 13.0.12
                    

When deployment is finished, check result in a browser.

Terraform approach

Pros:

• No dependencies needed except Terraform;
• No need to install additional agents;
• Most cloud resources are supported;
• Provides a way to specify dependency graph and ordering of deployment;
• Supports password generation;
• Can refer to terraform resource properties which is located in another state without explicit specification.

Cons:

• Need to know HCL language;
• Modifications take longer than Helm upgrade;
• Two artifacts for application: terraform module + helm chart.

Kubernetes / Helm approach

Pros:

• Everything is packaged in one artifact: Helm chart;
• Modifications are faster than in Terraform approach.

Cons:

• Need to know Go template;
• Doesn’t support password generation;
• No way to refer on properties of already deployed resources;
• --wait is not working with CRDs;
• There is a separate controller for each service type;
• Some properties may not exist in controller resources;
• Not so many cloud resources are supported. Example in AWS;
• No way to specify dependency graph and ordering of deployment;
• Some of properties needs to be explicitly set if they are related to resources that deployed in Terraform.

Summary

A comparison of these 2 approaches shows us that the Kubernetes / Helm approach can be used in small or medium organizations that require only basic resources, such as EC2, S3, etc. So, it will be easy to provision artifacts with all dependencies that are declared in the same manner. You can use it in projects like eCommerce, news portal, telegram bots, etc. On the other hand, the Terraform approach allows you to provision more resources and allows you to build a much more complex system. It can be used in projects of all sizes. With this approach you can easily cover requirements for Healthcare, Industrial, Big Data, IoT and other areas.

From our perspective, using the Terraform approach to manage cloud resources for applications is more efficient and comfortable than managing cloud resources in Kubernetes. But it’s ok to use the Kubernetes / Helm approach in applications that operate with simple cloud resources such as S3 buckets, EC2 instances.