author avatar
By Gopalakrishnan UnnikrishnanPrincipal Software Engineer

*Views, thoughts, and opinions expressed in this post belong solely to the author, and not necessarily to SemanticBits.

Introduction

DevOps is a practice that unifies the development (Dev) and operations (Ops) of software applications by enabling shorter development cycles and quick and seamless delivery of new versions and features of complex software applications. Applications that adopt a microservices architecture involve many pieces that are independently developed and deployed. It takes a lot of careful coordination for deployment and operation of those independent pieces to make the overall application work for the end user. It becomes even more elaborate when these complex applications are deployed in a cloud environment, as the cloud infrastructure adds additional complexities of resource management. All these complexities make it nearly impossible to manually deploy and manage without errors and omissions that will affect the availability of the application for end users. Thus, it is critical that we automate the deployment and delivery of applications to reduce manual errors and improve the overall availability of the application.

In this article we will look into a case study of how we automated the DevOps of a complex microservices-architecture-based web application deployed on the AWS cloud platform using Terraform, Ansible, and Jenkins. This two part article series will cover building the infrastructure-as-code (IaC) using Terraform and Ansible, as well as continuous integration (CI) and continuous deployment (CD) using Jenkins and SonarQube.

Terraform and Ansible

Terraform is an open-source tool that allows building IaC in a cloud-platform agnostic way. Terraform IaC is written as a JSON-like declarative configuration file that can be version controlled, tested, and executed repeatedly to stand up the infrastructure on multiple deployment tiers or to rebuild the infrastructure from scratch when needed.

Ansible is an open-source tool for deployment automation, configuration management, and orchestration. In this case, we use Ansible for automating, installing, and configuring an AWS server instance with Jenkins.

When these tools work together we can build an application platform from scratch and get it running without any manual steps.

Jenkins

Jenkins is an open-source build tool for implementing continuous integration/delivery (CI/CD). Adding the Jenkins pipeline helps us make the build definitions part of the source code and be version controlled. Jenkins integrated with a Git repository allows for automatic triggering of code builds and deployment on git pushes, branches, and pull-requests. It will also perform multi-branch builds, enabling building and validation of the code before it gets merged to the main release branches.

Infrastructure-as-Code

We considered both Amazon CloudFormation and Terraform for building our AWS IaC. But we decided to go with Terraform for having open framework and being cloud-agnostic. Terraform also allows multi-cloud deployment in a single Terraform script. This allows for mixing together resources from multiple cloud vendors in a single deployment plan to build an application that is more resilient to cloud service outages. In contrast CloudFormation is an Amazon technology and works only with AWS.

Terraform Deployment Configuration

Terraform deployment configuration is authored in a JSON-like text file format called HashiCorp Configuration Language (HCL). Configuration also can be written in plain JSON but HCF format is preferred. HCF configuration files have an extension of .tf.

Deployment configuration files contain one or more providers and resources. Providers are responsible for creation of resources and are targeted for a particular cloud service provider. A configuration can have more that one type of provider to support cross-platform infrastructure. Resources represent a particular resource that exists in the deployment (e.g., an EC2 server, an S3 bucket, etc.).

Modules are a collection of resources that are reusable in a single unit. Modules are used to build parameterized modular components of the deployment. Modules can be reused in the same deployment or across deployments with different parameters to build appropriately configured deployment of similar components. We heavily utilized modules to build a very extensible deployment configuration in this use case.

Terraform variables are used to provide values for parameters of the Terraform deployment configuration and modules, as well as for returning outputs from modules. Variable definitions for each Terraform module and the main module are separate variables.tf. Even though this is not mandatory, this makes it easier to manage the variables for each module instead of sprinkling all over the main Terraform configuration. At the deployment configuration execution, the values of Terraform variables can be passed in as command line arguments to Terraform command or specified in a variable files. Terraform variable files are usually given a .tfvars extension and the .tfvars file are specified on the Terraform execution command line.

Case Study Architecture

Shown below is the architecture of the case study application. The application will be deployed on AWS and utilizes S3, EC2, ECS, ELB, SQS and RDS AWS services. The application utilized microservices architecture.

This case study is a web application that allows users to upload files to a system that goes through review. Upon successful review, the data in the file gets uploaded to a RDBMS. Since the files are uploaded by a large user base, the files are uploaded directly into an isolated S3 bucket, and get scanned for viruses by the scanner service before getting used by the system. SQS is used to communicate between the S3 bucket and the virus scanner service and the rest of the system.

The architecture requires multiple instances of many resources (e.g., it requires three S3 buckets, two SQS queues, three ECS clusters). Except for a few, most attributes of these resources are same (e.g., in a single application tier, the VPC and subnet details for the ECS clusters are the same and only the name of the resources are different). To enable reuse of resource definitions, deployment code for each type of resource is created as a Terraform module.

Terraform configuration files for each module goes into its own subdirectory, and the main Terraform configuration (main.tf) at the root level composes the modules to build the complete infrastructure. Each module may be called multiple times from the main.tf with appropriate values for the input variables to create individual instances of the resource. Directory structure of the whole deployment configuration is given below:

  • main.tf
  • variables.tf
  • \ecr
    • ecr.tf
    • ecr.variables.tf
  • \ecs
    • ecs.tf
    • ecs.variables.tt
    • ecs_instance.tf
    • ecs_nlb.tf
  • \rds
    • sds.tf
    • sds.variables.tf
  • \s3
    • s3.tf
    • s3.variables.tf
  • \sqs
    • sqs.tf
    • sqs.variables.tf
  • \tiers
    • dev.tfvars
    • test.tfvars
    • stage.tfvars
    • prod.tfvars

Module configuration for each module is split into 2 terraform files (.tf). One with <module>.tf defines the actual resource(s) created by the component and another one named <module>.variables.tf defines all the input and output variables of the resource(s) created by the module. Any resource properties that need to be configured per deployment is defined as an input variable, and any attribute of the resource that may be required by the caller are defined as the output variable.

When a module creates a large number of resources like the ECS module in the example below, the configuration file could be split into multiple .tf files. Terraform automatically loads all .tf files in the folder when executing the Terraform commands, so there is no need for explicitly including/linking the multiple .tf files. Terraform also automatically identifies the dependencies between various resources based on the input/out variable dependencies, so we don’t need to specify the resource creation commands in any particular order.

AWS Resource Naming/Tagging Convention

With close to 50 AWS resources overall per deployment tier and 4 deployment tiers—namely dev test, stage, and prod—properly naming all these resources for easy identification and management is an absolute must. For this purpose we came up a naming and tagging convention for every single resource we create, so that they can be uniquely identified without confusion.

We came up with the following naming and tagging conventions for every AWS resource. The name or ID attribute is set to a value in format:

<project_name>-<tiername>-<component>-<resourcetype>, where

<project_name>: Name of the project

<tiername>: Name of the deployment tier, one of dev, test, stage or prod, and developer. We could also use the initials of the developer as an environment name to create temporary test environments for a developer to perform any dev testing during the development of a feature.

<component>: Application component that this resource belongs to or a name that identifies the service the resource provides. Basically, it should be something that identifies the purpose of the resource (e.g., UI, API, scan, Jenkins, Sonarqube). If the component is generic or is used across multiple components, a generic component name can be used (e.g., data).

<resourcetype>: A string representing the type of resource (e.g., server, RDS, bucket, SG, etc.). Use a term that is easily identifiable and use it consistently. The resource type value could also contain multiple keywords, if that helps identify the resource a bit better (e.g., for RDS dbs, we could include the database type in the resource type name, like rdspg, rdsmysql, etc.).

The following tags are set on every resource, so that we can filter and group them by tags and use various AWS resource management tools to manage the resources.

Project: project abbreviation, tf-demo for the case study project

Tier: <tiername> one of dev, test, stage, prod or abbreviation for the developer

Component: Name of the application component. When a resource is not used by a specific component, use a generic component name (e.g., data for a database, which will be used by all other components of the app).

Type: <resourcetype>, Use a meaning abbreviation or type name for the resource type

Name: <project_name>-<envname>-<component>-<resourcetype>

Closer Look at the Module Configuration Files

Let’s take a closer look at one of the module configuration files. I’ve chosen RDS module, as it is one of easiest and straightforward configurations.

Here is a module definition file for an RDS module and the variable definition file that declares all the variables needed by the RDS module. Both the RDS module definition (rds.tf) and the variable definition files (rds.variables.tf) are in the rds subfolder.

rds/rds.tf:

resource "aws_db_instance" "rds" {
 allocated_storage    = "${var.rds_allocated_storage}"
 storage_type         = "${var.rds_storage_type}"
 engine               = "${var.rds_engine}"
 engine_version       = "${var.rds_engine_version}"
 instance_class       = "${var.rds_instance_class}"
 name                 = "${var.project_name}_${var.env}_data_rds${var.rds_engine}"
 username             = "dbadmin"
 password             = "${var.rds_db_password}"
 multi_az             = false
 skip_final_snapshot  = true
 db_subnet_group_name = "${aws_db_subnet_group.rds_subnet.name}"
 vpc_security_group_ids = "${var.rds_vpc_security_group_ids}"
 identifier  = "${var.project_name}-${var.env}-data-rds${var.rds_engine}"


 tags {
   Project = "${var.project_name}"
   Env = "${var.env}"
   Type = "rds_${var.rds_engine}"
   Component = "data"
   Name = "${var.project_name}-${var.env}-data-rds${var.rds_engine}"
 }
}

Each Terraform configuration defines one or more AWS resource types using the resource configuration. A resource definition has 3 parts—the resource type (aws_db_instance) corresponding to the cloud service provider configured in the root module; a name for the resource definition (rds); and the properties of the resource definition. When the Terraform module is referenced in other parts of the Terraform configuration, the name of the resource is used. The properties control the behaviour of the resource and vary widely by the cloud resource being created. Name and identifier are two generic attributes that are available on most AWS resource. Tags set one or more tags on the resource. Name, identifier, and tags are added to the resource per the resource naming/tagging convention mentioned above. Resource definitions can refer to variables using notation ${var.<vartiable_name>}. All variables accessible for the resource are listed in the rds.variables.tf.

rds/rds.variables.tf:
variable "env" {
   description = "Environment name"
}

variable "project_name" {
 description = "Name of the project"
}

variable "rds_instance_class" {
 description = "RDS instance class"
}

variable "rds_storage_type" {
 description = "RDS storage type"
 default = "gp2"
}

variable "rds_allocated_storage" {
 description = "RDS allocated storage"
}

variable "rds_db_password" {
 description = "RDS DB Password"
}

variable "rds_subnet_ids" {
 type = "list"
 description = "RDS Subnet IDs"
}

variable "rds_vpc_security_group_ids" {
 type = "list"
 description = "RDS List of VPC security groups to associate"
}

variable "rds_engine" {
 description = "RDS The engine to use (postgres, mysql, etc)"
}

variable "rds_engine_version" {
 description = "RDS The engine version to use"
}

output "endpoint" {
 description = "DB Endpoint"
 value = "${aws_db_instance.rds.endpoint}"
}

output "port" {
 description = "DB port"
 value = "${aws_db_instance.rds.port}"
}

output "dbname" {
 description = "DB name"
 value = "${aws_db_instance.rds.name}"
}

The variables file defines all the input and output variables for the module. Input variables define all the values that can be specified when creating the module. Input variables are defined using the variable section. Variable definition can define a default value, which is used when no value is specified during the instantiation of the module. The type attribute specifies the data type of the variable. For common variables like env and project, we use the same name as in the root module. Module-specific variables are prefixed with the module name (e.g., rds_instance_class).

Output variables defined by the output section are the values that the module puts out. The value attribute specifies the property of the resource the module returns to the caller via the output variable. The caller of the module can access the output values of the module as module.<module_instancename>.<output var name> (e.g., ${module.rds.endpoint})

Instantiating Terraform Modules

Modules are instantiated in higher-level Terraform configurations using the module section. A module instantiation specifies the name of the instance and provides values for input variables. All input variables that do not have a default value must be provided a value. The source attribute on the module instantiation specifies a folder containing the configuration for the module definition. Here is an example instantiation of the RDS module:

module "rds" {
 source = "rds"
 env = "${var.env}"
 rds_instance_class = "${var.rds_instance_class}"
 rds_storage_type = "${var.rds_storage_type}"
 rds_allocated_storage = "${var.rds_allocated_storage}"
 rds_engine_version = "9.6.6"
 rds_db_password = "${data.vault_generic_secret.qio_vault.data["rds_password"]}"
 rds_subnet_ids = "${var.rds_subnet_ids}"
 rds_vpc_security_group_ids = "${var.rds_vpc_security_group_ids}"
 project_name = "${var.project_name}"
 rds_engine = "postgres"
}

This creates an instance of the RDS module and all the resources defined by the RDS module. The attributes in the module definition specify the values for input variables defined in the module definition script. The output values from the module as defined in the module definition can be accessed as module.rds.<outputvar_name> (e.g., ${module.rds.endpoint})

The same module can be instantiated any number of times in Terraform script by giving different names for each instance. Each instance of the module is referenced using the unique instance name.

Main Terraform Configuration

The main Terraform configuration, main.tf, instantiates all the modules needed to build the whole stack by providing appropriate values for the input variables required by each module. Main modules may create each module one or more times with different values for variables to build the differently configured instance of the same resources, as needed by the application. For example, the same SQS module is instantiate two times to create the two SQS needed by the app. The ECS module is instantiated four times to build the four ECS. Defining the SQS module and ECS modules—each of which requires a collection of resources to be created and linked together—makes it a very reusable modular component and avoids code duplication. Defining the variable for each module appropriately makes modules very reusable.

The main script also defines a bunch of variables that can be used to specify different values for attributes that vary by environment (e.g., the VPC and subnet ids are going to be different for different deployment environments, namely dev, test, impl, and prod). Making these attributes that vary by environment as variable at the top-level Terraform configuration makes the script usable for instantiating all environments.

The provider aws section on the main configuration specifies the cloud service provider that should be used for the Terraform configuration. Enabling that provider makes all the AWS-specific resource type definitions valid. It possible to configure more than one provider in a single Terraform config to use multiple services in a single infrastructure setup.

Providing Values for Input Variables

When executing the top-level script, we need to provide values for all variables defined used in the Terraform configuration. Values for variables can be specified by -var command line option on the Terraform command line or by specifying the variables file name using –var-file command line option. Terraform variables files are usually named .tfvars, and is basically an advanced properties file that allows specifying different types of values for Terraform variables.

Here is an example Terraform variables file from this case study:

#Common variables
project_name = "tf-demo"
gold_ami = "ami-747f12de" 

#ECS cluster related variables
ecs_subnet_ids=["subnet-bd259abc","subnet-ff7a0cdf"]
vpc_id="vpc-b5d69baa"

#RDS variables
rds_allocated_storage=20
rds_subnet_ids = ["subnet-d122abcd","subnet-a40a34df","subnet-e647fabc","subnet-e2a9dfed"]
rds_instance_class="db.m4.large"
rds_engine_version="9.6.5"
rds_vpc_security_group_ids=["sg-19c82adf"]

#ECS
ecs_lc_sg="sg-19c845df"
 

Executing Terraform

Terraform automatically keeps the Terraform configuration in sync with the infrastructure. Terraform does this by keeping the current state of the resource it created in a store that it maintains. Terraform can maintain this store in the local hard disk or a remote shared location like S3. To initialize a Terraform local store, run from the folder where the main Terraform script is present.

terraform init -backend-config 'key=tiers/<ENV>.tfstate'

This registers all the Terraform modules that are referenced by the Terraform script and also initializes the Terraform local store for the environment specified by ENV, where ENV is one of dev, test, stage or prod.

To execute the Terraform configuration and instantiate all the resources per the configuration, run:

terraform apply --var-file=<ENV>.tfvars

ENV is the environment on which we are building the stack. At this point Terraform will use the variable values specified in the <ENV>.tfvars file and instantiate the resource per the main Terraform script and the module that it refers to.

Terraform keeps record of all resources successfully created in its store. If we make changes to the Terraform configuration or the variables, and run the Terraform apply again, Terraform can identify the changes required on the existing resources to make it sync with the Terraform configuration.

To destroy or cleanup the resources created by the terraform can be done by running the destroy command. This will destroy all resources created by the terraform apply and registered in the terraform store.

terraform destroy --var-file=<ENV>.tfvars

Conclusion

In this case study, I tried to demonstrate how we used the Terraform modules and variables to effectively build a complex AWS infrastructure required for a complex application. Extensive use for modules with appropriate variables made the code very reusable and manageable in size. The attached zip file contains the complete Terraform config for the stack specified in this case study.

A second installment of this series will look into how we handle sensitive environment variables and using a remote store to handle collaborative building of the stack by multiple developers and configuration management using Ansible.