Terraweek#Day07-Advanced topics

What is Terraform Workspace?

When we create cloud resources using Terraform configuration language, the resources are known to be created in the default workspace. Workspace is a way to maintain multiple copies of deployments that can be created and destroyed on the go.

The information about all the resources managed by Terraform is stored in a state file. It is important to store this state file in a secure location. Every Terraform run is associated with a state file for validation and reference purposes. Any modifications to the Terraform configuration, planned or applied, are always first validated with references in the state files, and the execution result is updated back to it.

If you are not consciously using any workspace, then all of this already happens in a default workspace. Workspaces help you isolate independent deployments of the same Terraform config while using the same state file.

The general IaC development for AWS using Terraform follows the steps below.

Configure AWS CLI and credentials on Terraform host machine.
Configure and initialize AWS provider in Terraform.
Write Terraform configuration for the AWS resources to be created.
Configure a backend for state management that enables team collaboration.
Run terraform plan command to validate the configuration and generate a summary of changes that will be applied.
Run terraform apply command to actually apply the changes in configuration.
Run terraform destroy command to destroy all the AWS resources managed through Terraform configuration.

let’s take a look at the options available to us in the help.

let us consider a simple Terraform config that creates an EC2 instance with the below configuration. We are currently using three variables for AMI value, the type of instance to be created and the name tag.

resource "aws_instance" "my_vm" {
ami           = var.ami //Ubuntu AMI
instance_type = var.instance_type

tags = {
  Name = var.name_tag,
}
}

To check the current workspace we are in, run the below command.

terraform workspace show
default

The output here shows that we are currently in the workspace named default. To be sure about no other workspaces currently exist, run the list command as shown below.

terraform workspace list
* default

let us create another workspace and select the same. We can do this by running the new command with the desired name of the new workspace as below.

terraform workspace new test_workspace
Created and switched to workspace "test_workspace"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.

Here, I have selected the name of the new Terraform workspace as “test_workspace”. Note that running this command has created and switched to the new workspace. We can verify this selection is made by running the show command as below.

terraform workspace show
test_workspace

Of course, another way to verify it would be to run the list command and see where the asterisk (*) is pointing to.

terraform workspace list
  default
* test_workspace

Remote Operations

Terraform Cloud is designed as an execution platform for Terraform, and can perform Terraform runs on its own disposable virtual machines. This provides a consistent and reliable run environment, and enables advanced features like Sentinel policy enforcement, cost estimation, notifications, version control integration, and more.

Terraform runs managed by Terraform Cloud are called remote operations. Remote runs can be initiated by webhooks from your VCS provider, by UI controls within Terraform Cloud, by API calls, or by Terraform CLI. When using Terraform CLI to perform remote operations, the progress of the run is streamed to the user's terminal, to provide an experience equivalent to local operations.

Terraform Collaboration

There are three core elements to the Terraform collaboration experience:

State management: Store, secure, lock, and version Terraform state files
Centralized plans and applies: Safely run Terraform plans and applies in one location where collaborators can review and make decisions together
Module registry: Share reusable Terraform modules across a team

Managing a state file is the first collaboration hurdle with Terraform, and we are kicking off our commitment to collaboration for all by providing free remote state storage with:

No limits on users
No limits on workspaces
Encryption provided by HashiCorp Vault
Per-operation locking

The Importance of Terraform Collaboration

Terraform is a powerful tool when wielded by an individual, but when groups of people collaborate using Terraform, they can achieve real, transformational benefits from infrastructure as code.

We have always known that collaboration is essential for delivering infrastructure with Terraform. This is why we built world-class collaboration functionality into Terraform Enterprise, which became generally available in 2017. In partnership with our customers, we have been making Terraform Enterprise better with each release.

We've also been working closely with Terraform's open source user community, learning from our users working with Terraform in organizations large and small. Our original thesis was that generalized tooling for code collaboration (e.g. version control systems and CI systems) would be enough for basic collaboration on Terraform configuration. We now know that generalized tooling is not enough for effective collaboration on Terraform. There is a gap between the problems solved by the Terraform CLI and Terraform Enterprise.

For evidence of this gap, we need to look no further than the community trying to fill it. Several projects and products have shown up over the past year to provide supportive tooling for Terraform collaboration. It's clear to us now that collaboration is key for the whole Terraform community — individual practitioners, teams, and enterprises.

Remote Plans and Applies for Current Terraform Enterprise customers

Terraform Enterprise has an excellent collaboration workflow, and today we're announcing an improvement to that workflow with Remote Plans and Applies. Users can start runs in Terraform Enterprise with the standard terraform plan and terraform apply commands, and can watch the progress of the run without leaving their terminal. These runs execute remotely in Terraform Enterprise; they use variables from the appropriate workspace, enforce any applicable Sentinel policies, and can access Terraform Enterprise's private module registry and remote state inputs.

Best Terraform Practices:

Consistence File Structure

Terraform configurations files separation

Putting all code in main.tf is not a good idea, better having several files like:

main.tf - call modules, locals, and data sources to create all resources. variables.tf - contains declarations of variables used in main.tf outputs.tf - contains outputs from the resources created in main.tf versions.tf - contains version requirements for Terraform and providers. terraform.tfvars - contains variables values and should not be used anywhere.

Follow a standard module structure

Use separate directories for each application

Use separate directories for each environment Use separate directory for each environment (dev, qa, stage, prod).

Put static files in a separate directory

Common recommendations for structuring code

Use Consistence Naming & Code Style Conventions and Formatting General Naming Conventions Variables Conventions Outputs Conventions Use built-in formatting Better Security practices Setup backend state locking Don’t store secrets in state Minimize Blast Radius Run continuous audits Use Sensitive flag variables Use variable definitions (.tfvars) files Use Shared Modules Release tagged versions

Version control

Like application code, store infrastructure code in version control to preserve history and allow easy rollbacks.
Use a default branching strategy (such as GitFlow, GitHubFlow).
Encourage infrastructure stakeholders to submit merge requests as part of the change request process.
Use separate environment branches for root configurations if required.
Organize repositories based on team boundaries.

Use latest version of Terraform

Limit the complexity of expressions

Use Docker

Step-by-step process deploy your infrastructure in CI/CD using Terraform

In this section, we will go through the procedure for deploying infrastructure in CI/CD with the help of Terraform and other tools mentioned here:

Step #1: Create a Terraform configuration file

Use the HashiCorp Configuration Language (HCL) to create the Terraform configuration file like in the below example of a simple configuration file where an Amazon Web Services (AWS) Elastic Compute Cloud (EC2) instance is created:

provider “aws” {

region = “us-west-2”

}

resource “aws_instance” “example” {

ami = “ami-0ff8a91507f77f867”

instance_type = “t2.micro”

tags = {

Name = “example-instance”

}

Here, the “provider” component configures the AWS region. The “resource” block defines the EC2 instance, including the AMI, instance type, and name.

Terraform’s interpolation syntax can reference other resources in your configuration file, and modules can encapsulate and reuse configurations.

You must maintain configuration files in a version control system like git to track changes, roll back to previous versions, and manage interaction with teammates.

Step #2: Use Terraform to set up the environment and download the required providers

Follow the below process to initialize the environment and download any needful providers using Terraform:

Install Terraform on your local machine.
Create a new directory for your Terraform project and locate it in your terminal.
Run terraform init command to download the required providers and initialize the Terraform environment in the current directory.

Also, you may choose to specify providers (and their versions) in the providers block of your Terraform configuration file which is generally named main.tf and run the terraform init command.

For instance:

provider “aws” {

version = “~> 2.0”

}

This command will download the specified version of the AWS provider and initialize the Terraform environment.

Store your provider credentials in a terraform.tfvars file to avoid storing them in shared environments like a CI/CD pipeline.

Step #3: Plan the changes by running ‘terraform plan’

Follow the below procedure to plan the changes using Terraform:

Double-check whether you have a valid Terraform configuration file to define the resources you wish to create/modify.
Navigate to the directory containing your Terraform configuration file within your terminal.
Run the terraform plan command to get a preview of the results of creation/modification subsequent to terraform apply.

The Terraform plan compares your infrastructure’s present condition to your desired state and then creates an action plan and prompts confirmation.

For later use, add the -out flag to the plan command.

Step #4: Apply the changes by running ‘terraform apply’

Follow the below process to apply changes using Terraform:

Make sure you have a valid Terraform configuration file defining the resources you want to create/modify, and that you have run terraform plan and reviewed the plan.
Go to the directory containing your Terraform configuration file in your terminal.
Run the terraform apply command to create/modify the infrastructure as per your Terraform configuration.

Running the terraform apply command will promptTerraform to use the execution plan generated by your last terraform plan command for making the changes to your infrastructure.

You can also use the -auto-approve flag to skip the confirmation prompt:

terraform apply -auto-approve

Also, you can use a previously saved plan file with the help of -input=false -lock=false -refresh=false flags:

terraform apply tfplan

Note: Once you have run the terraform apply command, it is important to keep your Terraform configuration files up to date so that you can easily make changes or destroy the infrastructure in the future.

Also, it is in your best interests to use version control for your terraform configuration files, as it helps simplify keeping track of incremental changes you made over time and make rollbacks easier.

For destroying the redundant infrastructure, you can use the terraform destroy command (with -auto-approve flag) or target specific resources like in the below example:

terraform destroy -target=aws_instance.example

You must take notice of the fact that eliminating resources can potentially impact other resources/services with dependencies so you must be cautious when doing so.

It’s recommended to use CI/CD tools like Jenkins, TravisCI, or CircleCI to automate these steps and integrate with version control systems like git to version control your terraform code.

Terraform Cloud

Terraform Cloud, developed by HashiCorp, offers a centralized platform for managing Terraform code. Key features include version control integration, secure variable storage, remote state management, and policy enforcement, enabling organizations to efficiently maintain control over their cloud infrastructure.

Terraform Cloud vs. Terraform Enterprise

Another option Hashicorp offers for Terraform Cloud is its Enterprise solution. The main difference is that Terraform Enterprise is self-hosted and offered as a private installation rather than a SaaS solution. Terraform Enterprise offers most of the same features as the Terraform Business tier.

Terraform Enterprise

Terraform Enterprise is a self-hosted distribution of Terraform Cloud. All the features of Terraform Cloud and Terraform Enterprise are the same except additional features in Terraform Enterprise are audit logging, SAML single sign-on, private instance with no limits etc.

SAML Single sign on- Terraform Enterprise supports SAML 2.0, And it works with a variety of identity providers. Some IDP providers like ADFS, Azure Active Directory, Okta, OneLogin etc.
Administration of Terraform Enterprise instance- There are two main domains of administration of the instance, Infrastructure administration and application administration-
- Infrastructure administration: Maintenance tasks like upgrades, and operational tasks like backups and monitoring. Which take place outside the Terraform Cloud Application.
- Application administration: Administrative tasks and configuration within the application itself.

Terraform Registry

Overview

The Terraform registry in Artifactory allows you to create dedicated repositories for each of the following unique Terraform components:

Providers: A set of plugins that interact with cloud providers, SaaS providers, and other APIs.
Modules: Serve as containers for multiple resources that are used together. Modules contain a collection of

.tf

files kept together in a directory.

The Modules and Providers have different settings for local repositories but are the same when it comes to configuring remote and virtual repositories.

To learn more about the Terraform repository solution in the JFrog Platform, see Terraform Repositories.

Terraform Module Registry

Terraform Modules are Terraform configurations that can be called and configured by other configurations. They serve as containers for multiple resources that are used together. They contain a collection of.tf files kept together in a directory. To create a module, pack all your

.tf

files into a Zip archive and deploy to the Terraform local repository. For more information, see Creating Terraform Modules.

Terraform Module Repository Structure

The Terraform Module repository is a directory with a collection of Zip files, consisting of these main coordinates:

namespace
module
Provider
version.zip

Artifactory complies with the Terraform Module directory layout convention.

namespace/module-name/provider/version.zip

For example:

terraform-aws-modules/vpc/aws/2.0.0.zip
harshicorp/consul/aws/DEV-123.zip

Terraform Provider Registry

Providers are a set of plugins that interact with cloud providers, SaaS providers and other APIs.

Terraform Provider Repository Structure

The Terraform Provider repository is a directory with a collection of Zip files consisting of these main coordinates:

Namespace
Provider
Version.zip
Operating system
Architecture

Artifactory complies with the Terraform Provider directory layout convention.

namespace/provider/version/terraform-provider-{NAME}_{VERSION}_{OS}_{ARCH}.zip