Terraform

This article is about utilising a tool from HashiCorp called Terraform to build infrastructure as a code. Meaning to spin up AWS instances, setup security groups, VPC and any other cloud based infrastructure component.

Install terraform

wget https://releases.hashicorp.com/terraform/0.9.1/terraform_0.9.1_linux_amd64.zip
unzip terraform_0.9.1_linux_amd64.zip
sudo mv ./terraform /usr/local/bin

When terraform is run it looks for .tf file where configuration is stored. The look up process is limited to a flat directory and never leaves the directory that runs from. Therefore if you wish to address a common file a symbolic-link needs to be created within the directory you have .tf file.

vi example.tf 
provider "aws" {
	access_key = "AK01234567890OGD6WGA" 
	secret_key = "N8012345678905acCY6XIc1bYjsvvlXHUXMaxOzN"
	region = "eu-west-1"
}
resource "aws_instance" "webserver" {
	ami = "ami-405f7226"
	instance_type = "t2.nano"
}

The example above will create an Ubuntu instance.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
<...>

+ aws_instance.webserver
   ami:                         "ami-405f7226"
   associate_public_ip_address: "<computed>"
   availability_zone:           "<computed>"
   ebs_block_device.#:          "<computed>"
   ephemeral_block_device.#:    "<computed>"
   instance_state:              "<computed>"
   instance_type:               "t2.nano"
   ipv6_addresses.#:            "<computed>"
   key_name:                    "<computed>"
   network_interface_id:        "<computed>"
   placement_group:             "<computed>"
   private_dns:                 "<computed>"
   private_ip:                  "<computed>"
   public_dns:                  "<computed>"
   public_ip:                   "<computed>"
   root_block_device.#:         "<computed>"
   security_groups.#:           "<computed>"
   source_dest_check:           "true"
   subnet_id:                   "<computed>"
   tenancy:                     "<computed>"
   vpc_security_group_ids.#:    "<computed>"

Apply stage, if runs first time will create terraform.tfstate after all changes are done. This file should not be modified manually. It's used to compare what is out in cloud already so the next time APPLY stage runs it will look at the file and execute only necessary changes.

$ terraform apply
aws_instance.webserver: Creating...
 ami:                         "" => "ami-405f7226"
 associate_public_ip_address: "" => "<computed>"
 availability_zone:           "" => "<computed>"
 ebs_block_device.#:          "" => "<computed>"
 ephemeral_block_device.#:    "" => "<computed>"
 instance_state:              "" => "<computed>"
 instance_type:               "" => "t2.nano"
 ipv6_addresses.#:            "" => "<computed>"
 key_name:                    "" => "<computed>"
 network_interface_id:        "" => "<computed>"
 placement_group:             "" => "<computed>"
 private_dns:                 "" => "<computed>"
 private_ip:                  "" => "<computed>"
 public_dns:                  "" => "<computed>"
 public_ip:                   "" => "<computed>"
 root_block_device.#:         "" => "<computed>"
 security_groups.#:           "" => "<computed>"
 source_dest_check:           "" => "true"
 subnet_id:                   "" => "<computed>"
 tenancy:                     "" => "<computed>"
 vpc_security_group_ids.#:    "" => "<computed>"
aws_instance.webserver: Still creating... (10s elapsed)
aws_instance.webserver: Creation complete (ID: i-0eb33af34b94d1a78)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: 

$ terraform show
aws_instance.webserver:
 id = i-0eb33af34b94d1a78
 ami = ami-405f7226
 associate_public_ip_address = true
 availability_zone = eu-west-1c
 disable_api_termination = false
 ebs_block_device.# = 0
 ebs_optimized = false
 ephemeral_block_device.# = 0
 iam_instance_profile = 
 instance_state = running
 instance_type = t2.nano
 ipv6_address_count = 0
 ipv6_addresses.# = 0
 key_name = 
 monitoring = false
 network_interface_id = eni-2c6b1553
 private_dns = ip-172-31-7-109.eu-west-1.compute.internal
 private_ip = 172.31.7.109
 public_dns = ec2-34-249-29-115.eu-west-1.compute.amazonaws.com
 public_ip = 34.249.29.115
 root_block_device.# = 1
 root_block_device.0.delete_on_termination = true
 root_block_device.0.iops = 100
 root_block_device.0.volume_size = 8
 root_block_device.0.volume_type = gp2
 security_groups.# = 0
 source_dest_check = true
 subnet_id = subnet-92a4bbf6
 tags.% = 0
 tenancy = default
 vpc_security_group_ids.# = 1
 vpc_security_group_ids.1039819662 = sg-5201fb2b

$ terraform destroy
Do you really want to destroy?
 Terraform will delete all your managed infrastructure.
 There is no undo. Only 'yes' will be accepted to confirm.
 Enter a value: yes
aws_instance.webserver: Refreshing state... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Destroying... (ID: i-0eb33af34b94d1a78)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 10s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 20s elapsed)
aws_instance.webserver: Still destroying... (ID: i-0eb33af34b94d1a78, 30s elapsed)
aws_instance.webserver: Destruction complete

Destroy complete! Resources: 1 destroyed.

After the instance has been terminated the terraform.tfstate looks like below:

vi terraform.tfstate
{
    "version": 3,
    "terraform_version": "0.9.1",
    "serial": 1,
    "lineage": "c22ccad7-ff26-4b8a-bf19-819477b45202",
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {},
            "depends_on": []
        }
    ]
}

AWS credentials profiles and variable files

Instead to reference secret_access keys within .tf file directly we can use AWS profile file. This file will be look at for the profile variable we specify in variables.tf file.

$ vi ~/.aws/credentials    #create AWS profile file
[terraform-profile1]       #profile name
aws_access_key_id = AK01234567890OGD6WGA
aws_secret_access_key = N8012345678905acCY6XIc1bYjsvvlXHUXMaxOzN

Then we can now remove the secret_access keys from the main .tf file (example.tf) and amend as follows:

provider "aws" {                provider "aws" { 
  access_key = "AK012345678 =>    profile = "${var.profile}"
  secret_key = "N8012345678       region  = "${var.region }"
  region = "eu-west-1"

and create a variable file to reference it

$ vi variables.tf
variable "region" {
        default = "eu-west-1"
}
variable "profile" {}     #variable without a default value will prompt to type in the value. And that should be 'terraform-profile1'

Run terraform

$ terraform   #it it will prompt for a value
$ terraform plan -var 'profile=terraform-profile1'  #this way value can be set

Example

Prerequisites are:

• ~/.aws/credential file exists
• variables.tf exist, with context below:

$ vi variables.tf
variable "region"          { default = "eu-west-1"                                   }
variable "profile"         { description = "AWS credentials profile you want to use" }
variable "key_name"        { description = "Name of the AWS key pair"                }
variable "public_key_path" { description = <<DESCRIPTION
Path to the SSH public keys for authentication.
Example: ~./ssh/terraform.pub
DESCRIPTION }

Terraform .tf file

$ vi example.tf
provider "aws" {
  region = "${var.region}"
  profile = "${var.profile}"
}
resource "aws_vpc" "vpc" {
  cidr_block = "10.0.0.0/16"
}
# Create an internet gateway to give our subnet access to the open internet
resource "aws_internet_gateway" "internet-gateway" {
  vpc_id = "${aws_vpc.vpc.id}"
}
# Give the VPC internet access on its main route table
resource "aws_route" "internet_access" {
  route_table_id         = "${aws_vpc.vpc.main_route_table_id}"
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = "${aws_internet_gateway.internet-gateway.id}"
}
# Create a subnet to launch our instances into
resource "aws_subnet" "default" {
  vpc_id                  = "${aws_vpc.vpc.id}"
  cidr_block              = "10.0.1.0/24"
  map_public_ip_on_launch = true

  tags {
    Name = "Public"
  }
}
# Our default security group to access
# instances over SSH and HTTP
resource "aws_security_group" "default" {
  name        = "terraform_securitygroup"
  description = "Used for public instances"
  vpc_id      = "${aws_vpc.vpc.id}"

  # SSH access from anywhere
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  # HTTP access from the VPC
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }
  # outbound internet access
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1" # all protocols
    cidr_blocks = ["0.0.0.0/0"]
  }
}
resource "aws_key_pair" "auth" {
  key_name   = "${var.key_name}"
  public_key = "${file(var.public_key_path)}"
}
resource "aws_instance" "webserver" {
  ami = "ami-405f7226"
  instance_type = "t2.nano"

  key_name = "${aws_key_pair.auth.id}"
  vpc_security_group_ids = ["${aws_security_group.default.id}"]

  # We're going to launch into the public subnet for this.
  # Normally, in production environments, webservers would be in
  # private subnets.
  subnet_id = "${aws_subnet.default.id}"

  # The connection block tells our provisioner how to
  # communicate with the instance
  connection {
    user = "ubuntu"
  }
  # We run a remote provisioner on the instance after creating it 
  # to install Nginx. By default, this should be on port 80
  provisioner "remote-exec" {
    inline = [
      "sudo apt-get -y update",
      "sudo apt-get -y install nginx",
      "sudo service nginx start"
    ]
  }
}

Run a plan

$ terraform plan
var.key_name
  Name of the AWS key pair

  Enter a value: id_rsa        #name of the key_pair

var.profile
  AWS credentials profile you want to use

  Enter a value: terraform-profile   #aws profile in ~/.aws/credentials file

var.public_key_path
  Path to the SSH public keys for authentication.
  Example: ~./ssh/terraform.pub

  Enter a value: ~/.ssh/id_rsa.pub  #path to the matching public key

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

+ aws_instance.webserver
    ami:                         "ami-405f7226"
    associate_public_ip_address: "<computed>"
    availability_zone:           "<computed>"
    ebs_block_device.#:          "<computed>"
    ephemeral_block_device.#:    "<computed>"
    instance_state:              "<computed>"
    instance_type:               "t2.nano"
    ipv6_addresses.#:            "<computed>"
    key_name:                    "${aws_key_pair.auth.id}"
    network_interface_id:        "<computed>"
    placement_group:             "<computed>"
    private_dns:                 "<computed>"
    private_ip:                  "<computed>"
    public_dns:                  "<computed>"
    public_ip:                   "<computed>"
    root_block_device.#:         "<computed>"
    security_groups.#:           "<computed>"
    source_dest_check:           "true"
    subnet_id:                   "${aws_subnet.default.id}"
    tenancy:                     "<computed>"
    vpc_security_group_ids.#:    "<computed>"

+ aws_internet_gateway.internet-gateway
    vpc_id: "${aws_vpc.vpc.id}"

+ aws_key_pair.auth
    fingerprint: "<computed>"
    key_name:    "id_rsa"
    public_key:  "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfc piotr@ubuntu"

+ aws_route.internet_access
    destination_cidr_block:     "0.0.0.0/0"
    destination_prefix_list_id: "<computed>"
    egress_only_gateway_id:     "<computed>"
    gateway_id:                 "${aws_internet_gateway.internet-gateway.id}"
    instance_id:                "<computed>"
    instance_owner_id:          "<computed>"
    nat_gateway_id:             "<computed>"
    network_interface_id:       "<computed>"
    origin:                     "<computed>"
    route_table_id:             "${aws_vpc.vpc.main_route_table_id}"
    state:                      "<computed>"

+ aws_security_group.default
    description:                           "Used for public instances"
    egress.#:                              "1"
    egress.482069346.cidr_blocks.#:        "1"
    egress.482069346.cidr_blocks.0:        "0.0.0.0/0"
    egress.482069346.from_port:            "0"
    egress.482069346.ipv6_cidr_blocks.#:   "0"
    egress.482069346.prefix_list_ids.#:    "0"
    egress.482069346.protocol:             "-1"
    egress.482069346.security_groups.#:    "0"
    egress.482069346.self:                 "false"
    egress.482069346.to_port:              "0"
    ingress.#:                             "2"
    ingress.2165049311.cidr_blocks.#:      "1"
    ingress.2165049311.cidr_blocks.0:      "10.0.0.0/16"
    ingress.2165049311.from_port:          "80"
    ingress.2165049311.ipv6_cidr_blocks.#: "0"
    ingress.2165049311.protocol:           "tcp"
    ingress.2165049311.security_groups.#:  "0"
    ingress.2165049311.self:               "false"
    ingress.2165049311.to_port:            "80"
    ingress.2541437006.cidr_blocks.#:      "1"
    ingress.2541437006.cidr_blocks.0:      "0.0.0.0/0"
    ingress.2541437006.from_port:          "22"
    ingress.2541437006.ipv6_cidr_blocks.#: "0"
    ingress.2541437006.protocol:           "tcp"
    ingress.2541437006.security_groups.#:  "0"
    ingress.2541437006.self:               "false"
    ingress.2541437006.to_port:            "22"
    name:                                  "terraform_securitygroup"
    owner_id:                              "<computed>"
    vpc_id:                                "${aws_vpc.vpc.id}"

+ aws_subnet.default
    assign_ipv6_address_on_creation: "false"
    availability_zone:               "<computed>"
    cidr_block:                      "10.0.1.0/24"
    ipv6_cidr_block_association_id:  "<computed>"
    map_public_ip_on_launch:         "true"
    tags.%:                          "1"
    tags.Name:                       "Public"
    vpc_id:                          "${aws_vpc.vpc.id}"

+ aws_vpc.vpc
    assign_generated_ipv6_cidr_block: "false"
    cidr_block:                       "10.0.0.0/16"
    default_network_acl_id:           "<computed>"
    default_route_table_id:           "<computed>"
    default_security_group_id:        "<computed>"
    dhcp_options_id:                  "<computed>"
    enable_classiclink:               "<computed>"
    enable_dns_hostnames:             "<computed>"
    enable_dns_support:               "true"
    instance_tenancy:                 "<computed>"
    ipv6_association_id:              "<computed>"
    ipv6_cidr_block:                  "<computed>"
    main_route_table_id:              "<computed>"


Plan: 7 to add, 0 to change, 0 to destroy.

Terraform apply

$ terraform apply

Once this completes, you can preview the current status with

$ terraform show
aws_instance.webserver:
 id = i-09c1c665cef284235
 ami = ami-405f7226
 associate_public_ip_address = true
 availability_zone = eu-west-1c
 disable_api_termination = false
 ebs_block_device.# = 0
 ebs_optimized = false
 ephemeral_block_device.# = 0
 iam_instance_profile = 
 instance_state = running
 instance_type = t2.nano
 ipv6_address_count = 0
 ipv6_addresses.# = 0
 key_name = id_rsa
 monitoring = false
 network_interface_id = eni-9edda3e1
 private_dns = ip-10-0-1-21.eu-west-1.compute.internal
 private_ip = 10.0.1.21
 public_dns = 
 public_ip = 34.251.232.46
 root_block_device.# = 1
 root_block_device.0.delete_on_termination = true
 root_block_device.0.iops = 100
 root_block_device.0.volume_size = 8
 root_block_device.0.volume_type = gp2
 security_groups.# = 0
 source_dest_check = true
 subnet_id = subnet-6f4f510b
 tags.% = 0
 tenancy = default
 vpc_security_group_ids.# = 1
 vpc_security_group_ids.2119542656 = sg-b14bb1c8
aws_internet_gateway.internet-gateway:
 id = igw-05af2961
 vpc_id = vpc-9ba0b7ff
aws_key_pair.auth:
 id = id_rsa
 key_name = id_rsa
 public_key = ssh-rsa AAAAB3NzXXXXXXXXXXXXXXX piotr@ubuntu
aws_route.internet_access:
 id = r-rtb-8207b3e51080289494
 destination_cidr_block = 0.0.0.0/0
 destination_prefix_list_id = 
 egress_only_gateway_id = 
 gateway_id = igw-05af2961
 instance_id = 
 instance_owner_id = 
 nat_gateway_id = 
 network_interface_id = 
 origin = CreateRoute
 route_table_id = rtb-8207b3e5
 state = active
 vpc_peering_connection_id = 
aws_security_group.default:
 id = sg-b14bb1c8
 description = Used for public instances
 egress.# = 1
 egress.482069346.cidr_blocks.# = 1
 egress.482069346.cidr_blocks.0 = 0.0.0.0/0
 egress.482069346.from_port = 0
 egress.482069346.ipv6_cidr_blocks.# = 0
 egress.482069346.prefix_list_ids.# = 0
 egress.482069346.protocol = -1
 egress.482069346.security_groups.# = 0
 egress.482069346.self = false
 egress.482069346.to_port = 0
 ingress.# = 2
 ingress.2165049311.cidr_blocks.# = 1
 ingress.2165049311.cidr_blocks.0 = 10.0.0.0/16
 ingress.2165049311.from_port = 80
 ingress.2165049311.ipv6_cidr_blocks.# = 0
 ingress.2165049311.protocol = tcp
 ingress.2165049311.security_groups.# = 0
 ingress.2165049311.self = false
 ingress.2165049311.to_port = 80
 ingress.2541437006.cidr_blocks.# = 1
 ingress.2541437006.cidr_blocks.0 = 0.0.0.0/0
 ingress.2541437006.from_port = 22
 ingress.2541437006.ipv6_cidr_blocks.# = 0
 ingress.2541437006.protocol = tcp
 ingress.2541437006.security_groups.# = 0
 ingress.2541437006.self = false
 ingress.2541437006.to_port = 22
 name = terraform_securitygroup
 owner_id = 312345678906
 tags.% = 0
 vpc_id = vpc-9ba0b7ff
aws_subnet.default:
 id = subnet-6f4f510b
 assign_ipv6_address_on_creation = false
 availability_zone = eu-west-1c
 cidr_block = 10.0.1.0/24
 map_public_ip_on_launch = true
 tags.% = 1
 tags.Name = Public
 vpc_id = vpc-9ba0b7ff
aws_vpc.vpc:
 id = vpc-9ba0b7ff
 assign_generated_ipv6_cidr_block = false
 cidr_block = 10.0.0.0/16
 default_network_acl_id = acl-c54c9fa2
 default_route_table_id = rtb-8207b3e5
 default_security_group_id = sg-bd4bb1c4
 dhcp_options_id = dopt-36d84252
 enable_classiclink = false
 enable_dns_hostnames = false
 enable_dns_support = true
 instance_tenancy = default
 main_route_table_id = rtb-8207b3e5
 tags.% = 0

Visualise configuration

Create visualised file. You may need to install sudo apt-get install graphviz if it is not in your system.

$ terraform graph | dot -Tpng > example.png

Terraform destroy

Run destroy command to delete all resources that were created

$ terraform destroy

aws_key_pair.auth: Refreshing state... (ID: id_rsa)
aws_vpc.vpc: Refreshing state... (ID: vpc-9ba0b7ff)
aws_subnet.default: Refreshing state... (ID: subnet-6f4f510b)
aws_internet_gateway.internet-gateway: Refreshing state... (ID: igw-05af2961)
aws_security_group.default: Refreshing state... (ID: sg-b14bb1c8)
aws_route.internet_access: Refreshing state... (ID: r-rtb-820...80289494)
aws_instance.webserver: Refreshing state... (ID: i-09c1c665cef284235)
aws_instance.webserver: Destroying... (ID: i-09c1c665cef284235)
aws_route.internet_access: Destroying... (ID: r-rtb-820...80289494)
aws_route.internet_access: Destruction complete
aws_internet_gateway.internet-gateway: Destroying... (ID: igw-05af2961)
aws_instance.webserver: Still destroying... (ID: i-09c1c665cef284235, 10s elapsed)
aws_internet_gateway.internet-gateway: Still destroying... (ID: igw-05af2961, 10s elapsed)
aws_instance.webserver: Still destroying... (ID: i-09c1c665cef284235, 20s elapsed)
aws_internet_gateway.internet-gateway: Still destroying... (ID: igw-05af2961, 20s elapsed)
aws_instance.webserver: Destruction complete
aws_subnet.default: Destroying... (ID: subnet-6f4f510b)
aws_key_pair.auth: Destroying... (ID: id_rsa)
aws_security_group.default: Destroying... (ID: sg-b14bb1c8)
aws_internet_gateway.internet-gateway: Destruction complete
aws_key_pair.auth: Destruction complete
aws_subnet.default: Destruction complete
aws_security_group.default: Destruction complete
aws_vpc.vpc: Destroying... (ID: vpc-9ba0b7ff)
aws_vpc.vpc: Destruction complete

Destroy complete! Resources: 7 destroyed.