Aws terraform

From campisano.org
Jump to navigation Jump to search

AWS

Finally, there are really low cost AWS services for developers who want to explore the AWS ecosystem, like AWS Free Tier services and AWS Lightsail ($3.5 -> $5 USD/Month services).

Terraform getting start guide

Terraform allows to manage Infrastructure as Code, so that you can programmatically define (and versioning) your infrastructure using code and setup virtual servers and other resources in the Cloud in few minutes.

Install and configure AWS client

Install and configure aws cli for first use

aws cli v2

from https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

VERSION="2.0.61"
cd /tmp
curl -kLO "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${VERSION}.zip"
unzip "awscli-exe-linux-x86_64-${VERSION}.zip"
./aws/install -u -i ~/.local/aws -b "${HOME}"/bin
rm -rf aws "awscli-exe-linux-x86_64-${VERSION}.zip"

aws cli v1 (obsolete)

from https://github.com/aws/aws-cli

  • then:
python3 -m pip install --user --upgrade awscli

Create an admin-user (best practices) using web interface

from https://docs.aws.amazon.com/lambda/latest/dg/setup.html and https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html

-> Groups -> Create New Group

use "admin-group" name
attach "AdministratorAccess" policy
confirm "Create Group"

-> Users -> Add user

use "admin-user" name
select "Programmatic access"
select "admin-group"
confirm "Create user"

NOTE: keep the "Access key ID" and the "Secret access key"

Configure aws cli to use admin-user profile

NOTE: for this example we will use "eu-west-1" region

aws --profile admin-user configure
# put the "Access key ID"
# put the "Secret access key"
# use "eu-west-1"
# use "json"
aws --profile admin-user iam list-users

Install terraform

VERSION="0.13.5"
cd /tmp
curl -kLO "https://releases.hashicorp.com/terraform/${VERSION}/terraform_${VERSION}_linux_amd64.zip"
unzip "terraform_${VERSION}_linux_amd64.zip"
mv -f terraform ~/bin
rm -f "terraform_${VERSION}_linux_amd64.zip"

Setup a new EC2 machine

Create a custom ssh keypair

ssh-keygen -q -t rsa -b 2048 -N '' -f ~/.ssh/aws-keypair
chmod 400 ~/.ssh/aws-keypair

Use AWS lightsail simplified service

What do you need:

  • an email and a phone number for identity verification

For the Nano virtual machine (1CPU, 512Mb, 20GB) the first month is free.

Find a bundle virtual server

aws --profile admin-user lightsail get-bundles --no-include-inactive --output table --query 'bundles[*].{Type:instanceType,Cpu:cpuCount,Disk:diskSizeInGb,Memory:ramSizeInGb,Transfer:transferPerMonthInGb,Price:price,Id:bundleId} | sort_by([],&Memory) | sort_by([],&Cpu) | sort_by([],&Price)'
------------------------------------------------------------------------------
|                                 GetBundles                                 |
+-----+-------+-------------------+---------+--------+-----------+-----------+
| Cpu | Disk  |        Id         | Memory  | Price  | Transfer  |   Type    |
+-----+-------+-------------------+---------+--------+-----------+-----------+
|  1  |  20   |  nano_2_0         |  0.5    |  3.5   |  1024     |  nano     |
|  1  |  40   |  micro_2_0        |  1.0    |  5.0   |  2048     |  micro    |
|  1  |  30   |  nano_win_2_0     |  0.5    |  8.0   |  1024     |  nano     |
|  1  |  60   |  small_2_0        |  2.0    |  10.0  |  3072     |  small    |
|  1  |  40   |  micro_win_2_0    |  1.0    |  12.0  |  2048     |  micro    |
|  1  |  60   |  small_win_2_0    |  2.0    |  20.0  |  3072     |  small    |
|  2  |  80   |  medium_2_0       |  4.0    |  20.0  |  4096     |  medium   |
|  2  |  80   |  medium_win_2_0   |  4.0    |  40.0  |  4096     |  medium   |
|  2  |  160  |  large_2_0        |  8.0    |  40.0  |  5120     |  large    |
|  2  |  160  |  large_win_2_0    |  8.0    |  70.0  |  5120     |  large    |
|  4  |  320  |  xlarge_2_0       |  16.0   |  80.0  |  6144     |  xlarge   |
|  4  |  320  |  xlarge_win_2_0   |  16.0   |  120.0 |  6144     |  xlarge   |
|  8  |  640  |  2xlarge_2_0      |  32.0   |  160.0 |  7168     |  2xlarge  |
|  8  |  640  |  2xlarge_win_2_0  |  32.0   |  240.0 |  7168     |  2xlarge  |
+-----+-------+-------------------+---------+--------+-----------+-----------+

Find a blueprint image

aws --profile admin-user lightsail get-blueprints --no-include-inactive --output table --query 'blueprints[?(type==`os`)&&(platform==`LINUX_UNIX`)].{Name:name,Group:group,Version:version,Id:blueprintId} | sort_by([],&Version) | sort_by([],&Id)'
----------------------------------------------------------------------------------
|                                  GetBlueprints                                 |
+----------------+-------------------+-----------------+-------------------------+
|      Group     |        Id         |      Name       |         Version         |
+----------------+-------------------+-----------------+-------------------------+
|  amazon-linux  |  amazon_linux     |  Amazon Linux   |  2018.03.0.20200918.0   |
|  amazon_linux_2|  amazon_linux_2   |  Amazon Linux 2 |  2.0.20200917.0         |
|  centos        |  centos_7_1901_01 |  CentOS         |  7 1901-01              |
|  debian_10     |  debian_10        |  Debian         |  10.5                   |
|  debian        |  debian_8_7       |  Debian         |  8.7                    |
|  debian_9      |  debian_9_5       |  Debian         |  9.5                    |
|  freebsd       |  freebsd_12       |  FreeBSD        |  12.1                   |
|  opensuse      |  opensuse_15_1    |  openSUSE       |  15.1                   |
|  ubuntu        |  ubuntu_16_04_2   |  Ubuntu         |  16.04 LTS              |
|  ubuntu_18     |  ubuntu_18_04     |  Ubuntu         |  18.04 LTS              |
|  ubuntu_20     |  ubuntu_20_04     |  Ubuntu         |  20.04 LTS              |
+----------------+-------------------+-----------------+-------------------------+

Import the custom ssh keypair

aws --profile admin-user lightsail import-key-pair --key-pair-name aws-keypair --public-key-base64 file://~/.ssh/aws-keypair.pub

Create a terraform infrastructure

Create a virtual machine using terraform

from from https://droidbasement.com/db-blog/aws-terraform-provision-a-lightsail-instance-using-infrastructure-as-code/

  • configure terraform
cat > terraform.tf << 'EOF'
terraform {
  required_version = ">= 0.13"
}
EOF
  • define an aws provider
cat > provider.tf << 'EOF'
provider "aws" {
  version                 = ">= 3.0"
  region                  = var.aws.region
  shared_credentials_file = var.aws.credentials_file
  profile                 = var.aws.profile
}
EOF
  • create a module to manage multiple machines
mkdir -p modules/lightsail
cat > modules/lightsail/input.tf << 'EOF'
variable "name"         { type = string }
variable "zone"         { type = string }
variable "keypair_name" { type = string }
variable "blueprint_id" { type = string }
variable "bundle_id"    { type = string }
variable "static_ip"    {
  type = bool
  default = false
}
variable "init_script"  {
  type = string
  default = null
}
EOF
cat > modules/lightsail/main.tf << 'EOF'
data "template_file" "init_script" {
  count = var.init_script != null ? 1 : 0

  template = file(var.init_script)
}

resource "aws_lightsail_instance" "instance" {
  name              = var.name
  availability_zone = var.zone
  key_pair_name     = var.keypair_name
  blueprint_id      = var.blueprint_id
  bundle_id         = var.bundle_id
  user_data         = var.init_script != null ? data.template_file.init_script[0].rendered : null
}

resource "aws_lightsail_static_ip" "static_ip" {
  count = var.static_ip ? 1 : 0

  name = "${var.name}_static_ip"
}

resource "aws_lightsail_static_ip_attachment" "static_ip_att" {
  count = var.static_ip ? 1 : 0

  static_ip_name = aws_lightsail_static_ip.static_ip[0].name
  instance_name  = aws_lightsail_instance.instance.name
}
EOF
cat > modules/lightsail/output.tf << 'EOF'
output "static_ip" {
  value = var.static_ip ? aws_lightsail_static_ip.static_ip[0].ip_address : "null"
}
EOF
  • configure the module using external variables
cat > input.tf << 'EOF'
variable "aws"       { type = map(string) }
variable "lightsail" { type = map(any) }
EOF
cat > main.tf << 'EOF'
module "lightsail" {
  source = "./modules/lightsail"

  for_each = var.lightsail

  name         = each.key
  zone         = each.value.zone
  keypair_name = each.value.keypair_name
  blueprint_id = each.value.blueprint_id
  bundle_id    = each.value.bundle_id
  static_ip    = lookup(each.value, "static_ip", false)
  init_script  = lookup(each.value, "init_script", null)
}
EOF
cat > output.tf << 'EOF'
output "lightsail_instances" {
  value = {for key, val in module.lightsail : key => val.static_ip}
}
EOF
  • configure external variables
cat > vars.json << 'EOF'
{
    "aws": {
        "region": "eu-west-1",
        "profile": "admin-user",
        "credentials_file": "~/.aws/credentials"
    },
    "lightsail": {
        "instance_name": {
            "zone": "eu-west-1a",
            "keypair_name": "aws-keypair",
            "blueprint_id": "debian_10",
            "bundle_id": "nano_2_0",
            "static_ip": true,
            "init_script": "init_script.sh"
        }
    }
}
EOF
  • configure init_script.sh
cat > init_script.sh << 'EOF'
# START CUSTOM SECTION init_script.sh
#
# this script assume to be run in a Debian 10 AWS image (that will copy-paste this script in a bigger one)
# and keep just a minimal number of packages and its dependencies

export DEBIAN_FRONTEND=noninteractive

apt-get -y update

# mark all packages as "not requested by the user"
apt-mark auto `apt-mark showmanual`

# install or upgrade or at least mark the follow packages as "requested by the user"
apt-get -y -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install \
  apt apt-utils bash binutils bsdutils bzip2 coreutils cron debconf debianutils dialog dpkg findutils grep grub-pc gzip ifupdown init iptables isc-dhcp-client kmod less libc-bin locales login lsb-release lsof mount nano openssh-server passwd procps psmisc readline-common rsyslog sed sudo systemd sysvinit-utils tar util-linux

# remove all "not requested" packages
apt-get -y autoremove --purge
apt-get -y dist-upgrade
apt-get -y clean
sync

echo "custom init script completed, see /var/log/cloud-init-output.log for details" > /var/log/init_script.log

# restart in a minute, just giving the time to complete other stuffs for the first boot
shutdown -r +1

# END CUSTOM SECTION init_script.sh
EOF
  • execute creation
terraform init
terraform apply -input=false -var-file=vars.json
  • destroy all resources
terraform destroy -input=false -var-file=vars.json
  • Optionally, use a simple Makefile to setup the previous custom commands:
cat > Makefile << 'EOF'
export TF_PLUGIN_CACHE_DIR := ${HOME}/.terraform.d/plugin-cache

.PHONY: clean
clean:
	@mkdir -p ${TF_PLUGIN_CACHE_DIR} || true
	@rm -rf .terraform

.PHONY: init
init: clean
	terraform init

.PHONY: apply
apply:
	terraform apply -input=false -var-file=vars.json

.PHONY: destroy
destroy:
	terraform destroy -input=false -var-file=vars.json
EOF

Use AWS EC2 custom instance

Find a good minimal AMI

aws --profile admin-user ec2 describe-images --owners amazon --filters 'Name=block-device-mapping.volume-size,Values=2' 'Name=description,Values=*Minimal*' --output table --query 'Images[?(Public)&&(State==`available`)&&(Architecture==`x86_64`)&&(VirtualizationType==`hvm`)&&(RootDeviceType==`ebs`)&&(CreationDate>=`2020-06-01`)].{id:ImageId,name:Name,desc:Description,date:CreationDate,owner:ImageOwnerAlias,arch:Architecture,virt:VirtualizationType,type:RootDeviceType,size:to_string(BlockDeviceMappings[*].Ebs.VolumeSize)} | reverse(sort_by([],&date)) | reverse(sort_by([],&name))'
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                                                     DescribeImages                                                                                                      |
+--------+---------------------------+----------------------------------------------------------------+------------------------+--------------------------------------------------------+---------+-------+-------+-------+
|  arch  |           date            |                             desc                               |          id            |                         name                           |  owner  | size  | type  | virt  |
+--------+---------------------------+----------------------------------------------------------------+------------------------+--------------------------------------------------------+---------+-------+-------+-------+
|  x86_64|  2020-09-22T02:18:38.000Z |  Amazon Linux 2 AMI 2.0.20200917.0 x86_64 Minimal HVM ebs      |  ami-06e7a85d57337c9fa |  amzn2-ami-minimal-hvm-2.0.20200917.0-x86_64-ebs       |  amazon |  [2]  |  ebs  |  hvm  |
|  x86_64|  2020-09-04T02:37:57.000Z |  Amazon Linux 2 AMI 2.0.20200904.0 x86_64 Minimal HVM ebs      |  ami-09b8e6a77f149e6ec |  amzn2-ami-minimal-hvm-2.0.20200904.0-x86_64-ebs       |  amazon |  [2]  |  ebs  |  hvm  |
|  x86_64|  2020-09-01T18:21:15.000Z |  Amazon Linux 2 AMI 2.0.20200824.0 x86_64 Minimal HVM ebs      |  ami-01726776b01c7d9c2 |  amzn2-ami-minimal-hvm-2.0.20200824.0-x86_64-ebs       |  amazon |  [2]  |  ebs  |  hvm  |
|  x86_64|  2020-07-24T20:49:31.000Z |  Amazon Linux 2 AMI 2.0.20200722.0 x86_64 Minimal HVM ebs      |  ami-09eb2d6d439ae885e |  amzn2-ami-minimal-hvm-2.0.20200722.0-x86_64-ebs       |  amazon |  [2]  |  ebs  |  hvm  |
|  x86_64|  2020-06-23T06:18:00.000Z |  Amazon Linux 2 AMI 2.0.20200617.0 x86_64 Minimal HVM ebs      |  ami-0249782201f0ee367 |  amzn2-ami-minimal-hvm-2.0.20200617.0-x86_64-ebs       |  amazon |  [2]  |  ebs  |  hvm  |
|  x86_64|  2020-09-23T17:45:32.000Z |  Amazon Linux AMI 2018.03.0.20200918.0 x86_64 Minimal HVM ebs  |  ami-09c00af71f9f4a141 |  amzn-ami-minimal-hvm-2018.03.0.20200918.0-x86_64-ebs  |  amazon |  [2]  |  ebs  |  hvm  |
|  x86_64|  2020-09-04T02:17:52.000Z |  Amazon Linux AMI 2018.03.0.20200904.0 x86_64 Minimal HVM ebs  |  ami-0780ed21a81408761 |  amzn-ami-minimal-hvm-2018.03.0.20200904.0-x86_64-ebs  |  amazon |  [2]  |  ebs  |  hvm  |
|  x86_64|  2020-08-11T20:17:17.000Z |  Amazon Linux AMI 2018.03.0.20200729.0 x86_64 Minimal HVM ebs  |  ami-02428347d30e4e928 |  amzn-ami-minimal-hvm-2018.03.0.20200729.0-x86_64-ebs  |  amazon |  [2]  |  ebs  |  hvm  |
|  x86_64|  2020-07-21T18:39:43.000Z |  Amazon Linux AMI 2018.03.0.20200716.0 x86_64 Minimal HVM ebs  |  ami-0be996716886f5772 |  amzn-ami-minimal-hvm-2018.03.0.20200716.0-x86_64-ebs  |  amazon |  [2]  |  ebs  |  hvm  |
|  x86_64|  2020-06-16T06:36:13.000Z |  Amazon Linux AMI 2018.03.0.20200602.1 x86_64 Minimal HVM ebs  |  ami-09e12379867321151 |  amzn-ami-minimal-hvm-2018.03.0.20200602.1-x86_64-ebs  |  amazon |  [2]  |  ebs  |  hvm  |
+--------+---------------------------+----------------------------------------------------------------+------------------------+--------------------------------------------------------+---------+-------+-------+-------+

Create a simple EC2

  • It will use an elastic ip to see how resources are reused

Create

  • example using a basic Ubuntu 16.04 LTS AMI
mkdir example_org
cd example_org
cat > example_org.tf << 'EOF'
provider "aws" {
  profile           = "admin-user"
  region            = "eu-west-1"
}

resource "aws_instance" "example_org" {
  # Ubuntu 16.04 LTS
  ami               = "ami-03746875d916becc0"
  instance_type     = "t3.nano"
}
EOF
terraform init  # init terraform for the new project requirements
terraform plan  # see changes to be applyed
terraform apply # apply changes

Edit

  • example changing to Ubuntu 18.04 LST AMI
cat > example_org.tf << 'EOF'
provider "aws" {
  profile           = "admin-user"
  region            = "eu-west-1"
}

resource "aws_instance" "example_org" {
  #  # Ubuntu 16.04 LTS
  #  ami               = "ami-03746875d916becc0"
  # Ubuntu 18.04 AMI
  ami               = "ami-01e6a0b85de033c99"
  instance_type     = "t3.nano"
}
EOF
terraform apply
  • example: adding an elastic-ip
cat >> example_org.tf << 'EOF'

resource "aws_eip" "ip" {
  instance = "${aws_instance.example_org.id}"
}
EOF
terraform apply
  • example: changing AMI again to minimal, keeping the Elastic IP
cat > example_org.tf << 'EOF'
provider "aws" {
  profile           = "admin-user"
  region            = "eu-west-1"
}

resource "aws_instance" "example_org" {
  #  # Ubuntu 16.04 LTS
  #  ami               = "ami-03746875d916becc0"
  #  # Ubuntu 18.04 AMI
  #  ami               = "ami-01e6a0b85de033c99"
  # minimal hvm ebs x86_64 AMI 20190618
  ami               = "ami-0a3b59edf43c875be"
  instance_type     = "t3.nano"
}

resource "aws_eip" "ip" {
  instance = "${aws_instance.example_org.id}"
}
EOF
terraform apply


Delete

  • cleanup
terraform destroy

List existing instances

aws --profile admin-user ec2 describe-instances --output table --query 'Reservations[*].Instances[*].{Name:[Tags[?Key==`Name`].Value][0][0],Ip:PrivateIpAddress,Type:InstanceType,Zone:Placement.AvailabilityZone,State:State.Name,Launched:LaunchTime,VPC:VpcId,SN:SubnetId,SG:join(`,`, SecurityGroups[*].GroupId)}'
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                                                     DescribeInstances                                                                                                    |
+---------------+----------------------------+--------------------+------------------------------------------------------------------------------+------------------+----------+------------+---------------+--------------+
|      Ip       |         Launched           |       Name         |                                     SG                                       |       SN         |  State   |   Type     |      VPC      |    Zone      |
+---------------+----------------------------+--------------------+------------------------------------------------------------------------------+------------------+----------+------------+---------------+--------------+
|  172.31.20.253|  2020-05-04T02:14:52+00:00 |  ec2_campisano.org |  sg-0265edc543d39e5ce,sg-5c21bb23,sg-07f4afef9a2105c23,sg-09c096105eacbf55d  |  subnet-76c1a23e |  running |  t3a.micro |  vpc-0bc5e36d |  eu-west-1a  |
+---------------+----------------------------+--------------------+------------------------------------------------------------------------------+------------------+----------+------------+---------------+--------------+

References