Aws (Cloud)
Jump to navigation
Jump to search
AWS - Amazon Web Services
Finally, there are really low cost AWS services for developers who want to explore the AWS ecosystem, like AWS Free Tier services and AWS Lightsail ($3.5 -> $5 USD/Month services).
Configure an AWS account
- Create a Lightsail account.
- Create an admin-user (best practices) using web interface
from https://docs.aws.amazon.com/lambda/latest/dg/setup.html
and https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html
go to https://console.aws.amazon.com/iam/ click in Groups -> Create New Group use "admin-group" name attach "AdministratorAccess" policy confirm "Create Group"
go back to https://console.aws.amazon.com/iam/ click in Users -> Add user use "admin-user" name select "Programmatic access" select "admin-group" confirm "Create user"
NOTE: keep the "Access key ID" and the "Secret access key"
Install and configure AWS client
Install and configure aws cli for first use
Configure aws cli to use admin-user profile
NOTE: for this example we will use "eu-west-1" region
aws --profile admin-user configure # put the "Access key ID" # put the "Secret access key" # use "eu-west-1" # use "json" aws --profile admin-user iam list-users
Create a custom ssh keypair
ssh-keygen -q -t rsa -b 2048 -N '' -f ~/.ssh/aws-keypair chmod 400 ~/.ssh/aws-keypair
Install Terraform
Terraform allows to manage Infrastructure as Code, so that you can programmatically define (and versioning) your infrastructure using code and setup virtual servers and other resources in the Cloud in few minutes.
To install, follow Terraform_(Application) instructions.
Setup a new EC2 machine
Use AWS lightsail simplified service
Note: the source code is available at GitLab - terraform_aws_lightsail_module.
What do you need:
- an email and a phone number for identity verification
For the Nano virtual machine (1CPU, 512Mb, 20GB) the first month is free.
Find a bundle virtual server
aws --profile admin-user lightsail get-bundles --no-include-inactive --output table --query 'bundles[*].{Type:instanceType,Cpu:cpuCount,Disk:diskSizeInGb,Memory:ramSizeInGb,Transfer:transferPerMonthInGb,Price:price,Id:bundleId} | sort_by([],&Memory) | sort_by([],&Cpu) | sort_by([],&Price)'
------------------------------------------------------------------------------
| GetBundles |
+-----+-------+-------------------+---------+--------+-----------+-----------+
| Cpu | Disk | Id | Memory | Price | Transfer | Type |
+-----+-------+-------------------+---------+--------+-----------+-----------+
| 1 | 20 | nano_2_0 | 0.5 | 3.5 | 1024 | nano |
| 1 | 40 | micro_2_0 | 1.0 | 5.0 | 2048 | micro |
| 1 | 30 | nano_win_2_0 | 0.5 | 8.0 | 1024 | nano |
| 1 | 60 | small_2_0 | 2.0 | 10.0 | 3072 | small |
| 1 | 40 | micro_win_2_0 | 1.0 | 12.0 | 2048 | micro |
| 1 | 60 | small_win_2_0 | 2.0 | 20.0 | 3072 | small |
| 2 | 80 | medium_2_0 | 4.0 | 20.0 | 4096 | medium |
| 2 | 80 | medium_win_2_0 | 4.0 | 40.0 | 4096 | medium |
| 2 | 160 | large_2_0 | 8.0 | 40.0 | 5120 | large |
| 2 | 160 | large_win_2_0 | 8.0 | 70.0 | 5120 | large |
| 4 | 320 | xlarge_2_0 | 16.0 | 80.0 | 6144 | xlarge |
| 4 | 320 | xlarge_win_2_0 | 16.0 | 120.0 | 6144 | xlarge |
| 8 | 640 | 2xlarge_2_0 | 32.0 | 160.0 | 7168 | 2xlarge |
| 8 | 640 | 2xlarge_win_2_0 | 32.0 | 240.0 | 7168 | 2xlarge |
+-----+-------+-------------------+---------+--------+-----------+-----------+
Find a blueprint image
aws --profile admin-user lightsail get-blueprints --no-include-inactive --output table --query 'blueprints[?(type==`os`)&&(platform==`LINUX_UNIX`)].{Name:name,Group:group,Version:version,Id:blueprintId} | sort_by([],&Version) | sort_by([],&Id)'
----------------------------------------------------------------------------------
| GetBlueprints |
+----------------+-------------------+-----------------+-------------------------+
| Group | Id | Name | Version |
+----------------+-------------------+-----------------+-------------------------+
| amazon-linux | amazon_linux | Amazon Linux | 2018.03.0.20210126.1 |
| amazon_linux_2| amazon_linux_2 | Amazon Linux 2 | 2.0.20210126.0 |
| centos | centos_7_1901_01 | CentOS | 7 1901-01 |
| debian_10 | debian_10 | Debian | 10.5 |
| debian | debian_8_7 | Debian | 8.7 |
| debian_9 | debian_9_5 | Debian | 9.5 |
| freebsd | freebsd_12 | FreeBSD | 12.1 |
| opensuse | opensuse_15_1 | openSUSE | 15.1 |
| ubuntu | ubuntu_16_04_2 | Ubuntu | 16.04 LTS |
| ubuntu_18 | ubuntu_18_04 | Ubuntu | 18.04 LTS |
| ubuntu_20 | ubuntu_20_04 | Ubuntu | 20.04 LTS |
+----------------+-------------------+-----------------+-------------------------+
Import the custom ssh keypair
aws --profile admin-user lightsail import-key-pair --key-pair-name aws-keypair --public-key-base64 file://~/.ssh/aws-keypair.pub
Create a virtual machine using Terraform
- configure Terraform
cat > versions.tf << 'EOF'
terraform {
required_version = ">= 0.13"
required_providers {
aws = ">= 3.0"
}
}
EOF
- define an aws provider
cat > provider.tf << 'EOF'
provider "aws" {
region = var.aws_provider.region
shared_credentials_file = var.aws_provider.credentials_file
profile = var.aws_provider.profile
}
EOF
- create a module to manage multiple machines
mkdir -p modules/lightsail
cat > modules/lightsail/input.tf << 'EOF'
variable "aws_profile" { type = string }
variable "name" { type = string }
variable "zone" { type = string }
variable "keypair_name" { type = string }
variable "blueprint_id" { type = string }
variable "bundle_id" { type = string }
variable "static_ip" {
type = bool
default = false
}
variable "init_script_path" {
type = string
default = null
}
variable "public_ports_rules" {
type = string
default = null
}
EOF
cat > modules/lightsail/main.tf << 'EOF'
data "template_file" "init_script" {
count = var.init_script_path != null ? 1 : 0
template = file(var.init_script_path)
}
resource "aws_lightsail_instance" "instance" {
name = var.name
availability_zone = var.zone
key_pair_name = var.keypair_name
blueprint_id = var.blueprint_id
bundle_id = var.bundle_id
user_data = var.init_script_path != null ? data.template_file.init_script[0].rendered : null
}
resource "aws_lightsail_static_ip" "static_ip" {
count = var.static_ip ? 1 : 0
name = "${var.name}_static_ip"
}
resource "aws_lightsail_static_ip_attachment" "static_ip_att" {
count = var.static_ip ? 1 : 0
static_ip_name = aws_lightsail_static_ip.static_ip[0].name
instance_name = aws_lightsail_instance.instance.name
}
resource "null_resource" "firewall" {
count = var.public_ports_rules != null ? 1 : 0
provisioner "local-exec" {
command = "aws --profile ${var.aws_profile} lightsail put-instance-public-ports --instance-name=${aws_lightsail_instance.instance.name} --port-infos ${var.public_ports_rules}"
}
}
EOF
cat > modules/lightsail/output.tf << 'EOF'
output "static_ip" {
value = var.static_ip ? aws_lightsail_static_ip.static_ip[0].ip_address : "null"
}
EOF
- configure the module using external variables
cat > input.tf << 'EOF'
variable "aws_provider" { type = map(string) }
variable "lightsail_module" { type = map(any) }
EOF
cat > main.tf << 'EOF'
module "lightsail" {
source = "./modules/lightsail"
for_each = var.lightsail_module
aws_profile = var.aws_provider.profile
name = each.key
zone = each.value.zone
keypair_name = each.value.keypair_name
blueprint_id = each.value.blueprint_id
bundle_id = each.value.bundle_id
static_ip = lookup(each.value, "static_ip", false)
init_script_path = lookup(each.value, "init_script_path", null)
public_ports_rules = lookup(each.value, "public_ports_rules", null)
}
EOF
cat > output.tf << 'EOF'
output "lightsail_instances" {
value = {for key, val in module.lightsail : key => val.static_ip}
}
EOF
- configure external variables
cat > vars.json << 'EOF'
{
"aws_provider": {
"region": "eu-west-1",
"profile": "admin-user",
"credentials_file": "~/.aws/credentials"
},
"lightsail_module": {
"my_instance_name_1": {
"zone": "eu-west-1a",
"keypair_name": "aws-keypair",
"blueprint_id": "debian_10",
"bundle_id": "nano_2_0",
"static_ip": true,
"init_script_path": "init_script.sh"
}
}
}
EOF
- (optionally) configure init_script.sh
The script can be used to customize the O.S. image. In this example, I'm adapting the image to my objective: use this O.S. image for a successive (off-topic) use as host machine for multiple small containers. You may use it as example or skip it.
cat > init_script.sh << 'EOF' # START CUSTOM SECTION init_script.sh # # this script assume to be run in a Debian 10 AWS image (that will copy-paste this script in a bigger one) # and keep just a minimal number of packages and its dependencies export DEBIAN_FRONTEND=noninteractive apt-get -y update # mark all packages as "not requested by the user" apt-mark auto `apt-mark showmanual` # install or upgrade or at least mark the follow packages as "requested by the user" apt-get -y -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install \ apt apt-utils bash binutils bsdutils bzip2 coreutils cron debconf debianutils dialog dpkg findutils grep grub-pc gzip ifupdown init iptables isc-dhcp-client kmod less libc-bin locales login lsb-release lsof mount nano openssh-server passwd procps psmisc readline-common rsyslog sed sudo systemd sysvinit-utils tar util-linux # remove all "not requested" packages apt-get -y autoremove --purge apt-get -y dist-upgrade apt-get -y clean sync echo "custom init script completed, see /var/log/cloud-init-output.log for details" > /var/log/init_script.log # restart in a minute, just giving the time to complete other stuffs for the first boot shutdown -r +1 # END CUSTOM SECTION init_script.sh EOF
- execute creation
terraform init terraform apply -input=false -var-file=vars.json
- destroy all resources
terraform destroy -input=false -var-file=vars.json
- Optionally, use a simple Makefile to setup the previous custom commands:
cat > Makefile << 'EOF'
export TF_PLUGIN_CACHE_DIR := ${HOME}/.terraform.d/plugin-cache
.PHONY: clean
clean:
@mkdir -p ${TF_PLUGIN_CACHE_DIR} || true
@rm -rf .terraform
.PHONY: init
init: clean
terraform init
.PHONY: apply
apply:
terraform apply -input=false -var-file=vars.json
.PHONY: destroy
destroy:
terraform destroy -input=false -var-file=vars.json
EOF
Use AWS EC2 custom instance
Find a good minimal AMI
aws --profile admin-user ec2 describe-images --owners amazon --filters 'Name=block-device-mapping.volume-size,Values=2' 'Name=description,Values=*Minimal*' --output table --query 'Images[?(Public)&&(State==`available`)&&(Architecture==`x86_64`)&&(VirtualizationType==`hvm`)&&(RootDeviceType==`ebs`)&&(CreationDate>=`2020-06-01`)].{id:ImageId,name:Name,desc:Description,date:CreationDate,owner:ImageOwnerAlias,arch:Architecture,virt:VirtualizationType,type:RootDeviceType,size:to_string(BlockDeviceMappings[*].Ebs.VolumeSize)} | reverse(sort_by([],&date)) | reverse(sort_by([],&name))'
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| DescribeImages |
+--------+---------------------------+----------------------------------------------------------------+------------------------+--------------------------------------------------------+---------+-------+-------+-------+
| arch | date | desc | id | name | owner | size | type | virt |
+--------+---------------------------+----------------------------------------------------------------+------------------------+--------------------------------------------------------+---------+-------+-------+-------+
| x86_64| 2020-09-22T02:18:38.000Z | Amazon Linux 2 AMI 2.0.20200917.0 x86_64 Minimal HVM ebs | ami-06e7a85d57337c9fa | amzn2-ami-minimal-hvm-2.0.20200917.0-x86_64-ebs | amazon | [2] | ebs | hvm |
| x86_64| 2020-09-04T02:37:57.000Z | Amazon Linux 2 AMI 2.0.20200904.0 x86_64 Minimal HVM ebs | ami-09b8e6a77f149e6ec | amzn2-ami-minimal-hvm-2.0.20200904.0-x86_64-ebs | amazon | [2] | ebs | hvm |
| x86_64| 2020-09-01T18:21:15.000Z | Amazon Linux 2 AMI 2.0.20200824.0 x86_64 Minimal HVM ebs | ami-01726776b01c7d9c2 | amzn2-ami-minimal-hvm-2.0.20200824.0-x86_64-ebs | amazon | [2] | ebs | hvm |
| x86_64| 2020-07-24T20:49:31.000Z | Amazon Linux 2 AMI 2.0.20200722.0 x86_64 Minimal HVM ebs | ami-09eb2d6d439ae885e | amzn2-ami-minimal-hvm-2.0.20200722.0-x86_64-ebs | amazon | [2] | ebs | hvm |
| x86_64| 2020-06-23T06:18:00.000Z | Amazon Linux 2 AMI 2.0.20200617.0 x86_64 Minimal HVM ebs | ami-0249782201f0ee367 | amzn2-ami-minimal-hvm-2.0.20200617.0-x86_64-ebs | amazon | [2] | ebs | hvm |
| x86_64| 2020-09-23T17:45:32.000Z | Amazon Linux AMI 2018.03.0.20200918.0 x86_64 Minimal HVM ebs | ami-09c00af71f9f4a141 | amzn-ami-minimal-hvm-2018.03.0.20200918.0-x86_64-ebs | amazon | [2] | ebs | hvm |
| x86_64| 2020-09-04T02:17:52.000Z | Amazon Linux AMI 2018.03.0.20200904.0 x86_64 Minimal HVM ebs | ami-0780ed21a81408761 | amzn-ami-minimal-hvm-2018.03.0.20200904.0-x86_64-ebs | amazon | [2] | ebs | hvm |
| x86_64| 2020-08-11T20:17:17.000Z | Amazon Linux AMI 2018.03.0.20200729.0 x86_64 Minimal HVM ebs | ami-02428347d30e4e928 | amzn-ami-minimal-hvm-2018.03.0.20200729.0-x86_64-ebs | amazon | [2] | ebs | hvm |
| x86_64| 2020-07-21T18:39:43.000Z | Amazon Linux AMI 2018.03.0.20200716.0 x86_64 Minimal HVM ebs | ami-0be996716886f5772 | amzn-ami-minimal-hvm-2018.03.0.20200716.0-x86_64-ebs | amazon | [2] | ebs | hvm |
| x86_64| 2020-06-16T06:36:13.000Z | Amazon Linux AMI 2018.03.0.20200602.1 x86_64 Minimal HVM ebs | ami-09e12379867321151 | amzn-ami-minimal-hvm-2018.03.0.20200602.1-x86_64-ebs | amazon | [2] | ebs | hvm |
+--------+---------------------------+----------------------------------------------------------------+------------------------+--------------------------------------------------------+---------+-------+-------+-------+
Create a simple EC2
- It will use an elastic ip to see how resources are reused
Create
- example using a basic Ubuntu 16.04 LTS AMI
mkdir example_org
cd example_org
cat > example_org.tf << 'EOF'
provider "aws" {
profile = "admin-user"
region = "eu-west-1"
}
resource "aws_instance" "example_org" {
# Ubuntu 16.04 LTS
ami = "ami-03746875d916becc0"
instance_type = "t3.nano"
}
EOF
terraform init # init terraform for the new project requirements
terraform plan # see changes to be applyed
terraform apply # apply changes
Edit
- example changing to Ubuntu 18.04 LST AMI
cat > example_org.tf << 'EOF'
provider "aws" {
profile = "admin-user"
region = "eu-west-1"
}
resource "aws_instance" "example_org" {
# # Ubuntu 16.04 LTS
# ami = "ami-03746875d916becc0"
# Ubuntu 18.04 AMI
ami = "ami-01e6a0b85de033c99"
instance_type = "t3.nano"
}
EOF
terraform apply
- example: adding an elastic-ip
cat >> example_org.tf << 'EOF'
resource "aws_eip" "ip" {
instance = "${aws_instance.example_org.id}"
}
EOF
terraform apply
- example: changing AMI again to minimal, keeping the Elastic IP
cat > example_org.tf << 'EOF'
provider "aws" {
profile = "admin-user"
region = "eu-west-1"
}
resource "aws_instance" "example_org" {
# # Ubuntu 16.04 LTS
# ami = "ami-03746875d916becc0"
# # Ubuntu 18.04 AMI
# ami = "ami-01e6a0b85de033c99"
# minimal hvm ebs x86_64 AMI 20190618
ami = "ami-0a3b59edf43c875be"
instance_type = "t3.nano"
}
resource "aws_eip" "ip" {
instance = "${aws_instance.example_org.id}"
}
EOF
terraform apply
Delete
- cleanup
terraform destroy
List existing instances
aws --profile admin-user ec2 describe-instances --output table --query 'Reservations[*].Instances[*].{Name:[Tags[?Key==`Name`].Value][0][0],Ip:PrivateIpAddress,Type:InstanceType,Zone:Placement.AvailabilityZone,State:State.Name,Launched:LaunchTime,VPC:VpcId,SN:SubnetId,SG:join(`,`, SecurityGroups[*].GroupId)}'
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| DescribeInstances |
+---------------+----------------------------+--------------------+------------------------------------------------------------------------------+------------------+----------+------------+---------------+--------------+
| Ip | Launched | Name | SG | SN | State | Type | VPC | Zone |
+---------------+----------------------------+--------------------+------------------------------------------------------------------------------+------------------+----------+------------+---------------+--------------+
| 172.31.20.253| 2020-05-04T02:14:52+00:00 | ec2_campisano.org | sg-0265edc543d39e5ce,sg-5c21bb23,sg-07f4afef9a2105c23,sg-09c096105eacbf55d | subnet-76c1a23e | running | t3a.micro | vpc-0bc5e36d | eu-west-1a |
+---------------+----------------------------+--------------------+------------------------------------------------------------------------------+------------------+----------+------------+---------------+--------------+
References
- https://aws.amazon.com/ec2/pricing/
- https://aws.amazon.com/ec2/pricing/on-demand/
- https://aws.amazon.com/ec2/pricing/reserved-instances/pricing/
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/apply_ri.html
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-market-concepts-buying.html#reserved-instances-process
- https://aws.amazon.com/efs/pricing/
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html
- https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Addresses
- https://calculator.s3.amazonaws.com/index.html?nc2=h_ql_pr
- https://console.aws.amazon.com/billing/home?region=eu-west-1#/
- https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/
- https://ping.varunagw.com/aws
- https://stackoverflow.com/a/40932906 (comparison)
- https://stackoverflow.com/a/46214617 (comparison)