Работа с AWS RDS и Terraform в Unix/Linux
Amazon Relational Database Service (Amazon RDS) позволяет легко настраивать, использовать и масштабировать реляционные базы данных в облаке. Сервис обеспечивает экономичное и масштабируемое использование ресурсов при одновременной автоматизации трудоемких задач администрирования, таких как выделение аппаратного обеспечения, настройка базы данных, установка исправлений и резервное копирование. Это позволяет сосредоточить внимание на приложениях, чтобы обеспечить необходимые им высокую производительность, высокую доступность, безопасность и совместимость.
Amazon RDS доступен в виде инстансов базы данных нескольких типов: оптимизированные по использованию памяти, по производительности или выполнению операций ввода-вывода – и позволяет на выбор использовать шесть известных ядер баз данных, в том числе Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle и Microsoft SQL Server. Можно использовать AWS Database Migration Service, чтобы легко переносить или реплицировать существующие базы данных в Amazon RDS.
Установка terraform в Unix/Linux
Установка крайне примитивная и я описал как это можно сделать тут:
Установка terraform в Unix/Linux
Так же, в данной статье, я создал скрипт для автоматической установки данного ПО. Он был протестирован на CentOS 6/7, Debian 8 и на Mac OS X. Все работает должным образом!
Чтобы получить помощь по использованию команд, выполните:
$ terraform
Usage: terraform [--version] [--help] <command> [args]
The available commands for execution are listed below.
The most common, useful commands are shown first, followed by
less common or more advanced commands. If you're just getting
started with Terraform, stick with the common commands. For the
other commands, please read the help and docs before usage.
Common commands:
apply Builds or changes infrastructure
console Interactive console for Terraform interpolations
destroy Destroy Terraform-managed infrastructure
env Workspace management
fmt Rewrites config files to canonical format
get Download and install modules for the configuration
graph Create a visual graph of Terraform resources
import Import existing infrastructure into Terraform
init Initialize a Terraform working directory
output Read an output from a state file
plan Generate and show an execution plan
providers Prints a tree of the providers used in the configuration
push Upload this Terraform module to Atlas to run
refresh Update local state file against real resources
show Inspect Terraform state or plan
taint Manually mark a resource for recreation
untaint Manually unmark a resource as tainted
validate Validates the Terraform files
version Prints the Terraform version
workspace Workspace management
All other commands:
debug Debug output management (experimental)
force-unlock Manually unlock the terraform state
state Advanced state management
Приступим к использованию!
Работа с AWS RDS и Terraform в Unix/Linux
У меня есть папка terraform, в ней у меня будут лежать провайдеры с которыми я буду работать. Т.к в этом примере я буду использовать AWS, то создам данную папку и перейду в нее. Далее, в этой папке, стоит создать:
$ mkdir examples modules
В папке examples, я буду хранить так званые «плейбуки» для разварачивания различных служб, например — zabbix-server, grafana, web-серверы и так далее. В modules директории, я буду хранить все необходимые модули.
Начнем писать модуль, но для этой задачи, я создам папку:
$ mkdir modules/rds
Переходим в нее:
$ cd modules/rds
Открываем файл:
$ vim rds.tf
В данный файл, вставляем:
#---------------------------------------------------
# Create AWS RDS instance(s)
#---------------------------------------------------
resource "aws_db_instance" "db_instance" {
count = "${var.create_rds_cluster ? 0 : var.number_of_instances_in_the_cluster}"
identifier = "${lower(var.name)}-rds-${lower(var.environment)}-${count.index+1}"
allocated_storage = "${var.allocated_storage}"
storage_type = "${var.storage_type}"
iops = "${var.iops}"
engine = "${var.engine}"
engine_version = "${var.engine_version}"
instance_class = "${var.instance_class}"
replicate_source_db = "${var.replicate_source_db}"
backup_retention_period = "${var.backup_retention_period}"
backup_window = "${var.backup_window}"
copy_tags_to_snapshot = "${var.copy_tags_to_snapshot}"
skip_final_snapshot = "${var.skip_final_snapshot}"
final_snapshot_identifier = "${lower(var.name)}-rds-${lower(var.environment)}-${md5(timestamp())}"
name = "${var.db_name}"
username = "${var.db_username}"
password = "${var.db_password}"
port = "${lookup(var.default_ports, var.engine)}"
character_set_name = "${var.character_set_name}"
vpc_security_group_ids = ["${var.vpc_security_group_ids}"]
db_subnet_group_name = "${var.db_subnet_group_name == "" ? aws_db_subnet_group.db_subnet_group.id : var.db_subnet_group_name}"
parameter_group_name = "${length(var.parameter_group_name) > 0 ? var.parameter_group_name : aws_db_parameter_group.db_parameter_group.id}"
publicly_accessible = "${var.publicly_accessible}"
storage_encrypted = "${var.storage_encrypted}"
multi_az = "${var.multi_az}"
allow_major_version_upgrade = "${var.allow_major_version_upgrade}"
auto_minor_version_upgrade = "${var.auto_minor_version_upgrade}"
apply_immediately = "${var.apply_immediately}"
maintenance_window = "${var.maintenance_window}"
monitoring_interval = "${var.monitoring_interval}"
tags {
Name = "${var.name}"
Environment = "${var.environment}"
Orchestration = "${var.orchestration}"
Createdby = "${var.createdby}"
}
lifecycle {
create_before_destroy = true,
ignore_changes = ["final_snapshot_identifier", "replicate_source_db"],
}
depends_on = ["aws_db_subnet_group.db_subnet_group", "aws_db_parameter_group.db_parameter_group"]
}
#---------------------------------------------------
# Create AWS RDS cluster
#---------------------------------------------------
resource "aws_rds_cluster_instance" "rds_cluster_instance" {
count = "${var.create_rds_cluster ? var.number_of_instances_in_the_cluster : 0}"
identifier = "${lower(var.name)}-cluster-${lower(var.environment)}-${count.index+1}"
cluster_identifier = "${aws_rds_cluster.rds_cluster.id}"
instance_class = "${var.instance_class}"
db_subnet_group_name = "${var.db_subnet_group_name == "" ? aws_db_subnet_group.db_subnet_group.id : var.db_subnet_group_name}"
apply_immediately = "${var.apply_immediately}"
db_parameter_group_name = "${var.instance_parameter_group_name == "" ? aws_db_parameter_group.db_parameter_group.id : var.instance_parameter_group_name}"
tags {
Name = "${lower(var.name)}-cluster-${lower(var.environment)}-${count.index+1}"
Environment = "${var.environment}"
Orchestration = "${var.orchestration}"
Createdby = "${var.createdby}"
}
depends_on = ["aws_rds_cluster.rds_cluster", "aws_db_subnet_group.db_subnet_group", "aws_db_parameter_group.db_parameter_group"]
}
resource "aws_rds_cluster" "rds_cluster" {
count = "${var.create_rds_cluster ? 1 : 0}"
cluster_identifier = "${lower(var.name)}-cluster-${lower(var.environment)}"
engine = "${var.engine}"
engine_version = "${var.engine_version}"
backup_retention_period = "${var.backup_retention_period}"
preferred_backup_window = "${var.backup_window}"
skip_final_snapshot = "${var.skip_final_snapshot}"
final_snapshot_identifier = "${lower(var.name)}-cluster-${lower(var.environment)}-${md5(timestamp())}"
db_subnet_group_name = "${var.db_subnet_group_name == "" ? aws_db_subnet_group.db_subnet_group.id : var.db_subnet_group_name}"
vpc_security_group_ids = ["${var.vpc_security_group_ids}"]
storage_encrypted = "${var.storage_encrypted}"
apply_immediately = "${var.apply_immediately}"
db_cluster_parameter_group_name = "${length(var.db_cluster_parameter_group_name) > 0 ? aws_db_parameter_group.db_parameter_group.id : var.db_cluster_parameter_group_name}"
availability_zones = ["${split(",", (lookup(var.availability_zones, var.region)))}"]
database_name = "${var.db_name}"
master_username = "${var.db_username}"
master_password = "${var.db_password}"
tags {
Name = "${lower(var.name)}-cluster-${lower(var.environment)}"
Environment = "${var.environment}"
Orchestration = "${var.orchestration}"
Createdby = "${var.createdby}"
}
lifecycle {
create_before_destroy = true,
ignore_changes = ["final_snapshot_identifier"],
}
depends_on = ["aws_db_subnet_group.db_subnet_group", "aws_db_parameter_group.db_parameter_group"]
}
#---------------------------------------------------
# Create AWS DB subnet group
#---------------------------------------------------
resource "aws_db_subnet_group" "db_subnet_group" {
count = "${var.db_subnet_group_name == "" ? 1 : 0}"
name = "${lower(var.name)}-db_subnet_group-for-${var.create_rds_cluster ? "cluster" : "rds"}-${lower(var.environment)}"
description = "My ${lower(var.name)}-db_subnet_group-for-${var.create_rds_cluster ? "cluster" : "rds"}-${lower(var.environment)} group of subnets"
subnet_ids = ["${var.subnet_ids}"]
tags {
Name = "${lower(var.name)}-db_subnet_group-${lower(var.environment)}"
Environment = "${var.environment}"
Orchestration = "${var.orchestration}"
Createdby = "${var.createdby}"
}
}
#---------------------------------------------------
# Create AWS DB parameter group
#---------------------------------------------------
resource "aws_db_parameter_group" "db_parameter_group" {
count = "${length(var.parameter_group_name) > 0 ? 0 : 1}"
#name_prefix = "${lower(var.name)}-${lower(var.environment)}-"
name = "${lower(var.name)}-db-parameter-group-for-${var.create_rds_cluster ? "cluster" : "rds"}-${lower(var.environment)}"
description = "RDS ${lower(var.name)}-db_parameter_group-for-${var.create_rds_cluster ? "cluster" : "rds"}-${lower(var.environment)} parameter group for ${var.engine}"
family = "${var.db_group_family[var.engine]}"
parameter = "${var.default_db_parameters[var.engine]}"
tags {
Name = "${lower(var.name)}-db_parameter_group-${lower(var.environment)}"
Environment = "${var.environment}"
Orchestration = "${var.orchestration}"
Createdby = "${var.createdby}"
}
}
Открываем файл:
$ vim variables.tf
И прописываем:
#-----------------------------------------------------------
# Global or/and default variables
#-----------------------------------------------------------
variable "name" {
description = "Name to be used on all resources as prefix"
default = "TEST-RDS"
}
variable "region" {
description = "The region where to deploy this code (e.g. us-east-1)."
default = "us-east-1"
}
variable "environment" {
description = "Environment for service"
default = "STAGE"
}
variable "orchestration" {
description = "Type of orchestration"
default = "Terraform"
}
variable "createdby" {
description = "Created by"
default = "Vitaliy Natarov"
}
variable "create_rds_cluster" {
description = "If true, then rds cluster will create"
default = false
}
variable "number_of_instances_in_the_cluster" {
description = "Number of nodes in the cluster"
default = "1"
}
variable "db_cluster_parameter_group_name" {
description = "A cluster parameter group to associate with the cluster."
default = ""
}
variable "instance_parameter_group_name" {
description = "A instance parameter group to associate"
default = ""
}
variable "subnet_ids" {
description = "subnet IDs"
type = "list"
}
variable "identifier" {
description = "The name of the RDS instance, if omitted, Terraform will assign a random, unique identifier."
default = ""
}
variable "identifier_prefix" {
description = "Creates a unique identifier beginning with the specified prefix. Conflicts with identifer."
default = ""
}
variable "allocated_storage" {
description = "The allocated storage in gigabytes."
default = "20"
}
variable "storage_type" {
description = "One of 'standard' (magnetic), 'gp2' (general purpose SSD), or 'io1' (provisioned IOPS SSD). The default is 'io1' if iops is specified, 'standard' if not. Note that this behaviour is different from the AWS web console, where the default is 'gp2'."
default = "gp2"
}
variable "iops" {
description = "The amount of provisioned IOPS. Setting this implies a storage_type of 'io1', default is 0 if rds storage type is not io1"
default = "0"
}
variable "engine" {
description = "The database engine to use (mysql, postgres etc)"
default = "mysql"
}
variable "engine_version" {
description = "The engine version to use."
default = "5.6.37"
}
variable "instance_class" {
description = "The instance type of the RDS instance."
default = "db.t2.micro"
}
variable "db_name" {
description = "The name of the database to create when the DB instance is created. If this parameter is not specified, no database is created in the DB instance. Note that this does not apply for Oracle or SQL Server engines."
default = "db_name_test"
}
variable "db_username" {
description = "Username for the master DB user."
default = "root"
}
variable "db_password" {
description = "Password for the master DB user. Note that this may show up in logs, and it will be stored in the state file."
default = "ROot666roOT"
}
variable "default_ports" {
description = "Default database ports"
type = "map"
default = {
mysql = "3306"
postgres = "5432"
oracle = "1521"
}
}
#lookup parameters for specic RDS types
variable "default_db_parameters" {
default = {
mysql = [
{
name = "slow_query_log"
value = "1"
},
{
name = "long_query_time"
value = "1"
},
{
name = "general_log"
value = "0"
},
{
name = "log_output"
value = "FILE"
},
{
name = "character_set_server"
value = "utf8"
},
{
name = "character_set_client"
value = "utf8"
},
]
aurora = [
{
name = "slow_query_log"
value = "1"
},
{
name = "long_query_time"
value = "1"
},
{
name = "general_log"
value = "0"
},
{
name = "log_output"
value = "FILE"
},
]
postgres = []
oracle = []
}
}
variable "db_group_family" {
description = "Set DB group family"
type = "map"
default = {
mysql = "mysql5.6"
postgres = "postgres9.6"
oracle = "oracle-ee-12.1"
aurora = "aurora5.6"
}
}
variable "character_set_name" {
description = "The character set name to use for DB encoding in Oracle instances. This can't be changed"
#default = "utf8"
default = ""
}
variable "db_subnet_group_name" {
description = "Name of DB subnet group. DB instance will be created in the VPC associated with the DB subnet group. If unspecified, will be created in the default VPC, or in EC2 Classic, if available."
default = ""
}
variable "parameter_group_name" {
description = "Name of the DB parameter group to associate."
#default = "default.mysql5.6"
default = ""
}
variable "publicly_accessible" {
description = "Bool to control if instance is publicly accessible. Default is false."
default = "false"
}
variable "storage_encrypted" {
description = "Specifies whether the DB instance is encrypted. The default is false if not specified."
default = "false"
}
variable "vpc_security_group_ids" {
description = "List of VPC security groups to associate."
type = "list"
default = []
}
variable "iam_database_authentication_enabled" {
description = "Specifies whether or mappings of AWS Identity and Access Management (IAM) accounts to database accounts is enabled."
default = []
}
variable "availability_zones" {
description = "Availability zones for AWS ASG"
type = "map"
default = {
us-east-1 = "us-east-1b,us-east-1a"
us-east-2 = "us-east-2a,eu-east-2b,eu-east-2c"
us-west-1 = "us-west-1a,us-west-1c"
us-west-2 = "us-west-2a,us-west-2b,us-west-2c"
ca-central-1 = "ca-central-1a,ca-central-1b"
eu-west-1 = "eu-west-1a,eu-west-1b,eu-west-1c"
eu-west-2 = "eu-west-2a,eu-west-2b"
eu-central-1 = "eu-central-1a,eu-central-1b,eu-central-1c"
ap-south-1 = "ap-south-1a,ap-south-1b"
sa-east-1 = "sa-east-1a,sa-east-1c"
ap-northeast-1 = "ap-northeast-1a,ap-northeast-1c"
ap-southeast-1 = "ap-southeast-1a,ap-southeast-1b"
ap-southeast-2 = "ap-southeast-2a,ap-southeast-2b,ap-southeast-2c"
ap-northeast-1 = "ap-northeast-1a,ap-northeast-1c"
ap-northeast-2 = "ap-northeast-2a,ap-northeast-2c"
}
}
variable "backup_retention_period" {
description = "The backup retention period (in days)"
default = "0"
}
variable "backup_window" {
description = "The daily time range (in UTC) during which automated backups are created if they are enabled. Example: '09:46-10:16'. Must not overlap with maintenance_window."
## 12:00AM-12:30AM ET
default = "09:00-09:30"
}
variable "maintenance_window" {
description = "The daily time range (in UTC) during which maintenance window are enabled. Must not overlap with backup_window."
# SUN 12:30AM-01:30AM ET
default = "sun:04:30-sun:05:30"
}
variable "monitoring_interval" {
description = "To disable collecting Enhanced Monitoring metrics, specify 0. The default is 0. Valid Values: 0, 1, 5, 10, 15, 30, 60."
default = "0"
}
variable "replicate_source_db" {
description = "Specifies that this resource is a Replicate database, and to use this value as the source database. This correlates to the identifier of another Amazon RDS Database to replicate."
default = ""
}
variable "skip_final_snapshot" {
description = "Determines whether a final DB snapshot is created before the DB instance is deleted. If true is specified, no DBSnapshot is created. If false is specified, a DB snapshot is created before the DB instance is deleted, using the value from final_snapshot_identifier. Default is false."
default = "false"
}
variable "copy_tags_to_snapshot" {
description = "On delete, copy all Instance tags to the final snapshot (if final_snapshot_identifier is specified). Default is false."
default = "false"
}
#variable "final_snapshot_identifier" {
# description = "The name of your final DB snapshot when this DB instance is deleted. If omitted, no final snapshot will be made."
# default = "2018"
#}
variable "multi_az" {
description = "If the RDS instance is multi AZ enabled."
default = "false"
}
variable "allow_major_version_upgrade" {
description = "Indicates that major version upgrades are allowed. Changing this parameter does not result in an outage and the change is asynchronously applied as soon as possible."
default = "true"
}
variable "auto_minor_version_upgrade" {
description = "Indicates that minor engine upgrades will be applied automatically to the DB instance during the maintenance window. Defaults to true."
default = "false"
}
variable "apply_immediately" {
description = "Specifies whether any database modifications are applied immediately, or during the next maintenance window. Default is false"
default = "false"
}
Собственно в этом файле храняться все переменные. Спасибо кэп!
Открываем последний файл:
$ vim outputs.tf
и в него вставить нужно следующие строки:
output "rds_ids" {
value = "${aws_db_instance.db_instance.*.id}"
}
output "rds_arns" {
value = "${aws_db_instance.db_instance.*.arn}"
}
output "rds_addresses" {
value = "${aws_db_instance.db_instance.*.address}"
}
output "endpoints" {
value = "${aws_rds_cluster.rds_cluster.*.endpoint}"
}
output "reader_endpoints" {
value = "${aws_rds_cluster.rds_cluster.*.reader_endpoint}"
}
output "aws_db_subnet_group_ids" {
value = "${aws_db_subnet_group.db_subnet_group.*.id}"
}
output "db_parameter_groups" {
value = "aws_db_parameter_group.db_parameter_group.*.id"
}
output "hosted_zone_ids" {
value = "${aws_db_instance.db_instance.*.hosted_zone_id}"
}
Переходим теперь в папку aws/examples и создадим еще одну папку для проверки написанного чуда:
$ mkdir rds && cd $_
Внутри созданной папки открываем файл:
$ vim main.tf
И вставим в него следующий код:
#
# MAINTAINER Vitaliy Natarov "vitaliy.natarov@yahoo.com"
#
terraform {
required_version = "> 0.9.0"
}
provider "aws" {
region = "us-east-1"
# alias = "us-east-1"
shared_credentials_file = "${pathexpand("~/.aws/credentials")}"
# access_key = "${var.aws_access_key}"
# secret_key = "${var.aws_secret_key}"
}
module "iam" {
source = "../../modules/iam"
name = "TEST-AIM"
region = "us-east-1"
environment = "PROD"
aws_iam_role-principals = [
"ec2.amazonaws.com",
"monitoring.rds.amazonaws.com",
]
aws_iam_policy-actions = [
"cloudwatch:GetMetricStatistics",
"logs:DescribeLogStreams",
"logs:GetLogEvents",
"rds:*",
]
}
module "vpc" {
source = "../../modules/vpc"
name = "TEST-VPC"
environment = "PROD"
# VPC
instance_tenancy = "default"
enable_dns_support = "true"
enable_dns_hostnames = "true"
assign_generated_ipv6_cidr_block = "false"
enable_classiclink = "false"
vpc_cidr = "172.31.0.0/16"
private_subnet_cidrs = ["172.31.64.0/20"]
public_subnet_cidrs = ["172.31.0.0/20"]
availability_zones = ["us-east-1a", "us-east-1b"]
enable_all_egress_ports = "true"
allowed_ports = ["80", "3306", "8080", "443"]
map_public_ip_on_launch = "true"
#Internet-GateWay
enable_internet_gateway = "true"
#NAT
enable_nat_gateway = "false"
single_nat_gateway = "false"
#VPN
enable_vpn_gateway = "false"
#DHCP
enable_dhcp_options = "false"
# EIP
enable_eip = "false"
}
module "rds" {
source = "../../modules/rds"
name = "Test-DB"
region = "us-east-1"
environment = "PROD"
subnet_ids = ["subnet-11ed234b", "subnet-34eddf51"]
##subnet_ids = ["module.vpc.vpc-privatesubnet-ids"]
# Single node(s)
#create_rds_cluster = false
#engine = "mysql"
## For cluster
create_rds_cluster = true
## aurora or aurora-postgresql. !!!!Tested just for aurora!!!!
engine = "aurora"
instance_class = "db.t2.small"
}
PS: Тут имеются вспомогательные модули. Я описывал работу тут:
Работа с AWS IAM и Terraform в Unix/Linux
Работа с AWS VPC и Terraform в Unix/Linux
Еще полезности:
Работа с AWS EC2 и Terraform в Unix/Linux
Работа с AWS ASG(auto scaling group) и Terraform в Unix/Linux
Работа с AWS ELB и Terraform в Unix/Linux
Работа с AWS Route53 и Terraform в Unix/Linux
Работа с AWS S3 и Terraform в Unix/Linux
Все уже написано и готово к использованию. Ну что, начнем тестирование. В папке с вашим плейбуком, выполняем:
$ terraform init
Этим действием я инициализирую проект. Затем, подтягиваю модуль:
$ terraform get
PS: Для обновление изменений в самом модуле, можно выполнять:
$ terraform get -update
Проверим валидацию:
$ terraform validate
Запускем прогон:
$ terraform plan
Мне вывело что все у меня хорошо и можно запускать деплой:
$ terraform apply
Как видно с вывода, — все прошло гладко! Чтобы удалить созданное творение, можно выполнить:
$ terraform destroy
Весь материал аплоаджу в github аккаунт для удобства использования:
$ git clone https://github.com/SebastianUA/terraform.git
Вот и все на этом. Данная статья «Работа с AWS RDS и Terraform в Unix/Linux» завершена.