Работа с AWS RDS и Terraform в Unix/Linux

Работа с AWS RDS и Terraform в Unix/Linux

Amazon Relational Database Service (Amazon RDS) позволяет легко настраивать, использовать и масштабировать реляционные базы данных в облаке. Сервис обеспечивает экономичное и масштабируемое использование ресурсов при одновременной автоматизации трудоемких задач администрирования, таких как выделение аппаратного обеспечения, настройка базы данных, установка исправлений и резервное копирование. Это позволяет сосредоточить внимание на приложениях, чтобы обеспечить необходимые им высокую производительность, высокую доступность, безопасность и совместимость.

Amazon RDS доступен в виде инстансов базы данных нескольких типов: оптимизированные по использованию памяти, по производительности или выполнению операций ввода-вывода – и позволяет на выбор использовать шесть известных ядер баз данных, в том числе Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle и Microsoft SQL Server. Можно использовать AWS Database Migration Service, чтобы легко переносить или реплицировать существующие базы данных в Amazon RDS.

Установка terraform в Unix/Linux

Установка крайне примитивная и я описал как это можно сделать тут:

Установка terraform в Unix/Linux

Так же, в данной статье, я создал скрипт для автоматической установки данного ПО. Он был протестирован на CentOS 6/7, Debian 8 и на Mac OS X. Все работает должным образом!

Чтобы получить помощь по использованию команд, выполните:

$ terraform
Usage: terraform [--version] [--help] <command> [args]

The available commands for execution are listed below.
The most common, useful commands are shown first, followed by
less common or more advanced commands. If you're just getting
started with Terraform, stick with the common commands. For the
other commands, please read the help and docs before usage.

Common commands:
    apply              Builds or changes infrastructure
    console            Interactive console for Terraform interpolations
    destroy            Destroy Terraform-managed infrastructure
    env                Workspace management
    fmt                Rewrites config files to canonical format
    get                Download and install modules for the configuration
    graph              Create a visual graph of Terraform resources
    import             Import existing infrastructure into Terraform
    init               Initialize a Terraform working directory
    output             Read an output from a state file
    plan               Generate and show an execution plan
    providers          Prints a tree of the providers used in the configuration
    push               Upload this Terraform module to Atlas to run
    refresh            Update local state file against real resources
    show               Inspect Terraform state or plan
    taint              Manually mark a resource for recreation
    untaint            Manually unmark a resource as tainted
    validate           Validates the Terraform files
    version            Prints the Terraform version
    workspace          Workspace management

All other commands:
    debug              Debug output management (experimental)
    force-unlock       Manually unlock the terraform state
    state              Advanced state management

Приступим к использованию!

Работа с AWS RDS и Terraform в Unix/Linux

У меня есть папка terraform, в ней у меня будут лежать провайдеры с которыми я буду работать. Т.к в этом примере я буду использовать AWS, то создам данную папку и перейду в нее. Далее, в этой папке, стоит создать:

$ mkdir examples modules

В папке examples, я буду хранить так званые «плейбуки» для разварачивания различных служб, например — zabbix-server, grafana, web-серверы и так далее. В modules директории, я буду хранить все необходимые модули.

Начнем писать модуль, но для этой задачи, я создам папку:

$  mkdir modules/rds

Переходим в нее:

$ cd modules/rds

Открываем файл:

$ vim rds.tf

В данный файл, вставляем:

#---------------------------------------------------
# Create AWS RDS instance(s)
#---------------------------------------------------
resource "aws_db_instance" "db_instance" {
    count                       = "${var.create_rds_cluster ? 0 : var.number_of_instances_in_the_cluster}"

    identifier                  = "${lower(var.name)}-rds-${lower(var.environment)}-${count.index+1}"
    allocated_storage           = "${var.allocated_storage}"
    storage_type                = "${var.storage_type}"
    iops                        = "${var.iops}"
    engine                      = "${var.engine}"
    engine_version              = "${var.engine_version}"
    instance_class              = "${var.instance_class}"
    replicate_source_db         = "${var.replicate_source_db}"
    backup_retention_period     = "${var.backup_retention_period}"
    backup_window               = "${var.backup_window}"
    copy_tags_to_snapshot       = "${var.copy_tags_to_snapshot}"
    skip_final_snapshot         = "${var.skip_final_snapshot}"
    final_snapshot_identifier   = "${lower(var.name)}-rds-${lower(var.environment)}-${md5(timestamp())}"

    name                    = "${var.db_name}"
    username                = "${var.db_username}"
    password                = "${var.db_password}"
    port                    = "${lookup(var.default_ports, var.engine)}"
    character_set_name      = "${var.character_set_name}"

    vpc_security_group_ids  = ["${var.vpc_security_group_ids}"]
    db_subnet_group_name    = "${var.db_subnet_group_name == "" ? aws_db_subnet_group.db_subnet_group.id : var.db_subnet_group_name}"
    parameter_group_name    = "${length(var.parameter_group_name) > 0 ? var.parameter_group_name : aws_db_parameter_group.db_parameter_group.id}"
    publicly_accessible     = "${var.publicly_accessible}"
    storage_encrypted       = "${var.storage_encrypted}"
    multi_az                = "${var.multi_az}"

    allow_major_version_upgrade = "${var.allow_major_version_upgrade}"
    auto_minor_version_upgrade  = "${var.auto_minor_version_upgrade}"
    apply_immediately           = "${var.apply_immediately}"
    maintenance_window          = "${var.maintenance_window}"

    monitoring_interval = "${var.monitoring_interval}"

    tags {
        Name            = "${var.name}"
        Environment     = "${var.environment}"
        Orchestration   = "${var.orchestration}"
        Createdby       = "${var.createdby}"
    }

    lifecycle {
        create_before_destroy   = true,
        ignore_changes          = ["final_snapshot_identifier", "replicate_source_db"],
    }

    depends_on  = ["aws_db_subnet_group.db_subnet_group", "aws_db_parameter_group.db_parameter_group"]
}
#---------------------------------------------------
# Create AWS RDS cluster
#---------------------------------------------------
resource "aws_rds_cluster_instance" "rds_cluster_instance" {
    count               = "${var.create_rds_cluster ? var.number_of_instances_in_the_cluster : 0}"

    identifier          = "${lower(var.name)}-cluster-${lower(var.environment)}-${count.index+1}"
    cluster_identifier  = "${aws_rds_cluster.rds_cluster.id}"
    instance_class      = "${var.instance_class}"

    db_subnet_group_name    = "${var.db_subnet_group_name == "" ? aws_db_subnet_group.db_subnet_group.id : var.db_subnet_group_name}"
    apply_immediately       = "${var.apply_immediately}"
    db_parameter_group_name = "${var.instance_parameter_group_name == "" ? aws_db_parameter_group.db_parameter_group.id : var.instance_parameter_group_name}"

    tags {
        Name            = "${lower(var.name)}-cluster-${lower(var.environment)}-${count.index+1}"
        Environment     = "${var.environment}"
        Orchestration   = "${var.orchestration}"
        Createdby       = "${var.createdby}"
    }

    depends_on          = ["aws_rds_cluster.rds_cluster", "aws_db_subnet_group.db_subnet_group", "aws_db_parameter_group.db_parameter_group"]
}

resource "aws_rds_cluster" "rds_cluster" {
    count               = "${var.create_rds_cluster ? 1 : 0}"

    cluster_identifier  = "${lower(var.name)}-cluster-${lower(var.environment)}"
    engine                      = "${var.engine}"
    engine_version              = "${var.engine_version}"

    backup_retention_period     = "${var.backup_retention_period}"
    preferred_backup_window     = "${var.backup_window}"
    skip_final_snapshot         = "${var.skip_final_snapshot}"
    final_snapshot_identifier   = "${lower(var.name)}-cluster-${lower(var.environment)}-${md5(timestamp())}"

    db_subnet_group_name        = "${var.db_subnet_group_name == "" ? aws_db_subnet_group.db_subnet_group.id : var.db_subnet_group_name}"
    vpc_security_group_ids      = ["${var.vpc_security_group_ids}"]

    storage_encrypted               = "${var.storage_encrypted}"
    apply_immediately               = "${var.apply_immediately}"
    db_cluster_parameter_group_name = "${length(var.db_cluster_parameter_group_name) > 0 ? aws_db_parameter_group.db_parameter_group.id : var.db_cluster_parameter_group_name}"
    availability_zones  = ["${split(",", (lookup(var.availability_zones, var.region)))}"]

    database_name       = "${var.db_name}"
    master_username     = "${var.db_username}"
    master_password     = "${var.db_password}"

    tags {
        Name            = "${lower(var.name)}-cluster-${lower(var.environment)}"
        Environment     = "${var.environment}"
        Orchestration   = "${var.orchestration}"
        Createdby       = "${var.createdby}"
    }

    lifecycle {
        create_before_destroy   = true,
        ignore_changes          = ["final_snapshot_identifier"],
    }
    depends_on          = ["aws_db_subnet_group.db_subnet_group", "aws_db_parameter_group.db_parameter_group"]
}
#---------------------------------------------------
# Create AWS DB subnet group
#---------------------------------------------------
resource "aws_db_subnet_group" "db_subnet_group" {
    count       = "${var.db_subnet_group_name == "" ? 1 : 0}"

    name        = "${lower(var.name)}-db_subnet_group-for-${var.create_rds_cluster ? "cluster" : "rds"}-${lower(var.environment)}"
    description = "My ${lower(var.name)}-db_subnet_group-for-${var.create_rds_cluster ? "cluster" : "rds"}-${lower(var.environment)} group of subnets"
    subnet_ids  = ["${var.subnet_ids}"]

    tags {
        Name            = "${lower(var.name)}-db_subnet_group-${lower(var.environment)}"
        Environment     = "${var.environment}"
        Orchestration   = "${var.orchestration}"
        Createdby       = "${var.createdby}"
  }
}
#---------------------------------------------------
# Create AWS DB parameter group
#---------------------------------------------------
resource "aws_db_parameter_group" "db_parameter_group" {
    count       = "${length(var.parameter_group_name) > 0 ? 0 : 1}"

    #name_prefix = "${lower(var.name)}-${lower(var.environment)}-"
    name        = "${lower(var.name)}-db-parameter-group-for-${var.create_rds_cluster ? "cluster" : "rds"}-${lower(var.environment)}"
    description = "RDS ${lower(var.name)}-db_parameter_group-for-${var.create_rds_cluster ? "cluster" : "rds"}-${lower(var.environment)} parameter group for ${var.engine}"
    family      = "${var.db_group_family[var.engine]}"
    parameter   = "${var.default_db_parameters[var.engine]}"

    tags {
        Name            = "${lower(var.name)}-db_parameter_group-${lower(var.environment)}"
        Environment     = "${var.environment}"
        Orchestration   = "${var.orchestration}"
        Createdby       = "${var.createdby}"
    }

}

Открываем файл:

$ vim variables.tf

И прописываем:

#-----------------------------------------------------------
# Global or/and default variables
#-----------------------------------------------------------
variable "name" {
  description = "Name to be used on all resources as prefix"
  default     = "TEST-RDS"
}

variable "region" {
  description = "The region where to deploy this code (e.g. us-east-1)."
  default     = "us-east-1"
}

variable "environment" {
    description = "Environment for service"
    default     = "STAGE"
}

variable "orchestration" {
    description = "Type of orchestration"
    default     = "Terraform"
}

variable "createdby" {
    description = "Created by"
    default     = "Vitaliy Natarov"
}

variable "create_rds_cluster" {
    description = "If true, then rds cluster will create"
    default     = false
}

variable "number_of_instances_in_the_cluster" {
    description = "Number of nodes in the cluster"
    default     = "1"
}

variable "db_cluster_parameter_group_name" {
    description = "A cluster parameter group to associate with the cluster."
    default     = ""
}

variable "instance_parameter_group_name" {
    description = "A instance parameter group to associate"
    default     = ""
}

variable "subnet_ids" {
    description = "subnet IDs"
    type        = "list"
}

variable "identifier" {
    description = "The name of the RDS instance, if omitted, Terraform will assign a random, unique identifier."
    default     = ""
}

variable "identifier_prefix" {
    description = "Creates a unique identifier beginning with the specified prefix. Conflicts with identifer."
    default     = ""
}

variable "allocated_storage" {
    description = "The allocated storage in gigabytes."
    default     = "20"
}

variable "storage_type" {
    description = "One of 'standard' (magnetic), 'gp2' (general purpose SSD), or 'io1' (provisioned IOPS SSD). The default is 'io1' if iops is specified, 'standard' if not. Note that this behaviour is different from the AWS web console, where the default is 'gp2'."
    default     = "gp2"
}

variable "iops" {
    description = "The amount of provisioned IOPS. Setting this implies a storage_type of 'io1', default is 0 if rds storage type is not io1"
    default     = "0"
}

variable "engine" {
    description = "The database engine to use (mysql, postgres etc)"
    default     = "mysql"
}

variable "engine_version" {
    description = "The engine version to use."
    default     = "5.6.37"
}

variable "instance_class" {
    description = "The instance type of the RDS instance."
    default     = "db.t2.micro"
}

variable "db_name" {
    description = "The name of the database to create when the DB instance is created. If this parameter is not specified, no database is created in the DB instance. Note that this does not apply for Oracle or SQL Server engines."
    default     = "db_name_test"
}

variable "db_username" {
    description = "Username for the master DB user."
    default     = "root"
}

variable "db_password" {
    description = "Password for the master DB user. Note that this may show up in logs, and it will be stored in the state file."
    default     = "ROot666roOT"
}

variable "default_ports" {
    description = "Default database ports"
    type        = "map"
    default     = {
        mysql    = "3306"
        postgres = "5432"
        oracle   = "1521"
    }
}

#lookup parameters for specic RDS types
variable "default_db_parameters" {
    default = {
        mysql = [
        {
            name  = "slow_query_log"
            value = "1"
        },
        {
            name  = "long_query_time"
            value = "1"
        },
        {
            name  = "general_log"
            value = "0"
        },
        {
            name  = "log_output"
            value = "FILE"
        },
        {
            name  = "character_set_server"
            value = "utf8"
        },
        {
            name  = "character_set_client"
            value = "utf8"
        },
        ]
        aurora        = [
        {
            name  = "slow_query_log"
            value = "1"
        },
        {
            name  = "long_query_time"
            value = "1"
        },
        {
            name  = "general_log"
            value = "0"
        },
        {
            name  = "log_output"
            value = "FILE"
        },
        ]
        postgres      = []
        oracle        = []
    }
}
variable "db_group_family" {
    description = "Set DB group family"
    type        = "map"
    default     = {
        mysql     = "mysql5.6"
        postgres  = "postgres9.6"
        oracle    = "oracle-ee-12.1"
        aurora    = "aurora5.6"
    }
}

variable "character_set_name" {
    description = "The character set name to use for DB encoding in Oracle instances. This can't be changed"
    #default     = "utf8"
    default     = ""
}

variable "db_subnet_group_name" {
    description = "Name of DB subnet group. DB instance will be created in the VPC associated with the DB subnet group. If unspecified, will be created in the default VPC, or in EC2 Classic, if available."
    default     = ""
}

variable "parameter_group_name" {
    description = "Name of the DB parameter group to associate."
    #default     = "default.mysql5.6"
    default     = ""
}

variable "publicly_accessible" {
    description = "Bool to control if instance is publicly accessible. Default is false."
    default     = "false"
}

variable "storage_encrypted" {
    description = "Specifies whether the DB instance is encrypted. The default is false if not specified."
    default     = "false"
}

variable "vpc_security_group_ids" {
    description = "List of VPC security groups to associate."
    type        = "list"
    default     = []
}

variable "iam_database_authentication_enabled" {
    description = "Specifies whether or mappings of AWS Identity and Access Management (IAM) accounts to database accounts is enabled."
    default     = []
}

variable "availability_zones" {
    description = "Availability zones for AWS ASG"
    type        = "map"
    default     = {
        us-east-1      = "us-east-1b,us-east-1a"
        us-east-2      = "us-east-2a,eu-east-2b,eu-east-2c"
        us-west-1      = "us-west-1a,us-west-1c"
        us-west-2      = "us-west-2a,us-west-2b,us-west-2c"
        ca-central-1   = "ca-central-1a,ca-central-1b"
        eu-west-1      = "eu-west-1a,eu-west-1b,eu-west-1c"
        eu-west-2      = "eu-west-2a,eu-west-2b"
        eu-central-1   = "eu-central-1a,eu-central-1b,eu-central-1c"
        ap-south-1     = "ap-south-1a,ap-south-1b"
        sa-east-1      = "sa-east-1a,sa-east-1c"
        ap-northeast-1 = "ap-northeast-1a,ap-northeast-1c"
        ap-southeast-1 = "ap-southeast-1a,ap-southeast-1b"
        ap-southeast-2 = "ap-southeast-2a,ap-southeast-2b,ap-southeast-2c"
        ap-northeast-1 = "ap-northeast-1a,ap-northeast-1c"
        ap-northeast-2 = "ap-northeast-2a,ap-northeast-2c"
    }
}

variable "backup_retention_period" {
    description = "The backup retention period (in days)"
    default     = "0"
}

variable "backup_window" {
    description = "The daily time range (in UTC) during which automated backups are created if they are enabled. Example: '09:46-10:16'. Must not overlap with maintenance_window."
    ## 12:00AM-12:30AM ET
    default     = "09:00-09:30"
}

variable "maintenance_window" {
    description = "The daily time range (in UTC) during which maintenance window are enabled. Must not overlap with backup_window."
    # SUN 12:30AM-01:30AM ET
    default = "sun:04:30-sun:05:30"
}

variable "monitoring_interval" {
    description = "To disable collecting Enhanced Monitoring metrics, specify 0. The default is 0. Valid Values: 0, 1, 5, 10, 15, 30, 60."
    default     = "0"
}

variable "replicate_source_db" {
    description = "Specifies that this resource is a Replicate database, and to use this value as the source database. This correlates to the identifier of another Amazon RDS Database to replicate."
    default     = ""
}

variable "skip_final_snapshot" {
    description = "Determines whether a final DB snapshot is created before the DB instance is deleted. If true is specified, no DBSnapshot is created. If false is specified, a DB snapshot is created before the DB instance is deleted, using the value from final_snapshot_identifier. Default is false."
    default     = "false"
}

variable "copy_tags_to_snapshot" {
    description = "On delete, copy all Instance tags to the final snapshot (if final_snapshot_identifier is specified). Default is false."
    default     = "false"
}

#variable "final_snapshot_identifier" {
#    description = "The name of your final DB snapshot when this DB instance is deleted. If omitted, no final snapshot will be made."
#    default     = "2018"
#}

variable "multi_az" {
    description = "If the RDS instance is multi AZ enabled."
    default     = "false"
}

variable "allow_major_version_upgrade" {
    description = "Indicates that major version upgrades are allowed. Changing this parameter does not result in an outage and the change is asynchronously applied as soon as possible."
    default     = "true"
}

variable "auto_minor_version_upgrade" {
    description = "Indicates that minor engine upgrades will be applied automatically to the DB instance during the maintenance window. Defaults to true."
    default     = "false"
}

variable "apply_immediately" {
    description = "Specifies whether any database modifications are applied immediately, or during the next maintenance window. Default is false"
    default     = "false"
}

Собственно в этом файле храняться все переменные. Спасибо кэп!

Открываем последний файл:

$ vim outputs.tf

и в него вставить нужно следующие строки:

output "rds_ids" {
    value = "${aws_db_instance.db_instance.*.id}"
}

output "rds_arns" {
    value = "${aws_db_instance.db_instance.*.arn}"
}

output "rds_addresses" {
    value = "${aws_db_instance.db_instance.*.address}"
}

output "endpoints" {
    value = "${aws_rds_cluster.rds_cluster.*.endpoint}"
}

output "reader_endpoints" {
    value = "${aws_rds_cluster.rds_cluster.*.reader_endpoint}"
}

output "aws_db_subnet_group_ids" {
    value = "${aws_db_subnet_group.db_subnet_group.*.id}"
}

output "db_parameter_groups" {
    value = "aws_db_parameter_group.db_parameter_group.*.id"
}

output "hosted_zone_ids" {
    value = "${aws_db_instance.db_instance.*.hosted_zone_id}"
}

Переходим теперь в папку aws/examples и создадим еще одну папку для проверки написанного чуда:

$ mkdir rds && cd $_

Внутри созданной папки открываем файл:

$ vim main.tf

И вставим в него следующий код:

#
# MAINTAINER Vitaliy Natarov "vitaliy.natarov@yahoo.com"
#
terraform {
  required_version = "> 0.9.0"
}
provider "aws" {
    region  = "us-east-1"
    # alias = "us-east-1"
    shared_credentials_file = "${pathexpand("~/.aws/credentials")}"
     # access_key = "${var.aws_access_key}"
     # secret_key = "${var.aws_secret_key}"
}
module "iam" {
    source                          = "../../modules/iam"
    name                            = "TEST-AIM"
    region                          = "us-east-1"
    environment                     = "PROD"

    aws_iam_role-principals         = [
        "ec2.amazonaws.com",
        "monitoring.rds.amazonaws.com",
    ]
    aws_iam_policy-actions           = [
        "cloudwatch:GetMetricStatistics",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents",
        "rds:*",
    ]
}
module "vpc" {
    source                              = "../../modules/vpc"
    name                                = "TEST-VPC"
    environment                         = "PROD"
    # VPC
    instance_tenancy                    = "default"
    enable_dns_support                  = "true"
    enable_dns_hostnames                = "true"
    assign_generated_ipv6_cidr_block    = "false"
    enable_classiclink                  = "false"
    vpc_cidr                            = "172.31.0.0/16"
    private_subnet_cidrs                = ["172.31.64.0/20"]
    public_subnet_cidrs                 = ["172.31.0.0/20"]
    availability_zones                  = ["us-east-1a", "us-east-1b"]
    enable_all_egress_ports             = "true"
    allowed_ports                       = ["80", "3306", "8080", "443"]

    map_public_ip_on_launch             = "true"

    #Internet-GateWay
    enable_internet_gateway             = "true"
    #NAT
    enable_nat_gateway                  = "false"
    single_nat_gateway                  = "false"
    #VPN
    enable_vpn_gateway                  = "false"
    #DHCP
    enable_dhcp_options                 = "false"
    # EIP
    enable_eip                          = "false"
}
module "rds" {
    source              = "../../modules/rds"
    name                = "Test-DB"
    region              = "us-east-1"
    environment         = "PROD"

    subnet_ids          = ["subnet-11ed234b", "subnet-34eddf51"]
    ##subnet_ids          = ["module.vpc.vpc-privatesubnet-ids"]
    # Single node(s)
    #create_rds_cluster  = false
    #engine              = "mysql"

    ## For cluster
    create_rds_cluster  = true
    ## aurora or aurora-postgresql. !!!!Tested just for aurora!!!!
    engine              = "aurora"
    instance_class      = "db.t2.small"

}

PS: Тут имеются вспомогательные модули. Я описывал работу тут:

Работа с AWS IAM и Terraform в Unix/Linux

Работа с AWS VPC и Terraform в Unix/Linux

Еще полезности:

Работа с AWS EC2 и Terraform в Unix/Linux

Работа с AWS ASG(auto scaling group) и Terraform в Unix/Linux

Работа с AWS ELB и Terraform в Unix/Linux

Работа с AWS Route53 и Terraform в Unix/Linux

Работа с AWS S3 и Terraform в Unix/Linux

Все уже написано и готово к использованию. Ну что, начнем тестирование. В папке с вашим плейбуком, выполняем:

$ terraform init

Этим действием я инициализирую проект. Затем, подтягиваю модуль:

$ terraform get

PS: Для обновление изменений в самом модуле, можно выполнять:

$ terraform get -update

Проверим валидацию:

$ terraform validate

Запускем прогон:

$ terraform plan

Мне вывело что все у меня хорошо и можно запускать деплой:

$ terraform apply

Как видно с вывода, — все прошло гладко! Чтобы удалить созданное творение, можно выполнить:

$ terraform destroy

Весь материал аплоаджу в github аккаунт для удобства использования:

$ git clone https://github.com/SebastianUA/terraform.git

Вот и все на этом. Данная статья «Работа с AWS RDS и Terraform в Unix/Linux» завершена.

Добавить комментарий

Ваш e-mail не будет опубликован. Обязательные поля помечены *

Этот сайт использует Akismet для борьбы со спамом. Узнайте как обрабатываются ваши данные комментариев.