Интеграция Zabbix алертов с Jira, Slack в Unix/Linux

Интеграция Zabbix алертов с Jira, Slack в Unix/Linux

На текущем месте работы, я работаю больше чем год. За это время, я не моло сделал для своего клиента. Месяц назад у меня возникла идея как можно было бы оптимизировать L1 процесс и сократить SLA между реакцией на ивент и создание тикета из него же, с 15-30 минут, до 1 минуты (идеальный мир и без каких либо задержек).

Сервисы которые я буду использовать:

  • Zabbix — В качестве алертинга.
  • PagerDuty — в качестве серт-пати решения для централизированого сервиса перенаправления всех ивентов куда-либо.
  • Slack — Коммуникация, постинг полезной инфы.
  • Skype —  Коммуникация, постинг полезной инфы (на момент написания статьи — не используется, но планируется).
  • Python3 + пакетные зависимости — для создания скрипта.

Логика работы L1:

  • Срабатывает триггер в заббиксе.
  • Ивент перенаправляется в созданный сервис в PagerDuty. После получения любого ивента, L1 тима должна была нажать на «Acknowledge» и перейти к созданию тикета.
  • Собственно, как я и говорил, — время реакции на P1-P2 инциденты у команды варьировалось от 15 до 30 минут (в зависимости от нагрузки).
  • После создания тикета, — постится все необходимая информация в канал коммуникации или звонок L2-L3 команде (в зависимости от инстуркции).

Логика работы скрипта:

  • Как-то та и работает 😀

Переходим к реализации…

Реализация интеграции Zabbix алертов с Jira, Slack в Unix/Linux

Открываем файл:

# vim create_jira_ticket_and_poster.py

Сам скрипт, выглядит вот так:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import argparse
import re
import time
import requests

from datetime import datetime
from jira import JIRA


class Bgcolors:
    def __init__(self):
        self.get = {
            'PURPURE': '\033[95m',
            'BLUE': '\033[94m',
            'GREEN': '\033[92m',
            'YELLOW': '\033[93m',
            'RED': '\033[91m',
            'END': '\033[0m',
            'BOLD': '\033[1m',
            'UNDERLINE': '\033[4m'
        }


def change_time(t):
    if t == 'now':
        time_to_timestamp = int(time.mktime(datetime.now().timetuple()))
    elif t == 'now_utc':
        time_to_timestamp = int(time.mktime(datetime.utcnow().timetuple()))
    elif t == 'utc':
        time_to_timestamp = time.strftime("%H:%M:%S (%Y-%m-%d)",
                                          time.gmtime(int(time.mktime(datetime.now().timetuple()))))
    else:
        time_to_timestamp = int(time.mktime(datetime.strptime(t, "%Y-%m-%dT%H:%M:%S.000+0000").timetuple()))

    return time_to_timestamp


def post_to_slack(s_webhook, j_priority, j_project, t_status, t_number, j_assigner, t_summary, service):
    headers = {"Content-type": "application/json"}

    if j_project is not None:
        project = {'CCM': 'CM DevOps tasks', 'CMSSD': 'CM Shared Service Desk', 'ESD': 'ESD Service Desk',
                   'ELTM': 'ETM Service Desk', 'PSD': 'PSD Service Desk', 'QSD': 'QSD Service Desk',
                   'RSD': 'RSD Service Desk', 'NSD': 'NSD Service Desk',
                   'PSSD': 'PSSD Service Desk'}
    else:
        print(Bgcolors().get['RED'], 'Please add [--priority]', Bgcolors().get['END'])
        print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
        exit(0)

    if j_priority is not None:
        prior = {'Disaster': 'P1', 'High': 'P2', 'Average': 'P3'}
    else:
        print(Bgcolors().get['RED'], 'Please add [--priority]', Bgcolors().get['END'])
        print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
        exit(0)

    payload = {
        "channel": "#servicedesk", 
        "username": "ticket-poster",
        "icon_emoji": ":ghost:",
        "color": "#2eb886",
        "text": "🔥🔥🔥 Woops, apparently we are in trouble 🔥🔥🔥",
        "attachments": [
            {
                "color": "#BF0D0D",
                "fields": [
                    dict(title="Priority:", value=prior[j_priority], short=True),
                    {
                        "title": "Project:",
                        "value": project[j_project],
                        "short": True
                    },
                    {
                        "title": "Service:",
                        "value": service,
                        "short": True
                    },
                    {
                        "title": "Time:",
                        "value": "UTC: %s" % change_time('utc'),
                        "short": True
                    },
                    {
                        "title": "Ticket status:",
                        "value": t_status,
                        "short": True
                    },
                    {
                        "title": "Assigned:",
                        "value": "%s" % (j_assigner),
                        "short": True
                    }
                ],
                "title": "The ticket is https://jira.my_company.com/browse/%s" % t_number,
                "title_link": "https://jira.my_company.com/browse/%s" % t_number,
                "text": "\n%s\n" % t_summary,
            },
            {
                "color": "#2eb886",
                "title": "Synopsis",
                "text": "<!channel> This message has been posted from py-script by automatically way",
                "author_name": "Vitaliy Natarov",
                "author_icon": "https://cdn4.iconfinder.com/data/icons/music-icon-5/24/You-Rock-512.png",
                # "image_url": "https://www.channelfutures.com/sites/channelfutures.com/files/styles/"
                #             "article_featured_standard/public/DevOps-2018_0.jpg?itok=CYz0R8DF"
            },
            {
                "fallback": "Would you like to acknowledge or collect tickets to the one?",
                "title": "Would you like to acknowledge or collect tickets to the one?",
                "callback_id": "actions",
                "color": "#3AA3E3",
                "attachment_type": "default",
                "actions": [
                    {
                        "name": "Acknowledge",
                        "text": "Acknowledge",
                        "type": "button",
                        "style": "primary",
                        "value": change_time('utc'),
                        # "url": "https://my_company.pagerduty.com/"
                        "response_url": "https://hooks.slack.com/services/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",
                    },
                    {
                        "name": "Collect tickets",
                        "text": "Collect tickets",
                        "style": "danger",
                        "type": "button",
                        "value": "Collect tickets",
                        "confirm": {
                            "title": "Are you sure?",
                            "text": "Would you like to collect tickets to the one?",
                            "ok_text": "Yes",
                            "dismiss_text": "No"
                        },
                    },
                ],
            },
            {
                "image_url": "http://my-website.com/path/to/image.jpg",
                "thumb_url": "http://example.com/path/to/thumb.png",
                "footer": "Ticket posted",
                "footer_icon": "https://cdn3.iconfinder.com/data/icons/people-avatar/30/superhero-512.png",
                "ts": change_time('now')
            }
        ]
    }

    send_request = requests.post(s_webhook, json=payload, headers=headers)
    if send_request.status_code == 200 or send_request.reason == "OK":
        print('Posted to slack')
    else:
        print('Whoops... something is wrong.... [%s]' % send_request.status_code)

    return post_to_slack


def login_to_jira(url, user, password):
    auth_jira = JIRA(url, auth=(user, password))

    return auth_jira


def show_jira_projects(url, user, password, j_project):
    auth_jira = login_to_jira(url, user, password)

    projects = auth_jira.projects()
    for project in projects:
        project_id = project.id
        project_name = project.name
        project_key = project.key
        if j_project is None:
            print('[%s] : [%s] : [%s]' % (project_key, project_name, project_id))
        else:
            if (project_name == j_project) or (project_key == j_project):
                print('[%s] : [%s] : [%s]' % (project_key, project_name, project_id))

    return show_jira_projects


def create_jira_ticket(url, user, password, j_project, j_priority, j_reporter,
                       j_assignee, j_issue, s_webhook, service):
    auth_jira = login_to_jira(url, user, password)

    if j_project is None:
        print(Bgcolors().get['RED'], 'Please add [--project]', Bgcolors().get['END'])
        print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
        exit(0)
    else:
        for project in auth_jira.projects():
            if (project.name == j_project) or (project.key == j_project):
                project_found = True
                break
            else:
                project_found = False
        if j_priority is not None:
            prior = {'Disaster': 'Blocker', 'High': 'Critical', 'Average': 'Major'}
        else:
            print(Bgcolors().get['RED'], 'Please add [--priority]', Bgcolors().get['END'])
            print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
            exit(0)

        if j_issue is None:
            print(Bgcolors().get['RED'], 'Please add [--issue]', Bgcolors().get['END'])
            print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
            exit(0)

        allfields = auth_jira.fields()
        name_map = {field['name']: field['id'] for field in allfields}

        if project_found:
            issue_dict = {
                'CCM': {
                    'project': {'key': j_project},
                    'summary': j_issue,
                    'description': 'Look into this one',
                    'priority': {'name': prior[j_priority]},
                    'issuetype': {'name': 'Bug'},
                    'assignee': {'name': j_assignee},
                    'labels': ['Alert'],
                },
                'CMSSD': {
                    'project': {'key': j_project},
                    'summary': j_issue,
                    'description': 'Look into this one',
                    'priority': {'name': prior[j_priority]},
                    'issuetype': {'name': 'Incident'},
                    'reporter': {'name': j_reporter},
                    'assignee': {'name': j_assignee},
                    'labels': ['Alert'],
                    name_map['Origin']: {'value': 'Application'},
                    name_map['Affected AWS Services']: ['EC2'],
                    name_map['Affected components']: [service]
                },
                'QSD': {
                    'project': {'key': j_project},
                    'summary': j_issue,
                    'description': 'Look into this one',
                    'priority': {'name': prior[j_priority]},
                    'issuetype': {'name': 'Incident'},
                    'reporter': {'name': j_reporter},
                    'assignee': {'name': j_assignee},
                    name_map['Origin']: {'value': 'Application'},
                    name_map['Affected AWS Services']: ['EC2'],
                    name_map['Affected components']: [service]
                },
                'RSD': {
                    'project': {'key': j_project},
                    'summary': j_issue,
                    'description': 'Look into this one',
                    'priority': {'name': prior[j_priority]},
                    'issuetype': {'name': 'Incident'},
                    'reporter': {'name': j_reporter},
                    'assignee': {'name': j_assignee},
                    'labels': ['Alert'],
                    name_map['Origin']: {'value': 'Application'},
                    name_map['Affected AWS Services']: ['EC2'],
                    name_map['Affected components']: [service]
                },
                'ESD': {
                    'project': {'key': j_project},
                    'summary': j_issue,
                    'description': 'Look into this one',
                    'priority': {'name': prior[j_priority]},
                    'issuetype': {'name': 'Incident'},
                    'reporter': {'name': j_reporter},
                    'assignee': {'name': j_assignee},
                    'labels': ['Alert'],
                    name_map['Origin']: {'value': 'Application'},
                    name_map['Affected AWS Services']: ['EC2'],
                    name_map['Affected components']: [service]
                },
                'PSD': {
                    'project': {'key': j_project},
                    'summary': j_issue,
                    'description': 'Look into this one',
                    'priority': {'name': prior[j_priority]},
                    'issuetype': {'name': 'Incident'},
                    'reporter': {'name': j_reporter},
                    'assignee': {'name': j_assignee},
                    'labels': ['Alert'],
                    name_map['Origin']: {'value': 'Application'},
                    name_map['Affected AWS Services']: ['EC2'],
                    name_map['Affected components']: [service]
                },
                'NSD': {
                    'project': {'key': j_project},
                    'summary': j_issue,
                    'description': 'Look into this one',
                    'priority': {'name': prior[j_priority]},
                    'issuetype': {'name': 'Incident'},
                    'reporter': {'name': j_reporter},
                    'assignee': {'name': j_assignee},
                    'labels': ['Alert'],
                    name_map['Origin']: {'value': 'Application'},
                    name_map['Affected AWS Services']: ['EC2'],
                    name_map['Affected components']: [service]
                },
                'PSSD': {
                    'project': {'key': j_project},
                    'summary': j_issue,
                    'description': 'Look into this one',
                    'priority': {'name': prior[j_priority]},
                    'issuetype': {'name': 'Issue'},
                    'reporter': {'name': j_reporter},
                    'assignee': {'name': j_assignee},
                    'labels': ['Alert'],
                    name_map['Origin']: {'value': 'Application'},
                    name_map['Affected AWS Services']: ['EC2'],
                    name_map['Affected components']: [service]
                },
                'ELTM': {
                    'project': {'key': j_project},
                    'summary': j_issue,
                    'description': 'Look into this one',
                    'priority': {'name': prior[j_priority]},
                    'issuetype': {'name': 'Incident'},
                    'reporter': {'name': j_reporter},
                    'assignee': {'name': j_assignee},
                    'labels': ['Alert'],
                    name_map['Origin']: {'value': 'Application'},
                    name_map['Affected AWS Services']: ['EC2'],
                    name_map['Affected components']: [service]
                },
            }

            new_issue = auth_jira.create_issue(fields=issue_dict[j_project])
            assigner = new_issue.fields.assignee
            print("JIRA issue created: {}".format(new_issue))
            print("https://jira.my_company.com/browse/" + str(new_issue))

            # Change workflow
            transition_issue_workflow(url, user, password, str(new_issue), transitions=['Acknowledgment'])

            # Add description to the ticket if it's will be needed!

            # Add comment to the ticket if it's will be needed!
            add_comment_to_jira_ticket(url, user, password, str(new_issue), j_issue, comment_details=None)

            # Post to Slack
            ticket_status = "Created"
            ticket_number = str(new_issue)
            ticket_summary = issue_dict[j_project]['summary']

            post_to_slack(s_webhook, j_priority, j_project, ticket_status, ticket_number, assigner,
                          ticket_summary, service)
        else:
            print(Bgcolors().get['RED'], 'Please use correct project key!', Bgcolors().get['END'])
            print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
            exit(0)
    return create_jira_ticket


def add_comment_to_jira_ticket(url, user, password, j_ticket, j_issue, comment_details):
    auth_jira = login_to_jira(url, user, password)

    if comment_details is None:
        details = '''\n\nHi all,
The [%s] alert was received, processed with SOP and escalated for further investigation.
\n\nRegards,
Service Desk''' % j_issue
    else:
        details = comment_details

    auth_jira.add_comment(j_ticket, details)

    return add_comment_to_jira_ticket


def transition_issue_workflow(url, user, password, j_issue, transitions):
    auth_jira = login_to_jira(url, user, password)
    for transition in transitions:
        auth_jira.transition_issue(j_issue, transition=transition)

    return transition_issue_workflow


def update_jira_ticket(url, user, password, j_project, j_issue, j_priority, j_ticket, s_webhook, service):
    auth_jira = login_to_jira(url, user, password)

    if j_project is None:
        print(Bgcolors().get['RED'], 'Please add [--project]', Bgcolors().get['END'])
        print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
        exit(0)
    else:
        for project in auth_jira.projects():
            if (project.name == j_project) or (project.key == j_project):
                issues = auth_jira.search_issues('project=%s' % j_project)
                for issue in issues:
                    if str(issue.key) == j_ticket:
                        if str(issue.fields.status) == 'Closed':
                            add_comment_to_jira_ticket(url, user, password, j_ticket, j_issue, comment_details=None)
                            # add workflow
                            transition_issue_workflow(url, user, password, issue,
                                                      transitions=['Reopen', 'Reopened', 'Acknowledgment'])
                        elif str(issue.fields.status) == 'Resolved':
                            add_comment_to_jira_ticket(url, user, password, j_ticket, j_issue, comment_details=None)
                            # add workflow
                            transition_issue_workflow(url, user, password, issue,
                                                      transitions=['Closure', 'Reopen', 'Reopened', 'Acknowledgment'])
                        else:
                            # print('Woops.... [%s]' % issue.fields.status)
                            add_comment_to_jira_ticket(url, user, password, j_ticket, j_issue, comment_details=None)
                        # Post to Slack
                        ticket_status = "Updated"
                        ticket_number = j_ticket
                        assigner = issue.fields.assignee
                        comments = auth_jira.comments(j_ticket)
                        comment = auth_jira.comment(j_ticket, comments[-1])
                        post_to_slack(s_webhook, j_priority, j_project, ticket_status, ticket_number,
                                      assigner, comment.body, service)

    return update_jira_ticket


def check_jira_ticket(url, user, password, j_project, j_priority, j_reporter,
                      j_assignee, j_issue, j_issue_time, s_webhook):
    auth_jira = login_to_jira(url, user, password)

    if j_project is None:
        print(Bgcolors().get['RED'], 'Please add [--project]', Bgcolors().get['END'])
        print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
        exit(0)
    else:
        for project in auth_jira.projects():
            if (project.name == j_project) or (project.key == j_project):
                issues = auth_jira.search_issues('project=%s' % j_project, maxResults=1)
                for issue in issues:
                    try:
                        service = j_issue.split('PROD)')[1].split('(')[1].split(')')[0]
                    except IndexError:
                        service = j_issue.split('_')[-1].split('"')[0]

                    third_part_of_alert = j_issue[:round(int(len(j_issue) / 3), 0)]

                    if re.search(r'%s' % str(third_part_of_alert), str(j_issue)) and \
                            re.search(r'.*(.*?PROD).*(%s)' % str(service), str(j_issue)):
                        issue_regex = '.*(.*?PROD).*(%s)' % str(service)
                    else:
                        issue_regex = j_issue

                    if re.search(issue_regex, str(issue.fields.summary)):
                        # check time and status ticket
                        current_time = int(change_time('now_utc'))
                        if str(issue.fields.status) == 'Open' or str(issue.fields.status) == 'Opened':
                            issue_updated_time = int(change_time(issue.fields.updated))
                            diff_times = int(round((current_time - issue_updated_time) / 60, 1))

                        elif str(issue.fields.status) == 'Closed' or str(issue.fields.status) == 'Resolved':
                            issue_resolution_time = int(change_time(issue.fields.resolutiondate))
                            diff_times = int(round((current_time - issue_resolution_time) / 60, 1))

                        else:
                            if issue.fields.resolutiondate is not None:
                                issue_resolution_time = int(change_time(issue.fields.resolutiondate))
                                diff_times = int(round((current_time - issue_resolution_time) / 60, 1))
                            else:
                                issue_updated_time = int(change_time(issue.fields.updated))
                                diff_times = int(round((current_time - issue_updated_time) / 60, 1))

                        if diff_times >= int(j_issue_time):
                            print(
                                Bgcolors().get['RED'], 'Im going to create a new ticket', Bgcolors().get['END'])
                            create_jira_ticket(url, user, password, j_project, j_priority,
                                               j_reporter, j_assignee, j_issue, s_webhook, service)
                        else:
                            print(
                                Bgcolors().get['GREEN'], 'Im going to update the created ticket [%s]' % issue.key,
                                Bgcolors().get['END'])
                            update_jira_ticket(url, user, password, j_project, j_issue, j_priority,
                                               issue.key, s_webhook, service)

                    else:
                        print(Bgcolors().get['RED'],
                              'Ticket didnt find in the [%s] ticket. Creating a new ticket' % j_project,
                              Bgcolors().get['END'])
                        create_jira_ticket(url, user, password, j_project, j_priority, j_reporter,
                                           j_assignee, j_issue, s_webhook, service)

    return check_jira_ticket


def main():
    start__time = time.time()
    parser = argparse.ArgumentParser(prog='python3 script_name.py -h',
                                     usage='python3 script_name.py {ARGS}',
                                     add_help=True,
                                     prefix_chars='--/',
                                     epilog='''created by Vitalii Natarov''')
    parser.add_argument('--version', action='version', version='v0.2.0')
    # Jira
    parser.add_argument('--url', '--url', dest='url', help='URL', default='https://jira.my_company.com')
    parser.add_argument('--login', dest='login', help='A login to Jira', default='zabbix-jira-slack')
    parser.add_argument('--password', dest='password', help='A password of Jira user', default='pJ16666Z@cDDdfdsfvxcvfdsdK')

    parser.add_argument('--project', dest='project', help='Set project to find it or create ticket. Ex: CCM, RSD, ESD',
                        default=None)
    parser.add_argument('--priority', dest='priority', help='Set priority for a ticket. Ex: Average, High, Disaster',
                        default=None)
    parser.add_argument('--reporter', dest='reporter', help='Set reporter for a ticket. Ex: natarovv',
                        default='zabbix-jira-slack')
    parser.add_argument('--assignee', dest='assignee', help='Set assignee for a ticket. Ex: natarovv',
                        default='natarovv')
    parser.add_argument('--time', dest='time', help='Set time in minutes to check a ticket. Ex: 60', default='60')
    parser.add_argument('--issue', dest='issue',
                        help='Set issue to create a new ticket or check if needs to update it',
                        default='New issue created from python script')
    # Slack
    parser.add_argument('--webhook', dest='webhook', help='Set webhook URL for slack',
                        default="https://hooks.slack.com/services/T5DS9XFD/BC9QfHHN0M/KkckWtkgffds77uVu9pA2xKB40O")

    # Functions
    group = parser.add_mutually_exclusive_group(required=False)
    group.add_argument('--show', dest='show', help='Show projects', action='store_true')
    group.add_argument('--s', dest='show', help='Show projects', action='store_true')

    results = parser.parse_args()
    url = results.url
    login = results.login
    password = results.password
    project = results.project
    priority = results.priority
    reporter = results.reporter
    assignee = results.assignee
    time_check = results.time
    issue = results.issue
    webhook = results.webhook

    if results.show:
        if (url is not None) and (login is not None) and (password is not None):
            show_jira_projects(url, login, password, project)
        else:
            print('Please add [--url] or [--login] or [--password]')
            print('For help, use: script_name.py -h')
            exit(0)
    elif not results.show:
        if (url is not None) and (login is not None) and (password is not None):
            check_jira_ticket(url, login, password, project, priority, reporter, assignee, issue, time_check, webhook)
        else:
            print('Please add [--url] or [--login] or [--password]')
            print('For help, use: script_name.py -h')
            exit(0)
    else:
        print('For help, use: script_name.py -h')
        exit(0)

    end__time = round(time.time() - start__time, 2)
    print("--- %s seconds ---" % end__time)

    print(
        Bgcolors().get['GREEN'], "============================================================",
        Bgcolors().get['END'])
    print(
        Bgcolors().get['GREEN'], "==========================FINISHED==========================",
        Bgcolors().get['END'])
    print(
        Bgcolors().get['GREEN'], "============================================================",
        Bgcolors().get['END'])


if __name__ == '__main__':
    main()

Скрипт готов, нужно залить на заббикс-сервер(я использую стандартную папку, в ней лежат все скрипты). Открываем заббикс и переходим в Configuration->Actions->triggers. Создаем триггер и прописываем все необходимое, у меня он выглядит следующим образом:

Jira-poster

Вкладка «Operations»:

Собственно в «Operations» прописываем:

/usr/bin/python3 /usr/lib/zabbix/externalscripts/create_jira_ticket_and_poster.py --project CCM --time 60 --priority {TRIGGER.SEVERITY} --issue "{TRIGGER.NAME}"

Сохраняем. Настраиваем заббикс триггеры на Average, High, Disaster (по логики используется только 3 приоритета).

С заббикс уже все. Перейдем к настройке слака. Открываем найтройки->Administrations->Manage Apps:

Создаение вебхука в слак

В поле «search» ввобдим «Incoming WebHooks» добавляем некоторые настройки (нужно выбрать канал для алертинга), чтобы получить вебхук. Данный вебхук нужно прописать в коде самого скрипта или переопределить в командной строке.

Настроив все необходимое, из заббикс-алертов будет создаватся тикет и постится в слак:

Честно, это очень драфтовая версия скрипта и у меня на нее были очень большие планы по реализации. Но учитывая что на все это нужно время — имеем, что имеем….

Планирую добавить пару нужных кнопок в слак, например чтобы нажимать акновледж в PD — для этого стоит уделить немного времени, а именно, — изучить flask или django. Коллектить тикети в один. Постинг созданного тикета в скайп и другие меседжеры.

Вот и все, статья «Интеграция Zabbix алертов с Jira, Slack в Unix/Linux» завершена.

Добавить комментарий

Ваш e-mail не будет опубликован. Обязательные поля помечены *

Этот сайт использует Akismet для борьбы со спамом. Узнайте как обрабатываются ваши данные комментариев.