Интеграция Zabbix алертов с Jira, Slack в Unix/Linux
На текущем месте работы, я работаю больше чем год. За это время, я не моло сделал для своего клиента. Месяц назад у меня возникла идея как можно было бы оптимизировать L1 процесс и сократить SLA между реакцией на ивент и создание тикета из него же, с 15-30 минут, до 1 минуты (идеальный мир и без каких либо задержек).
Сервисы которые я буду использовать:
- Zabbix — В качестве алертинга.
- PagerDuty — в качестве серт-пати решения для централизированого сервиса перенаправления всех ивентов куда-либо.
- Slack — Коммуникация, постинг полезной инфы.
- Skype — Коммуникация, постинг полезной инфы (на момент написания статьи — не используется, но планируется).
- Python3 + пакетные зависимости — для создания скрипта.
Логика работы L1:
- Срабатывает триггер в заббиксе.
- Ивент перенаправляется в созданный сервис в PagerDuty. После получения любого ивента, L1 тима должна была нажать на «Acknowledge» и перейти к созданию тикета.
- Собственно, как я и говорил, — время реакции на P1-P2 инциденты у команды варьировалось от 15 до 30 минут (в зависимости от нагрузки).
- После создания тикета, — постится все необходимая информация в канал коммуникации или звонок L2-L3 команде (в зависимости от инстуркции).
Логика работы скрипта:
- Как-то та и работает 😀
Переходим к реализации…
Реализация интеграции Zabbix алертов с Jira, Slack в Unix/Linux
Открываем файл:
# vim create_jira_ticket_and_poster.py
Сам скрипт, выглядит вот так:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import argparse
import re
import time
import requests
from datetime import datetime
from jira import JIRA
class Bgcolors:
def __init__(self):
self.get = {
'PURPURE': '\033[95m',
'BLUE': '\033[94m',
'GREEN': '\033[92m',
'YELLOW': '\033[93m',
'RED': '\033[91m',
'END': '\033[0m',
'BOLD': '\033[1m',
'UNDERLINE': '\033[4m'
}
def change_time(t):
if t == 'now':
time_to_timestamp = int(time.mktime(datetime.now().timetuple()))
elif t == 'now_utc':
time_to_timestamp = int(time.mktime(datetime.utcnow().timetuple()))
elif t == 'utc':
time_to_timestamp = time.strftime("%H:%M:%S (%Y-%m-%d)",
time.gmtime(int(time.mktime(datetime.now().timetuple()))))
else:
time_to_timestamp = int(time.mktime(datetime.strptime(t, "%Y-%m-%dT%H:%M:%S.000+0000").timetuple()))
return time_to_timestamp
def post_to_slack(s_webhook, j_priority, j_project, t_status, t_number, j_assigner, t_summary, service):
headers = {"Content-type": "application/json"}
if j_project is not None:
project = {'CCM': 'CM DevOps tasks', 'CMSSD': 'CM Shared Service Desk', 'ESD': 'ESD Service Desk',
'ELTM': 'ETM Service Desk', 'PSD': 'PSD Service Desk', 'QSD': 'QSD Service Desk',
'RSD': 'RSD Service Desk', 'NSD': 'NSD Service Desk',
'PSSD': 'PSSD Service Desk'}
else:
print(Bgcolors().get['RED'], 'Please add [--priority]', Bgcolors().get['END'])
print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
exit(0)
if j_priority is not None:
prior = {'Disaster': 'P1', 'High': 'P2', 'Average': 'P3'}
else:
print(Bgcolors().get['RED'], 'Please add [--priority]', Bgcolors().get['END'])
print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
exit(0)
payload = {
"channel": "#servicedesk",
"username": "ticket-poster",
"icon_emoji": ":ghost:",
"color": "#2eb886",
"text": "🔥🔥🔥 Woops, apparently we are in trouble 🔥🔥🔥",
"attachments": [
{
"color": "#BF0D0D",
"fields": [
dict(title="Priority:", value=prior[j_priority], short=True),
{
"title": "Project:",
"value": project[j_project],
"short": True
},
{
"title": "Service:",
"value": service,
"short": True
},
{
"title": "Time:",
"value": "UTC: %s" % change_time('utc'),
"short": True
},
{
"title": "Ticket status:",
"value": t_status,
"short": True
},
{
"title": "Assigned:",
"value": "%s" % (j_assigner),
"short": True
}
],
"title": "The ticket is https://jira.my_company.com/browse/%s" % t_number,
"title_link": "https://jira.my_company.com/browse/%s" % t_number,
"text": "\n%s\n" % t_summary,
},
{
"color": "#2eb886",
"title": "Synopsis",
"text": "<!channel> This message has been posted from py-script by automatically way",
"author_name": "Vitaliy Natarov",
"author_icon": "https://cdn4.iconfinder.com/data/icons/music-icon-5/24/You-Rock-512.png",
# "image_url": "https://www.channelfutures.com/sites/channelfutures.com/files/styles/"
# "article_featured_standard/public/DevOps-2018_0.jpg?itok=CYz0R8DF"
},
{
"fallback": "Would you like to acknowledge or collect tickets to the one?",
"title": "Would you like to acknowledge or collect tickets to the one?",
"callback_id": "actions",
"color": "#3AA3E3",
"attachment_type": "default",
"actions": [
{
"name": "Acknowledge",
"text": "Acknowledge",
"type": "button",
"style": "primary",
"value": change_time('utc'),
# "url": "https://my_company.pagerduty.com/"
"response_url": "https://hooks.slack.com/services/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",
},
{
"name": "Collect tickets",
"text": "Collect tickets",
"style": "danger",
"type": "button",
"value": "Collect tickets",
"confirm": {
"title": "Are you sure?",
"text": "Would you like to collect tickets to the one?",
"ok_text": "Yes",
"dismiss_text": "No"
},
},
],
},
{
"image_url": "http://my-website.com/path/to/image.jpg",
"thumb_url": "http://example.com/path/to/thumb.png",
"footer": "Ticket posted",
"footer_icon": "https://cdn3.iconfinder.com/data/icons/people-avatar/30/superhero-512.png",
"ts": change_time('now')
}
]
}
send_request = requests.post(s_webhook, json=payload, headers=headers)
if send_request.status_code == 200 or send_request.reason == "OK":
print('Posted to slack')
else:
print('Whoops... something is wrong.... [%s]' % send_request.status_code)
return post_to_slack
def login_to_jira(url, user, password):
auth_jira = JIRA(url, auth=(user, password))
return auth_jira
def show_jira_projects(url, user, password, j_project):
auth_jira = login_to_jira(url, user, password)
projects = auth_jira.projects()
for project in projects:
project_id = project.id
project_name = project.name
project_key = project.key
if j_project is None:
print('[%s] : [%s] : [%s]' % (project_key, project_name, project_id))
else:
if (project_name == j_project) or (project_key == j_project):
print('[%s] : [%s] : [%s]' % (project_key, project_name, project_id))
return show_jira_projects
def create_jira_ticket(url, user, password, j_project, j_priority, j_reporter,
j_assignee, j_issue, s_webhook, service):
auth_jira = login_to_jira(url, user, password)
if j_project is None:
print(Bgcolors().get['RED'], 'Please add [--project]', Bgcolors().get['END'])
print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
exit(0)
else:
for project in auth_jira.projects():
if (project.name == j_project) or (project.key == j_project):
project_found = True
break
else:
project_found = False
if j_priority is not None:
prior = {'Disaster': 'Blocker', 'High': 'Critical', 'Average': 'Major'}
else:
print(Bgcolors().get['RED'], 'Please add [--priority]', Bgcolors().get['END'])
print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
exit(0)
if j_issue is None:
print(Bgcolors().get['RED'], 'Please add [--issue]', Bgcolors().get['END'])
print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
exit(0)
allfields = auth_jira.fields()
name_map = {field['name']: field['id'] for field in allfields}
if project_found:
issue_dict = {
'CCM': {
'project': {'key': j_project},
'summary': j_issue,
'description': 'Look into this one',
'priority': {'name': prior[j_priority]},
'issuetype': {'name': 'Bug'},
'assignee': {'name': j_assignee},
'labels': ['Alert'],
},
'CMSSD': {
'project': {'key': j_project},
'summary': j_issue,
'description': 'Look into this one',
'priority': {'name': prior[j_priority]},
'issuetype': {'name': 'Incident'},
'reporter': {'name': j_reporter},
'assignee': {'name': j_assignee},
'labels': ['Alert'],
name_map['Origin']: {'value': 'Application'},
name_map['Affected AWS Services']: ['EC2'],
name_map['Affected components']: [service]
},
'QSD': {
'project': {'key': j_project},
'summary': j_issue,
'description': 'Look into this one',
'priority': {'name': prior[j_priority]},
'issuetype': {'name': 'Incident'},
'reporter': {'name': j_reporter},
'assignee': {'name': j_assignee},
name_map['Origin']: {'value': 'Application'},
name_map['Affected AWS Services']: ['EC2'],
name_map['Affected components']: [service]
},
'RSD': {
'project': {'key': j_project},
'summary': j_issue,
'description': 'Look into this one',
'priority': {'name': prior[j_priority]},
'issuetype': {'name': 'Incident'},
'reporter': {'name': j_reporter},
'assignee': {'name': j_assignee},
'labels': ['Alert'],
name_map['Origin']: {'value': 'Application'},
name_map['Affected AWS Services']: ['EC2'],
name_map['Affected components']: [service]
},
'ESD': {
'project': {'key': j_project},
'summary': j_issue,
'description': 'Look into this one',
'priority': {'name': prior[j_priority]},
'issuetype': {'name': 'Incident'},
'reporter': {'name': j_reporter},
'assignee': {'name': j_assignee},
'labels': ['Alert'],
name_map['Origin']: {'value': 'Application'},
name_map['Affected AWS Services']: ['EC2'],
name_map['Affected components']: [service]
},
'PSD': {
'project': {'key': j_project},
'summary': j_issue,
'description': 'Look into this one',
'priority': {'name': prior[j_priority]},
'issuetype': {'name': 'Incident'},
'reporter': {'name': j_reporter},
'assignee': {'name': j_assignee},
'labels': ['Alert'],
name_map['Origin']: {'value': 'Application'},
name_map['Affected AWS Services']: ['EC2'],
name_map['Affected components']: [service]
},
'NSD': {
'project': {'key': j_project},
'summary': j_issue,
'description': 'Look into this one',
'priority': {'name': prior[j_priority]},
'issuetype': {'name': 'Incident'},
'reporter': {'name': j_reporter},
'assignee': {'name': j_assignee},
'labels': ['Alert'],
name_map['Origin']: {'value': 'Application'},
name_map['Affected AWS Services']: ['EC2'],
name_map['Affected components']: [service]
},
'PSSD': {
'project': {'key': j_project},
'summary': j_issue,
'description': 'Look into this one',
'priority': {'name': prior[j_priority]},
'issuetype': {'name': 'Issue'},
'reporter': {'name': j_reporter},
'assignee': {'name': j_assignee},
'labels': ['Alert'],
name_map['Origin']: {'value': 'Application'},
name_map['Affected AWS Services']: ['EC2'],
name_map['Affected components']: [service]
},
'ELTM': {
'project': {'key': j_project},
'summary': j_issue,
'description': 'Look into this one',
'priority': {'name': prior[j_priority]},
'issuetype': {'name': 'Incident'},
'reporter': {'name': j_reporter},
'assignee': {'name': j_assignee},
'labels': ['Alert'],
name_map['Origin']: {'value': 'Application'},
name_map['Affected AWS Services']: ['EC2'],
name_map['Affected components']: [service]
},
}
new_issue = auth_jira.create_issue(fields=issue_dict[j_project])
assigner = new_issue.fields.assignee
print("JIRA issue created: {}".format(new_issue))
print("https://jira.my_company.com/browse/" + str(new_issue))
# Change workflow
transition_issue_workflow(url, user, password, str(new_issue), transitions=['Acknowledgment'])
# Add description to the ticket if it's will be needed!
# Add comment to the ticket if it's will be needed!
add_comment_to_jira_ticket(url, user, password, str(new_issue), j_issue, comment_details=None)
# Post to Slack
ticket_status = "Created"
ticket_number = str(new_issue)
ticket_summary = issue_dict[j_project]['summary']
post_to_slack(s_webhook, j_priority, j_project, ticket_status, ticket_number, assigner,
ticket_summary, service)
else:
print(Bgcolors().get['RED'], 'Please use correct project key!', Bgcolors().get['END'])
print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
exit(0)
return create_jira_ticket
def add_comment_to_jira_ticket(url, user, password, j_ticket, j_issue, comment_details):
auth_jira = login_to_jira(url, user, password)
if comment_details is None:
details = '''\n\nHi all,
The [%s] alert was received, processed with SOP and escalated for further investigation.
\n\nRegards,
Service Desk''' % j_issue
else:
details = comment_details
auth_jira.add_comment(j_ticket, details)
return add_comment_to_jira_ticket
def transition_issue_workflow(url, user, password, j_issue, transitions):
auth_jira = login_to_jira(url, user, password)
for transition in transitions:
auth_jira.transition_issue(j_issue, transition=transition)
return transition_issue_workflow
def update_jira_ticket(url, user, password, j_project, j_issue, j_priority, j_ticket, s_webhook, service):
auth_jira = login_to_jira(url, user, password)
if j_project is None:
print(Bgcolors().get['RED'], 'Please add [--project]', Bgcolors().get['END'])
print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
exit(0)
else:
for project in auth_jira.projects():
if (project.name == j_project) or (project.key == j_project):
issues = auth_jira.search_issues('project=%s' % j_project)
for issue in issues:
if str(issue.key) == j_ticket:
if str(issue.fields.status) == 'Closed':
add_comment_to_jira_ticket(url, user, password, j_ticket, j_issue, comment_details=None)
# add workflow
transition_issue_workflow(url, user, password, issue,
transitions=['Reopen', 'Reopened', 'Acknowledgment'])
elif str(issue.fields.status) == 'Resolved':
add_comment_to_jira_ticket(url, user, password, j_ticket, j_issue, comment_details=None)
# add workflow
transition_issue_workflow(url, user, password, issue,
transitions=['Closure', 'Reopen', 'Reopened', 'Acknowledgment'])
else:
# print('Woops.... [%s]' % issue.fields.status)
add_comment_to_jira_ticket(url, user, password, j_ticket, j_issue, comment_details=None)
# Post to Slack
ticket_status = "Updated"
ticket_number = j_ticket
assigner = issue.fields.assignee
comments = auth_jira.comments(j_ticket)
comment = auth_jira.comment(j_ticket, comments[-1])
post_to_slack(s_webhook, j_priority, j_project, ticket_status, ticket_number,
assigner, comment.body, service)
return update_jira_ticket
def check_jira_ticket(url, user, password, j_project, j_priority, j_reporter,
j_assignee, j_issue, j_issue_time, s_webhook):
auth_jira = login_to_jira(url, user, password)
if j_project is None:
print(Bgcolors().get['RED'], 'Please add [--project]', Bgcolors().get['END'])
print(Bgcolors().get['GREEN'], 'For help, use: script_name.py -h', Bgcolors().get['END'])
exit(0)
else:
for project in auth_jira.projects():
if (project.name == j_project) or (project.key == j_project):
issues = auth_jira.search_issues('project=%s' % j_project, maxResults=1)
for issue in issues:
try:
service = j_issue.split('PROD)')[1].split('(')[1].split(')')[0]
except IndexError:
service = j_issue.split('_')[-1].split('"')[0]
third_part_of_alert = j_issue[:round(int(len(j_issue) / 3), 0)]
if re.search(r'%s' % str(third_part_of_alert), str(j_issue)) and \
re.search(r'.*(.*?PROD).*(%s)' % str(service), str(j_issue)):
issue_regex = '.*(.*?PROD).*(%s)' % str(service)
else:
issue_regex = j_issue
if re.search(issue_regex, str(issue.fields.summary)):
# check time and status ticket
current_time = int(change_time('now_utc'))
if str(issue.fields.status) == 'Open' or str(issue.fields.status) == 'Opened':
issue_updated_time = int(change_time(issue.fields.updated))
diff_times = int(round((current_time - issue_updated_time) / 60, 1))
elif str(issue.fields.status) == 'Closed' or str(issue.fields.status) == 'Resolved':
issue_resolution_time = int(change_time(issue.fields.resolutiondate))
diff_times = int(round((current_time - issue_resolution_time) / 60, 1))
else:
if issue.fields.resolutiondate is not None:
issue_resolution_time = int(change_time(issue.fields.resolutiondate))
diff_times = int(round((current_time - issue_resolution_time) / 60, 1))
else:
issue_updated_time = int(change_time(issue.fields.updated))
diff_times = int(round((current_time - issue_updated_time) / 60, 1))
if diff_times >= int(j_issue_time):
print(
Bgcolors().get['RED'], 'Im going to create a new ticket', Bgcolors().get['END'])
create_jira_ticket(url, user, password, j_project, j_priority,
j_reporter, j_assignee, j_issue, s_webhook, service)
else:
print(
Bgcolors().get['GREEN'], 'Im going to update the created ticket [%s]' % issue.key,
Bgcolors().get['END'])
update_jira_ticket(url, user, password, j_project, j_issue, j_priority,
issue.key, s_webhook, service)
else:
print(Bgcolors().get['RED'],
'Ticket didnt find in the [%s] ticket. Creating a new ticket' % j_project,
Bgcolors().get['END'])
create_jira_ticket(url, user, password, j_project, j_priority, j_reporter,
j_assignee, j_issue, s_webhook, service)
return check_jira_ticket
def main():
start__time = time.time()
parser = argparse.ArgumentParser(prog='python3 script_name.py -h',
usage='python3 script_name.py {ARGS}',
add_help=True,
prefix_chars='--/',
epilog='''created by Vitalii Natarov''')
parser.add_argument('--version', action='version', version='v0.2.0')
# Jira
parser.add_argument('--url', '--url', dest='url', help='URL', default='https://jira.my_company.com')
parser.add_argument('--login', dest='login', help='A login to Jira', default='zabbix-jira-slack')
parser.add_argument('--password', dest='password', help='A password of Jira user', default='pJ16666Z@cDDdfdsfvxcvfdsdK')
parser.add_argument('--project', dest='project', help='Set project to find it or create ticket. Ex: CCM, RSD, ESD',
default=None)
parser.add_argument('--priority', dest='priority', help='Set priority for a ticket. Ex: Average, High, Disaster',
default=None)
parser.add_argument('--reporter', dest='reporter', help='Set reporter for a ticket. Ex: natarovv',
default='zabbix-jira-slack')
parser.add_argument('--assignee', dest='assignee', help='Set assignee for a ticket. Ex: natarovv',
default='natarovv')
parser.add_argument('--time', dest='time', help='Set time in minutes to check a ticket. Ex: 60', default='60')
parser.add_argument('--issue', dest='issue',
help='Set issue to create a new ticket or check if needs to update it',
default='New issue created from python script')
# Slack
parser.add_argument('--webhook', dest='webhook', help='Set webhook URL for slack',
default="https://hooks.slack.com/services/T5DS9XFD/BC9QfHHN0M/KkckWtkgffds77uVu9pA2xKB40O")
# Functions
group = parser.add_mutually_exclusive_group(required=False)
group.add_argument('--show', dest='show', help='Show projects', action='store_true')
group.add_argument('--s', dest='show', help='Show projects', action='store_true')
results = parser.parse_args()
url = results.url
login = results.login
password = results.password
project = results.project
priority = results.priority
reporter = results.reporter
assignee = results.assignee
time_check = results.time
issue = results.issue
webhook = results.webhook
if results.show:
if (url is not None) and (login is not None) and (password is not None):
show_jira_projects(url, login, password, project)
else:
print('Please add [--url] or [--login] or [--password]')
print('For help, use: script_name.py -h')
exit(0)
elif not results.show:
if (url is not None) and (login is not None) and (password is not None):
check_jira_ticket(url, login, password, project, priority, reporter, assignee, issue, time_check, webhook)
else:
print('Please add [--url] or [--login] or [--password]')
print('For help, use: script_name.py -h')
exit(0)
else:
print('For help, use: script_name.py -h')
exit(0)
end__time = round(time.time() - start__time, 2)
print("--- %s seconds ---" % end__time)
print(
Bgcolors().get['GREEN'], "============================================================",
Bgcolors().get['END'])
print(
Bgcolors().get['GREEN'], "==========================FINISHED==========================",
Bgcolors().get['END'])
print(
Bgcolors().get['GREEN'], "============================================================",
Bgcolors().get['END'])
if __name__ == '__main__':
main()
Скрипт готов, нужно залить на заббикс-сервер(я использую стандартную папку, в ней лежат все скрипты). Открываем заббикс и переходим в Configuration->Actions->triggers. Создаем триггер и прописываем все необходимое, у меня он выглядит следующим образом:
Вкладка «Operations»:
Собственно в «Operations» прописываем:
/usr/bin/python3 /usr/lib/zabbix/externalscripts/create_jira_ticket_and_poster.py --project CCM --time 60 --priority {TRIGGER.SEVERITY} --issue "{TRIGGER.NAME}"
Сохраняем. Настраиваем заббикс триггеры на Average, High, Disaster (по логики используется только 3 приоритета).
С заббикс уже все. Перейдем к настройке слака. Открываем найтройки->Administrations->Manage Apps:
В поле «search» ввобдим «Incoming WebHooks» добавляем некоторые настройки (нужно выбрать канал для алертинга), чтобы получить вебхук. Данный вебхук нужно прописать в коде самого скрипта или переопределить в командной строке.
Настроив все необходимое, из заббикс-алертов будет создаватся тикет и постится в слак:
Честно, это очень драфтовая версия скрипта и у меня на нее были очень большие планы по реализации. Но учитывая что на все это нужно время — имеем, что имеем….
Планирую добавить пару нужных кнопок в слак, например чтобы нажимать акновледж в PD — для этого стоит уделить немного времени, а именно, — изучить flask или django. Коллектить тикети в один. Постинг созданного тикета в скайп и другие меседжеры.
Вот и все, статья «Интеграция Zabbix алертов с Jira, Slack в Unix/Linux» завершена.