Recently, I was looking what it would take to notify a Zabbix server whenever an error is encountered in a Python web server. Ultimately (before I realized that Zabbix has log monitoring and that we weren’t going to be installing a Zabbix agent on the server), we went with Zabbix’s log monitoring, but before that I started looking at sending the notification directly from the Python code itself. The quick and dirty lesson here is to thoroughly research the features of monitoring servers (and discuss them with your system admins) before writing code to solve a problem you don’t actually have. The other lesson is here’s how I learned to send notifications directly to Zabbix servers directly from your Python code.
After doing some Googling around, I came across this StackOverflow question and answer about surfacing Zabbix events from Python. The zabbix_sender binary is part of a zabbix_agent installation, so you don’t have to do anything special on that front. So, from there all you need is some python code that will call zabbix_sender. A simple version of this code would be something like:
import logging
import os
def zabbix_sender(key, output):
"""
Sends a message to the Zabbix monitoring server to update the given key
with the given output. This is designed to be only called whenever
the service encounters an error.
Zabbix should be configured with an Zabbix Trapper
item for any key passed in, and a trigger for any instance where the output
value of the message has a string length greater than 0. Since this method
should only be called when something goes wrong, the Zabbix setup for
listening for this key should be "any news is bad news"
@param key
The item key to use when communicating with Zabbix. This should match a key
configured on the Zabbix monitoring server.
@param output
The data to display in Zabbix notifying TechOps of a problem.
"""
# When I actually did this at work, I had the server and hostname set in an
# external configuration file. That's probably how you want to do this as
# opposed to hard-coding it into the script.
server = "zabbix-server-name"
hostname = "http://zabbix-server.com"
cmd = "zabbix_sender -z " + server + " -s " + hostname + " -k " + key +\
" -o \"" + output +"\""
os.system(cmd)
At this point, you can just import the zabbix_sender method from your python file, and then call it with whatever key/message you want:
from my_zabbix_wrapper import zabbix_sender
zabbix_sender("key", "My message to appear in Zabbix")
If you're looking for a more "proper" zabbix_sender wrapper, here's the final version of what I wound up writing:
import logging
import select
import subprocess
def zabbix_sender(key, output):
"""
Sends a message to the Zabbix monitoring server to update the given key
with the given output. This is designed to be only called whenever
the service encounters an error.
Zabbix should be configured with an Zabbix Trapper
item for any key passed in, and a trigger for any instance where the output
value of the message has a string length greater than 0. Since this method
should only be called when something goes wrong, the Zabbix setup for
listening for this key should be "any news is bad news"
@param key
The item key to use when communicating with Zabbix. This should match a key
configured on the Zabbix server for the service.
@param output
The data to display in Zabbix notifying TechOps of a problem.
"""
# When I actually did this at work, I had the server and hostname set in an
# external configuration file. That's probably how you want to do this as
# opposed to hard-coding it into the script.
server = "zabbix-server-name"
hostname = "http://zabbix-server.com"
cmd = "zabbix_sender -z " + server + " -s " + hostname + " -k " + key +\
" -o \"" + output +"\""
zabbix_send_is_running = lambda: zabbix_send_process.poll() is None
zabbix_send_process = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True)
error_buffer = ""
# Submit a message with the given key and output information to the Zabbix
# server.
while zabbix_send_is_running():
rlist, wlist, xlist = select.select([zabbix_send_process.stdout,
zabbix_send_process.stderr],
[],
[],
1)
# Although the zabbix_send_process is grabbing data from both standard
# output and standard error, really all we care about is standard
# error.
if zabbix_send_process.stderr in rlist:
error_buffer += zabbix_send_process.stderr.read(1024)
final_out, final_error = zabbix_send_process.communicate()
# A return code of anything other than 0 means something went wrong. Log
# an error and raise an exception.
if zabbix_send_process.returncode != 0:
logging.exception("Could not send message to Zabbix monitoring " +
"server! The message was: %s" % (cmd))
logging.exception(error_buffer + final_error)
raise Exception("Could not send message to Zabbix server! Check " +
"the logs for full details!")
This zabbix_sender() method is called the same way as the os.system() version, and isn’t much better than the shorter, os.system() call above, as all I’m doing is blocking and collecting any error messages, and then just dumping those out to the application log, but you can always tweak this to fit your needs.
No matter whether you code for the “short” or “full” version of pushing notifications from Python to Zabbix, you still need to configure Zabbix to listen for the messages. Make sure you create an item with the key same you’re using when you call zabbix_sender. After that, create a trigger from that Zabbix item so you can be notified whenever your code sends the event.
None of this really did any good with the code at work, but hopefully this will help anyone looking to push events out to Zabbix from their Python code. If nothing else, hopefully it can give a couple people some ideas about how to raise Zabbix events from their own code.