Hosts Subnets Monitor (HSM): get notified when hosts’ subnets change
November 30, 2010 Leave a comment
In the recent past I have been tasked to block traffic from a LAN segment toward some public websites by blackholing their subnets.
While this approach may be not fully convincing, it is easy to implement and with few impacts on the infrastructure. The real problem is the management overhead it introduces, since websites may change the IP subnet they are run on.
How does it work?
In order to ease this task I wrote a little script which, given a hosts list, resolves IP addresses and gets the most specific subnet they fall on. Public whois databases are used to acquire subnet information: RIPE RIS, TeamCymru IP to ASN Mapping, RIRs databases. Some of them (RIPE RIS and TeamCymru IP-to-ASN) are based on BGP feeds collected around the world, others (RIRs databases) are based on LIRs allocations and assignments; the configuration section of the script allows to set which databases have to be used and how.
Requirements
The script is written in Linux Bash and uses some basic programs and an optional (but really recommended) Perl addition (implemented in the hsm-utils.pl file).
The programs used by the script are dig (to resolve hostnames in IP addresses), whois (the improved Whois client by Marco D’Itri, to get data from whois databases) and sendmail, alias of exim4 (to send email notifications when subnets change).
The hsm-utils.pl script needs the NetAddr::IP::Lite module, which can be installed using the CPAN installer:
LINUX:~#perl -MCPAN -e shell cpan>install NetAddr::IP::Lite
How to use it
To be used it just needs the “hosts” file in the working directory (/var/local/hsm by default) containing a list of hosts to be monitored:
LINUX:/var/local/hsm#cat hosts www.facebook.com www.blogspot.com www.youtube.com www.twitter.com
In order to receive email notifications the EMAIL_TO parameter has to be set with a working email address.
The script
Following is an excerpt from the script, I suggest you to read it since it contains some (I think) useful notes and configuration options; in the bottom of this post you may find the link to download it.
##################################################################################
# HOW DOES IT WORK?
# --------------------------------------------------------------------------------
#
# Everything is done in the DATA_DIR directory (default to /var/local/hsm); HSM
# reads a list of hosts to monitor from the 'hosts' file and, for each of them, it
# resolves the IP address and gets its subnet.
#
# In order to identify the subnet, it queries public databases using the whois
# client: it may be configured to use the following sources (see CONFIGURATION):
#
# - RIPE RIS database: http://www.ripe.net/ris/
# - TeamCymru IP to ASN Mapping: http://www.team-cymru.org/Services/ip-to-asn.html
# - general RIRs databases
#
# Subnets are stored in the 'subnets' file; at the end of the execution, if there
# are new subnets HSM notifies them in the output. It also notifies expired
# subnets, that is subnets appeared in the past which seem to be not binded to
# hosts anymore.
#
# Output is written to the 'output' file; you may let HSM to send the output by
# email too.
#
# The script may be scheduled to be run periodically through the crontab file.
##################################################################################
# REQUIREMENTS AND DEPENDENCIES
# --------------------------------------------------------------------------------
#
# dig Used to resolve hostnames
#
# whois I used the improved Whois client by Marco D'Itri:
# http://www.linux.it/~md/software/
#
# sendmail Used as alias of exim4; optional, only if EMAIL_TO is set
#
# awk, grep, Some basic utilities
# tail, sed
#
# Highly Recommended:
#
# Perl with NetAddr::IP::Lite module, in order to execute the hsm-utils.pl script.
# Please see CHANGE LOG & KNOWN ISSUES and USEHSMUTILS in the CONFIGURATION
# section for more details about it.
#
# Developed and tested under Debian GNU/Linux 4.0 (Etch).
##################################################################################
# CHANGE LOG & KNOWN ISSUES
# --------------------------------------------------------------------------------
#
# Date Ver. Note
# 2010-11-15 0.1 First release
#
# If you configure HSM to not use the hsm-utils.pl script (see CONFIGURATION for
# more details) you have to consider the following issues:
#
# - subnets are stored as they appear on the whois output; that is
# "192.168.0.0/24" is different from "192.168.0.0 - 192.168.0.255". This may lead
# to a wrong behaviour when the same IP address is queried against a whois
# database which returns information in a format different from the previous one.
#
# - when GENERAL whois is used, or when GETLONGESTMATCH = 1, there are no
# guarantees that the more specific subnet is choosen among those returned.
##################################################################################
# CONFIGURATION
# --------------------------------------------------------------------------------
# USEHSMUTILS
# ------------------------------------
# If USEHSMUTILS = 1 then HSM uses the hsm-utils.pl file.
# It is a Perl script which implements some functions
# performing subnets normalization and selection. It is
# needed to solve some issues reported in the CHANGE LOG
# & KNOWN ISSUES section. If you can't run a Perl script
# or you prefer to avoid it you may set USEHSMUTILS to 0.
USEHSMUTILS=1
HSMUTILSPATH=`dirname $0`/hsm-utils.pl
# DATA_DIR
# ------------------------------------
# Where files are stored:
# - the file containing the input hosts list ('hosts')
# - one file for each IP address resolved by hostnames
# - the subnets file ('subnets'), where subnets information are stored
# - the output file ('output')
# - temporary files
# No trailing slash.
DATA_DIR=/var/local/hsm
# USECACHE and CACHE_TIME
# ------------------------------------
# If USECACHE = 1 then IP addresses resolved by hostnames
# are checked against whois databases only if they were
# checked before CACHE_TIME days ago.
USECACHE=1
CACHE_TIME=3
# DNS_QUERIES
# ------------------------------------
# HSM sends this number of DNS queries in order to resolve
# hostnames IP addresses. It may be useful to discover IP
# address of hostnames with round-robin records.
DNS_QUERIES=3
# SUBNET_EXPIRY
# ------------------------------------
# When a subnet is not seen for more than SUBNET_EXPIRY days
# it is removed from the subnets file and a notification
# is written in the output.
SUBNET_EXPIRY=15
# WHOIS_LIST and GETLONGESTMATCH
# ------------------------------------
# WHOIS_LIST contains a list of sources to be used to get
# subnets information from IP addresses.
# The following sources are allowed:
# TEAMCYMRU, RIPERIS, GENERAL.
# Sources are used in the order they appear in the list.
# If GETLONGESTMATCH = 0, as soon as HSM succeds to obtain
# the subnet it stops searching.
# If a source is not working properly it uses the next one.
# If GETLONGESTMATCH = 1 HSM grabs results from all the
# listed sources, then uses the most specific one.
WHOIS_LIST="RIPERIS TEAMCYMRU GENERAL"
GETLONGESTMATCH=1
# EMAIL
# ------------------------------------
# Set EMAIL_TO with your email address if you want the output
# to be sent by email. The other parameters are optional.
EMAIL_TO=
EMAIL_FROM=
EMAIL_SUBJECT=
Download and installation
Here you can download the script – Licensed under the terms of the GNU General Public License.
To run it:
LINUX:/usr/local/bin#tar -xf hsm.tar # extract it LINUX:/usr/local/bin#nano hsm # do your configuration LINUX:/usr/local/bin#nano /var/local/hsm/hosts # edit hosts you want to monitor LINUX:/usr/local/bin#./hsm # run it
You may also schedule it using crontab:
LINUX:/usr/local/bin#cat /etc/crontab # /etc/crontab: system-wide crontab SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # m h dom mon dow user command 53 22 * * * root /usr/local/bin/hsm #
Any feedback, comment or suggestion is appreciated! 🙂
Pierky's Blog 