The Zabbix built-in SMS notification system seemed to have a bug, which cause a triple notification to be sent on every trigger action (bug ID ZBX-3507), so I preferred to use an external (custom) script and a third party tool: gammu-smsd.

This daemon connects to the GSM modem and listens for an outgoing queue; when you need to send a message, you just have to “inject” it into its queue, then it does the rest.

Installation

As soon as I plugged the USB key into the server, Ubuntu recognized it:

Oct 27 13:57:08 MyMachineName kernel: [764858.260009] usb 4-1: new full speed USB device number 2 using uhci_hcd
Oct 27 13:57:08 MyMachineName kernel: [764858.481158] cdc_acm 4-1:1.0: ttyACM0: USB ACM device
Oct 27 13:57:08 MyMachineName kernel: [764858.484118] usbcore: registered new interface driver cdc_acm
Oct 27 13:57:08 MyMachineName kernel: [764858.484121] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters

Then I installed gammu-smsd and edited its configuration file:

apt-get install gammu-smsd

cat /etc/gammu-smsdrc

# Configuration file for Gammu SMS Daemon

# Gammu library configuration, see gammurc(5)
[gammu]
# Please configure this!
port = /dev/ttyACM0
connection = at
# Debugging
logformat = textall

# SMSD configuration, see gammu-smsdrc(5)
[smsd]
service = files
logfile = /var/log/gammu-smsd
# Increase for debugging information
debuglevel = 0
ReceiveFrequency = 300
# Paths where messages are stored
inboxpath = /var/spool/gammu/inbox/
outboxpath = /var/spool/gammu/outbox/
sentsmspath = /var/spool/gammu/sent/
errorsmspath = /var/spool/gammu/error/

The port parameter is the device where the modem is mapped. This may change on other system.

I changed the logfile parameter also, because by default it is configured to log to syslog. You need to give the right permissions to it:

chmod a+rw /var/log/gammu-smsd

The ReceiveFrequency is used to tell gammu-smsd how often to check for incoming SMS; by default it is every 1 second, but in my case I didn’t need to receive SMS, so I raised it to 5 minutes.

An important parameter is PIN on the [smsd] section: in my case I removed the PIN check from my SIM card, so I didn’t use it in the configuration file.

In order to avoid “Cannot open file” errors, I also granted access to the device to everyone:

chmod a+rw /dev/ttyACM0

At this time I reloaded gammu-smsd and tested the configuration:

/etc/init.d/gammu-smsd restart
echo "Test message" | gammu-smsd-inject TEXT 335123456

(replace 335123456 with your mobile phone number!)

The gammu-smsd-monitor utility let you to check your modem status too.

In order to rotate gammu-smsd log file, I wrote a logrotate.d configuration file too:

cat /etc/logrotate.d/gammu-smsd
/var/log/gammu-smsd {
    daily
    rotate 7
    compress
    missingok
    notifempty
}

Zabbix configuration

As first I configured a new media type using the Zabbix administration front-end:

Administration / Media types -> Create Media Type

Description: SMS-via-gammu
Type: Script
Script name: sendsms

sendsms is the name of the external script I used for my custom notification system.

Then I configured the new media type for my Zabbix user:

Administration / Users -> select Users in the top right corner – edit your user Media: Add

Type: SMS-via-gammu
Send to: 335123456
When active, Use if severity: as you wish
Status: Enabled

Of course, I had an Action configured too, but for more information I suggest you to read about it on the official documentation page.

In order to configure Zabbix to use external notification scripts, I verified that the AlertScriptsPath was present in the configuration file:

cat /etc/zabbix/zabbix_server.conf | grep AlertScriptsPath
AlertScriptsPath=/etc/zabbix/alert.d/

/etc/zabbix/alert.d is the directory where notifications script have to be.

Then I added the zabbix user to the gammu group, in order to give it the right permissions to inject messages into the outgoing gammu queue…

adduser zabbix gammu

… then I restarted Zabbix:

/etc/init.d/zabbix-server restart

This is the script I used to inject Zabbix notifications into the gammu outgoing queue (/etc/zabbix/alert.d/sendsms):

#!/bin/sh

# $1    recipient
# $2    subject
# $3    message

TMPFILE=`mktemp -t`

echo "$3" >> $TMPFILE

cat $TMPFILE | gammu-smsd-inject TEXT $1

rm $TMPFILE

chmod a+x /etc/zabbix/alert.d/sendsms

At this time I tested it by creating a new trigger with fake values (temperature < 100 °C, or everything else could let an action to be raised).

References

Zabbix.com: Configuration page on the official documentation site

gammu-smsd: Gammu SMSD

The scenario I worked on had two switches, a 3560 and a 2960, with a 802.1q (etherchannel) trunk between them; the 3560 was the gateway for the VLAN 100 while on the 2960 I only had some access ports and the management interface.

3560:

! Port-channel toward 2960, 802.1q trunk carrying VLAN 100
interface Port-channel1
 description 3560-to-2960
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 100
 switchport trunk allowed vlan 100
 switchport mode trunk
 switchport nonegotiate
end
!
! Native VLAN tagging
vlan dot1q tag native
!
! VLAN 100 declaration
vlan 100
!
! Layer3 interface for VLAN 100
interface Vlan100
 description SVI100
 ip address 10.0.100.1 255.255.255.0
end

2960:

! Port-channel toward 3560, 802.1q trunk carrying VLAN 100
interface Port-channel1
 description 2960-to-3560
 switchport trunk native vlan 100
 switchport trunk allowed vlan 200
 switchport mode trunk
 switchport nonegotiate
end
!
! VLAN 100 declaration
vlan 100
!
! Default management interface is shutdown
interface Vlan1
 no ip address
 no ip route-cache
 shutdown
end
!
! Management interface
interface Vlan100
 ip address 10.0.100.2 255.255.255.0
 no ip route-cache
end

A ping from the 3560 toward the 2960 (where I ran a debug ip icmp) showed that ICMP echo requests was coming to the switch, replies were crafted by 2960 but they never arrived to 3560.

When I focused on the native VLANs topic, I found they were aligned on both switches: I thought that frames leaving 2960 toward 3560 were untagged (because of the switchport trunk native vlan 100 command) but on 3560 side they should be accepted thanks to the same command. Here I was wrong! I missed the vlan dot1q tag native full behaviour, which means that every untagged ingress frame is dropped, even if it matches the configured native VLAN.

In order to get this configuration to work properly, I had to ensure that every 2960 egress frame was tagged, but it seems 2960s don’t support native VLAN tagging: here I had not the vlan dot1q tag native global configuration capability, nor the switchport trunk native vlan tag interface command, so I removed the switchport trunk native vlan 100 command and everything worked.

References

Cisco.com: Command Lookup Tool

Cisco Support Community: cat2960 native vlan tagged on trunk discussion

R1 and R2 were BGP speakers, R1 was announcing both IPv4 and IPv6 prefixes to R2 and I wanted to set, let’s say, different weights on those prefixes: weight = 4 to the IPv4 prefix and weight = 6 to the IPv6 prefix. It’s stupid, of course, but it’s just to simplify the real task I had to face.

My goal was to have a single route-map for both IPv6 and IPv4 routes manipulation: a single route-map applied to the two neighbor statements in ipv4 and ipv6 address-family:

So, on R2 I wrote the route-map and applied it to the neighbor statement:

R2:

ip prefix-list IPv4 seq 5 permit 192.168.1.1/32
!
ipv6 prefix-list IPv6 seq 5 permit 2001:DB8:1::/64
!
route-map RouteMapIN permit 10
 match ip address prefix-list IPv4
 set weight 4
!
route-map RouteMapIN permit 20
 match ipv6 address prefix-list IPv6
 set weight 6
!
route-map RouteMapIN permit 100
!
router bgp 65535
 address-family ipv4
  neighbor 192.168.0.1 route-map RouteMapIN in
 exit-address-family
 !
 address-family ipv6
  neighbor 2001:DB8::1 route-map RouteMapIN in
 exit-address-family
!

This is what I expected:

I thought it was fine to match only the IPv4 prefix on sequence number 10, and only the IPv6 prefix on sequence number 20, but that was incorrect:

R2#clear bgp all 65535
R2#show bgp all
For address family: IPv4 Unicast
BGP table version is 6, local router ID is 192.168.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i192.168.1.1/32   192.168.0.1              0    100      4 i

For address family: IPv6 Unicast
BGP table version is 6, local router ID is 192.168.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i2001:DB8:1::/64  2001:DB8::1              0    100      4 i

Both prefixes had weight = 4!

So I tried to exchange the route-map entries’ position to confirm my doubt:

R2:

no route-map RouteMapIN
!
route-map RouteMapIN permit 10
 match ipv6 address prefix-list IPv6
 set weight 6
!
route-map RouteMapIN permit 20
 match ip address prefix-list IPv4
 set weight 4
!
route-map RouteMapIN permit 100

R2#clear bgp all 65535
R2#show bgp all
For address family: IPv4 Unicast
BGP table version is 8, local router ID is 192.168.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i192.168.1.1/32   192.168.0.1              0    100      6 i

For address family: IPv6 Unicast
BGP table version is 8, local router ID is 192.168.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i2001:DB8:1::/64  2001:DB8::1              0    100      6 i

As supposed, both prefixes had weight = 6!

Finally, I decided to use more restrictive statements, by matching the specific prefix I wanted to match and explicitly denying the other…

R2:

! Here I deny every IPv4 prefix
ip prefix-list NoIPv4 seq 5 deny 0.0.0.0/0 le 32
!
! Here I deny every IPv6 prefix
ipv6 prefix-list NoIPv6 seq 5 deny ::/0 le 128
!
no route-map RouteMapIN
!
route-map RouteMapIN permit 10
 ! Match the IPv4 prefix...
 match ip address prefix-list IPv4
 ! ... but not IPv6 prefixes.
 match ipv6 address prefix-list NoIPv6
 set weight 4
!
route-map RouteMapIN permit 20
 ! Do not match IPv4 prefixes...
 match ip address prefix-list NoIPv4
 ! ... but match only the wanted IPv6 prefix.
 match ipv6 address prefix-list IPv6
 set weight 6
!
route-map RouteMapIN permit 100

… and I got the right result:

R2#clear bgp all 65535
R2#show bgp all
For address family: IPv4 Unicast
BGP table version is 10, local router ID is 192.168.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i192.168.1.1/32   192.168.0.1              0    100      4 i

For address family: IPv6 Unicast
BGP table version is 10, local router ID is 192.168.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i2001:DB8:1::/64  2001:DB8::1              0    100      6 i

The IPv4 prefix had weight = 4, and the IPv6 one weight = 6!

What did I miss? Any suggestion is strongly appreciated!

While this approach may be not fully convincing, it is easy to implement and with few impacts on the infrastructure. The real problem is the management overhead it introduces, since websites may change the IP subnet they are run on.

How does it work?

In order to ease this task I wrote a little script which, given a hosts list, resolves IP addresses and gets the most specific subnet they fall on. Public whois databases are used to acquire subnet information: RIPE RIS, TeamCymru IP to ASN Mapping, RIRs databases. Some of them (RIPE RIS and TeamCymru IP-to-ASN) are based on BGP feeds collected around the world, others (RIRs databases) are based on LIRs allocations and assignments; the configuration section of the script allows to set which databases have to be used and how.

Requirements

The script is written in Linux Bash and uses some basic programs and an optional (but really recommended) Perl addition (implemented in the hsm-utils.pl file).

The programs used by the script are dig (to resolve hostnames in IP addresses), whois (the improved Whois client by Marco D’Itri, to get data from whois databases) and sendmail, alias of exim4 (to send email notifications when subnets change).

The hsm-utils.pl script needs the NetAddr::IP::Lite module, which can be installed using the CPAN installer:

LINUX:~#perl -MCPAN -e shell
cpan>install NetAddr::IP::Lite

How to use it

To be used it just needs the “hosts” file in the working directory (/var/local/hsm by default) containing a list of hosts to be monitored:

LINUX:/var/local/hsm#cat hosts
www.facebook.com
www.blogspot.com
www.youtube.com
www.twitter.com

In order to receive email notifications the EMAIL_TO parameter has to be set with a working email address.

The script

Following is an excerpt from the script, I suggest you to read it since it contains some (I think) useful notes and configuration options; in the bottom of this post you may find the link to download it.

##################################################################################
# HOW DOES IT WORK?
# --------------------------------------------------------------------------------
#
# Everything is done in the DATA_DIR directory (default to /var/local/hsm); HSM
# reads a list of hosts to monitor from the 'hosts' file and, for each of them, it
# resolves the IP address and gets its subnet.
#
# In order to identify the subnet, it queries public databases using the whois
# client: it may be configured to use the following sources (see CONFIGURATION):
#
# - RIPE RIS database: http://www.ripe.net/ris/
# - TeamCymru IP to ASN Mapping: http://www.team-cymru.org/Services/ip-to-asn.html
# - general RIRs databases
#
# Subnets are stored in the 'subnets' file; at the end of the execution, if there
# are new subnets HSM notifies them in the output. It also notifies expired
# subnets, that is subnets appeared in the past which seem to be not binded to
# hosts anymore.
#
# Output is written to the 'output' file; you may let HSM to send the output by
# email too.
#
# The script may be scheduled to be run periodically through the crontab file.

##################################################################################
# REQUIREMENTS AND DEPENDENCIES
# --------------------------------------------------------------------------------
#
# dig           Used to resolve hostnames
#
# whois         I used the improved Whois client by Marco D'Itri:
#               http://www.linux.it/~md/software/
#
# sendmail      Used as alias of exim4; optional, only if EMAIL_TO is set
#
# awk, grep,    Some basic utilities
# tail, sed
#
# Highly Recommended:
#
# Perl with NetAddr::IP::Lite module, in order to execute the hsm-utils.pl script.
# Please see CHANGE LOG & KNOWN ISSUES and USEHSMUTILS in the CONFIGURATION
# section for more details about it.
#
# Developed and tested under Debian GNU/Linux 4.0 (Etch).

##################################################################################
# CHANGE LOG & KNOWN ISSUES
# --------------------------------------------------------------------------------
#
# Date          Ver.    Note
# 2010-11-15    0.1     First release
#
# If you configure HSM to not use the hsm-utils.pl script (see CONFIGURATION for
# more details) you have to consider the following issues:
#
# - subnets are stored as they appear on the whois output; that is
# "192.168.0.0/24" is different from "192.168.0.0 - 192.168.0.255". This may lead
# to a wrong behaviour when the same IP address is queried against a whois
# database which returns information in a format different from the previous one.
#
# - when GENERAL whois is used, or when GETLONGESTMATCH = 1, there are no
# guarantees that the more specific subnet is choosen among those returned.

##################################################################################
# CONFIGURATION
# --------------------------------------------------------------------------------

# USEHSMUTILS
# ------------------------------------
# If USEHSMUTILS = 1 then HSM uses the hsm-utils.pl file.
# It is a Perl script which implements some functions
# performing subnets normalization and selection. It is
# needed to solve some issues reported in the CHANGE LOG
# & KNOWN ISSUES section. If you can't run a Perl script
# or you prefer to avoid it you may set USEHSMUTILS to 0.

USEHSMUTILS=1
HSMUTILSPATH=`dirname $0`/hsm-utils.pl

# DATA_DIR
# ------------------------------------
# Where files are stored:
# - the file containing the input hosts list ('hosts')
# - one file for each IP address resolved by hostnames
# - the subnets file ('subnets'), where subnets information are stored
# - the output file ('output')
# - temporary files
# No trailing slash.

DATA_DIR=/var/local/hsm

# USECACHE and CACHE_TIME
# ------------------------------------
# If USECACHE = 1 then IP addresses resolved by hostnames
# are checked against whois databases only if they were
# checked before CACHE_TIME days ago.

USECACHE=1
CACHE_TIME=3

# DNS_QUERIES
# ------------------------------------
# HSM sends this number of DNS queries in order to resolve
# hostnames IP addresses. It may be useful to discover IP
# address of hostnames with round-robin records.

DNS_QUERIES=3

# SUBNET_EXPIRY
# ------------------------------------
# When a subnet is not seen for more than SUBNET_EXPIRY days
# it is removed from the subnets file and a notification
# is written in the output.

SUBNET_EXPIRY=15

# WHOIS_LIST and GETLONGESTMATCH
# ------------------------------------
# WHOIS_LIST contains a list of sources to be used to get
# subnets information from IP addresses.
# The following sources are allowed:
# TEAMCYMRU, RIPERIS, GENERAL.
# Sources are used in the order they appear in the list.
# If GETLONGESTMATCH = 0, as soon as HSM succeds to obtain
# the subnet it stops searching.
# If a source is not working properly it uses the next one.
# If GETLONGESTMATCH = 1 HSM grabs results from all the
# listed sources, then uses the most specific one.

WHOIS_LIST="RIPERIS TEAMCYMRU GENERAL"
GETLONGESTMATCH=1

# EMAIL
# ------------------------------------
# Set EMAIL_TO with your email address if you want the output
# to be sent by email. The other parameters are optional.

EMAIL_TO=
EMAIL_FROM=
EMAIL_SUBJECT=

Download and installation

Here you can download the script – Licensed under the terms of the GNU General Public License.

To run it:

LINUX:/usr/local/bin#tar -xf hsm.tar # extract it
LINUX:/usr/local/bin#nano hsm # do your configuration
LINUX:/usr/local/bin#nano /var/local/hsm/hosts # edit hosts you want to monitor
LINUX:/usr/local/bin#./hsm # run it

You may also schedule it using crontab:

LINUX:/usr/local/bin#cat /etc/crontab
# /etc/crontab: system-wide crontab

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
53 22   * * *   root    /usr/local/bin/hsm
#

Any feedback, comment or suggestion is appreciated!

Something like this:

CLIENT -> HOST_A (proxy) -> HOST_B (real server)

Finally I found sudppipe, by Luigi Auriemma, and I want to share it, with the hope this post could raise its visibility and lower others’ search time.

Here is the author’s description:

advanced UDP proxy/datapipe/packets forwarder and modifier with multiple functions.
a datapipe is like a minimalistic proxy which acts as a bridge for connecting to a specific host, so the input connection can be any UDP client while the output is ever the same IP:port (clients->stcppipe->target).
it supports multiple clients allowed, creation of tcpdump capture files (like a sniffer), packets forwarding (chat style, each packet is forwarded to all the other clients and server connected), packets injection (the tool opens a specific UDP port to which is possible to send the customized packets that will be sent to the server), support for multiple target hosts plus some interesting options for controlling the outgoing sockets, hexadecimal visualization and plugins support for the modification and visualization of the packets (supports also the mysendto and myrecvfrom functions of the plugins of Proxocket).
in the package there are also some example plugins: Zdaemon huffman, Doom huffman, Half-life decoding and one for the replacement of the text strings.
the plugins (which work on both Windows and Linux) are very basic to create and use, just take a look at example_sudp.c for more informations.

Thanks Luigi!

The book covers every aspect of Zabbix, from the installation process to distributed monitoring, with a in-depth coverage of hosts monitoring techniques and user notification system. Many practical examples and screenshots help the reader to configure the system and have a fully working setup in a few minutes.

A whole chapter is dedicated to the SNMP integration, with a good explaination of the protocol and very practical tips about the use of Net-SNMP suite and MIBs. The author also included a good tutorial about the setup of a traps handling procedure with hosts binding.

Two more noteworthy chapters are about the use of templates and macros to simplify the configuration of Zabbix, and advanced monitoring, such as using external scripts or data already gathered by other systems.

If you plan to use Zabbix and you are starting from scratch, or even if you already use it but you want to deepen some aspects, I really suggest you to have a look at this book! In the meantime, here you can find a sample chapter: Getting your first Notification (PDF).

My two pennies worth

Details

Title: Zabbix 1.8 Network Monitoring
Language: English
Paperback: 428 pages [ 235mm x 191mm ]
Release Date: March 2010
ISBN: 184719768X
ISBN 13: 978-1-847197-68-9
Author(s): Rihards Olups
More info: Packt Publishing web site

In this post I don’t want to cover the mechanisms of this protocol extension, if you want to understand them I please you to follow some links in the References section; I prefer to show some interactions between OLD 2-byte ASN and NEW 4-byte ASN BGP speakers. I built a little GNS3/Dynamips lab using a couple of 7200s with IOS 12.2(33)SRE as NEW speakers, and a couple of 3640s as OLD peers.

In the topology every router announces any subnet to the others; BGP peering sessions follow the physical topology. Green ASs support 4-byte AS, while gray do not.
For the sake of readibility I’ll use asdot notation in this post, with the exception of some configuration blocks where I’ll use asplain just to show both notation usage.

I also uploaded two packet captures on PacketLife.net: they show BGP UPDATES with and without the NEW_AS_PATH attributes. They are 4-byte_AS_numbers_Full_Support.cap and 4-byte_AS_numbers_Mixed_Scenario.cap.

Partial 4-byte ASN support: A-B routers

Router A is on AS 10.1 / 655361 (asdot / asplain notation):

A#sh run | sec bgp
router bgp 655361
 no synchronization
 bgp log-neighbor-changes
 network 10.0.0.0
 neighbor 172.16.3.2 remote-as 2
 no auto-summary
B#sh run | sec bgp
router bgp 2
 no synchronization
 bgp log-neighbor-changes
 network 20.0.0.0
 neighbor 172.16.3.1 remote-as 23456
 no auto-summary

As you can see, on router B neighbor statement we can’t use the real ASN to peer with A, so we have to use the AS_TRANS (23456). Let’s wait until our sessions come up, then show BGP neighbors and tables:

A#sh ip bgp neighbors 172.16.3.2
BGP neighbor is 172.16.3.2,  remote AS 2, external link
  BGP version 4, remote router ID 20.0.0.1
  BGP state = Established, up for 00:00:27
  Last read 00:00:27, last write 00:00:27, hold time is 180, keepalive interval is 60 seconds
  Neighbor sessions:
    1 active, is not multisession capable
  Neighbor capabilities:
    Route refresh: advertised and received(new)
    Four-octets ASN Capability: advertised
    Address family IPv4 Unicast: advertised and received
    Multisession Capability: advertised
...

Please note the Four-octets ASN Capability: advertised line: A advertised this capability but did not receive it back from B.

A#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         0.0.0.0                  0         32768 i
*> 20.0.0.0         172.16.3.2               0             0 2 i

As expected, A receives the B‘s route but…

B#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.3.1               0             0 23456 i
*> 20.0.0.0         0.0.0.0                  0         32768 i

… B only sees A‘s subnet as from AS_TRANS (23456).

This is because B does not support 4-byte ASN, and router A knows this (capabilities exchange during session setup), so it just sends AS_PATH attributes containing the 16-bit AS_TRANS in place of the real 32-bit AS number.

You can find a similar UPDATE on the first capture I sent on PacketLife.net (4-byte_AS_numbers_Mixed_Scenario.cap), packet number 2.

Full 4-byte ASN support: A-D routers

Let’s setup the BGP session between routers A and D; they both support 4-byte ASN.

A#sh run | sec bgp
router bgp 655361
 no synchronization
 bgp log-neighbor-changes
 network 10.0.0.0
 neighbor 172.16.1.2 remote-as 2621441
 neighbor 172.16.3.2 remote-as 2
 no auto-summary
D#sh run | sec bgp
router bgp 2621441
 no synchronization
 bgp log-neighbor-changes
 network 40.0.0.0
 neighbor 172.16.1.1 remote-as 655361
 no auto-summary

When sessions come up…

A#sh ip bgp neighbors 172.16.1.2
BGP neighbor is 172.16.1.2,  remote AS 40.1, external link
  BGP version 4, remote router ID 40.0.0.1
  BGP state = Established, up for 00:04:04
  Last read 00:00:46, last write 00:00:53, hold time is 180, keepalive interval is 60 seconds
  Neighbor sessions:
    1 active, is multisession capable
  Neighbor capabilities:
    Route refresh: advertised and received(new)
    Four-octets ASN Capability: advertised and received
    Address family IPv4 Unicast: advertised and received
    Multisession Capability: advertised and received
...

Here A both sent and received the 4-byte ASN capability to/by B, because it supports this feature.

A#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         0.0.0.0                  0         32768 i
*> 20.0.0.0         172.16.3.2               0             0 2 i
*> 40.0.0.0         172.16.1.2               0             0 2621441 i
D#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.1.1               0             0 655361 i
*> 20.0.0.0         172.16.1.1                             0 655361 2 i
*> 40.0.0.0         0.0.0.0                  0         32768 i

Both A and D have subnets with the full 32-bit AS number.

You can see the capture on the file 4-byte_AS_numbers_Full_Support.cap on PacketLife.net.

As I already said, let’s switch to asdot notation now, just to improve readibility:

D#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
D(config)#router bgp 40.1
D(config-router)#bgp asnotation dot
D(config-router)#do show ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.1.1               0             0 10.1 i
*> 20.0.0.0         172.16.1.1                             0 10.1 2 i
*> 40.0.0.0         0.0.0.0                  0         32768 i
A#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
A(config)#router bgp 10.1
A(config-router)#bgp asnotation dot

Ok, 10.1 is better than 655361, and 40.1 than 2621441!!

Mixed scenario: D-B routers

Let’s have more fun now and bring D-B session up!

D(config-router)#neighbor 172.16.2.1 remote-as 2
B(config-router)#neighbor 172.16.2.2 remote-as 23456

As we already saw, B doesn’t support 4-byte ASN, so we have to use AS_TRANS to peer with D.

D#show ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*  10.0.0.0         172.16.2.1                             0 2 10.1 i
*>                  172.16.1.1               0             0 10.1 i
*> 20.0.0.0         172.16.2.1               0             0 2 i
*                   172.16.1.1                             0 10.1 2 i
*> 40.0.0.0         0.0.0.0                  0         32768 i

The interesting thing we can see now on D is the presence of A‘s subnet (10.0.0.0) advertised by B: even if B does not support 4-octect ASN, D receives it with the real 32-bit path. When A advertises the subnet to B it makes the AS_PATH attribute up using AS_TRANS, but it also adds the transitive attribute NEW_AS_PATH, containing the full 32-bit AS numbers; when B advertises the subnet to D it adds this attribute and so D can rebuild the real 32-bit path.

As before, you can find a similar UPDATE on the capture I sent on PacketLife.net; the file is 4-byte_AS_numbers_Mixed_Scenario.cap, packet number 3.

Another interesting aspect of 4-byte AS lack of support can be seen in the following table:

B#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*  10.0.0.0         172.16.2.2                             0 23456 23456 i
*>                  172.16.3.1               0             0 23456 i
*> 20.0.0.0         0.0.0.0                  0         32768 i
*> 40.0.0.0         172.16.2.2               0             0 23456 i
*                   172.16.3.1                             0 23456 23456 i

Here B has two entries for the 10.0.0.0 subnet: they both seem to be originated from the same AS, but that’s not true: the first comes from D (AS 40.1), while the second (the selected one) comes directly from A.

Similar behaviour is for the 40.0.0.0 subnet. Of course, this is not a really big problem, but may lead to false assumptions and compromise traffic patterns analysis.

16-bit stub router

Just to complete all the possible scenarios, here is C configuration:

B#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
B(config)#router bgp 2
B(config-router)#neighbor 172.16.4.2 remote-as 3
C#sh run | sec bgp
router bgp 3
 no synchronization
 bgp log-neighbor-changes
 network 30.0.0.0
 neighbor 172.16.4.1 remote-as 2
 no auto-summary
C#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.4.1                             0 2 23456 i
*> 20.0.0.0         172.16.4.1               0             0 2 i
*> 30.0.0.0         0.0.0.0                  0         32768 i
*> 40.0.0.0         172.16.4.1                             0 2 23456 i

It has no idea of what 4-byte ASN are, so it sees both 10.0.0.0 and 40.0.0.0 subnets as originated from the same AS.

Thanks

I have to say thanks to Marco Rizzi for his kind support on helping me to build this lab! And, of course, I suggest you to visit his blog: Marco Rizzi Blog – networking with passion!!!

Download the lab

To download the lab and configuration files click here.

References

Report on consumption of AS Numbers: http://www.potaroo.net/tools/asns/

BGP Support for Four-octet AS Number Space: RFC 4893

APRICOT 2007: 4-Byte AS Numbers (PDF)

Cisco.com: Migration Guide for Explaining 4-Byte Autonomous System

Cisco.com: Cisco IOS BGP 4-Byte ASN Support

Cisco.com: Explaining 4-Byte Autonomous System (AS) ASPLAIN and ASDOT Notation for Cisco IOS

Components used in the solution

As it was for the previous solution, two main components are involved, one to acquire NetFlow data, the other to analyze them in an easy way.

NFDUMP is a suite of tools composed by many programs; nfcapd is one of them and it is the daemon which listens for incoming NetFlow data. It is launched by NfSen.

NfSen is the web based front end we use to analyze NFDUMP NetFlow data. It is splitted up in two parts: a Perl program which runs in the background and launches nfcapd, and some PHP web pages.

The biggest difference between a NFDUMP/NfSen based solution and a flow-tools/FlowViewer solution is that the former does support NetFlow v9, while the latter may result in a simpler and easier tool to work with. Anyway, there is not one perfect solution for all the needs!

In this post I’ll use NFDUMP 1.6.1 and NfSen 1.3.2, the latest stable releases I can find on SourceForge while I’m writing this blog entry; you can find their web pages in the References section of this post. I’ll build them from source code.

NFDUMP

Let’s add some tools we’ll need to build the package:

apt-get install gcc flex librrd-dev make

We are ready to start! The steps are very simple: download source code and extract the tar:

cd /usr/local/src/
wget http://sourceforge.net/projects/nfdump/files/stable/nfdump-1.6.1/nfdump-1.6.1.tar.gz/download
gzip -dc nfdump-1.6.1.tar.gz | tar -xf -
cd nfdump-1.6.1

In order to use NFDUMP with NfSen we need to enable nfprofile in the configure step; then we have to make && make install it:

./configure --enable-nfprofile 
make
make install

Now it is ready to be ran.

If we want to use it without NfSen we can stop here and create a simple init.d script to launch NFDUMP at startup; it will acquire NetFlow data and we can use command line tools (such as nfdump) to analyze them. You can find a basic init.d script at the end of this post (“Annex A: stand-alone NFDUMP”).

But that’s not our goal, we want NfSen!

NfSen

As said, NfSen has a PHP front-end so it needs a web server with PHP support in order to be used. It also uses RRD to store data and paint nice graphs and some Perl modules to produce alerts: let’s satisfy its prerequisites:

apt-get install apache2 libapache2-mod-php5 php5-common libmailtools-perl rrdtool librrds-perl

Now it’s time to download and configure the program:

cd /usr/local/src/
wget http://sourceforge.net/projects/nfsen/files/stable/nfsen-1.3.2/nfsen-1.3.2.tar.gz/download
gzip -dc nfsen-1.3.2.tar.gz | tar -xf -
cd nfsen-1.3.2

Once extracted, we have to copy the config file into /etc and modify it:

cp etc/nfsen-dist.conf /etc/nfsen.conf
nano /etc/nfsen.conf

As you can see from the support web page it presents a lot of parameters, but many of them may be left unchanged; in this example I chose to leave as many parameters as possible at their default settings, so we have all the files in a single directory: /data/nfsen. Here the parameters I changed:

$USER    = "www-data";
$WWWUSER  = "www-data";
$WWWGROUP = "www-data";

%sources = (
    'MYROUTER'    => { 'port'    => '9995', 'col' => '#0000ff', 'type' => 'netflow' },
);

$MAIL_FROM   = 'MYEMAIL@MYDOMAIN.COM';
$SMTP_SERVER = 'MY.SMTPSERVER.COM';

For a better setup I suggest you to read the configuration guide on the project’s web site.

Let’s make the destination directory and complete the installation by running the install.pl script:

mkdir -p /data/nfsen
./install.pl /etc/nfsen.conf

We are ready to run the nfsen program and let it executes nfcapd daemon:

cd /data/nfsen/bin
./nfsen start

We can also schedule it to be ran at startup:

ln -s /data/nfsen/bin/nfsen /etc/init.d/nfsen
update-rc.d nfsen defaults 20

It’s ready: point your browser at http://YOUR_IP_ADDRESS/nfsen/nfsen.php and enjoy! Of course, this is just a very basic setup, both NFDUMP and NfSen have many options you can set and use to have better performances and to improve stability and scalability. If you want to go deep into their configuration I suggest you to use the links you can find at the end of this post.

P.S.: Don’t forget to configure your routers to export NetFlow data!

Annex A: stand-alone NFDUMP

As said, NFDUMP may be used in a stand-alone way, without the help of NfSen.

We just have to build a destination directory for NetFlow data…

mkdir -p /var/flows/MYROUTER
mkdir -p /var/flows/MYSECONDROUTER

and add an init.d script like the following one:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          nfcapd
# Required-Start:    $network
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: netflow capture daemon
# Description:       nfcapd is the netflow capture daemon of the nfdump tools.
### END INIT INFO

# Author: Erik Wenzel <erik@debian.org>
# Edited by Pierky for the blog post "NetFlow: installation and configuration of 
# NFDUMP and NfSen on Debian" on http://pierky.wordpress.com

# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin
DESC="netflow capture daemon"
NAME=nfcapd
DAEMON=/usr/local/bin/$NAME
DATA_BASE_DIR="/var/flows"
PIDFILE=/var/run/$NAME.pid
DAEMON_ARGS="-D -w -S 1 -P $PIDFILE -n MYROUTER,192.168.0.1,$DATA_BASE_DIR/MYROUTER"
SCRIPTNAME=/etc/init.d/nfdump

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
                || return 1
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
                $DAEMON_ARGS \
                || return 2
        # Add code here, if necessary, that waits for the process to be ready
        # to handle requests from services started subsequently which depend
        # on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        #
        # Disabled second call, because is kills nfsen controlled nfcapd
        #start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        #[ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
        return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
        #
        # If the daemon can reload its configuration without
        # restarting (for example, when it is sent a SIGHUP),
        # then implement that here.
        #
        start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
        return 0
}

case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
                0|1) [ q"$VERBOSE" != qno ] && log_end_msg 0 ;;
                2)   [ p"$VERBOSE" != pno ] && log_end_msg 1 ;;
        esac
        ;;
  stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
                0|1) [ "r$VERBOSE" != rno ] && log_end_msg 0 ;;
                2) [ "s$VERBOSE" != sno ] && log_end_msg 1 ;;
        esac
        ;;
  #reload|force-reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        #log_daemon_msg "Reloading $DESC" "$NAME"
        #do_reload
        #log_end_msg $?
        #;;
  restart|force-reload)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
          0|1)
                do_start
                case "$?" in
                        0|1) log_end_msg 0 ;;
                        *) log_end_msg 1 ;; # Failed to start
                esac
                ;;
          *)
                # Failed to stop
                log_end_msg 1
                ;;
        esac
        ;;
  *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac
exit 0

I edited this script starting from the one included in the Debian’s package.

Please note the DATA_BASE_DIR and the DAEMON_ARGS variables. The first contains the base destination directory for our NetFlow data. The second represents arguments we want to use when launching nfcapd; you can add as many -n options as you want, one for each NetFlow source you want to collect data from. Remember: man nfcapd is your friend!

Finally:

chmod a+x /etc/init.d/nfdump
update-rc.d nfdump defaults 20
/etc/init.d/nfdump start

References

Cisco.com: NetFlow version 9

NFDUMP: http://nfdump.sourceforge.net/

NfSen: http://nfsen.sourceforge.net/