Example Puppet 2.7 git pre-commit script

I had a hard time finding a decent pre-commit script for puppet 2.7. This is composed of snippets I found at code.seas.harvard.edu. The pre-commit script will check the puppet syntax of the changed .pp files, and also check if an attempt has been made to properly document the .pp file.


#!/bin/sh
#
# install this as .git/hooks/pre-commit to check Puppet manifests
# for errors before committing changes.

rc=0

[ "$SKIP_PRECOMMIT_HOOK" = 1 ] && exit 0

# Make sure we're at top level of repository.
cd $(git rev-parse --show-toplevel)

trap 'rm -rf $tmpdir $tmpfile1 $tmpfile2' EXIT INT HUP
tmpdir=$(mktemp -d precommitXXXXXX)
tmpfile1=$(mktemp errXXXXXX)
tmpfile2=$(mktemp errXXXXXX)

echo "$(basename $0): Validating changes."

# Here we copy files out of the index into a temporary directory. This
# protects us from a the situation in which we have staged an invalid
# configuration using ``git add`` but corrected the changes in the
# working directory. If we checked the files "in place", we would
# fail to detect the errors.

git diff-index --cached --name-only HEAD |
grep '\.pp$' |
git checkout-index --stdin --prefix=$tmpdir/

find $tmpdir -type f -name '*.pp' |
while read manifest; do
puppet parser validate $manifest | sed "s#$tmpdir/##" >> $tmpfile1 2>&1

if ! head -1 $manifest | grep -q '^#'; then
echo $manifest | sed "s#$tmpdir/##" >> $tmpfile2
fi
done

if [ -s "$tmpfile1" ]; then
echo
echo Error: Puppet parse problem:
echo ----------------------------
cat $tmpfile1
echo ----------------------------
echo

rc=1
fi

if [ -s "$tmpfile2" ]; then
echo
echo Error: missing manifest documentation
echo see http://projects.puppetlabs.com/projects/puppet/wiki/Puppet_Manifest_Documentation
echo for more information.
echo Files with problems:
echo -------------------------------------
cat $tmpfile2
echo -------------------------------------
echo

rc=1
fi

exit $rc


LISP is serious business

Some perspective with regards to the maturity of the protocol specifications and implementations of the Locator/Identifier Separation Protocol (LISP):

The first LISP specification was published in January 2007 as an individual submission. After 13 revisions the Internet-Draft was adopted as an IETF LISP Working Group document. Within the LISP Working Group there have been 12 versions of the main Internet-Draft. Literally hundreds of contributors from a lot of different companies have made suggestions and fixed bugs to make the LISP specification what it is today.

The first implementation started at the Prague IETF conference in 2007. As of today there are about 10 implementations: Linux, Android, FreeBSD (OpenLISP), Zlisp, LISP-Click, FNSC FITELnet G21, IOS, NX-OS, IOS-XR and IOS-XE. Please note that not all of them have yet been released or are of production quality. I recommend using the implementations developed by Cisco because they are the most mature and feature rich implementations.

Better yet, recently Cisco announced to the world its first production software releases 15.1(4)M and 15.1(2)S which support the Locator/Identifier Separation Protocol (LISP). Cisco has committed to make LISP, as an emerging standard, available on all its major routers and switches in mainstream trains including the Nexus 7000.

So what does all this mean?

The LISP specification has received a lot of attention and review inside the IETF. This is a very good thing because it makes the protocol stronger and development less prone to for example tunnel vision. The academic world has developed various proof-of-concept implementations to discover its scalability properties and other testing purposes. Unlike other protocols such as ILNP, HIP, SHIM, and IRON-RANGER which have received significantly less review or have almost no real-world deployment experience.

Big vendors are investing lots of resources into the development of their respective LISP implementations, which means LISP is here to stay.

Nagios and IPv6 made easy with the mknagconf configuration generator

This article describes how to install Nagios3 and my mknagconf tool and how to use them. It should take about 30 minutes to install nagios3 and mknagconf and set it up to monitor a few hosts. The following has been tested with Ubuntu 10.04, 10.10 and 11.04 on an amd64 platform.

Nagios3 is an excellent monitoring engine, but the stock Nagios has some limitations in regard to dual-stack hosts. In the Nagios universe, one host is one ip address, and a secondary ipv6 address would require an extra host definition.

The Nagios packages which you are about to install have been patched to support this concept "one host = 1 ipv4 address + 1 optional ipv6 address". The mknagconf script makes it easier to maintain your Nagios installation. mknagconf takes small short, and simple definition files, parses them and generate the configuration files for Nagios. This will be explained after installing the required software.

Step 1: Install all dependencies

apt-get install apache2-mpm-prefork apache2-utils apache2.2-bin \  apache2.2-common bsd-mailx libapache2-mod-php5 libapr1 \  libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap \  libgd2-noxpm libjpeg62 libperl5.10 nagios-plugins-basic \  php5-common postfix ssl-cert nagios-plugins-standard \  nagios-plugins-extra git-core make

Step 2: obtain PGP key, configure apt to fetch updates from my repo

Add my PGP key to your apt setup and install the Nagios packages from my repository:

apt-key adv --keyserver keyserver.stack.nl --recv-keys 7E5BEC10echo "deb http://snijders-it.nl/deb natty main" \> /etc/apt/sources.list.d/snijders-it.listapt-get update

Step 3: install nagios

During the following step the Nagios installation will ask you for a password for the user 'nagiosadmin'. Don't forget to use a strong password!

apt-get install nagios3 nagios3-core nagios3-doc \    nagios3-common nagios3-cgi

Step 4: download and install mknagconf

This step downloads the latest version of the script and example configuration files from github. After downloading you can run the ./install.sh script to copy all files to the correct place. This script will overwrite the current nagios configuration from Step 3.

cd /usr/srcgit clone git://github.com/SnijdersIT/mknagconf.gitcd mknagconfsudo ./install.sh

Step 5 (optional): setup local repo for configs

It's usually good to do nagios stuff in some kind of repository, an example would be:

cd /etc/nagios3git initgit config --global user.name "your name"git config --global user.email you@example.netecho "conf.d/*_autogenerated.cfg" >> .git/info/excludegit add *git commit -am 'first nagios config'

Step 6: edit the hosts.def definition file

Open the following file in your texteditor:

/etc/nagios3/definitions/hosts.def

You will see a line like:

hpserv;192.168.23.4;2001:db8:1::65;coregate;central HP server

Which will be interpreted as: host 'hpserv' with ipv4 address "192.168.23.4" and ipv6 address "2001:db8:1::65", has 1 parent namely: "coregate" and is also called the "central HP server".

The ipv6 address, parents and human name are optional. You can specify multiple parents if you separate them with a comma. Don't forget to create a line specifying the parent itself!

Remove the hpserv line and add a (dual-stack) host you would like to monitor. Define one host per line. In the next step you can define what services you want to monitor on that host.

Step 7: edit the services.def definition file

Open the following file in your texteditor:

/etc/nagios3/definitions/services.def

You will see a line like:

hpserv = ping, ping6, ftp, ssh6

Which will be interpreted as: perform an IPv4 ping check, IPv6 ping check, an FTP connect over IPv4 and a SSH connect over IPv6 to host "hpserv". Define one host per line, and all services that should be checked on that host on the same line.

Step 8: configure correct contact

Open the following file and modify it according to your situation:

/etc/nagios3/conf.d/contacts.cfg

Step 9: generate configuration, reload nagios

Change to the nagios3 configuration directory, run the make commands to clean up the auto-generated files, compile the nagios configurations from the definition files and check if the configurations makes any sense.

cd /etc/nagios3make cleanmakemake check && /etc/init.d/nagios3 reloadgit commit -am 'added my first hosts' # optional step

For more information open the various files in /etc/nagios3/definitions/ and read their headers. Further recommended reading: the README file in /usr/src/mknagconf/

If at some later point you want to add more hosts or delete old ones, just go through step 6, 7 and 9.

That's it! Have fun using Nagios :-)

How Network Operators can cooperate: the NLNOG RING

In December 2010 I started a project with a few friends to make life for network engineers in the Netherlands better.

I noticed that there are a lot of friendly 'shell access' exchange deals between dutch network operators. This makes it easier for parties to debug network issues and troubleshoot from the outside. A point of view outside your network is absolutely essential, seeing what others see is a useful thing for a variety of network problems. Well known examples are "it works for even numbered ip address, but not for odd numbered ip address via this and this route".

The NLNOG RING tries to do this in a more organized way, basically the deal is "donate 1 machine, and gain access to all other machines in the ring". So far already 10 organisations are participating.

How useful is the ring exactly? A very nice example is executing a traceroute from ten different autonomous systems: nlnog ring example.

More information about the NLNOG RING can be found on the website we've launched today: ring.nlnog.net.

New Cisco IOS releases in an RSS feed

Over the last few days we've been spending some time on an RSS feed generator which can help you stay on top of new IOS releases. It takes regular expressions as input and can be useful for a quick search or generating an RSS feed for your favorite news reader.

The database is built upon a public md5 database Cisco publishes roughly every week. I cannot vouch for the accuracy of this information.

You can find the quick search & rss generator at http://snijders-it.nl/ios-rss/.

LISP + GETVPN as alternative for DMPVN+OSPF+GETVPN

Originally LISP was developed to address the issues and concerns raised by the growth of the internet routing table, but LISP turns out to possess appealing features that can be of interest to Service Providers like my friends at InTouch.

At the Cisco NAG2010 conference in San Jose I talked about using LISP as a transport mechanism instead of regular manual GRE tunnels or a DMVPN design. I believe that provisioning and debugging a LISP based virtual private network will be easier and simpler than current approaches.

Some fair warnings are in order here: this setup runs on beta IOS and NXOS images, this design and the configuration syntax are very likely to change a little with every IOS release, for Cisco's LISP implementation is under very active development. The most important aspect of this design is that it's not a multi-tenant architecture. Multi-tenancy will probably be available in a few months, after which I'll post an updated version with more comments on the specifics.

View the slides online at slideshare: LISP+GETVPN or download the PDF from my website: Job_Snijders-InTouch-LISP_GETVPN.pdf.

libvirt & KVM & unnumbered bridge setup

This is an ubuntu/debian recipe to use an 'unnumbered bridge' to save on the amount of IP addresses needed to connect your virtual machines on a libvirt host to other networks.

I'm assuming you want the host to be a router between the VM's and the external network. The advantage of this is that you can firewall traffic between the virtual machines and other networks on the host.

A setup like this can be used if your ISP provides you with a /29 and you want to be able to use every IP address out of that /29, and not waste IP's on the network, broadcast and gateway address.

This image shows the various elements involved:

The /etc/network/interfaces file on the host:

auto virbr0
iface virbr0 inet manual
  bridge-ports none
  bridge_stp off
  bridge_maxwait 1
  post-up ip route add 10.10.10.0/29 dev virbr0

The above configuration will configure a bridge interface without an IPv4 address and route the /29 that was assigned to you by your ISP to that interface. This will force Linux to ARP for every IP from that /29 on this particular virbr0 interface.

The following virsh commands will remove the default network settings, you can enter them on a root prompt:

virsh net-destroy default
virsh net-undefine default

Virtlib's /etc/qemu-ifup script needs to be replaced with the following:

#!/bin/sh
switch="virbr0"
/sbin/ifconfig $1 0.0.0.0 up
/usr/sbin/brctl addif ${switch} $1

The above will make sure that any tap interface associated with a virtual machine will be added to the correct bridge. The default /etc/qemu-ifup file will try and guess the correct bridge, which is not desirable.

You can edit the XML describing the virtual machines interface with the following command: 'virsh edit NAME_OF_VM'. You could also just type this command:

virsh attach-interface NAME_OF_VM bridge virbr0

This is what the XML could like like:

<interface type="'bridge'">
  <mac address="'52:54:00:51:8b:ad'/">
  <source bridge="'virbr0'/">
  <target dev="'vnet0'/">
</interface>

The last step is to configure the network settings inside the virtual machine (VM1). This is an example /etc/network/interfaces. Notice the pointtopoint statement and the subnet mask:

# The primary network interface
auto eth0
iface eth0 inet static
       address 10.10.10.1
       netmask 255.255.255.255
       broadcast 10.10.10.1
       gateway 10.10.9.2
       pointopoint 10.10.9.2

To wrap it all up:

• /etc/network/interfaces on the host takes care of creating the virbr0 bridge
• libvirt will make a vnetX interface (so don't worry about it)
• /etc/qemu-ifup adds tap interfaces to the correct bridge when booting the virtual machine
• thanks to ARP the machines will figure out who is where
• /etc/network/interfaces inside the virtual machine will force traffic towards the host

I'd like to thank Sten Spans for providing some essential hints.