5 minutes collectd + facette setup

I recently added a fantastic graphing tool named facette to pkgsrc.
Facette knows how to pull data sources from various backends, and among them, the famous collectd.

In this article, we will see how to setup both on NetBSD but keep in mind it should also work for any platform supported by pkgsrc.

First up, collectd installation. It can be done either with pkgin (binary installation) or pkgsrc (source installation):

  • with pkgin
1
$ sudo pkgin in collectd collectd-rrdtool
  • with pkgsrc
1
2
3
4
$ cd /usr/pkgsrc/sysutils/collectd
$ sudo make install clean
$ cd ../collectd-rrdtool
$ sudo make install clean

Tune up a minimal collectd configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Hostname    "myname"
BaseDir "/var/db/collectd"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/pkg/lib/collectd"
LoadPlugin syslog
LoadPlugin cpu
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin rrdcached
LoadPlugin rrdtool
<Plugin rrdtool>
DataDir "/var/db/collectd/rrd"
CreateFilesAsync false
CacheTimeout 120
CacheFlush 900
WritesPerSecond 50
</Plugin>

Enable and start collectd

1
2
# echo "collectd=YES" >> /etc/rc.conf
# /etc/rc.d/collectd start

Wait a couple of minutes for collectd to actually collect some datas, you should see them appear in /var/db/collectd/rrd.

As I write these lines, facette is not yet available as a binary package, it will probably be available in the next pkgsrc release, so we’ll have to install it using pkgsrc

1
2
3
4
5
$ cd /usr/pkgsrc/sysutils/facette
$ sudo make install clean
$ sudo -s
# echo "facette=YES" >> /etc/rc.conf
# /etc/rc.d/facette start

That’s right, no configuration, I’ve setup the package so it is working out-of-the-box with a basic collectd installation.

If everything went well, you should be admiring a facette console by connecting with your web browser to the port 12003

facette

Simpler postfix + dspam

I have read a shitload of overcomplicated setups to bring up a postfix / dspam SMTP + antispam server, and finally came to a much lighter and simpler configuration by basically reading documentation and real life examples.
Note this is suitable for a personnal and basic environment, no database, no virtual setup. Basic stuff.

The target system is NetBSD but this short doc should apply to pretty much any UNIX / Linux.

On dspam‘s side, I added the following parameters:

1
2
3
4
5
6
7
8
9
10
# really postfix
TrustedDeliveryAgent "/usr/sbin/sendmail"
[...]
# add involved users
Trust dspam
Trust postfix
[...]
# declare UNIX socket
ServerDomainSocketPath "/tmp/dspam.sock"
ClientHost /tmp/dspam.sock

On postfix‘s main.cf side:

1
2
3
4
5
# don't overwhelm dspam, only one message at a time
dspam_destination_recipient_limit = 1
smtpd_client_restrictions =
permit_sasl_authenticated
check_client_access regexp:/etc/postfix/dspam_filter_access

Warning, I used regexp: instead of pcre: because that’s what NetBSD base’s postfix supports.
The dspam_filter_access pipes the message to dspam‘s socket by matching everything:

1
2
$ cat /etc/postfix/dspam_filter_access
/./ FILTER dspam:unix:/tmp/dspam.sock

The only remaining piece is to declare the dspam service in postfix‘s master.cf file:

1
2
dspam     unix  -       n       n       -       10      pipe
flags=Ru user=dspam argv=/usr/pkg/bin/dspam --deliver=innocent,spam -i -f ${sender} --user ${user} -- ${recipient}

The final delivery method is up to you, but I chose procmail, mostly because I have written my rules a while ago and am too lazy to adapt them to sieve :)

1
mailbox_command = /usr/pkg/bin/procmail

Sources:

Start pkgsrc's nginx with systemd

Not so long ago, I wrote about using pkgsrc on Debian GNU/Linux, and assumed you’d start an installed service using rc.d. When I setup the new iMil.net server, I decided to give a try to kvm as it is easier to maintain, has good performances (sometimes better than Xen), nice administration tools, plus NetBSD now has a good VirtIO driver but no PVHVM support yet.

The first thing I do when setting up a Debian Jessie server is getting rid of systemd, whose philosophy and quality don’t match my personnal taste; but in that case, I wanted to use libvirtd so I could manage my virtual machines with virt-manager, and as a matter of fact, libvirtd has a hard dependency on systemd. There was no escape this time, I had to learn and use it.

Once nginx installed through pkgsrc, I wrote a unit file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cat /etc/systemd/system/nginx.service 
[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network.target

[Service]
Type=forking
ExecStartPre=/usr/pkg/sbin/nginx -t
ExecStart=/usr/pkg/sbin/nginx
ExecReload=/usr/pkg/sbin/nginx -s reload
ExecStop=/usr/pkg/sbin/nginx -s quit

[Install]
WantedBy=multi-user.target

Then enabled the nginx service:

1
$ sudo systemctl enable nginx

And finally started it:

1
$ sudo systemctl start nginx

Witness everything went as expected:

1
2
3
4
5
6
7
8
9
10
11
$ sudo systemctl status nginx -l
● nginx.service - The NGINX HTTP and reverse proxy server
Loaded: loaded (/etc/systemd/system/nginx.service; enabled)
Active: active (running) since Mon 2016-02-08 11:54:48 CET; 2 weeks 5 days ago
Main PID: 23512 (nginx)
CGroup: /system.slice/nginx.service
├─11453 nginx: worker proces
└─23512 nginx: master process /usr/pkg/sbin/ngin

Feb 08 11:54:48 starkiller nginx[23508]: nginx: the configuration file /usr/pkg/etc/nginx/nginx.conf syntax is ok
Feb 08 11:54:48 starkiller nginx[23508]: nginx: configuration file /usr/pkg/etc/nginx/nginx.conf test is successful

Note that this does not prevent from using nginx‘s -t or -s flags.

(not-so) new website!

If you’re used to this website you might have noticed the layout has somewhat changed. Actually the engine itself has changed, iMil.net is no more powered by wordpress, instead I switched to a static website generator called hexo. While it can be tricky sometimes, the tool is nicely organized and easy to handle.

On the service side, this web site defaults to HTTPS and is natively IPv6 ready, it is served by an nginx server contained in a sailor ship. Of course, the virtual machine runs NetBSD, on a kvm hypervisor, hosted on Debian GNU/Linux.

Hope you like it!

GRE tunnel PREROUTING

Here’s a simple solution in order to forward GRE tunnels to a server, or here a virtual machine, which is located behind a GNU/Linux gateway:

1
2
# iptables -t nat -A PREROUTING -i eth0 -p gre -j DNAT --to-destination 192.168.0.1
# modprobe nf_conntrack_proto_gre

No need for complex PREROUTING / POSTROUTING / FORWARD combinations as I could read here and there.

In my case, the virtual machine is a NetBSD domU where I created the following gre(4) interface:

1
2
3
4
# cat /etc/ifconfig.gre0
create
tunnel 1.2.3.4 192.168.0.1 up
inet 172.16.0.1 172.16.0.2 netmask 255.255.255.252

1.2.3.4 being the remote public IP address
192.168.0.1 is the domU private IP address
172.16.0.1 and 2 are the tunnel endpoints.

NetBSD/amd64 7.0 on kvm

If you recently tried to install NetBSD 7.0 using Linux KVM you might have encountered the following failure:

This bug have been recently fixed on the 7-branch but the official ISO images are not yet updated, so you’ll have to use NetBSD daily builds mini-ISO which includes Christos fix to bus_dma.c

For the record, here’s the virt-install command I use:

1
2
3
4
5
6
7
8
9
10
11
sudo virt-install
--virt-type kvm
--name korriban
--ram 4096 --disk path=/dev/vms/korriban,bus=virtio
--vcpus 2
--network bridge:br0,model=virtio
--graphics vnc
--accelerate
--noautoconsole
--cdrom /home/imil/iso/boot.iso
--cpu host

Performances are really good, the host is a Debian GNU/Linux 8.0 amd64 running on Online’s Dedibox Classic 2015.

YAML and markdown based website rendering with AngularJS

A couple of weeks ago, Clark / @jeaneymerit told me he was digging into AngularJS, and as I’m working on a private project where a static website is involved, I thought this framework could help me make that website lightweight in terms of external dependencies.

The site I’m working on contains exclusively static content, and most of it is text, I wanted a simple and elegant method in order to manipulate that content easily, so I wrote a basic website generator in python based on jinja2, for the record it’s available here.

The principle is that a basic YAML file will contain the text of a page in a key / value fashion. For my own needs, there are two kinds of text, simple or markdown, the file look like this:

1
2
3
4
5
welcome: 'welcome!'
md_about: |
**Yes**! this _new_ website is powered by [markdown][1]

[1]: http://daringfireball.net/projects/markdown/

Now, with modern frameworks poping all around the web, I felt that “building” the website, even if it was a really simple task, was an oldschool approach, plus I needed a pretext to get my hands on AngularJS :)

It turns out there were plenty of modules to handle the YAML / markdown mechanism, I picked js-yaml and marked after trying a couple of other similar modules which were less effective and / or buggy.

Now the interesting part. I won’t explain AngularJS principles in this blog post as there are very informative and well written documentation on the subject, among them this quick tutorial from w3schools and of course the official AngularJS documentation; instead, I’ll drop here the main component of the AngularJS-based website, the controller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
var mySite = angular.module('mySite', ["ngSanitize"]);

mySite.controller('contentCtrl', function($scope, $parse, $http, $rootScope) {

$http.get('content/index_' + $scope.lang + '.yaml')
.success(function(response) {
angular.forEach(jsyaml.load(response), function(v, k) {
if (k.startsWith('md_') == true) {
$parse('md.' + k.substring(3)).assign($scope, marked(v));
} else {
$parse(k).assign($scope, v);
}
if (k == 'title') {
$rootScope.pageTitle = v;
}
});
});
})

That’s right, in less than 20 lines of code, we read our YAML file, converted it to good old JSON and passed the content of both simple and markdown values to the HTML view.

A word of explanation on this controller. I chose that the YAML file name would correspond to the language the site is displayed with, for example, if lang == 'en', content/index_en.yaml will be read. The distinction between simple and markdown content is made by reading the first 3 characters of the key, if it begins with md_, then it’s markdown and it should be interpreted by marked.

There are two tricky traps here. First about the rendered markdown. For security reasons, AngularJS will not display the produced HTML when the expression is called with double braces {{ .. }} within the HTML view, instead it will show the HTML code escaped. It is mandatory to reference the key variable through a ng-bind-html tag, and as the documentation explains, you’ll have to add angularjs-sanitize.js to the list of loaded modules.

<span ng-bind-html="md.about"></span>

Now about the scope variables organization. I learned that an AngularJS expression will not be interpreted as a scope variable if it is the result of a string manipulation, for example, if foo == 'about':

<span ng-bind-html="'md_' + foo"></span>

will only print md_foo, it will not reference the md_about scope variable. In order to play with dynamic naming, I was forced to organize the markdown variables in an array, thus the following trick in the controller:

$parse('md.' + k.substring(3)).assign($scope, marked(v));

so I can have the following code in the HTML view:

<span ng-bind-html="md[foo]"></span>

Why so much pain? Because thanks to this little trick, I’ll be able to create loops for similar blocks in the HTML view, such as:

<div class="col-md-4" ng-repeat="cat in ['foo', 'bar', 'baz']">
  <img class="center-block" src="media/images/undefined.png" />
  <h3 class="text-center"></h3>
  <span ng-bind-html="md[cat]"></span>
</div>

I like to factorize.

I’m yet a total AngularJS newbie, if something in this blogpost seem awful to you, ng-guru, please feel free to contact me!

Annoying WiFi captive portal

The hotel I stay in Ibiza is awesome, it’s in front of the sea, the staff is nice and welcoming, there’s this amazing pool, AND… they have pretty good WiFi for the Island standards.

But this year, they’ve put those D-Link DWC-1000 repeaters all around the hotel which disconnects you every now and then, you have to re-register via a web page everytime, which is quite annoying. So I came up with this very simple python script that checks if we can reach the outside world, and if not, will use python mechanize in order to register automatically to the captive portal. Not the greatest script I’ve released, but hey, first beaches, parties and beers, then only comes IT :)

#!/usr/bin/env python

import os
import time
import mechanize

def recon():
    br = mechanize.Browser()

    br.set_handle_robots(False)
    br.open('http://192.168.1.1/platform.cgi?page=captivePortal.htm')

    response = br.response()
    headers = response.info()
    # that crappy repeater doesn't even reply the correct content-type
    headers["Content-type"] = "text/html; charset=utf-8"

    br.select_form(nr=0)

    br.form['CaptivePortalSession.UserName'] = 'myuser'
    br.form['CaptivePortalSession.Password'] = 'mypasswd'

    br.submit()

    print br.response().info()

while True:
    # yeah well, I'm not running for the best script of the year ok?
    r = os.system('ping -c 1 8.8.8.8')
    if r != 0:
        recon()
    time.sleep(5)

HTH!

Using pkgsrc on debian GNU/Linux

While I tend to appreciate debian GNU/Linux, its tendency to be quite late on software versionning is sometimes annoying. Also, as a pkgsrc developer, I am used to have greater control over the packages I install, for example regarding the options I’d like to include.

For these reasons and a couple more, I sometimes choose to use pkgsrc along with apt to deal with particular packages. In this article, I’ll show how to achieve that task.

First install build pre-requisite packages:

1
# apt-get install cvs libncurses5-dev gcc g++ zlib1g-dev zlib1g libssl-dev libudev-dev

Then fetch pkgsrc:

1
# cd /usr && cvs -d anoncvs@anoncvs3.de.NetBSD.org:/cvsroot co pkgsrc

export the SH environment variable to /bin/bash:

1
# export SH=/bin/bash

And bootstrap pkgsrc:

1
2
# cd /usr/pkgsrc/bootstrap
# ./bootstrap

From now on, you’ll have a /usr/pkg directory filled with necessary bits for building packages from pkgsrc.

If you are to install services from pkgsrc packages, you’ll have to copy NetBSD‘s /etc/rc.subr to debian‘s /etc directory:

1
# wget -O/etc/rc.subr "http://cvsweb.netbsd.org/bsdweb.cgi/~checkout~/src/etc/rc.subr?rev=1.96&content-type=text/plain"

Create an ad-hoc rc.d directory:

1
# mkdir /usr/pkg/etc/rc.d

Let’s say you’d like to install nginx out of pkgsrc, possibly because debian‘s version is outdated or it does not contain your favorite module. Add the desired option to pkgsrc's options file, /usr/pkg/etc/mk.conf:

1
PKG_OPTIONS.nginx+= naxsi spdy

Build the software:

1
2
# cd /usr/pkgsrc/www/nginx
# /usr/pkg/bin/bmake install clean clean-depends

Copy the startup script:

1
# cp /usr/pkg/share/examples/rc.d/nginx /usr/pkg/etc/rc.d/

Enable the service:

1
# echo "nginx=YES" &gt;&gt; /etc/rc.conf

And start it:

1
# /usr/pkg/etc/rc.d/nginx start

Now, how you integrate services start to your favorite init system is up to you!