But most of all, I’ve been preparing my ${DAYJOB} infrastructure for Salt, and I must say this has been much easier than I thought, thanks to this beautifully designed piece of code.
One aspect I’d like to share is the simple way I found to make a minion dynamically configured, through custom-made grains.

Like many companies, we have an Information System, which has a bunch of informations about everything on our network, and among those, the roles of the virtual machines which are started. Those informations are made available to a minion through an HTTP server, which will recognize the minion’s REMOTE_ADDR and tell him what are its «roles».

The webserver is our always-favorite nginx, which interfaces a WSGI server, uWSGI.

nginx‘s configuration is simple:

location / {
        include uwsgi_params;
        uwsgi_pass unix:///var/run/uwsgi/app/is/socket;
}

uwsgi application’s .ini is pretty trivial too:

[uwsgi]
workers = 2
log-date = true
plugins = python
chdir = /var/www/is
module = is

A very simple python script using web.py resides in /var/www/is and gives the minion which will query him a YAML structured output containing, among other data, the minion‘s roles.

I wrote a basic grain module which retrieves those informations and make them available as grains for the minions:

$ cat _grains/is_http.py 

import requests
import yaml

def more_infos():
        '''
        Returns the minion's roles and stuff.
        '''
        r = requests.get('http://private.fqdn/')

        return yaml.load(r.text)

After deploying that grain module via salt '*' saltutil.sync_grains, the minions are now aware of what they’re meant to do, thus making a generic top.sls easy to write.

Thanks to Jinja2, a sls file can be made more dynamic, and we can write a very complex scenario within a couple of lines:

$ cat top.sls 
base:

  {% set states = salt['cp.list_states'](env) %}

  '*':
    - common
    {% if 'roles' in grains %}
    {% for role in grains['roles'] %}
    {% if role in states %}
    - {{ role }}
    {% endif %}  # state exists
    {% endfor %} # for roles
    {% endif %}  # role exists

  'virtual_subtype:Xen PV DomU':
    - match: grain
    - domus

  {% if grains['host'] in states %}
  {{ grains['host'] }}:
    - {{ grains['host'] }}
  {% endif %}

There we go, for every role a minion has, we can add a sls to match that role globally, as we can also add a very specific, host-related sls. Beautiful and handy, that’s what Salt is.

Update for Salt 0.15.3

Since Salt 0.15.3, a sls file can’t reference an non-existent state anymore, thus the additional checks in order to get sure the target state actually exists.

The post SaltStack: dynamic sls (updated for 0.15.3) appeared first on Emile "iMil" Heitor 's home.

Only drawback was it had poor NetBSD support. Was :)

It’s been a long time since I’ve dug into python, so it took me a little bit of effort, but Salt now has full support of pkgin in its generic packaging functions, knows how to handle NetBSD services and is capable of dealing with NetBSD‘s sysctl(8) and sysctl.conf.

Those pieces of code have been merged upstream, I hope they’ll be available in version 0.16!

Some examples:

$ cat packages/init.sls 
mypkgs:
  pkg.installed:
    - pkgs:
      - vim
      - tmux
      - bash
      - bash-completion
      - sudo

$ sudo salt '*' state.sls packages
watto:
----------
    State: - pkg
    Name:      mypkgs
    Function:  installed
        Result:    True
        Comment:   All specified packages are already installed.
        Changes:   
korriban:
----------
    State: - pkg
    Name:      mypkgs
    Function:  installed
        Result:    True
        Comment:   All specified packages are already installed.
        Changes:   
tatooine:
----------
    State: - pkg
    Name:      mypkgs
    Function:  installed
        Result:    True
        Comment:   All specified packages are already installed.
        Changes:   
coruscant:
----------
    State: - pkg
    Name:      mypkgs
    Function:  installed
        Result:    True
        Comment:   All specified packages are already installed.
        Changes:   
ragnos:
----------
    State: - pkg
    Name:      mypkgs
    Function:  installed
        Result:    True
        Comment:   All specified packages are already installed.
        Changes:   
exar:
----------
    State: - pkg
    Name:      mypkgs
    Function:  installed
        Result:    True
        Comment:   All specified packages are already installed.
        Changes:

$ sudo salt '*' cmd.run 'uname -a'
tatooine:
    Linux tatooine 3.2.0-4-686-pae #1 SMP Debian 3.2.41-2 i686 GNU/Linux
watto:
    NetBSD watto.home.imil.net 6.1_RC4 NetBSD 6.1_RC4 (GENERIC) i386
exar:
    NetBSD exar 6.0_STABLE NetBSD 6.0_STABLE (EXAR) #0: Sun Nov 25 12:39:12 CET 2012  root@exar:/usr/src/sys/arch/i386/compile/EXAR i386
coruscant:
    NetBSD coruscant 6.0 NetBSD 6.0 (XEN3_DOM0) amd64
korriban:
    NetBSD korriban.imil.net 6.0_STABLE NetBSD 6.0_STABLE (KORRIBAN) #0: Tue Jan  1 23:20:36 CET 2013  root@korriban.imil.net:/usr/src/sys/arch/amd64/compile/KORRIBAN amd64
ragnos:
    NetBSD ragnos 6.0 NetBSD 6.0 (RAGNOS) #2: Wed Oct 17 11:33:31 CEST 2012  root@ragnos:/usr/src/sys/arch/i386/compile/RAGNOS i386

$ sudo salt '*' pkg.version vim   
watto:
    7.3.762
exar:
    7.3.762
korriban:
    7.3.712
coruscant:
    7.3.762
ragnos:
    7.3.762
tatooine:
    2:7.3.547-7

$ sudo salt '*' service.status sshd
tatooine:
    False
watto:
    True
coruscant:
    True
exar:
    True
korriban:
    True
ragnos:
    True

If you whish to use these modules without tainting your Salt package installation, simply copy them to a _modules directory within the file_roots.

Happy Salting!

The post NetBSD configuration management appeared first on Emile "iMil" Heitor 's home.

In order to understand what was happening, I started estd with the -o flag, which outputs the CPU-frequencies as they are set. I then realized that the “ligh watermark percentage” and “low watermark percentage” default values were way too high (respectively 40 and 80) and were never reached, so the CPU speed was never changed.

With lower values, I was able to see the CPU speed increasing and lowering as expected. So I added the following line to the /etc/rc.conf file:

estd_flags="-l 5 -h 15 -a -m 800 -d"

meaning that the low watermark is set at 5 and the high watermark at 15, which were the values I’ve considered being the right ones while watching estd -o -a console output.

Since then, whenever a CPU intensive operation occurs, I can see the CPU speed rising with the following conky parameter:

CPU Frequency: ${alignr}${exec /sbin/sysctl -n machdep.powernow.frequency.current}

The post CPU dynamic scaling on NetBSD appeared first on Emile "iMil" Heitor 's home.

As written on the comments:

Colin Watson Says:
May 17th, 2013 at 7:12 pm

I uploaded 6.2 packages to Debian a week or so after you posted this, so you can/should now just use those instead. I expect they should build fine on wheezy.

As a matter of fact, the following is now deprecated

At ${DAYWORK}, we used to have our own OpenSSH debian package which included the famous OpenSSH LPK patch, which permits the use of an OpenLDAP server as an SSH public key provider.

I’ve been using OpenSSH-LPK for years, as this is a really handy solution and no valid alternative existed… until a couple of months.

OpenSSH 6.2 has a new configuration item called “AuthorizedKeysCommand”. The value associated to that key permits to call any executable as a public key provider. Yes, that is sexy.

Debian only have OpenSSH 6.1p1 packages available and tagged as “experimental”, so we had to hack a little bit in order to build 6.2 packages, here’s how:

• Fetch experimental source package

# echo "deb-src http://ftp2.fr.debian.org/debian/ experimental main contrib non-free" > /etc/apt/sources.list.d/experimental.list
# apt-get update
$ mkdir openssh && cd openssh
$ apt-get source openssh

• Bump the release

$ wget http://ftp.fr.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-6.2p1.tar.gz
$ cd openssh-6.1p1
$ uupdate -v 6.2p2 ../openssh-6.2p1.tar.gz
$ cd ../openssh-6.2p1
$ dch -i # enter changelog informations

• Get rid of conflicting patches

As expected, many patches from debian don’t apply anymore, and I was not brave enough to backport them, I’ve just commented them in debian/patches/series:

#gssapi.patch
#selinux-role.patch
#copy-id-restorecon.patch
#ssh-vulnkey.patch
#consolekit.patch
#user-group-modes.patch
#max-startups-default.patch
#package-versioning.patch
#debian-banner.patch
#lintian-symlink-pickiness.patch
#openbsd-docs.patch
#ssh-argv0.patch
#doc-upstart.patch

• Remove uninstalled files

In order not to check some files that will not be present as we commented the patches which creates them, we’ll have to remove the following lines from debian/openssh-client.install

usr/bin/ssh-vulnkey
usr/share/man/man1/ssh-vulnkey.1

And the following one from debian/openssh-client.docs

ChangeLog.gssapi

Finally, we just comment out the use of the vulnerable_host_keys shell function in debian/openssh-server.postinst.in:

fix_doc_symlink
create_sshdconfig
create_keys
#vulnerable_host_keys
fix_statoverride

That’s it! You can now happily build the brand new OpenSSH version using debuild as usual.

After installing it, you’ll have access to the AuthorizedKeysCommand option.

Thanks gaston, davromaniak and SliX from #GCU for the help.

The post Debian backport of OpenSSH 6.2 appeared first on Emile "iMil" Heitor 's home.

" inside screen / tmux
map <Esc>[C <C-Right>
map <Esc>[D <C-Left>
" insert mode
map! <Esc>[C <C-Right>
map! <Esc>[D <C-Left>
" no screen
map <Esc>[1;5D <C-Left>
map <Esc>[1;5C <C-Right>
" insert mode
map! <Esc>[1;5D <C-Left>
map! <Esc>[1;5C <C-Right>

nnoremap <C-t> :tabnew<CR>
nnoremap <C-w> :tabclose<CR>
nnoremap <C-right> :tabnext<CR>
nnoremap <C-left> :tabprevious<CR>
" insert mode
inoremap <C-t> <Esc>:tabnew<CR>
inoremap <C-w> <Esc>:tabclose<CR>
inoremap <C-right> <Esc>:tabnext<CR>
inoremap <C-left> <Esc>:tabprevious<CR>

The post vim tabs, tmux and Control-arrows appeared first on Emile "iMil" Heitor 's home.

Anyway, GitHub interface may be sexy, they used to have some kind of “upload” section which has been dropped. That may sound like a simple story, but the fact is when it comes to packaging a GitHub-hosted application, things are not that simple when the author has not explicitly tagged a specific release. Another use case, in which I actually am, is when you have an ongoing development, like pkgin in pkgsrc WIP and do not want to tag every test-release.

The way I found to handle that case with pkgsrc is to use GitHub’s commit archives. In short, I will use that kind of URL:

https://github.com/NetBSDfr/pkgin/archive/34b823c158e62e4d347de74499a075a2259382c5.tar.gz

which is redirected like this by GitHub:

HTTP/1.1 302 Found
Server: GitHub.com
Date: Sun, 21 Apr 2013 21:05:17 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Status: 302 Found
Cache-Control: max-age=0, private
Strict-Transport-Security: max-age=2592000
X-Frame-Options: deny
Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Thu, 21-Apr-2033 21:05:17 GMT; HttpOnly
Location: https://nodeload.github.com/NetBSDfr/pkgin/tar.gz/34b823c158e62e4d347de74499a075a2259382c5
X-Runtime: 13
Content-Length: 156
Vary: Accept-Encoding

HTTP/1.1 200 OK
Server: GitHub.com
Date: Sun, 21 Apr 2013 21:05:18 GMT
Content-Type: application/x-gzip
Connection: keep-alive
Content-Length: 187510
Content-Disposition: attachment; filename=pkgin-34b823c158e62e4d347de74499a075a2259382c5.tar.gz
Vary: Accept-Encoding

and permits to point to a particular commit, no matter if it has been tagged or not.

A typical pkgsrc Makefile will look like this:

VERSION=                34b823c158e62e4d347de74499a075a2259382c5
DISTNAME=               ${VERSION}
PKGNAME=                pkgin-20130412
CATEGORIES=             pkgtools
MASTER_SITES=           https://github.com/NetBSDfr/pkgin/archive/
FETCH_USING=            curl

# [...]

WRKSRC=                 ${WRKDIR}/pkgin-${VERSION}

# [...]

Note that FETCH_USING= curl is mandatory here in order to follow redirect codes along with https.

There you go, happy GitHub packaging!

Update

Here’s another approach pointed out by Amitai Schlair (schmonz@):

GIT_COMMIT=	dd51ac5

DISTNAME=	${GIT_COMMIT}
PKGNAME=	p5-App-Prove-Plugin-ProgressBar-0.01
CATEGORIES=	devel perl5
MASTER_SITES=	-http://nodeload.github.com/Ovid/App-Prove-Plugin-ProgressBar/tar.gz/${GIT_COMMIT}

# [...]

WRKSRC=		${WRKDIR}/App-Prove-Plugin-ProgressBar-${GIT_COMMIT}

Here, Amitai doesn’t use HTTPS so specifying curl as the fetch method is not mandatory. The dash before the URL in the MASTER_SITES line means that DISTNAME will not be appended when fetching, which is very handy when it comes to GitHub archives.

The post pkgsrc and github archives appeared first on Emile "iMil" Heitor 's home.

• 3NMP: NetBSD, Nginx, Naxsi, MySQL, PHP
• Nouvelles commandes et nouveaux démons dans NetBSD 6.0

Enjoy!

The post GLMF 159 appeared first on Emile "iMil" Heitor 's home.

A while ago, a couple of pkgin users told me it was a shame that /usr/pkg/etc/pkgin/repositories.conf was still pointing to a 5.0 URL when pkgin is freshly installed. Thing is, pkgin does support the $osrelease variable, but on NetBSD, the result of kern.osrelease can be 6.0_SOMETHING, which would lead to:

ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/6.0_SOMETHING/All

and this does dot exists on the repository.

So in the REPOSITORIES file, which is used to generate the right repositories.conf entry, I added that line:

ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/@OSREL@/All

And wrote the following trick in pkgtools/pkgin‘s Makefile:

.if ${OPSYS} == "NetBSD"
SUBST_CLASSES+=         osrel
SUBST_STAGE.osrel=      pre-configure
SUBST_MESSAGE.osrel=    Adjusting repository OS release
SUBST_FILES.osrel=      REPOSITORIES
SUBST_SED.osrel=        -e "s|@OSREL@|${OS_VERSION:C/_.*//}|"
.endif

The SUBST framework, part of pkgsrc, will then use (a portable) sed to replace @OSREL@ with the variable ${OS_VERSION}, which is defined in mk/bsd.prefs.mk (included in the Makefile), but will erase anything like the regexp “_.*”. This magic is done by the :C modifier, which behaves like the :S modifier but is capable of replacing regexps instead of simple strings.

Learn to know the power of pkgsrc…

The post pkgtools/pkgin, quick fix appeared first on Emile "iMil" Heitor 's home.

access_log /var/log/nginx/$host.access_log;

But I quickly noticed that some unwanted FQDN’s where appearing on the log directory. In order to keep control on the created files, I figured out a simple way to ensure unwanted domains to be logged in the general access.log:

if ($host ~ gcu) {
        set $log_fqdn $host;
}

access_log      /var/log/nginx/$log_fqdn.access_log;

And no, you can’t include the access_log keyword in the if statement, I tried :)

Anyway, isn’t it just beautiful? mmmm nginx…

The post auto-FQDN logging appeared first on Emile "iMil" Heitor 's home.

Instead, I’ll point out here all the resources that helped me switching rather quickly:

• Generic hands-on https://wiki.archlinux.org/index.php/Tmux
• Commented example http://www.doknowevil.net/2010/10/18/sorry-screen-tmux-is-better-but-here-are-some-screen-lik-hotkeys/
• Playing with the statusbar http://jasonwryan.com/blog/2011/06/10/setting-tmux-statusbar-if-in-x/, this author has other very good resources on the topic
• Fixing the 256colors mess (without using tmux -2) http://benjaminthomas.org/2012-03-28/256-colors-in-tmux.html
• Another well documented configuration file http://www.sheevaboite.fr/articles/show-me-your-tmux-conf
• Screen-to-tmux helper (in french, but easy to comprehend) http://wiki.soolbox.net/wiki/SysadminTips/tmux
• Very well written customization guide http://blog.hawkhost.com/2010/07/02/tmux-%E2%80%93-the-terminal-multiplexer-part-2/ (author has also written a comprehensive guide on how to basically use tmux)
• Enable mouse pane focus and resize http://tangledhelix.com/blog/2012/07/16/tmux-and-mouse-mode/

Here’s my ~/.tmux.conf, which is pretty much the standard screen-keys.conf available with the package, with only some small customizations like the status bar and some fixes:

# Set the prefix to ^Q.
unbind C-b
set -g prefix ^Q
bind q send-prefix

# Bind appropriate commands similar to screen.
# lockscreen ^X x 
unbind ^X
bind ^X lock-server
unbind x
bind x lock-server

# screen ^C c 
unbind ^C
bind ^C new-window
unbind c
bind c new-window

# detach ^D d
unbind ^D
bind ^D detach

# displays * 
unbind *
bind * list-clients

# next ^@ ^N sp n 
unbind ^@
bind ^@ next-window
unbind ^N
bind ^N next-window
unbind " "
bind " " next-window
unbind n
bind n next-window

# title A
unbind A
bind A command-prompt "rename-window %%"

# other ^A
unbind ^A
bind ^A last-window

# prev ^H ^P p ^? 
unbind ^H
bind ^H previous-window
unbind ^P
bind ^P previous-window
unbind p
bind p previous-window
unbind BSpace
bind BSpace previous-window

# windows ^W w 
unbind ^W
bind ^W list-windows
unbind w
bind w list-windows

# quit \ 
unbind \
bind \ confirm-before "kill-server"

# kill K k 
unbind K
bind K confirm-before "kill-window"
unbind k
bind k confirm-before "kill-window"

# redisplay ^L l 
unbind ^L
bind ^L refresh-client
unbind l
bind l refresh-client


# :kB: focus up
unbind Tab
bind Tab select-pane -t:.+
unbind BTab
bind BTab select-pane -t:.-

# split
unbind |
bind | split-window -h
unbind -
bind - split-window -v

# " windowlist -b
unbind '"'
bind '"' choose-window

unbind r
bind r source ~/.tmux.conf

set -g terminal-overrides 'xterm*:smcup@:rmcup@:colors=256'
# Status Bar
set -g default-terminal "screen"

# default statusbar colors
set -g status-fg white
set -g status-bg colour235

# current or active window in status bar
set-window-option -g window-status-current-bg colour53
set-window-option -g window-status-current-fg white
set-window-option -g window-status-current-format '[ #W ]'

# alerted window in status bar (bell, activity or content).
set-window-option -g window-status-alert-bg colour235
set-window-option -g window-status-alert-fg colour53
set-window-option -g window-status-alert-attr reverse

set -g status-right-length 40
set -g status-left-length 40
#set -g status-left '#[fg=purple]#H #[fg=black,bright]#[default]'
set -g status-left '#[fg=colour153]#H #[default]'
set -g status-right '#[fg=colour141]#(sysctl vm.loadavg|cut -f2-4 -d" ")#[default] | #[fg=colour117]%H:%M '

In order to pick up the right colors, I used a shell snippet I took from this github repository:

for i in {0..255} ; do
    printf "\x1b[38;5;${i}mcolour${i}\n"
done

Mandatory screenshot:

The post A new multiplexed world appeared first on Emile "iMil" Heitor 's home.