Emile "iMil" Heitor 's home

Ce matin, j’ai mis à jour le dom0 Debian d’une de mes machines. Passionnant me direz-vous. L’opération a consisté en la migration de Lenny vers Squeeze. De plus en plus interessant hein ? L’upgrade s’est effectué sans trop de peine, après quelques apt-get -f install et autres réinstallations de packages ayant sauté dans la bataille, rien de palpitant. Me voici donc avec un kernel 2.6.35-2 sur un Xen 4 flambant neuf.

Ce dom0 accueille des domU NetBSD. Des NetBSD 5.0.2 pour être précis. Et c’est le drame: Le PR 44743 indique en effet:

Subject: Network doesn’t work on DomU NetBSD 5.1 which is run on Debian Squeeze Dom0 (Xen 4)

Ce à quoi Dieu^WManuel Bouyer répond:

No, copy mode support was added to the backend (dom0) before 5.1, but
it was added to the frontend (domU) after 5.1 was released. So you need
something newer from the netbsd-5 branch.

Vous voyez poindre le bordel ? “Mais migre, bigre de couillon” me direz-vous, et vous aurez raison, seulement cette machine construit des packages (via bulk build, voir le post précédent) pour un certain nombre d’autres machines, elles aussi en 5.0.2. Bref, rien n’est simple. J’ai donc adopté une méthode “alternative” (aussi appelée la méthode *rhon* *rhon* *huiiiiii*): patcher le noyau 5.0.2 avec les bits and pieces nécessaires en provenance de netbsd-5. Finalement, cela n’a pas été si compliqué. Il suffit de cvs up -rnetbsd-5 -dP dans src/sys/arch/xen et src/sys/arch/x86 puis de relancer un config, make depend, make pour obtenir un noyau domU 5.0.2 muni du support copy mode.

“All went better than expected”.

Pour une partie de mon parc de machines, je fais mon propre bulk build. Ce dernier ne construit pas l’ensemble des packages, mais un petit subset (environ 600 packages) avec mes propres préférences. Parmi elles, il en est une qui fout un merdier sans nom dans le build, converters/libiconv. Comme je l’expliquais ici il y a quelques temps, j’ai besoin de construire converters/php-iconv avec la version pkgsrc de la libiconv. Cet impératif a un impact non négligeable dans la configuration de mon /etc/mk.conf, aussi je vous livre ce dernier, final et fonctionnel:

.ifdef BSD_PKG_MK

# no base X11
MKX11=no
X11_TYPE=modular
# clean dependencies when the "clean" target is called
CLEANDEPENDS=yes
# everybody likes vim
ACCEPTABLE_LICENSES+=vim-license

USE_BUILTIN.iconv=      no
# A built-in gettext is always going to use a built-in iconv.
USE_BUILTIN.gettext=    no

PKG_RCD_SCRIPTS=        yes

PKG_OPTIONS.irssi=      perl inet6
PKG_OPTIONS.mplayer=    oss

DSPAM_STORAGE_DRIVER=   mysql
PKG_OPTIONS.dspam+=     graphs
MYSQL_VERSION_DEFAULT=  50
PHP_VERSION_DEFAULT=    52
PKG_OPTIONS.php=        -cgi fastcgi

PKG_OPTIONS.rtorrent=           xmlrpc

UPDATE_TARGET=package-install
DEPENDS_TARGET=         bulk-install
BATCH=                  yes

ACCEPTABLE_LICENSES+= socks5-license
ACCEPTABLE_LICENSES+= sendmail-license
ACCEPTABLE_LICENSES+= openmotif-license
ACCEPTABLE_LICENSES+= idea-license

PKG_OPTIONS.dovecot=    ssl ldap dovecot-sieve dovecot-managesieve
PKG_OPTIONS.nagios-nrpe = ssl tcpwrappers

PKGCHK_CONF?=   /usr/pkgsrc/pkgchk.conf
BULK_PREREQ+=   converters/libiconv
#
# Parse pkgchk.conf and supply list of packages for the bulk build framework.
#
.if defined(SPECIFIC_PKGS)
PKGLIST!=               awk '{print $$1}' ${PKGCHK_CONF}
.  for _pkg_ in ${PKGLIST}
HOST_SPECIFIC_PKGS+=    ${_pkg_}
.  endfor
.endif

.endif # BSD_PKG_MK

À noter, donc, les particularités suivantes:

• Xorg modular pour les dépendances relatives à X11
• libiconv en provenance de pkgsrc, il est impératif de faire de même pour gettext
• irssi est compilé avec le support perl et IPv6
• storage MySQL pour dspam
• MySQL 5.0
• PHP 5.2
• Options ssl ldap dovecot-sieve et dovecot-managesieve pour dovecot
• Options ssl et tcpwrappers pour nagios-nrpe
• On ajoute libiconv comme pré-requis pour la construction bulk

J’utilise, pour générer tout ce petit monde, l’excellent script du sieur orgrim, disponible ici, avec sa note explicative.

I’ve just commited 0.5.2.1. As the version shows, it is a bugfix release; is@ got a very nasty bug that took me a while to figure out. For 3 years, the only pkg_summary(5) format I’ve seen was:

PKGNAME=foo
[...]

or, in case of conflicting packages:

CONFLICTS=bar
CONFLICTS=baz
PKGNAME=foo
[...]

Well it turns out that we can also find:

PKGNAME=foo
CONFLICTS=bar
[...]

which is pretty annoying when it comes to stick to an anchor. Until now, I did the following:

static int
chk_pkgname(char *field)
{
        if (strncmp(field, "PKGNAME=", 8) == 0 ||
                strncmp(field, "CONFLICTS=", 10) == 0)
                return 1;

        return 0;
}
[...]
                /* browse entries following PKGNAME and build the SQL query */
                while (*psum != NULL && !chk_pkgname(*psum)) {
                        update_col(sum, pkgid, *psum);
                        psum++;
                }

But with that new case, I had to check a little bit further:

static int
chk_pkgname(char *field, char *last_field)
{
	if (strncmp(field, "PKGNAME=", 8) == 0)
		return 1;
	/* in some very rare cases, CONFLICTS appears *after* PKGNAME */
	if (strncmp(last_field, "PKGNAME=", 8) != 0 &&
		/* never seen many CONFLICTS after PKGNAME, but just in case... */
		strncmp(last_field, "CONFLICTS=", 10) != 0 &&
		strncmp(field, "CONFLICTS=", 10) == 0)
		return 1;

	return 0;
}
[...]
		/* browse entries following PKGNAME and build the SQL query */
		while (*psum != NULL && !chk_pkgname(*psum, *(psum - 1))) {
			update_col(sum, pkgid, *psum);
			psum++;
		}

I really don’t like the way it’s done but I’ve no sexier way in mind at the moment. Maybe after some glasses of wine…

Finally, here comes a release ! Read the full announcement and changelog on pkgsrc-users.

And here we are for week 2 !

There have been some fixes last week, issues were mainly spotted by orgrim, thanks to him !

The big one concerns a type of dewey/glob I’ve never been through before:

libao-[a-z]*-[0-9]*

Until now, I was able to handle the following (from pkg_str.c)

/*                                                                              
 * AFAIK, here are the dewey/glob we can find as dependencies                   
 *                                                                              
 * foo>=1.0 - 19129 entries                                                     
 * foo<1.0 - 1740 entries (only perl)                                           
 * foo>1.0 - 44 entries                                                         
 * foo< =2.0 - 1                                                                 
 * {foo>=1.0,bar>=2.0}                                                          
 * foo>=1.0<2.0                                                                 
 * foo{-bar,-baz}>=1.0                                                          
 * foo{-bar,-baz}-[0-9]*                                                        
 * foo-{bar,baz}                                                                
 * foo-1.0{,nb[0-9]*} - 260                                                     
 * foo-[0-9]* - 3214                                                            
 * foo-1.0 - 20                                                                 
 */

As there was no “easy” way of handling those double-globs dependencies, I took a simple approach: resolve dependency with a matching package when the pattern is too complex (sqlite_callback.c):

    /* map corresponding pkgname */
    if ((pkg_map = map_pkg_to_dep(plisthead, deptree->depend)) != NULL)
        XSTRDUP(deptree->name, pkg_map->name);
    else
        /* some dependencies just don't match anything */
        XSTRDUP(deptree->name, DEPS_PKGNAME);

Only drawback, this takes a bit more time, but at last, there’s no possible error on dependency resolution. Maybe there’s some more optimizations I can do to reduce time taken with this mapping.

As planned, pkgin now depends on pkgsrc’s pkg_install, and the latter is pushed on top of the upgrade-list if there’s a new version.
I also fixed the many-versions-of-the-same-package problem with the following query (pkgindb_queries.c)

const char DIRECT_DEPS[] = /* prefer higher version */
    "SELECT REMOTE_DEPS_DEWEY, REMOTE_DEPS_PKGNAME "
    "FROM REMOTE_DEPS WHERE PKG_ID = "
    "(SELECT PKG_ID FROM REMOTE_PKG WHERE PKGNAME = '%s' "
    "ORDER BY FULLPKGNAME DESC LIMIT 1);";

Needs to be done:

• Tests, tests, tests !
• reproduce and fix 2 different bugs two users had; I’ve been unable to reproduce them at the moment
• test Minix 3.2.0