Great doc from http://blog.innerewut.de/articles/2005/08/25/freebsd-jails

FreeBSD jails

Posted by Jonathan 42 days ago

I manage several jails on one of my FreeBSD servers. FreeBSD jails are virtual servers that all run on one hosts machine. Each jail has its own complete userland, only the kernel is shared between jails and the host system. I use one jails for each development project with its own webserver, subversion repository and users. If a jail is hacked, the others are not affected.

What I usually do to install a new one is to install it from the compiled sources:

    # cd /usr/src
    # make installworld DESTDIR=/usr/jails/my_new_jail
    # cd /usr/src/etc
    # make distribution DESTDIR=/usr/jails/my_new_jail 

As the jail doe not have it's own kernel, create a fake one:

    # cd /usr/jails/my_new_jail
    # ln -sf dev/null kernel 

For this to work you need a mounted devfs filesystem, so before creating the fake kernel mount it:

    # mount_devfs devfs /usr/jails/my_new_jail/dev 

In order to start the jail you need to have an IP address for the jail and make sure that the host system does not bind services to all IP addresses. Yes, each jail need it's own IP address. If you do not have enough IP addresses you can just populate the 127.0.0.0 network and NAT it through the host system (e.g. "nat on $ext_if from $loop_net to any -> ($ext_if)"). Then you can use redirections of higher ports to the jail's ports.

So in the host system make sure to have some aliases for the additional IP addresses and restrict the services to the host's IP.

From /etc/rc.conf

    ifconfig_lo0_alias0="inet 127.0.0.2 netmask 0xffffffff"
    ifconfig_lo0_alias0="inet 127.0.0.3 netmask 0xffffffff"
    syslogd_flags="-ss" # Syslogd flags to not bind an inet socket
    sendmail_enable="NO" # Sendmail only binds to to localhost
    inetd_flags="-wW -a 192.168.0.1" # bind to specific address 

When these settings are in place, edit /etc/hosts.conf on both servers (/etc/hosts and /usr/jails/my_new_jail/etc/hosts ) and edit the jail's settings in /usr/jails/my_new_jail/etc to fit your needs.

To first start a jail use the jail command:

    # jail /usr/jails/my_new_jail jail_hostname 127.0.0.2 /bin/sh 

This will give you a shell on the jail so that you can fine tune the settings and change the password. For a permanent start use

    # jail /usr/jails/my_new_jail jail_hostname 127.0.0.2 /bin/sh /etc/rc 

Make sure to have the alias and the devfs in place.

Jail management

In order to manage several jails you can use /etc/rc.conf on the host machine:

    # general settings
    jail_enable="YES"
    jail_list="one two" # add to list for more jails

    # jail specific settings for jail "one"
    jail_one_rootdir="/usr/jails/my_new_jail/"
    jail_one_hostname="jail_hostname"
    jail_one_ip="127.0.0.2"
    jail_one_exec="/bin/sh /etc/rc"
    jail_one_devfs_enable="YES"

    # jail specific settings for jail "two"
    jail_two_rootdir="/usr/jails/my_other_jail/"
    jail_two_hostname="number_two"
    jail_two_ip="127.0.0.2"
    jail_two_exec="/bin/sh /etc/rc"
    jail_two_devfs_enable="YES" 

I hope that the pattern is visible. The jail_list defines which jails exist and the jail_NAME_* directives define the specific settings.

You can now start and stop jail with the /etc/rc.d/jail script:

    # /etc/rc.d/jail start
    # /etc/rc.d/jail stop
    # /etc/rc.d/jail start one
    # /etc/rc.d/jail stop two 

With the command jls you can see which jail are started and get their jail-id in order to execute commands in them with the jexec program.

Update a jail

In order to update a jail, I use the same mechanism that I use to update the host system. Install a new world and use mergemaster.

    # cd /usr/src
    # make buildworld
    # mergemaster -p -D /usr/jails/my_new_jail
    # make installworld DESTDIR=/usr/jails/my_new_jail
    # mergemaster -D /usr/jails/my_new_jail 

Make sure to keep the host system (or at least the host system's kernel) and the jail in sync. If the userland is "newer" than the kernel and some ABI changes went in, the kernel will not be able the execute the binaries.

Delete a jail

If you want to delete a jail, just remove the alias and settings from /etc/rc.conf and delete all the jail's files. Make sure that you can actually delete then as some files are protected by file flags.

    # cd /usr/jails/my_new_jail
    # chflags -R noschg *
    # rm -rf * 

Further information

You show also have a look at the various jail-sysctls:

    # sysctl -a | grep jail
    security.jail.set_hostname_allowed: 1
    security.jail.socket_unixiproute_only: 1
    security.jail.sysvipc_allowed: 0
    security.jail.getfsstatroot_only: 1
    security.jail.allow_raw_sockets: 0
    security.jail.chflags_allowed: 0
    security.jail.jailed: 0 

The security.jail.sysvipc_allowed is important if you need SystemV IPC inside the jails like for example for PostgreSQL.

A very good german HOWTO for jails can be found here. Also there are several jails related ports that are worth examining.

UPDATE:
I just discovered a webmin-module for jails while browsing bsdforen.de. This module helps you with installing & managing jails. There is also a screenshot.