website/howto/jails.md

title	author	lang
Howto: Setting up Jails	Mina Galić	en

We'll be setting up jails roughly folowing antranigv's VNET Jail Howto We'll be show-casing how the jail serving https://alpha.pkgbase.live/ is setup.

ZFS Setup

Unlike with the host system, in jails we cannot use bectl(8), but we still can and should use ZFS:

~ $ zfs list -r zroot/isolates
NAME                             USED  AVAIL     REFER  MOUNTPOINT
zroot/isolates                  3.26G  22.8G       96K  /isolates
zroot/isolates/scratchpad       2.17G  22.8G      136K  /isolates/scratchpad
zroot/isolates/scratchpad/root  2.17G  22.8G     2.17G  /isolates/scratchpad/root
zroot/isolates/webserver        1.09G  22.8G      120K  /isolates/webserver
zroot/isolates/webserver/root   1.09G  22.8G     1.09G  /isolates/webserver/root

For each jail, we create two ZFS Filesystems which we can snapshot separately when needed:

/isolates/webserver will hold some basic configuration files, and /isolates/webserver/root the actual root filesystem.

Install

Into this file system, we'll install our base-system, consisting of the following packages:

~ $ sudo pkg -r /isolates/webserver/root install -r FreeBSD-base FreeBSD-clibs \
    FreeBSD-clibs-dev FreeBSD-libexecinfo FreeBSD-rc \
    FreeBSD-runtime FreeBSD-utilities FreeBSD-vi \
    FreeBSD-zoneinfo FreeBSD-pkg-bootstrap

This webserver will be running Apache httpd, so let's set that up too:

~ $ sudo pkg -r /isolates/webserver/root install -r FreeBSD apache24 \
    ap24-mod_security git-lite mime-support pkg puppet6 rubygem-ffi

and because I wouldn't configure any server by hand, I'm installing puppet.

Note that some packages, or rather, their post-install scripts, don't play well with pkg -r. However, once we have the base-system and pkg itself in place, we can fix those packages up with pkg -j, once the jail exists.

Jail Config

The following jail.conf(5) helps create and tear-down this jail:

# General rules, these could live in jail.conf
persist;
mount.devfs;
path = "/isolates/$name/root";
host.hostname = $name;

exec.clean;
exec.stop = "/bin/sh /etc/rc.shutdown jail";

vnet;
$bridge = "bridge0";

# rules specific to this jail:
webserver {
    $id      = "10";

    $jepair  = "epair${id}b";
    $ipaddr  = "192.168.17.${id}/24";
    $ip6addr = "2a01:4f9:c010:c64c::${id}/64";
    $gw      = "192.168.17.1";
    $gw6     = "2a01:4f9:c010:c64c::1";

    mount.fstab = "/isolates/$name/fstab";

    vnet.interface = "$jepair";

    exec.prestart   = "ifconfig epair${id} create up";
    exec.prestart  += "ifconfig epair${id}a up descr vnet-${name}";
    exec.prestart  += "ifconfig $bridge addm epair${id}a up";

    exec.start      = "/sbin/ifconfig lo0 127.0.0.1 up";
    exec.start     += "/sbin/ifconfig epair${id}b ${ipaddr}";
    exec.start     += "/sbin/ifconfig epair${id}b inet6 ${ip6addr}";
    exec.start     += "/sbin/route add default ${gw}";
    exec.start     += "/sbin/route add -inet6 default ${gw6}";
    exec.start     += "/bin/sh /etc/rc";

    exec.prestop    = "ifconfig epair${id}b -vnet ${name}";

    exec.poststop   = "ifconfig $bridge deletem epair${id}a";
    exec.poststop  += "ifconfig epair${id}a destroy";
}

You may notice the mount.fstab option, which points to /isolates/webserver/fstab, and looks like so:

/poudriere/data/images /isolates/webserver/root/srv/images nullfs ro 0 0

This fstab(5) formatted file helps us null mount our poudriere build result into the webserver jail from where it is served publicly over IPv4 and IPv6.

Network Config

In the above jail.conf(5), we can already see the parts of the network config that need to happen in the jail.

The IPv6 address is assigned directly into Jail, but it is routed via the host system. The IPv4 address is a private IP addresses, so it has to be NATed via the host system.

Both cases are handled via a single if_bridge(4) configured in rc.conf(5):

cloned_interfaces="bridge0"

Let's assign the IPv4 address, and enable gateway:

# jail NAT and Network access
ifconfig_bridge0="inet 192.168.17.1/24"
gateway_enable="YES"

# firewall (well, NAT for now)
pf_enable="YES"

For the IPv4 part, we'll handle NAT with the following pf.conf:

scrub in all fragment reassemble
nat pass on vtnet0 inet from 192.168.17.0/24 to any -> (vtnet0:0)

webserver = 192.168.17.10

rdr on vtnet0 inet proto tcp from any to any port 80 -> $webserver port 80
rdr on vtnet0 inet proto tcp from any to any port 443 -> $webserver port 443

As for IPv6, since it's directly put into the jail, and we only do the routing, this is all we have to do on the host:

# working IPv6 setup needs link-local addresses (according to the spec)
ipv6_activate_all_interfaces="YES"
ifconfig_bridge0_ipv6="inet6 2a01:4f9:c010:c64c::1/64 auto_linklocal"
ipv6_defaultrouter="fe80::1%vtnet0"
# enable IPv6 gateway
ipv6_gateway_enable="YES"

Config in the Jail

Since in the jail's exec.start is set to /etc/rc, we want to disable a few things we don't need:

cron_enable="NO"
syslogd_enable="NO"
netif_enable="NO"

And because we want our jail to work with either IPv6 or IPv4, even if one of the two is broken, we need a resolv.conf(5) that represents that:

nameserver 2a01:4f8:0:1::add:1010
nameserver 213.133.100.100
nameserver 213.133.98.98
nameserver 2a01:4f8:0:1::add:9898

Finally, we can enable the jail in the host's rc.conf:

# jails setup
jail_enable="YES"
jail_list="webserver scratchpad"
# including syslog!
syslogd_flags="-sH"
altlog_jaillist="webserver scratchpad"

Note that the syslogd changes necessary for the above config aren't yet merged.

Starting

Note that many of these networking changes will need a service netif restart and service routing restart, or a reboot.

~ $ sudo service syslogd restart
~ $ sudo service jail start webserver

To actually configure my jails, I have a provision.sh script in /isolates/$name/ in the latest config with r10k, null-mounts it into the jail with a dedicated fstab(5) file:

/usr/local/etc/puppet /isolates/webserver/root/usr/local/etc/puppet nullfs ro 0 0

and then runs puppet:

#!/bin/sh
echo "run r10k to update data before running puppet"
r10k deploy environment --verbose --puppetfile

echo "webserver isolate: mounting puppet config provision.fstab"
mount -F /isolates/webserver/provision.fstab -a

echo "webserver isolate: running puppet apply"
jexec -l webserver puppet apply --verbose /usr/local/etc/puppet/environments/production/manifests/site.pp

echo "webserver isolate: umounting puppet config provision.fstab"
umount -F /isolates/webserver/provision.fstab -a