They are vastly different creatures as you know, but in my experience on long-lived-but-must-keep-running services, fat-jails are a MUST.
OS package mismatches will bite you in the ass years down the road if you try to maintain thin-jails.
I manage alot of sites that Management refuses to invest any dev-time to upgrade old Python2.7 infrastructure. Having fat-Jails allows me to keep everything else upgraded.
pf is a fantastic firewall, but the syntax in OpenBSD is different then in FreeBSD.
VNET is worth it's weight in gold. Each jails has it's own network stack. That let's you avoid funky localhost workarounds.
iocage is a great framework for jails management (but it isn't necessary) you can do everything from jail.conf. Personally I prefer it for it's ZFS integration.
Previous job, we were running almost everything in "normal" jails (as opposed to fat). We even stayed with ezjail, which is dead simple, because we were so comfortable with it and could understand how to manually fiddle with stuff when there was a need.
There are very very few things that cannot be told to bind to the real jail IP address instead of localhost. Most stuff will just work. And that's it. But there are also added benefits here:
* Parts of the filesystem cannot be altered, period. You update your base from your jailhost and every jail takes the change. That was an added security benefit.
* Pkg software can be individually installed per jail, goes into /usr/local. FreeBSD has already been brilliant in separating the base system from the third party software.
* For minor and major version bumps, we developed a simple set of steps and tools to upgrade a jailhost but leave the jails in their previous versions, maintaining multiple versions of base, like /var/jails/basejail104, /var/jails/basejail113, and so on. So the downtime is minimized, and each jail can be upgraded later in its own pace. This was a huge relief compared to having to upgrade everything only to find that some piece of software doesn't quite work correctly in the next version. And rollback is possible, by just copying the jail.
* Jails can likewise be very easily copied between jailhosts with a couple of rsync commands and some simple script.
Have used iocage and zfs and it is brilliant, but even without the benefits of zfs clone and snapshot, jails were (and I believe still are) a very pleasant experience to work with.
/usr/sbin/jail /jails/dns dns 1.2.3.5 /unbound/unbound -c /unbound/unbound.conf
So /jails/dns is a holder for two different dns servers and they run, jailed, from the same root. It's a little unorthodox but in this case I find it to be elegant.
I completely agree that the capability of running jails like that is awesome, and truth be told my original intention.
For some of my jails that is how I run them too. Especially where I am being especially paranoid about security. I was trying to answer about 'general use/unexpected gotchas' scenario.
We deal with the hands we are given. I wish I could run it as tight as that on all of my boxes, but the bosses I deal with do not want to waste money on upgrading old infrastructure but still get mad when it isn't running. So fat jails are my best compromise. My other option was a virtualbox/vmware setup.
I agree. I would run jails with very few files. A static executable, a config file, a log file, and some pipes maybe. Don't need anything else, don't want anything else (log file could be avoided with syslog, perhaps)
Bonus points: chflag schg the exe and config files such that they cannot be altered ... or better yet, make it a zfs filesystem and mount it read-only ...
OS package mismatches will bite you in the ass years down the road if you try to maintain thin-jails.
I manage alot of sites that Management refuses to invest any dev-time to upgrade old Python2.7 infrastructure. Having fat-Jails allows me to keep everything else upgraded.
pf is a fantastic firewall, but the syntax in OpenBSD is different then in FreeBSD.
VNET is worth it's weight in gold. Each jails has it's own network stack. That let's you avoid funky localhost workarounds.
iocage is a great framework for jails management (but it isn't necessary) you can do everything from jail.conf. Personally I prefer it for it's ZFS integration.