Actually, Bjoern Zeeb (bz@) did a lot of work around VIMAGE (sometimes also called VNET) in order to track down these bugs and fix them for 11.0-RELEASE. If you haven't, you should give it a try, because it's now the way to use per-jail IP stack.
I did and during my tests i found no more panics, but I ran into a bug in the IPFW tables logic. The fix hasn't been backported to FreeBSD 11.0 as an errata. Also the epair pseudo-interface used for connecting IP stacks tops out at about 4Gb/s while the loopback interface can handle >40Gb/s on the same hardware, because it uses just one kernel thread (I don't remember the pps).
I hope we'll get VIMAGE into GENERIC for FreeBSD 11.1. He completely refactored the IP stack shutdown logic. Now the layer are shut down from top to bottom instead of the other way around. This avoids most of the nasty locking problems draining the higher levels with their pointers to the lower layer resources (e.g. routes, interfaces) first. The remaining bugs won't be found without a lot more exposure.
I did get panics but I haven't been able to reproduce them. The only thing I can think of is create a jail without an assigned IP, start it, assign one, shut it down and start it up again.
