LXD, ZFS and bridged networking on Ubuntu 16.04 LTS+


(Jason Bayton) #1

Originally published at: https://bayton.org/docs/linux/lxd/lxd-zfs-and-bridged-networking-on-ubuntu-16-04-lts/

Network changes in Ubuntu 17.10+ This guide has been updated for netplan, introduced in 17.10. Please test the configuration and let me know if you have any issues with it (easiest via tweet, @jasonbayton). LXD works perfectly fine with a directory-based storage backend, but both speed and reliability are greatly improved when ZFS is used instead. 16.04 LTS saw the first officially supported release of ZFS for Ubuntu and having just set up a fresh LXD host on Elastichosts utilising both ZFS and bridged networking, I figured it’d be a good time to document it. In this article I’ll walk through the installation of LXD, ZFS and Bridge-Utils on Ubuntu 16.04 and configure LXD to use either a physical ZFS partition or loopback device combined with a bridged networking setup allowing for containers to pick up IP addresses via DHCP on the (v)LAN rather than a private subnet. Before we begin This walkthrough assumes you already have a Ubuntu 16.04 server host set up and ready to work with. If you do not, please download and install it now. You’ll also need a spare disk, partition or adequate space on-disk to support a loopback file for your ZFS filesystem. Finally this guide is reliant on the command line and some familiarity with the CLI would be advantageous, though the objective is to make this a copy & paste article as much as possible. Part 1: Installation To get started, let’s install our packages. They can all be installed with one…


(Will Coquillette) #2

Thank you for the amazing article! I’m considering migrating my existing server using this guide. I’m currently running ESXi with my 6-disk RAID-Z2 passed through to a FreeNAS VM as described in the guide here but on ESXi 6.5 and FreeNAS 11.x.

My goal is to install Ubuntu 16.04 to a 120 GB SSD and create a partition on the SSD that will be dedicated to ZFS. My containers will live there. I need to figure out how to import my existing RAID-Z2 zpool, as that’s where my media lives.

The LXD zpool will host a Plex container, and that Plex container will mount its media folders from the existing 6-disk zpool.

Things I have yet to figure out as I lab this out in advance:

  1. Whether LXD is even the right application for my Plex container or if Docker is better suited.
  2. How I’m going to mount the media folders in the Plex container. Currently, I have Plex installed on an Ubuntu VM in ESXi that uses nfs mounts in fstab to mount the FreeNAS shares.

Once I migrate using your tutorial, I shouldn’t need NFS mounts since they would be “local” storage if I’m understanding things correctly. This will be nice, as I can get away from having an internal “storage-only” network with 10Gbe VMXNET3 adapters and jumbo frames in addition to the standard, internet-routable virtual NICs at MTU 1500.

Looking forward to putting your post into action!


(Jason Bayton) #3

ZFS is pretty good with exports/imports and send/receive as long as the remote system can accept a pool of the size you’re trying to push :slight_smile: Check out an example here.

Not dramatically dissimilar from my setup!

  • LXD containers reside on a dedicated SSD
  • Media accessed from an 8-disk ZFS mirror (though I’m all about Emby :slight_smile:)

I need to add in a 2nd SSD at some point for LXD container redundancy. If that SSD dies I’ll need to do a lot of restoring.

1 is really entirely your call, LXD containers AFAIK are light enough I run pretty much one for each service I use:

Docker is more than adequate if that’s what you’re happier with though… and requires no update maintenance.

For 2, I use LXD devices to push folders into containers… though you could well instead utilise a FUSE filesystem like SSFS to mount directly within the containers. If pushing the folders, you’ll need to ensure the UID/GID mapping is correct, else the container won’t be able to write to them. See this.


(Will Coquillette) #4

Thanks for the tips! Considering you’re using a dedicated SSD for LXD while I’m using a partition on my Ubuntu SSD for LXD, should I just go ahead and make the entire SSD use ZFS? Is that possible? I haven’t even begun this project, so there’s still so much to plan. At the very least, I’d like to explore the option of setting up my home partition in ZFS (or moving it to ZFS post-install if I can’t do it while installing Ubuntu).

My goal in all of this is to have an environment can be re-created very easily if my single point of failure (the SSD) were to fail. Ideally, I’d like to be in a position where all of the LXC containers in my Ubuntu SSD’s LXD zpool are backed up to my media’s zpool (6x3TB RAID-Z2).

Then if my SSD were to fail, I could simply get a replacement SSD, Install Ubuntu, install LXD, and run a few commands to import my media’s zpool, which then would allow me to import a backup of my LXD zpool onto the new SSD’s LXD partition. If I understand this concept correctly, I’d be back in business in under an hour.

Can you expand upon this?

When I was using Proxmox a year ago, My LXC containers would have mountpoints in their configuration files. They’d look like this:
mp0: /tank/media/movies,mp=/mnt/movies
mp1: /tank/media/tv,mp=/mnt/tv

Would I not use an identical (similar?) system for mounting media folders into the Plex container? Or do you have some kind of LXC container that mounts /tank/media directories into your Plex container so the Plex container is actually mount-agnostic?


(Jason Bayton) #5

A little bit of Googling will highlight how difficult it is to get ZFS running as the root FS for Ubuntu. I’d stick with EXT4, or BTRFS if you’re feeling more adventurous, and retain the partition for ZFS method instead… or if you’ve got the space, pop in another SSD :wink:

LXD backups aren’t super easy from experience to date, primarily because LXD creates a pool for every container, image, and snapshot:

What I tend to do is lxc copy my snapshots to a 2nd host as a container, like lxc copy container1/snap2 remote:containerbackup2 which likely isn’t a very efficient way to do it either, but I have the machines running to accommodate it. Restoring LXD isn’t super easy either, it’s not just a case of reimporting the LXD pool as you’ve got configuration files and databases to consider too.

LXD utilises devices rather than raw editing. I’m sure you could work around it, but easier not to TBH. Devices are reusable and attach to containers really easily, see:


(Will Coquillette) #6

You inspired me. I made a tutorial that is part your guide and part the rest of the internet. See post here: Ubuntu 18.04 LXD/LXC, ZFS, Docker, and Advanced Networking.

Will