xhyve + ubuntu 14.04 netinstall

Today, I’ve been playing with xhyve, a port of the FreeBSD bhyve hypervisor to OS X.
Here are some steps to easily create a bare Ubuntu Server-based image. Coincidentally, this will also serve as my future cheat sheet regarding xhyve + ubuntu, so I’ll just keep adding stuff that might be remotely useful in this context.

BIG FAT WARNING

If you have run VirtualBox older than the new v5 after your last reboot, prepare to reboot first. There’s a weird bug in older VirtualBox versions, that triggers a kernel panic when starting xhyve.

Preparations

This guide assumes we’re going to keep all files related to virtual machines in the ~/VMs/ folder (this usually expands to /Users/username/VMs/ on a mac), and that this folder exists. If it does not, please create it:

mkdir ~/VMs

Install xhyve

I installed xhyve through brew, which will only give you the xhyve binary. Most early documentation and howto’s I ran into focus on the image supplied in the github repository. Installing via brew will not supply you with those example files.

To install via brew, type:

brew install xhyve

To install manually, use:

git clone git@github.com:mist64/xhyve.git
cd xhyve
make
mkdir -p ~/local/bin
echo 'export PATH="${PATH}:~/local/bin"' >> ~/.bash_profile
cp build/xhyve ~/local/bin/xhyve

Please close the current terminal, and open a new one for the changes to take effect.

Set up working directory and download needed files

mkdir ~/VMs/ubuntu-installer
cd ~/VMs/ubuntu-installer
wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz
wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux
wget http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/netboot/mini.iso

Create target disk image

The following will create a 16GB blank disk image:

mkdir ~/VMs/ubuntu
cd ~/VMs/ubuntu
dd if=/dev/zero of=hdd.img bs=1g count=0 seek=16

Start the installer

If you installed from source, you’ll need to provide the full path to the xhyve binary.
Sudo is currently needed for network access, so please type your user password to continue when asked.

cd ~/VMs/ubuntu-installer && \
sudo xhyve -m 512M -c 1 -s 2:0,virtio-net \
  -s 3,ahci-cd,mini.iso -s 4,virtio-blk,../ubuntu/hdd.img \
  -s 0:0,hostbridge -s 31,lpc -l com1,stdio \
  -f "kexec,linux,initrd.gz,earlyprintk=serial \
  console=ttyS0 acpi=off root=/dev/vda1 ro"

Installing Ubuntu 14.04 Server

The install process will look quite… arcane. This is because we’re dealing with the restrictions of an old-fashioned serial console. Xhyve does not (yet) support graphical output of any sort, and serial is the next best thing for text-based applications.

Please follow the installer as you would, but hold at the last screen.

Recover initrd and kernel

The final message after installation reads “Installation is complete, so it is time to boot into your new system.” At this point we’re going to stray a bit from the install process, as we have to recover the kernel and initrd from the install image:

  • Select “Go Back” to interrupt the installation process.
  • Select “Execute a shell”, and select “Continue”
    This will open up a very minimal shell prompt. Use the commands below to discover the container’s ip address, and copy over files.
  • Open a new terminal window

In the vm:

ip addr show dev eth0 | grep 'inet ' | cut -d" " -f6 | cut -d/ -f1 # this will output the ipaddress
cd /target
tar cf - boot/initrd.img-* boot/vmlinuz-* | nc -l -p 1234

At this point, nothing else should happen.

On your mac (in the new terminal window):

cd ~/VMs/ubuntu
nc <ipaddress> 1234 | tar xf -

This will create a copy of the container’s /boot/ folder outside the container, containing the new initrd and kernel.

A shell prompt should now show on the container again. Type “exit” to exit, and select “Finish the installation”. A few more questions might appear, possibly even ones that were asked before.

Launching the container

Manually:

The following command will launch the container manually:

cd ~/VMs/ubuntu && \
sudo xhyve -m 512M -c 1 -s 2:0,virtio-net \
  -s 4,virtio-blk,hdd.img -s 0:0,hostbridge \
  -s 31,lpc -l com1,stdio -f "kexec,$(ls boot/vmlinuz-* | \
    sort -u | tail -1),$(ls boot/initrd.img-* | sort -u | \
    tail -1),earlyprintk=serial console=ttyS0 acpi=off \
    root=/dev/vda1 ro"

The container will now start up, and eventually greet you with a login prompt. Log in with the user account and password you created earlier.

This user can use sudo to gain root access. To gain full root access, use the following command

sudo su -

This will ask you once more for your user password, and will then drop you in a root shell.

Setting up a startup script

The following script can be used to start the container:

#!/bin/sh

cd "$(dirname $0)"

MEM="-m 512M"
SMP="-c 1"

KERNEL="$(ls boot/vmlinuz-* | sort -u | tail -1)"
INITRD="$(ls boot/initrd.img-* | sort -u | tail -1)"
CMDLINE="earlyprintk=serial console=ttyS0 acpi=off root=/dev/vda1 ro"

NET="-s 2:0,virtio-net"
IMG_HDD="-s 4,virtio-blk,./hdd.img"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
LPC_DEV="-l com1,stdio"

sudo xhyve ${MEM} ${SMP} ${PCI_DEV} ${LPC_DEV} ${NET} ${IMG_HDD} -f "kexec,${KERNEL},${INITRD},${CMDLINE}"

the MEM and SMP parameters indicate the amount of memory and cores a container has access to fslxfir.

Save this file as “~/VMs/ubuntu/run.sh”. Then type:

chmod 755 ~/VMs/ubuntu/run.sh

to make it executable.

you can use sudo to run the command:

~/VMs/ubuntu/run.sh

to launch the container.

Cleaning up

After you have created your container, you can clean up the install files. Type the following to do so:

rm -rf ~/VMs/ubuntu-installer/

Maintenance

To keep the container up-to-date, you might want to utilise apt every now and then. by default, Ubuntu will not update kernels via apt-get upgrade, however it will when using aptitude or apt-get dist-upgrade. When kernels are upgraded, please repeat the kernel and initrd recovery procedure, with the minor difference that /target must now be /. When installing Avahi (see below) you can forego on the ip address lookup, and just use the hostname ubuntu.local instead of the ip.

Extras

Install Avahi

For ease of use, install avahi-daemon in your container

sudo apt-get install avahi-daemon

This will add easy name lookup support. From your mac you can now type:

ssh username@ubuntu.local

Replace <username> with your container username

Use screen

You can use screen to not have your vm hogging an active terminal. Start your container from within screen:

screen -S ubuntu ~/VMs/ubuntu/run.sh

You can now detach the console using the “ctrl-a” “d” key sequence. To re-attach type:

screen -rd ubuntu

Enable autologin on console

sed -i 's,^exec /sbin/getty -L ,exec /sbin/getty -a root -L ,' /etc/init/ttyS0 important site.conf

Enable passwordless sudo for xhyve (DANGEROUS)

This command will enable all local admin-type accounts to run “sudo xhyve <parameters>” without needing to type a password every time you start a vm. This could well be a security risk, so please think of the implications this might have for you.

sudo sh -c 'echo "%admin ALL=NOPASSWD: /usr/local/bin/xhyve" >> /etc/sudoers'

References