How to boot a diskless VM with iPXE over NFS using only IPv6

I have encountered many issues and countless hours trying to figure this out, but I finally succeeded. Reading source code, strace’ing binaries trying to figure out what was wrong with them.

iPXE

iPXE will do IPv4 DHCP first and then IPv6 SLAAC/DHCP. This is not a problem, it’s just annoying. IPv4 DHCP has a default timeout in iPXE of 10 seconds.

What you need to do in iPXE:

• enable serial console -> src/config/console.h -> #define CONSOLE_SERIAL
• enable IPv6 -> src/config/general.h -> #define NET_PROTO_IPV6
• mangle IPv4 DHCP timeout -> src/config/dhcp.h -> #define DHCP_DISC_END_TIMEOUT_SEC 1

Debian/Ubuntu ipxe-qemu patching

apt-get update
apt-get source ipxe-qemu
apt-get build-dep ipxe-qemu
cd ipxe-1.0.0+git-20180124.fbe8c52d # or whatever your version is
dch -n "add support for serial console / IPv6"
# apply the modifications above
debuild -b -uc -us
cd ..
dpkg -i ipxe-qemu_*.deb

Kernel options

Make sure you pass the right kernel options for initramfs/initrd to know exactly what you want to do. (replace eth0 with the name of your PXE interface)

• ip=:::::eth0:off (disable IPv4 on the PXE interface)
• ip6=eth0 (enable IPv6 on the PXE interface)
• nfsroot=<hostname>:<path> (mounting IPv6 NFS shares using a literal IPv6 WILL FAIL, you MUST use a hostname)

Those options will tell the initrd script which interface and protocol is desired.

Hooks in initramfs

NOTE: Be aware that this will increase the size of initrd. In my case it went to ~16MB which is still way less than on a physical host (~60MB).

A few files must be copied to the initramfs in order to make this work. First we need DNS to work and secondly we need to replace the busybox nfsmount with mount.nfs.

In /etc/initramfs-tools/hooks/ create an executable script with the name of your choice. Here’s the content:

#!/bin/bash
. /usr/share/initramfs-tools/hook-functions

# copy all the name libraries (for DNS to work)
cp -fpL /lib/x86_64-linux-gnu/libns* ${DESTDIR}/lib/x86_64-linux-gnu/

# copy helper files needed by mount.nfs
for file in /etc/protocols /etc/netconfig
do cp $file ${DESTDIR}${file}; done

# uncomment to embed in your setup, can be useful for debugging
# copy_exec /sbin/dhclient /sbin
# copy_exec /bin/ping /bin
# copy_exec /usr/bin/strace /bin

# replace the busybox nfsmount with mount.nfs (this way we don't need to modify the included nfs script which calls nfsmount)
cp /sbin/mount.nfs ${DESTDIR}/bin/nfsmount

# this little hack will find all libraries needed by mount.nfs, locate them and copy them.
for lib in `ldd /sbin/mount.nfs | awk '{print $1}'`
do
    found=0
    for path in /lib /lib/x86_64-linux-gnu /usr/lib /usr/lib/x86_64-linux-gnu
    do
        if [ -r "${path}/${lib}" ]
        then
            cp -L ${path}/${lib} ${DESTDIR}/${path}/${lib}
            found=1
        fi
    done

    if [ ! $found ]
    then
        echo "ERROR: Cannot find ${lib}"
        exit 1;
    fi
done

After that you’ll need to update-initramfs -u

Boot

Boot and it should just work! Happy IPv6!

Final notes

iPXE & NFS

Due to incompatible limiting requirements of iPXE and initramfs, both must be run on separate subnets. You will need one interface for iPXE and one for NFS.

radvd.conf

# iPXE needs SLAAC and DHCPv6 options
interface bridge0 {
    IgnoreIfMissing on;
    AdvSendAdvert on;
    AdvOtherConfigFlag on;
    AdvDefaultLifetime 0;
    AdvLinkMTU 9000;
    AdvCurHopLimit 64;
    AdvReachableTime 0;
    MaxRtrAdvInterval 600;
    MinRtrAdvInterval 198;
    AdvDefaultPreference low;
    AdvRetransTimer 0;
    AdvManagedFlag off;
    prefix <your PXE /64>::/64 { };
    RDNSS <your dnsmasq server (yourself) IP, must be in the same subnet>:: { };
};

# busybox/initramfs needs DHCPv6
interface bridge2 {
    IgnoreIfMissing on;
    AdvSendAdvert on;
    AdvOtherConfigFlag off;
    AdvDefaultLifetime 0;
    AdvLinkMTU 9000;
    AdvCurHopLimit 64;
    AdvReachableTime 0;
    MaxRtrAdvInterval 600;
    MinRtrAdvInterval 198;
    AdvDefaultPreference low;
    AdvRetransTimer 0;
    AdvManagedFlag off;
    prefix <your NFS /64>::/64 {
        AdvAutonomous off;
    };
};

isc-dhcp-server6 dhcpd6.conf

# PXE
subnet6 <your PXE /64>::/64 {
	option dhcp6.bootfile-url "http://<pxe server hostname>?uuid=${uuid}&op=ipxe";
}
# NFS
subnet6 <your NFS /64>::/64 {
	range6 <your NFS /64>::1 <your NFS /64>::ffff;
	option dhcp6.name-servers <your NFS server>::;
}

dnsmasq

# leave everything to defaults except
interface=bridge1
interface=bridge2
no-dhcp-interface=bridge1
no-dhcp-interface=bridge2

/etc/hosts

DO NOT MAKE YOUR VMs depend on VMs to boot !!! If your DNS server is a VM, make sure that the hostnames used to boot (pxe server hostname, nfs server hostname) can be resolved by dnsmasq, the easiest way to ensure this is to add them to /etc/hosts which dnsmasq will use by default.