How to boot a diskless VM with iPXE over NFS using only IPv6

I have encountered so many issues and countless hours trying to figure this out, but I finally succeeded. Reading source code, strace’ing binaries trying to figure out what was wrong with them.

iPXE

iPXE will do IPv4 DHCP first and then IPv6 SLAAC/DHCP. This is not a problem, it’s just annoying. IPv4 DHCP has a default timeout in iPXE of 10 seconds.

What you need to do in iPXE:

• enable serial console -> src/config/console.h -> #define CONSOLE_SERIAL
• enable IPv6 -> src/config/general.h -> #define NET_PROTO_IPV6
• mangle IPv4 DHCP timeout -> src/config/dhcp.h -> #define DHCP_DISC_END_TIMEOUT_SEC 1

Kernel options

Make sure you pass the right kernel options for initramfs/initrd to know exactly what you want to do. (replace eth0 with the name of your PXE interface)

• ip=:::::eth0:off (disable IPv4 on the PXE interface)
• ip6=eth0 (enable IPv6 on the PXE interface)
• nfsroot=<hostname>:<path> (mounting IPv6 NFS shares using a literal IPv6 WILL FAIL, you MUST use a hostname)

Those options will tell the initrd script which interface and protocol is desired.

Hooks in initramfs

NOTE: Be aware that this will increase the size of initrd. In my case it went to ~16MB which is still way less than on a physical host (~60MB).

A few files must be copied to the initramfs in order to make this work. First we need DNS to work and secondly we need to replace the busybox nfsmount with mount.nfs.

In /etc/initramfs-tools/hooks/ create an executable script with the name of your choice. Here’s the content:

#!/bin/bash
. /usr/share/initramfs-tools/hook-functions

# copy all the name libraries (for DNS to work)
cp -fpL /lib/x86_64-linux-gnu/libns* ${DESTDIR}/lib/x86_64-linux-gnu/

# copy helper files needed by mount.nfs
for file in /etc/protocols /etc/netconfig
do cp $file ${DESTDIR}${file}; done

# uncomment to embed in your setup, can be useful for debugging
# copy_exec /sbin/dhclient /sbin
# copy_exec /bin/ping /bin
# copy_exec /usr/bin/strace /bin

# replace the busybox nfsmount with mount.nfs (this way we don't need to modify the included nfs script which calls nfsmount)
cp /sbin/mount.nfs ${DESTDIR}/bin/nfsmount

# this little hack will find all libraries needed by mount.nfs, locate them and copy them.
for lib in `ldd /sbin/mount.nfs | awk '{print $1}'`
do
    found=0
    for path in /lib /lib/x86_64-linux-gnu /usr/lib /usr/lib/x86_64-linux-gnu
    do
        if [ -r "${path}/${lib}" ]
        then
            cp -L ${path}/${lib} ${DESTDIR}/${path}/${lib}
            found=1
        fi
    done

    if [ ! $found ]
    then
        echo "ERROR: Cannot find ${lib}"
        exit 1;
    fi
done

After that you’ll need to update-initramfs -u

Boot

Boot and it should just work! Happy IPv6!