What does this Raspberry Pi PXE Boot tutorial cover?

This Raspberry Pi PXE Boot tutorial walks you through netbooting a Raspberry Pi 4 without an SD card. We use another Raspberry Pi 4 with an SD card as the netboot server. Allocate 90-120 minute for completing this tutorial end to end. It can faster if you already familiar with some of the material.

Why I wrote this tutorial

Does the world need another Raspberry Pi PXE boot tutorial? I read many amazing docs, forum posts and blog posts on the topic before starting this project. However they all have some gaps I filled in myself. So I decided to write a tutorial that addresses the following gaps.

Why PXE boot or netboot a Raspberry Pi?

I am embarking on an IOT project using Raspberry Pis in a Kubernetes cluster. 10 Pis will be in the cluster for running containerized workloads. I want to make provisioning and re-provisioning the cluster nodes easy as pie (pun intended). As a result of this my first stage of the project is figuring out how to PXE boot the Raspberry Pi 4. Which led me to creating this tutorial.

My goals are:

Netbooting is a good path to achieve these. For example, when you netboot a Pi it does not require an SD card to boot. The OS and file system live on a central server. Because most of the provisioning happens on a central server I can eventually automate it via scripts.

What is PXE, How does it work?

This is a basic overview of PXE. If you want to dive deeper on PXE we suggest you read out post What is PXE? How does it work?

PXE stands for Preboot Execution Environment. At a high level PXE is a standard for network booting a computer. It uses standard networking protocols to achieve network booting. Specifically IP, UDP, DHCP and TFTP. PXE is typically used in one of two ways:

The diagram below shows the high level flow of the PXE boot process. Understanding the flow will help in the event you need to troubleshoot a boot failure.

The PXE boot flow. Implementations can differ. Also the server components can be spread across multiple hosts.

Overview of the PXE flow

Inventory

Assumptions

Phase 1 – PXE Boot Client Configuration

The Raspberry Pi 4 has an EEPROM. The EEPROM is capable of network booting. Unfortunately the only way I have found to configure network booting is from Linux. Hence you must boot the system at least once with an SD card to configure it.

Install Raspbian on an SD card and install needed tools

Let’s start configuring your client system for netboot. This is the Raspberry Pi that will eventually boot without a micro SD card installed.

sudo dd if=2019-09-26-raspbian-buster-lite.img of=/dev/sdX bs=4M
sudo apt-get update
sudo apt-get full-upgrade
sudo apt-get install rpi-eeprom

Configure the Rasperry Pi 4 bootloader to PXE boot

Next lets examine your boot loader configuration using this command:

vcgencmd bootloader_config

Here is the output on my fresh out of the box Raspberry Pi 4:

[email protected]:~ $ vcgencmd bootloader_config
 BOOT_UART=0
 WAKE_ON_GPIO=1
 POWER_OFF_ON_HALT=0
 FREEZE_VERSION=0

We need to modify the boot loader config to boot off the network using the BOOT_ORDER parameter. To do that we must extract it from the EEPROM image. Once extracted, make our modifications to enable PXE boot. Finally install it back into the boot loader.

We do that with these steps:

cd /lib/firmware/raspberrypi/bootloader/beta/
cp pieeprom-2019-11-18.bin new-pieeprom.bin
rpi-eeprom-config new-pieeprom.bin > bootconf.txt
rpi-eeprom-config --out netboot-pieeprom.bin --config bootconf.txt new-pieeprom.bin
sudo rpi-eeprom-update -d -f ./netboot-pieeprom.bin

Disabling automatic rpi-eeprom-update

As pointed out by a reddit user, rpi-update will update itself by default. The rpi-eeprom-update job does this. Considering that we are using beta features, a firmware update could disable PXE boot in the eeprom. You can disable automatic updates by masking the rpi-eeprom-update via systemctl. You can manually update the eeprom by running rpi-eeprom-update when desired. See the Raspberry Pi docs on rpi-eeprom-update for more details.

sudo systemctl mask rpi-eeprom-update

Phase 1 Conclusion

Congratulations! We are half way to first net boot. Our Raspberry Pi net boot client is configured for PXE boot. Before you shut down the Pi 4 please make note of ethernet interface MAC address. You can do this by running ip addr show eth0 and copying the value from the link/ether field. In my case it was link/ether dc:a6:32:1c:6a:2a.

Unplug and put aside your Raspberry Pi PXE boot client for now. We are moving on to configuring the server. Now is also a good time to remove the SD card. It is no longer needed now that the Pi will net boot.

Phase 2 – Raspberry Pi PXE Boot Server Configuration

If you completed the client configuration you can use the same SD card for the server or use a second one. For example I use two different micro SD cards in case I need to boot the client off micro SD for debugging purposes.

Are you are using two micro SD cards? Make sure install Raspbian on the second card as well. Follow the the instructions earlier in the tutorial. Then boot your server off the SD card. Some of the initial server configuration steps will be familiar. Boot the server connected to an Internet connection. We need the Internet connection to update and install packages. Later in this phase we will remove it from the Internet and plug directly into the other Raspberry Pi.

Update Raspbian and install rpi-eeprom, rsync and dnsmasq

Update the Raspbian OS via apt-get and install the rpi-config program. Note this step can take a while. Time will vary based on the speed of your Internet connection.

sudo apt-get update
sudo apt-get full-upgrade
sudo apt-get install rpi-eeprom

Install rsync and dnsmasq. We will use rsync to make a copy of the base os and we will use dnsmasq as the DHCP and TFTP server. NFS will be used expose the root file system to the client.

sudo apt-get install rsync dnsmasq nfs-kernel-server

Create the NFS, tftp boot directories and create our base netboot filesystem

Make the NFS and tftpboot directories. The /nfs/client1 directory will be the root of the file system for your client Raspberry Pi. If you add more Pis you will need to add more client directories. The /tftpboot directory will be used by all your netbooting Pis. It contains the bootloader and files needed to boot the system.

sudo mkdir -p /nfs/client1
sudo mkdir -p /tftpboot
sudo chmod 777 /tftpboot

Copy your Pi’s OS filesystem in the /nfs/client1 directory. We are going to exclude some files from the rsync. This is a preventative measure in case you run this command again after configuring the network and dnsmasq. This command takes some time due to the IO characteristics of SD cards. They are slow 🙂

sudo rsync -xa --progress --exclude /nfs/client1 \
    --exclude /etc/systemd/network/10-eth0.netdev \
    --exclude /etc/systemd/network/11-eth0.network \
    --exclude /etc/dnsmasq.conf \
    / /nfs/client1

Now we use chroot to change root into that directory. But before we chroot we need to bind mount the required virtual filesystems into the base client directory.

Once in the chroot we delete server SSH keys. Next we reconfigure the openssh server package which will regenerate the keys. Additionally we enable the ssh server so we can remotely login when the client comes online.

cd /nfs/client1
sudo mount --bind /dev dev
sudo mount --bind /sys sys
sudo mount --bind /proc proc
sudo chroot . rm /etc/ssh/ssh_host_*
sudo chroot . dpkg-reconfigure openssh-server
sudo chroot . systemctl enable ssh
sudo umount dev sys proc

Configure the PXE server to use a static IP

Our PXE server is a DHCP server. Meaning it assigns IP addresses and network configuration to clients which request them. In this case our Raspberry Pi PXE boot client. If we do not want the PXE boot server itself to run the DHCP client. Therefore we should disable the DHCP client. Let’s do that now. Create a new systemd file to disable the DHCP client on eth0. The path for the file we wish to create is /etc/systemd/network/10-eth0.netdev. Its contents should be:

[Match]
Name=eth0
[Network]
DHCP=no

Create the /etc/systemd/network/11-eth0.network file with the following contents. Please note that I am specifying 192.168.2.1 as the DNS server and gateway address. In this tutorial I do not have a gateway or DNS server at that address. Further, none are needed for this tutorial. I have them there as a place holder so if I want to connect this system I can drop a router on the network at that address. You can probably leave DNS and Gateway out if you prefer.

[Match]
Name=eth0

[Network]
Address=192.168.2.100/24
DNS=192.168.2.1
Gateway=192.168.2.1

No we are going to disable to the dhcp client service dhcpcd that is enabled by default on raspbian. Please pay extra careful attention to the fact that is “dhcpcd” and not “dhcpd”. The first is a DHCP client, the second a server.

sudo systemctl stop dhcpcd
sudo systemctl disable dhcpcd

Configure dnsmasq for PXE boot

This step configures dnsmasq to support our PXE boot. Replace your /etc/dnsmasq.conf file with the following contents:

interface=eth0
no-hosts
dhcp-range=192.168.2.101,192.168.2.200,12h
log-dhcp
enable-tftp
tftp-root=/tftpboot
pxe-service=0,"Raspberry Pi Boot"

Next we copy the boot files from our /boot directory into the tftpboot directory.

sudo cp -r /boot/* /tftpboot/

Enable systemd-networkd and dnsmasq. Restart dnsmasq to confirm the config is valid. Finally reboot and ensure the Pi comes up with the network configured properly.

sudo systemctl enable systemd-networkd
sudo systemctl enable dnsmasq.service
sudo systemctl restart dnsmasq.service
sudo reboot

Now we must update the cmdline.txt file in /tftpboot. This file contains the kernel parameters that are passed to our client Raspberry Pi at boot time. Edit /tftpboot/cmdline.txt replace it with:

console=serial0,115200 console=tty1 root=/dev/nfs 
nfsroot=192.168.2.100:/nfs/client1,vers=3 rw ip=dhcp rootwait elevator=deadline

Configure the NFS exports on the PXE boot server

This steps configures the exports. Exports are file systems that are being shared or exported via NFS. To do this we must configure the /etc/exports service and the restart the NFS related services.

The contents of /etc/exports should be as follows.

/nfs/client1 *(rw,sync,no_subtree_check,no_root_squash)
/tftpboot *(rw,sync,no_subtree_check,no_root_squash)

Configure the /etc/fstab to mount via NFS

We are almost done! One last step to modify the /etc/fstab file in our client’s file system. This will tell the client to mount its root volume off the NFS server on our PXE Boot server Raspberry Pi. Put the following into /nfs/client1/etc/fstab.

proc       /proc        proc     defaults    0    0
192.168.2.100:/tftpboot /boot nfs defaults,vers=3 0 0

Finally enable and restart NFS related services.

sudo systemctl enable rpcbind
sudo systemctl restart rpcbind
sudo systemctl enable nfs-kernel-server
sudo systemctl restart nfs-kernel-server

Now do one last reboot on the server for good measure. Take a look at the system logs and systemctl statuses to see if everything started correctly.

Complete. Does it work?

Nice work getting through the tutorial. Now is the final test. Plug your client Raspberry Pi into the network or directly to the server via ethernet. Now connect a keyboard and LCD screen to your client Raspberry Pi. Power on and wait. Hopefully you will see the following after a few moments!

Raspberry Pi PXE Troubleshooting Guide

Hopefully you are up and running. But if you are experiencing problems this section can help you debug your kit. The trickiest part of troubleshooting this setup is that the graphical console on the client emits no information until the OS kernel starts booting. As a result I had to do all troubleshooting on the server side.

It is possible the client does emit some useful information via serial console. But I have not tried because I don’t have the right equipment today.

Troubleshooting Tools

sudo systemctl status dnsmasq.service
sudo systemctl status rpcbind.service
sudo systemctl status nfs-mountd.service
sudo nfsstat
sudo tail -f /var/log/daemon.log
tcpdump -n -i eth0 
tcpdump -n -i eth0 port 67

What stage is the failure?

The key to troubleshooting PXE boot problems is figuring out where in the workflow it is failing. Hence if you are new to PXE, re-reading the earlier section of this post (What is PXE, How does it work?) will help.

The first question you need to answer is: “What stage is the failure in?” It could be in the following stages:

DHCP Stage

If your client is properly configured it should be making a DHCP request at boot time. Lets see if DHCP is working.

TFTP Stage

NFS Stage

If all else fails

Try again, the network boot is a beta feature and could have bugs. For example, reports on the Raspberry Pi site indicate a reboot can be required if it not working.

Room for improvement

This process is hacky. In other words, plenty of room for improvement. If time permits I will implement the following improvements.

Feedback

I want to make this guide as thorough as possible. Please provide feedback to this post in the comments. with any feedback. Constructive feedback will be worked into future edits.

Donations and Tips

We provide content like this tutorial free of charge. But people often ask how they can support the site. If you find this tutorial useful and wish to support the site you can donate here. These tips help to pay for hosting costs and time spent writing content.

Make a one-time donation

Your contribution is appreciated.

Donate

Make a monthly donation

Your contribution is appreciated.

Donate monthly

Make a yearly donation

Your contribution is appreciated.

Donate yearly

Suggested Products

Credits

The following resources were instrumental in this project.

Update Log

23 Responses

  1. This is a great tutorial!! I had one question though. Have you tried to PXE boot a Pi 4 client with the PiServer tool? It works on Pi 3 and to some extent Pi 2 ( It’s very slow and you still need an SD card with bootcode.bin on it) . I’m trying to get it to work with Pi 4, but my server doesn’t seem to be detecting my client. Thanks.

    1. I have not tried it. Looks interesting though. I will try to check it out in the next couple of weeks. If I had to guess what your problem is I would wager that Pi boot is using the dhcp/pxe client identifier string to decide if it should net boot it and it probably does not have the pi 4 identifier string in its configuration.

    2. If you change the /etc/dnsmasq.conf at the bottom, and add:
      dhcp-mac=set:client_is_a_rpi4,DC:A6:32:*:*:*
      just aftert the existing one, and you have the PI4 set up with the beta netboot firmware, and you have images after 9/26/2019, it works very well.

  2. Great work on the write up; I’m looking forward to working on packer for image builds now that I got PXE working with your guide. I think it’s worth noting the DHCP options the pi4 requires to boot successfully. This is gave me some hangs ups trying to use a different DHCP server, but a packet capture revealed some things I was missing. The clients BOOTREQUEST was also requesting options 128 – 135 but these don’t appear necessary. Found this on a RPF forum thread and wanted to include this here for your readers.

    Option 43 (vendor-encapsulation) – important – needs to be set to colon separated hex values 6:1:3:a:4:0:50:58:45:9:14:0:0:11:52:61:73:70:62:65:72:72:79:20:50:69:20:42:6f:6f:74:ff or string value “ Raspberry Pi Boot”

    Option 60 (vendor-class-identifier) – needs to be set to string value “PXEClient”

    Option 66 (tftp-server) – IP address for tftp server

    Option 67 (boot-file) – set to /path_to/bootcode.bin

    1. Thanks for the notes. Yes, in the dnsmasq.conf its not obvious what DHCP options the line ‘pxe-service=0,”Raspberry Pi Boot”‘ translate to in other DHCP servers. I might write a more generic DHCP guide that for this that can be used to make configurations for other servers.

      With regards to the “PXEClient” for option 60, I believe that is only needed if you want to specify a different configuration based on the identifier. In this case I was trying to keep it as simple as possible. So I built the config with the assumption that all hosts would get a DHCP lease that includes the PXE boot options.

      What version of Raspberry Pi are you using? I tested it this on 4 which should not require the bootcode.bin file if I remember correctly

      1. I’m using a Pi 4. And it appears you are correct; I deactivated the boot-file from my DHCP Server config and it it appears to boot fine.

        I Just added another node and Im confused how I would point another client to a different directory in the /tftpboot/cmdline.txt config. Both boot and mount the same rootfs as is, and it’s definitely a split brain.

        1. Hi Randall,

          This should be possible by creating a directory in /tftpboot that is named after the Pi’s serial number and putting a custom cmdline.txt in that directory. On boot the Pi will look for that directory and if it exists it will download files from there. I am not near my computer at the moment so I can’t fully test. But I have played around with it in the past and it should work.

          1. Playing around with it I got it working; thanks for the pointer. For those wondering you can get the serial as follows:

            [email protected]:~ $ cat /proc/cpuinfo | grep Serial
            Serial : 10000000ba8c150a

            When you create a directory its going to expect the name to be the last 8 characters of the serial.

            [email protected]:/tftpboot/ba8c150a $ pwd
            /tftpboot/ba8c150a

            I found it expects all of the files to be present including the kernel.img. I’m going to try and figure a work around for this; ideally i’d have them all sharing 1 kernel image for making updates easier but I should be able to work around with scripts.

  3. Hello,
    I’m trying to boot a few rpi’s from the same folder on server and it boots without any problem but when Xorg and chromium starts on both of them it crash. Can I use one folder for bunch of rpi’s or it will crash because of file overwriting? I don’t want to waste space on server and create multiple copies of the same image.

    1. Hi Maciej, the intent is one folder per host. Most programs do not expect to be running in an environment where multiple instances or hosts are sharing the file system.

  4. Hi Ken, very detailed blog post and write-up. I noticed that your original driving force for documenting this method was to run Kubernetes on your Raspberry Pi cluster. You haven’t mentioned it since in the post or on your recent articles, so I wondered how you got around the issue with Docker (overlayfs) not working with NFS?

    I ran into this several years ago with the RPi3. https://blog.alexellis.io/the-state-of-netbooting-raspberry-pi/

    At the time, the community was suggesting an iSCSI target.

    Feel free to email me [email protected] if you’d like to talk more.

    1. Hi Alex, I ended up shelving this project for a while. I had gotten k3s up and running but not on NFS root yet. My initial thought was to see if I could use tmpfs with really small container images. I realize RPI is not a great platform for that given the ram constraints but figured I would try it. ISCSI would be another option might be NBD but I haven’t tried it.

  5. Looking to PXEBoot for RPi4 project at work. Can the boot file directory be some machine name base vs the serial number?
    IE… PublicKiosk1, PublicKiosk2. That way when they fail I just load the initial configuration and settings to convert it from a SDCard operation to PXEBoot. Would need no server side changes to account for the new device replacing the old.

  6. You mention setting up a client from a downloaded image. Here are my notes if you’d like to incorporate them. /srv/rpi/diskless is my nfs export base.

    unzip $IMG.zip
    kpartx -av $IMG.img
    mkdir -p p1
    mount /dev/mapper/loop0p1 p1
    mkdir -p p2
    mount /dev/mapper/loop0p2 p2
    mkdir /srv/rpi/diskless/$NEWHOST
    cp -a p2/. /srv/rpi/diskless/$NEWHOST/.
    cp -a p1/. /srv/rpi/diskless/$NEWHOST/boot/.
    umount p1
    umount p2
    kpartx -d $IMG.img

    This is a minimal-ish setup for a raspian-lite image (July ’20)

    cd /srv/rpi/diskless/$NEWHOST/
    # Make it ssh
    touch boot/ssh
    # Don’t try to mount sd stuff
    sed -i /UUID/d etc/fstab
    # nfsboot please
    echo “console=serial0,115200 console=tty root=/dev/nfs nfsroot=$NFS:/srv/rpi/diskless/$NEWHOST,vers=3 rw ip=dhcp rootwait elevator=deadline” > boot/cmdline.txt
    # don’t look for the SD card all the time
    echo dtparam=sd_poll_once=on >> boot/config.txt
    # Don’t resize or try to add swap
    rm -fv etc/init.d/resize2fs_once etc/rc*.d/*resize2fs_once
    rm -fv etc/init.d/dphys-swapfile etc/rc*.d/*dphys-swapfile

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.