Build a Raspberry Pi image with Packer – packer-builder-arm

Introduction: Building a Raspberry Pi image with Packer

Today we are test driving packer-builder-arm, this tool enables you to build a Raspberry Pi image with Packer (in addition to other ARM platforms). Packer-builder-arm is a plugin for Packer. It extends Packer to support ARM platforms including Raspberry Pi. Packer is tool from Hashicorp for automating OS image builds. Additionally packer-builder-arm enables you to build these image on your local machine, cloud server or other x86 hardware. This means you don’t need your Raspberry Pi handy to build a Raspbian ARM image. To do this it leverages arm emulation available in QEMU. Specifically it copies a statically built QEMU arm emulator into the image which allows us to run files compiled for ARM inside the chroot on an X86 system.

packer-builder-arm github screenshot of boards directory
The boards directory of the packer-builder-arm project show some of the boards you can build images for.

Why use Packer?

Why build your Raspberry Pi image with Packer? Projects with embedded devices such as a Raspberry Pi often need tweaks to the OS installation. For example you might change config files or install some packages. Manually running these commands for one device is no big deal. However repeating this process is time consuming and error prone. It does not scale well to many devices.

A common solution is to customize your OS install and then clone the SD card or storage device. This works well enough. However each time you want to tweak your configuration you still need to manually re-run the process. Iteration is manual.

This is where Packer comes in. Packer enables you to codify your OS configuration and customization. Packer builds your image for you applying your customization. You can store your Packer files in git and now you have a repeatable process. If you need to add a new package, just add it to your packer build files and re-run packer. Presto! You have a new image. You can take this even further by setting up CI/CD pipelines to run your builds automatically.

How does Packer work?

Packer Overview

Packer automates a simple but powerful concept. Start with base installation media or a base image of operating system. Boot or chroot into the image and run commands or scripts to customize it. Then capture the output in an image artifact. You can now take that image and re-use it on multiple machines.

Packer is commonly used in large scale cloud environments. In these environments you often have thousands of machines. OS image builds need to be automated and tested in an environments of this scale. The only reasonable way to achieve this is with the automation that tools like Packer provide.

Packer Plugins

Packer supports plugins for extending its functionality. Packer’s plugin architecture is quite simple. Packer plugins are just stand alone binaries or scripts that Packer executes.

Two examples of plugin types for Packer are builders and provisioners. Builders are focused on setting up the infrastructure required to build the image. Provisioners handle the changes you make at the OS level such as install package. Read the Packer plugins page for more details.

Today we are going to explore a builder plugin called packer-arm-builder.

packer-builder-arm

Packer-builder-arm extends Packer to build arm based images. Specifically it does the following:

  • Fetch a base ARM os image such as Raspbian or Arch Linux.
  • Run the commands you specify in a chroot environment with ARM emulation.
  • Save the customized chroot environment to a new image.

It achieves this by copying the QEMU static ARM emulator into the chroot environment. This allows ARM compiled binaries in the chroot to execute as if they were running on an ARM machine. It is worth noting that this can be fairly slow compared to running directly on an ARM CPU. However it is really handy because you don’t to run on ARM hardware.

Building and installing packer-builder-arm

Prerequisites

  • A modern linux system. We are using Ubuntu 18.04.
  • A recent version of Go. If you need help installing Go see our tutorial: How to install Go on Linux.
  • Access to the internet to install dependencies.

Step 1 – Install dependencies

Install Go. We are testing this with Go 1.13. See our tutorial: How to install Go on Linux if you need more help.

Now that we have Go installed we install the following packages:

  • git
  • unzip
  • qemu-user-static
  • e2fsprogs
  • dosfstools
  • bsdtar

We are using Ubuntu 18.04. Hence we will use apt to install our dependencies. If you are using a different distribution you will need to adapt these commands to install the packages on your platform. In our case we run:

sudo apt-get install git
sudo apt-get install unzip
sudo apt-get install qemu-user-static
sudo apt-get install e2fsprogs
sudo apt-get install dosfstools
sudo apt-get install bsdtar

Step 2 – Install Packer

We chose to install Packer directly from the Packer website instead of using packages available in the Ubuntu package repositories. We have a reason for this. Packer moves pretty quickly so the version of Packer in the Ubuntu repos is far behind the latest Packer. Given the rate at which cloud native tools change we want to make sure we have the latest code base. The best way to do this is get Packer directly from Hashicorp.

To install Packer we will first download the zipped binary from the Packer website using the wget command line HTTP client.

wget https://releases.hashicorp.com/packer/1.4.5/packer_1.4.5_linux_amd64.zip

Version 1.4.5 is the latest as of this writing. However I suggest you check website to see if there is a newer Packer release available.

Now that we have the Packer zip file we need to unzip it:

unzip packer_1.4.5_linux_amd64.zip 

You should now have a packer binary in your current working directory. Lets move it to a more permanent location. We can do this putting the packer binary in your /usr/local/bin directory:

sudo mv packer /usr/local/bin/

More than likely you already have /usr/local/bin in your PATH environment variable. If you do then you should be able to run packer --help and see a similar output to what I have below.

packer --help
Usage: packer [--version] [--help] <command> [<args>]

Available commands are:
    build       build image(s) from template
    console     creates a console for testing variable interpolation
    fix         fixes templates from old versions of packer
    inspect     see components of a template
    validate    check that a template is valid
    version     Prints the Packer version

Congratulations! You have Packer installed. Time to move onto packer-builder-arm.

Step 3 – Install packer-builder-arm

In this step we are going to get the latest code for the packer-builder-arm plugin from Github. Next we will build it and then finally install it.

To clone the source code from github, use the following git command:

git clone https://github.com/mkaczanowski/packer-builder-arm

After cloning the source code we will need to change directories into the working directory, fetch go modules and build the go source code.

cd packer-builder-arm
go mod download
go build

After the go build command you should have a packer-builder-arm file in your current directory.

ubuntu@packer-test:~/packer-builder-arm$ ls -l
total 32484
-rw-rw-r--  1 ubuntu ubuntu    11357 Dec 13 22:04 LICENSE
-rw-rw-r--  1 ubuntu ubuntu     5097 Dec 13 22:04 README.md
drwxrwxr-x 11 ubuntu ubuntu     4096 Dec 13 22:04 boards
drwxrwxr-x  2 ubuntu ubuntu     4096 Dec 13 22:04 builder
drwxrwxr-x  2 ubuntu ubuntu     4096 Dec 13 22:04 config
-rw-rw-r--  1 ubuntu ubuntu      818 Dec 13 22:04 go.mod
-rw-rw-r--  1 ubuntu ubuntu    45835 Dec 13 22:04 go.sum
-rw-rw-r--  1 ubuntu ubuntu      268 Dec 13 22:04 main.go
-rwxrwxr-x  1 ubuntu ubuntu 33173413 Dec 13 22:23 packer-builder-arm

At this point we have a couple of options:

  • Run packer from within this directory. Packer will check the current directory for plugins.
  • Move the packer-builder-arm binary to /usr/local/bin/. Packer will also look for plugins in the same directory as the packer binary.
  • Move the packer-builder-arm binary to $HOME/.packer.d/plugins.

We are going to go with the first one. If you intend to use this a permanent setup, investigate the other two.

Ok. We are almost there. Time to move onto actually building the image!

Step 4 – Build the Raspbian image

This is where the power of Packer really shines. We are going to use one of the existing configurations in packer-builder-arm to build a Raspbian image. But before we do that, we need to fix a small bug that I came across with the raspian.json file.

Fix a bug in raspbian.json

Edit the file in packer-image-arm/boards/raspberry-pi/raspbian.json.

We need to update the value for image_chroot_env. Specifically we change the following line:

"image_chroot_env": ["PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/sbin"],

to include the /usr/sbin directory. The line should now look like this:

"image_chroot_env":["PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"],

The issue I ran into is that packer was unable to find the chroot command on my system. Chroot is in /usr/sbin so we add it the PATH variable that is being passed to image_chroot_env.

Understand the contents of raspbian.json

From within the packer-image-arm working directory, we will run the following packer command with sudo after looking at the raspbian.json file:

sudo packer build boards/raspberry-pi/raspbian.json

Before running the build lets figure out what the Raspbian.json is doing:

{
  "variables": {},
  "builders": [{
    "type": "arm",
    "file_urls" : ["https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip"],
    "file_checksum_url": "https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip.sha256",
    "file_checksum_type": "sha256",
    "file_target_extension": "zip",
    "image_build_method": "reuse",
        "image_path": "raspberry-pi.img",
        "image_size": "2G",
    "image_type": "dos",
        "image_partitions": [
                {
                        "name": "boot",
                        "type": "c",
                        "start_sector": "8192",
                        "filesystem": "vfat",
                        "size": "256M",
            "mountpoint": "/boot"
                },
                {
                        "name": "root",
                        "type": "83",
                        "start_sector": "532480",
                        "filesystem": "ext4",
                        "size": "0",
            "mountpoint": "/"
                }
        ],
    "image_chroot_env": ["PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"],
        "qemu_binary_source_path": "/usr/bin/qemu-arm-static",
        "qemu_binary_destination_path": "/usr/bin/qemu-arm-static"
  }],
  "provisioners": [
    {
      "type": "shell",
      "inline": [
          "touch /tmp/test"
      ]
    }
  ]
}

In the raspbian.json file we have two high level objects, builder and provisioner. The builder in this file has the following keys or parameters:

  • "type": "arm" – The builder type we are using.
  • "file_urls": ... – The url of the image we are using as a base.
  • "file_checksum_url"... – The url of the image’s checksum
  • "file_checksum_type": "sha256" – The hashing algorithm used to generate the checksum.
  • "file_target_extension": "zip" – The extension of the image file.
  • "image_build_method": "reuse" – This tells the plugin if we want to reuse the disk image or create a new one from scratch.
  • "image_path": "raspberry-pi.img" – The name of the image we will create.
  • "image_size": "2G" – The size of the image
  • "image_type": "dos"
  • "image_partitions":... – Contains the specifications of the partitions in the image.
  • "image_chroot_env"... – Shell environment that is passed to the chroot command.
  • "qemu_binary_source_path": & "qemu_binary_destination_path": The source where we will find the qemu static binary and the the destination path inside the chroot where we will copy it.

Additionally there is a the provisioner section which is much shorter:

  • "type": "shell" – This tells packer we are using the shell provisioner to configure the image.
  • "inline"... – Inline specifies an array of commands that get passed to the shell provisioner. In this case we pass one command “touch /tmp/test“. You can add additional commands here. See the shell provisioner doc for other options.

Build your Raspberry Pi image with Packer

Ok. Almost there! Let’s run the build now:

sudo packer build boards/raspberry-pi/raspbian.json

Here is the output from my packer run:

ubuntu@packer-test:~/packer-builder-arm$ sudo packer build boards/raspberry-pi/raspbian.json
arm output will be in this color.

==> arm: Retrieving rootfs_archive
==> arm: Trying https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false
==> arm: Trying https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false&checksum=sha256%3A2c4067d59acf891b7aa1683cb1918da78d76d2552c02749148d175fa7f766842
==> arm: https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false&checksum=sha256%3A2c4067d59acf891b7aa1683cb1918da78d76d2552c02749148d175fa7f766842 => /home/ubuntu/packer-builder-arm/packer_cache/665ae48e54b8fac4541688233efc4a82b75de3ce.zip
    arm: unpacking /home/ubuntu/packer-builder-arm/packer_cache/665ae48e54b8fac4541688233efc4a82b75de3ce.zip to raspberry-pi.img
    arm: searching for empty loop device (to map raspberry-pi.img)
    arm: mapping image raspberry-pi.img to /dev/loop2
    arm: mounting /dev/loop2p2 to /tmp/495303497
    arm: mounting /dev/loop2p1 to /tmp/495303497/boot
    arm: running extra setup
    arm: mounting /dev with: [mount --bind /dev /tmp/495303497/dev]
    arm: mounting /devpts with: [mount -t devpts /devpts /tmp/495303497/dev/pts]
    arm: mounting proc with: [mount -t proc proc /tmp/495303497/proc]
    arm: mounting binfmt_misc with: [mount -t binfmt_misc binfmt_misc /tmp/495303497/proc/sys/fs/binfmt_misc]
    arm: mounting sysfs with: [mount -t sysfs sysfs /tmp/495303497/sys]
    arm: binfmt setup found at: /proc/sys/fs/binfmt_misc/qemu-arm
    arm: copying qemu binary from /usr/bin/qemu-arm-static to: /tmp/495303497/usr/bin/qemu-arm-static
    arm: running the provision hook
==> arm: Provisioning with shell script: /tmp/packer-shell936190241
==> arm: ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
==> arm: ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORubuntu@packer-test:~/packer-builder-arm$ sudo packer build boards/raspberry-pi/raspbian.json
arm output will be in this color.

==> arm: Retrieving rootfs_archive
==> arm: Trying https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false
==> arm: Trying https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false&checksum=sha256%3A2c4067d59acf891b7aa1683cb1918da78d76d2552c02749148d175fa7f766842
==> arm: https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip?archive=false&checksum=sha256%3A2c4067d59acf891b7aa1683cb1918da78d76d2552c02749148d175fa7f766842 => /home/ubuntu/packer-builder-arm/packer_cache/665ae48e54b8fac4541688233efc4a82b75de3ce.zip
    arm: unpacking /home/ubuntu/packer-builder-arm/packer_cache/665ae48e54b8fac4541688233efc4a82b75de3ce.zip to raspberry-pi.img
    arm: searching for empty loop device (to map raspberry-pi.img)
    arm: mapping image raspberry-pi.img to /dev/loop2
    arm: mounting /dev/loop2p2 to /tmp/495303497
    arm: mounting /dev/loop2p1 to /tmp/495303497/boot
    arm: running extra setup
    arm: mounting /dev with: [mount --bind /dev /tmp/495303497/dev]
    arm: mounting /devpts with: [mount -t devpts /devpts /tmp/495303497/dev/pts]
    arm: mounting proc with: [mount -t proc proc /tmp/495303497/proc]
    arm: mounting binfmt_misc with: [mount -t binfmt_misc binfmt_misc /tmp/495303497/proc/sys/fs/binfmt_misc]
    arm: mounting sysfs with: [mount -t sysfs sysfs /tmp/495303497/sys]
    arm: binfmt setup found at: /proc/sys/fs/binfmt_misc/qemu-arm
    arm: copying qemu binary from /usr/bin/qemu-arm-static to: /tmp/495303497/usr/bin/qemu-arm-static
    arm: running the provision hook
==> arm: Provisioning with shell script: /tmp/packer-shell936190241
==> arm: ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
==> arm: ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
==> arm: ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
==> arm: ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
Build 'arm' finished.

==> Builds finished. The artifacts of successful builds are:
--> arm: raspberry-pi.img
ubuntu@packer-test:~/packer-builder-arm$ 
M}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
==> arm: ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
==> arm: ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
Build 'arm' finished.

==> Builds finished. The artifacts of successful builds are:
--> arm: raspberry-pi.img
ubuntu@packer-test:~/packer-builder-arm$ 

As you can see some errors were reported, specifically:

==> arm: ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.

You can safely ignore this error if you received it. It is due to the $PLATFORM variable not being set in the environment. This does not impact the image build.

I now have a Raspbian image for my Raspberry Pi. Lets check the size using the du command:

ubuntu@packer-test:~/packer-builder-arm$ du -hs raspberry-pi.img 
3.6G    raspberry-pi.img

You can now dd this image to an SD card and boot it on your Raspberry Pi.

Conclusion

To build a Raspberry Pi image with Packer and packer-builder-arm takes a bit of upfront work. However once working it is a very powerful tool. For a next step I suggest adding your customizations to the shell provisioner section to build and image that meets your needs. I plan on using this to automate the image build for my Raspberry Pi PXE boot tutorial.

Learn more about Packer

If you are interested in learning more about Packer we suggest you check out James Turnbull’s book on Packer.

How to install Go on Linux using official binary releases

Installing Go on Linux

In this tutorial we walk through how to install the Go programming language on Linux. We are using the binary distribution you can download from Golang.org. Additionally we talk about why we prefer this method over other methods for installing Go.

GoLang Language - gophers and people on computers

Why install go on Linux via this method?

Linux distributions have their own package management systems. For example apt on Debian variants and yum on Fedora variants. Package managers are powerful tools enabling you to easily install, track and upgrade software you run on your system.

However for faster moving software and tools the packages typically lag behind what is current. For example the current version of Go is 1.13 as of this writing. If I install Go via apt on Ubuntu 18.04 I will get version 1.10. I can use the –dry-run parameter with apt-get install to see what apt will install.

apt-get --dry-run install golang

If I look through the output I see that 1.10 will be installed.

Inst golang-1.10 (1.10.4-2ubuntu1~18.04.1 Ubuntu:18.04/bionic-updates [all])

1.10 is too old for my needs. When I need Go on a system I prefer to install official binaries released by Golang.org. The result is that I can get whatever version I want. Additionally I am using the builds that are supported and tested by the GoLang community.

Alternative methods to install Go on Linux?

Installing the Golang.org binary release is not the only way to get the latest Go. For example there are non-official distributions and packages built by users in the community. You can install a later version Go on Ubuntu via these methods:

When considering these option remember the Go project does not support or test these other distributions. Additionally you are trusting the entities that maintain these builds. If the Go binaries were to get hacked, I am more confident the Golang community will catch it sooner and notify me sooner because so many people are using their distribution. I know I can always get the latest version without hoping a package maintainer is going to keep their builds up to date. Hence I choose to use official Go binaries and do not trust the others options for production use. I think its great that they package up Go in easy to consume packages. But it does not meet my requirements.

The Go Github repo has an Ubuntu focused wiki page with more details about these options. Check it out if you are interested in investigating them further.

Install the official Go binary release

Step 1 Download

The first thing we need to do is download the latest Go binary release from the Go download site.

I will download version 1.13.5 for Linux. In this case I am getting the X86 64bit version. I use the following wget command to fetch the archive file.

wget https://dl.google.com/go/go1.13.5.linux-amd64.tar.gz

Step 2 Extract

Next I need to explode the archive file. I will do this using a tar command which decompresses the file and untars it at the same time.

 tar -xvzf go1.13.5.linux-amd64.tar.gz 

The flags to the tar command in this case mean the following:

  • x – extract. The command to extract files.
  • v – verbose. This gives us detailed about about what the command is doing.
  • z – decompress using gzip. This tells the tar command to decompress the file using gzip.
  • f – file. This tells tar that we are giving it a file path parameters. Versus for example using stdin.

When you run this command you get a bunch of output displaying the files being extracted from the archive. Now I should have a directory called go.

ubuntu@packer-test:~$ ls -l
total 117264
drwxr-xr-x 10 ubuntu ubuntu      4096 Dec  4 22:53 go
-rw-rw-r--  1 ubuntu ubuntu 120074076 Dec  5 01:25 go1.13.5.linux-amd64.tar.gz

Step 3 move the go binaries

Next we will move this go directory into /usr/local. We are doing this so the Go binaries are located in a system wide location. We could also move them into our home directory if our intention to is to only make it available for one user. Lets assume we want to put it in a central location. We must use sudo to move the go directory.

sudo mv go /usr/local

Step 4 Update PATH environment Variable

At this point the installation is done. The last remaining step is to update your PATH environment variable to include the go/bin directory. You can do this on the fly by setting the PATH environment variable from the command like. Do it like so:

PATH=$PATH:/usr/local/go/bin

However this change will only be good for your current session and will not persist. You should update your .bashrc file to include the following line:

PATH=$PATH:/usr/local/go/bin

If you are thinking that line looks like the command we just ran you are correct! We are just leveraging the .bashrc to run the command to set the variable for us. Because the .bashrc file is loaded every time we start a new shell. You can immediately load your new .bashrc file like so:

source ~/.bashrc

Step 5 Validate

Now lets confirm we can run go. Do this by running go help.

ubuntu@packer-test:~$ go help
Go is a tool for managing Go source code.

Usage:

        go <command> [arguments]

The commands are:

        bug         start a bug report
        build       compile packages and dependencies
        clean       remove object files and cached files
        doc         show documentation for package or symbol
        env         print Go environment information
        fix         update packages to use new APIs
        fmt         gofmt (reformat) package sources
        generate    generate Go files by processing source
        get         download and install packages and dependencies
        install     compile and install packages and dependencies
        list        list packages or modules
        mod         module maintenance
        run         compile and run Go program
        test        test packages
        tool        run specified go tool
        version     print Go version
        vet         report likely mistakes in packag

Conclusion

We suggest you consider using official Go binaries when installing Go on your system. By using official binary distributions you ensure you can always install the latest version. Typically newer than what is available via the operating system’s package manager. Additionally you are not relying on untrusted parties maintaining a PPA or similar archive. If you are interested in installing and managing multiple versions of the Go binary release for development purposes, check out GVM. GVM is the Go Version Manager.

PXE Boot, What is PXE? How does it work?

PXE Boot – Introduction

What can you expect to learn about PXE from this post?

  • High level overview of PXE boot process.
  • Use cases for PXE boot.
  • Detailed end to end overview of the PXE boot process.
  • Technical details of each stage.

What is PXE?

In this post we are deep diving into PXE boot. PXE stands for preboot execution environment. It is standards base and can be implemented using open source software or vendor supported products. PXE is a key part of data center infrastructure because it enables automated provisioning of servers or work stations over a network. An in depth understanding of the PXE stack benefits anyone working on infrastructure deployment of bare metal servers, embedded devices and IOT devices.

Authors background

I first implemented a PXE boot environment in a production data center 15 years ago. Installing operating systems from CDROM was painfully slow and we desired an automated solution. The knowledge I gained from that project increased in value through out my career. Since then I have worked with PXE in large scale deployments provisioning thousands and thousands of hosts in data centers across the globe. I am excited to share what I have learned through years of hands on experience.

Why did I write this guide?

PXE is often seems like a dark art. Typically only a handful of people in the team truly know how the environment’s PXE infrastructure boot works. Additionally debugging it is hard, debugging remotely even harder. Therefore, I wrote this guide to help demystify PXE boot by explaining it a simple, thorough and interesting fashion.

High level overview of PXE boot

PXE Use Case, What problem does it solve?

PXE solves a problem large enterprises face. How do you automate provisioning or installation of operating systems on large quantities of machines?

Operating system such as Windows or Linux have mechanisms to automate installation. Typically you create a seed file or configuration. The seed file provides answers to the questions asked by the OS installer. In the linux world examples of this are debian preseed files or Redhat kickstart files. But still you need access to the installation media on CD/DVD-ROM or a USB drive. A human running around with the usb drive touching every server does not scale. Its time consuming and error prone. Lets imagine a world where a human puts a server in the rack, powers it on and is done. This has many benefits:

  • Installers can be less technical.
  • Reduced time spent per server.
  • Less error prone due to automation.
  • OS installation tools are centralized and easier to update.

This is where PXE comes in. PXE is a standards based approached to solving the problem of getting the OS onto the system without a human being putting media (USB, CD/DVD-ROM) in the system. It does this by bootstrapping the machine over the network.

In a fully automated environment the human installing the server does the following:

  • Installs server in the rack.
  • Connects power and network.
  • Walks away.

The powered on server automatically fetches a network boot file (NBF) to boot itself up and provisions an operating system. It is a beautiful thing when its working properly 🙂

How does it work?

It all starts with the NIC

The start of a PXE workflow is booting network interface card (NIC). In a typical PC or laptop the NIC will not do anything until the operating system boots and loads the proper driver. However network booting requires a PXE enabled NIC. The NIC contains firmware with a tiny network stack. This firmware is capable of connecting to the network and fetching a file to boot, commonly referred to as the network boot file (NBF). The file could be a kernel or it could be network enabled boot loader.

The server boots the file downloaded off the network. Typically the boot image kicks off an automated installation of an operating system. Now lets dive into the components that make this process possible.

PXE boot components

A typical PXE environment has the following components.

PXE enabled NICs

Not all NICs are equal. Many consumer grade network cards do not have a PXE capabilities. Although that is rapidly changing as advances make it easier to include more features in cheaper devices. PXE enabled NICs are the defacto standard in data center grade servers. We suggest you double check before you buy. However I would be surprised if any major server manufacturer ships a NIC without PXE capability these days.

Some of the PXE enabled NICs even use open source PXE firmware. IPXE is an open source firmware often installed on data center NICs.

DHCP Server

DHCP stands for Dynamic Host Configuration Protocol. There are two types of actors in DHCP. The DHCP server and the DHCP client.

A DHCP server provides a network configuration to clients. Specifically, DHCP provides an IP network configuration to a client. A DHCP client runs on computers that join the network and need a configuration.

An example of real world DHCP use you are probably familiar with is connecting to your office LAN. Your laptop has no idea what IP addresses are in use on the network it has joined. The DHCP client on your laptop sends a broadcast to the network indicating it is looking for a DHCP server. A response is sent from the the server to announce its availability. Your client acknowledges this by sending a request for a DHCP lease. The DHCP server sees this request and finds an unused IP address. Your laptop gets a DHCP lease offer from the server. The lease offer among other things includes the IP address you will use. Your laptop’s DHCP client accepts the offer and begins using the IP address to talk to on the network. As lease expiration time approaches your laptop will ask to renew.

In a PXE boot environment there is always a DHCP server. The machines that are being provisioned are DHCP clients. The PXE enabled NIC has a DHCP client built into its firmware.

DHCP supports a wide range of options that can be provided to network clients. But typically it consists of an IP address for use by the client, a default gateway address and DNS servers to use for name resolution. In the case of PXE, an option that contains the IP address of the server to download its boot files from.

TFTP Server

TFTP stands for trivial file transfer protocol. TFTP is a simple UDP based protocol for getting or sending a file. It’s simplicity lends well to being implemented in firmware environments where resources are limited. Due to its simple nature TFTP has no bells or whistles. Getting and putting files are supported, that’s it. There is no directory listing, you must know the exact path of the file you want to download. Additionally there is no authentication or authorization.

While TFTP is still commonly used in PXE environments, advances is in technology has resulted in some PXE implementations supporting more complex protocols like HTTP or ISCSI. For example the IPXE firmware supports:

  • HTTP
  • ISCSI Storage Area Networks (SAN)
  • Fiber channel over ethernet (FCOE) Storage Area Networks (SAN)
  • ATA over etherent (AOE)

Putting it all together

This diagram illustrates the PXE boot flow from power on to network boot file download.

The above diagram illustrates a basic PXE workflow. Lets review each of the steps.

PXE DHCP Steps

  • Client PXE enabled NIC powers on and boots firmware.
  • Firmware’s DHCP client sends a broadcast packet to the local area network indicating it needs a network configuration from the DHCP server.
  • The DHCP server responds with what is called an “offer”. The offer contains the network configuration as specified by the DHCP protocol specification.
  • The DHCP client, happy with the result now sends a DHCP request. This request basically means “I got the offer, I want to confirm before moving forward”.
  • The DHCP server then responds with a unicast packet directed at the assigned IP address. Note that up until this point all packets have been broadcast.
  • The DHCP client gets the response and starts using the network configuration.

PXE TFTP Steps

At this point the NIC firmware in the PXE client has an IP configuration. Part of that configuration should have been what is referred to “next-server” option. The next-server option is a DHCP option that tells the client where it should go to download the network boot file.

  • NIC firmware makes a TFTP request to the server using the IP or name specified in the next-server option of the DHCP lease.
  • TFTP server sends the requested file in a udp data stream.
  • NIC firmware receives the file storing it in memory.
  • Server then executes the downloaded file.

Next steps after TFTP

What happens at this point will vary depending on the environment and goal of the PXE boot configuration. Some examples are OS installation or full network boot.

OS Installation Use Case

The system boots up an automated OS installer image that installs an OS to the local drive. After the installation a reboot is performed to reboot into the local OS.

Full Network Boot Use Case

In this use case the server boots entirely over the network on every boot. Typically the root file system is mounted via NFS. Pros of this configuration are the servers can run with no local storage. Cons are that the network needs to be functional to boot the server and performance may not be as good as local storage.

Variations

The PXE environment we just described is a simple and common configuration. It is a good starting point for newcomers trying to understand PXE for the first time.

Here are some variations you will see in the real world. Especially in enterprises.

  • DHCP relay or “helper”. The relay forwards DHCP request to a DHCP server not on the local LAN. This functionality is common on enterprise routers.
  • PXE proxy or relay. This is often used when one does not have the access required to modify the DHCP server configuration. In this case the relay responds to the DHCP request with just the server and filename of the network boot file. Letting the existing DHCP server provide the standard IP configuration.
  • HTTP or HTTPS instead of TFTP for retrieval for the network boot file.


Conclusion

In conclusion PXE is a very powerful tool for automating and managing the provisioning and updates of data center infrastructure, embedded devices, IOT devices and even workstations. We have covered the basics and hope you walk away from this article with a better understanding.

Appendix & further reading

Feedback

We appreciate feedback. If you have ideas on how we can make this article or site better please leave a comment.