In a previous blog post we covered booting a full iso image using gPXE for etherboot. I was asked by some blog readers why, since development of gPXE stopped in 2010, did I not rather use the recommended iPXE boot method.

Back then, the reason for my choice was that getting iPXE to work with pfSense’s built in Dnsmasq server via chainloading was non trivial and not possible without hacking the configuration files (ymmv). Most iPXE instructions out there talk about using ISC DHCP with a “(if) or “(else)” chainloading configuration statement. Which ensures that once iPXE has booted and it send a second DHCP request, it gets served your custom iPXE script rather than “undionly.kpxe”. Without this in place, it will go into and endless DHCP request loop.

I have since figured out the trick to getting this to work with pfSense’s Dnsmasq is to recompile a custom version of “undionly.kpxe” which includes your embedded script. This script tells iPXE to load your iPXE menu from specific URL. In this article, we will cover booting the SmartOS nodes using a method that is much more ram conservative.

As most of the sites I work on are behind dedicated pfSense perimeter devices, it makes sense for me to use them as the DHCP server for the “admin” network that the SmartOS nodes are connected to.

For the purposes of today article, I will be using a mini “fit2PC” computer with FreeBSD 9.1 installed on it. It uses minimal power and has a 60GB Intel SSD in it. Below is a pic of it on my desk at home with dramatic effect to emphasise its size. You could however very easily install all of this on your pfSense server itself or on another OS such as OmniOS.



TFTP Server Setup

Lets add the following line to inetd.conf to enable FreeBSD’s built in TFTP server.

Filename : inetd.conf

tftp dgram udp wait root /usr/libexec/tftpd tftpd -p -s /usr/tftpboot

Next we create the directory and enable the TFTP service.

mkdir /usr/tftpboot
echo 'inetd_enable="YES"' >> /etc/rc.conf
/etc/rc.d/inetd start

iPXE Configuration

The setup of iPXE is incredibly simple. You can either download the source and compile it yourself with your custom script included. Alternatively you could use the super easy method using a very handy web service to compile it in for you. You simply go to http://rom-o-matic.eu paste in your custom script, click “proceed” and a few seconds later, your custom “undionly.kpxe” file is ready to download. Save the file into the /usr/tftpboot directory.




pfSense DHCP Setup

The next step is to tell our DHCP server that when it hands out DHCP leases to a PXE boot enabled network card, it must send our tftp server details as well as the name of the image to load.

Login to the pfsense web interface and under the “Services” section click “DHCP Server”. Select the network interface you want to run the service on and fill in the ip address of your tftp server. Apply your settings as per the below screenshot.

pfSense Screenshot


Lets review the process we have setup thus far. A computer on the network is switched on. If the network card is set to PXE boot it will request a DHCP lease. The pfSense server which is running the DHCP service will hand out an ip address to the server. In addition the server will be given the ip of our tftp server ( and the name of the file to boot “undionly.kpxe”. The computer on the network will load the undionly.kpxe PXE firmware, and once its loaded it will try and connect via http and launch the script we specified which is called “bootstrap.ipxe”.


Web Server Configuration

All that is left to do is put the files onto a webserver so that they can be served via http. iPXE supports http which has many benefits, such as allowing you to PXE boot over the Internet from remote web servers as well as much better speeds. Unlike serving over tftp, its very easy to scale web servers and make them really fast. This guide assumes you know how to setup and copy files to a web server. In essence you could use any web server as long as its reachable via http. In this setup, we have the web server (nginx) and the tftp server on the same physical box. Our nginx document root is /usr/local/www/nginx-dist/

Lets grab the SmartOS images and our iPXE boot menu script and put them into a directory that is being served by our web server. We create a sub directory for each individual SmartOS platform version we want available.
Important: We have to rename each SmartOS version directory to “platform” or else the pxe boot will not work.

mkdir /usr/local/www/nginx-dist/smartos/
mkdir /usr/local/www/nginx-dist/smartos/20130808T195337Z
cd /usr/local/www/nginx-dist/smartos/20130808T195337Z
fetch https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/20130808T195337Z/platform-20130808T195337Z.tgz
tar -xvzf platform-20130808T195337Z.tgz
mv platform-20130808T195337Z platform

Next we create the file bootstrap.ipxe which contains our custom iPXE boot menu. The code snippet below are the settings that I use.

mkdir /usr/local/www/nginx-dist/ipxe
vi /usr/local/www/nginx-dist/ipxe/bootstrap.ipxe

Filename : bootstrap.ipxe


set remote-root
set smartos-build 20130808T195337Z
set smartos-build2 20130405T010449Z
menu Welcome to iPXE's Boot Menu
item --gap -- ------------------------- SmartOS option 1 ------------------------------
item smartos            Boot SmartOS (${smartos-build})
item smartos_noimport   Boot SmartOS (${smartos-build}) noimport
item --gap -- -------------------------------------------------------------------------
item --gap -- ------------------------- SmartOS option 2 ------------------------------
item smartos2            Boot SmartOS (${smartos-build2})
item smartos_noimport2   Boot SmartOS (${smartos-build2}) noimport
item --gap -- ------------------------------ Utilities ---------------------------------
item memtest    Memtest86+ 5.01
item shell      Enter iPXE shell
item reboot     Reboot
item exit       Exit (boot local disk)

choose --default smartos --timeout 30000 target &&; goto ${target}

echo Type exit to get the back to the menu
set menu-timeout 0
goto start



kernel ${remote-root}/smartos/${smartos-build}/platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$rbGKkqrW$c3rJPrkCPAc.Idnw0Tv0Grc3A6oHJuL131C2H4ykuXA',smartos=true
module ${remote-root}/smartos/${smartos-build}/platform/i86pc/amd64/boot_archive
boot || goto start

kernel ${remote-root}/smartos/${smartos-build}/platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$rbGKkqrW$c3rLPrkPCNc.Idnw0Tv0Grc3A6oHJuL131C2H4ykuXA',smartos=true,noimport=true
module ${remote-root}/smartos/${smartos-build}/platform/i86pc/amd64/boot_archive
boot || goto start

kernel ${remote-root}/smartos/${smartos-build2}/platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$rbGKkqrW$c3rLPrkPCNc.Idnw0Tv0Grc3A6oHJuL131C2H4ykuXA',smartos=true
module ${remote-root}/smartos/${smartos-build2}/platform/i86pc/amd64/boot_archive
boot || goto start

kernel ${remote-root}/smartos/${smartos-build2}/platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$rbGKkqrW$c3rLPrkPCNc.Idnw0Tv0Grc3A6oHJuL131C2H4ykuXA',smartos=true,noimport=true
module ${remote-root}/smartos/${smartos-build2}/platform/i86pc/amd64/boot_archive
boot || goto start

kernel ${remote-root}/memtest86/memtest86-5.01
boot || goto start

PXE Boot in Action

Thats it!, we are done!, lets power on the computer and watch the whole process unfold.

Stage 1

Stage 2

Stage 3

Stage 4