In a previous blog post we covered booting a full iso image using gPXE for etherboot. I was asked by some blog readers why, since development of gPXE stopped in 2010, did I not rather use the recommended iPXE boot method.
Back then, the reason for my choice was that getting iPXE to work with pfSense’s built in Dnsmasq server via chainloading was non trivial and not possible without hacking the configuration files (ymmv). Most iPXE instructions out there talk about using ISC DHCP with a “(if) or “(else)” chainloading configuration statement. Which ensures that once iPXE has booted and it send a second DHCP request, it gets served your custom iPXE script rather than “undionly.kpxe”. Without this in place, it will go into and endless DHCP request loop.
I have since figured out the trick to getting this to work with pfSense’s Dnsmasq is to recompile a custom version of “undionly.kpxe” which includes your embedded script. This script tells iPXE to load your iPXE menu from specific URL. In this article, we will cover booting the SmartOS nodes using a method that is much more ram conservative.
As most of the sites I work on are behind dedicated pfSense perimeter devices, it makes sense for me to use them as the DHCP server for the “admin” network that the SmartOS nodes are connected to.
For the purposes of today article, I will be using a mini “fit2PC” computer with FreeBSD 9.1 installed on it. It uses minimal power and has a 60GB Intel SSD in it. Below is a pic of it on my desk at home with dramatic effect to emphasise its size. You could however very easily install all of this on your pfSense server itself or on another OS such as OmniOS.
fit2PC
TFTP Server Setup
Lets add the following line to inetd.conf to enable FreeBSD’s built in TFTP server.
Filename : inetd.conf
tftp dgram udp wait root /usr/libexec/tftpd tftpd -p -s /usr/tftpboot
Next we create the directory and enable the TFTP service.
mkdir /usr/tftpboot echo 'inetd_enable="YES"' >> /etc/rc.conf /etc/rc.d/inetd start
iPXE Configuration
The setup of iPXE is incredibly simple. You can either download the source and compile it yourself with your custom script included. Alternatively you could use the super easy method using a very handy web service to compile it in for you. You simply go to http://rom-o-matic.eu paste in your custom script, click “proceed” and a few seconds later, your custom “undionly.kpxe” file is ready to download. Save the file into the /usr/tftpboot directory.
Script
#!ipxe dhcp chain http://10.1.1.7/ipxe/bootstrap.ipxe
pfSense DHCP Setup
The next step is to tell our DHCP server that when it hands out DHCP leases to a PXE boot enabled network card, it must send our tftp server details as well as the name of the image to load.
Login to the pfsense web interface and under the “Services” section click “DHCP Server”. Select the network interface you want to run the service on and fill in the ip address of your tftp server. Apply your settings as per the below screenshot.
pfSense Screenshot
Lets review the process we have setup thus far. A computer on the network is switched on. If the network card is set to PXE boot it will request a DHCP lease. The pfSense server which is running the DHCP service will hand out an ip address to the server. In addition the server will be given the ip of our tftp server (10.1.1.7) and the name of the file to boot “undionly.kpxe”. The computer on the network will load the undionly.kpxe PXE firmware, and once its loaded it will try and connect via http and launch the script we specified which is called “bootstrap.ipxe”.
Web Server Configuration
All that is left to do is put the files onto a webserver so that they can be served via http. iPXE supports http which has many benefits, such as allowing you to PXE boot over the Internet from remote web servers as well as much better speeds. Unlike serving over tftp, its very easy to scale web servers and make them really fast. This guide assumes you know how to setup and copy files to a web server. In essence you could use any web server as long as its reachable via http. In this setup, we have the web server (nginx) and the tftp server on the same physical box. Our nginx document root is /usr/local/www/nginx-dist/
Lets grab the SmartOS images and our iPXE boot menu script and put them into a directory that is being served by our web server. We create a sub directory for each individual SmartOS platform version we want available.
Important: We have to rename each SmartOS version directory to “platform” or else the pxe boot will not work.
mkdir /usr/local/www/nginx-dist/smartos/ mkdir /usr/local/www/nginx-dist/smartos/20130808T195337Z cd /usr/local/www/nginx-dist/smartos/20130808T195337Z fetch https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/20130808T195337Z/platform-20130808T195337Z.tgz tar -xvzf platform-20130808T195337Z.tgz mv platform-20130808T195337Z platform
Next we create the file bootstrap.ipxe which contains our custom iPXE boot menu. The code snippet below are the settings that I use.
mkdir /usr/local/www/nginx-dist/ipxe vi /usr/local/www/nginx-dist/ipxe/bootstrap.ipxe
Filename : bootstrap.ipxe
#!ipxe
set remote-root http://10.1.1.7
set smartos-build 20130808T195337Z
set smartos-build2 20130405T010449Z
dhcp
:start
menu Welcome to iPXE's Boot Menu
item
item --gap -- ------------------------- SmartOS option 1 ------------------------------
item smartos Boot SmartOS (${smartos-build})
item smartos_noimport Boot SmartOS (${smartos-build}) noimport
item --gap -- -------------------------------------------------------------------------
item
item --gap -- ------------------------- SmartOS option 2 ------------------------------
item smartos2 Boot SmartOS (${smartos-build2})
item smartos_noimport2 Boot SmartOS (${smartos-build2}) noimport
item --gap -- ------------------------------ Utilities ---------------------------------
item memtest Memtest86+ 5.01
item shell Enter iPXE shell
item reboot Reboot
item
item exit Exit (boot local disk)
choose --default smartos --timeout 30000 target &&; goto ${target}
:shell
echo Type exit to get the back to the menu
shell
set menu-timeout 0
goto start
:reboot
reboot
:exit
exit
:smartos
kernel ${remote-root}/smartos/${smartos-build}/platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$rbGKkqrW$c3rJPrkCPAc.Idnw0Tv0Grc3A6oHJuL131C2H4ykuXA',smartos=true
module ${remote-root}/smartos/${smartos-build}/platform/i86pc/amd64/boot_archive
boot || goto start
:smartos_noimport
kernel ${remote-root}/smartos/${smartos-build}/platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$rbGKkqrW$c3rLPrkPCNc.Idnw0Tv0Grc3A6oHJuL131C2H4ykuXA',smartos=true,noimport=true
module ${remote-root}/smartos/${smartos-build}/platform/i86pc/amd64/boot_archive
boot || goto start
:smartos2
kernel ${remote-root}/smartos/${smartos-build2}/platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$rbGKkqrW$c3rLPrkPCNc.Idnw0Tv0Grc3A6oHJuL131C2H4ykuXA',smartos=true
module ${remote-root}/smartos/${smartos-build2}/platform/i86pc/amd64/boot_archive
boot || goto start
:smartos_noimport2
kernel ${remote-root}/smartos/${smartos-build2}/platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$rbGKkqrW$c3rLPrkPCNc.Idnw0Tv0Grc3A6oHJuL131C2H4ykuXA',smartos=true,noimport=true
module ${remote-root}/smartos/${smartos-build2}/platform/i86pc/amd64/boot_archive
boot || goto start
:memtest
kernel ${remote-root}/memtest86/memtest86-5.01
boot || goto start
PXE Boot in Action
Thats it!, we are done!, lets power on the computer and watch the whole process unfold.
Stage 1
Stage 2
Stage 3
Stage 4
