One of the biggest benefits of SmartOS is that it does not require installation on a hard drive. It can be booted off CD, USB  device and over the network with PXE boot.

In this article we are going to cover setting up a PXE Boot server that will boot our SmartOS Nodes over the network. Using PXE boot, there is no need for CD or USB boot media, and as long as the Node supports boot from LAN it will boot from our PXE boot server. This means if a new version of SmartOS is released, we just change the version that our PXE server distributes. Once changed all that is needed is to reboot the nodes and all of them come back up running latest version. PXE boot means you do not have to worry about USB or CDROM devices failing. The only potential downside is that should the PXE Server fail prior to your SmartOS nodes booting, you could land up with a single point of failure. In the unlikely event this ever happens you could have USB boot media available as a backup option.

In a production environment, I suggest you ensure that each SmartOS node has at least 2 network interfaces. Each NIC should be plugged into a separate Switch. NIC 1 (admin nic) is plugged into the “admin” switch and NIC 2 (external nic) is plugged into the “external” switch. We will only provide DHCP and PXE Boot on the “admin” Network.

This setup uses FreeBSD as our Host operating system, but you could of course set this up on the Operating System of your choice. You do not strictly need to run all the below services on the same server,  but for my environments this is the most practical setup.

Our PXE head node will be setup with the following software:

  1. FreeBSD 9.0
  2. ISC DHCP Server – Serving as a dhcp server for our “admin” network.
  3. TFTP Server (built into FreeBSD) – To send PXE clients files.
  4. gpxelinux.0 – from the SYSLINUX project will be used as a PXE boot loader.
  5. Apache Web Server (optional) – much faster than TFTP to distribute large file or images over network.

The end result will be a boot menu as seen below.

Lets get started.

This guide assumes that you have a new install of FreeBSD 9 which has been setup with a static ip address. In this setup our PXE Server will have the ip address of 10.1.1.233

To start off lets fetch the Freebsd Ports tree.

portsnap fetch extract

DHCP Server Configuration

cd /usr/ports/net/isc-dhcp41-server && make install clean
nano /usr/local/etc/dhcpd.conf

“dhcpd.conf”

allow booting;
allow bootp;
ddns-update-style none;
log-facility local7;
default-lease-time -1;
max-lease-time 7200;
authoritative;
option space gpxe;
option gpxe-encap-opts code 175 = encapsulate gpxe;
option gpxe.bus-id code 177 = string;

subnet 10.1.1.0 netmask 255.255.255.0 {
use-host-decl-names on;
range 10.1.1.45 10.1.1.80;
option routers 10.1.1.1;
option broadcast-address 10.1.1.255;
option domain-name-servers 10.1.1.1, 8.8.8.8;
option root-path "10.1.1.233:/usr/tftpboot/";
next-server 10.1.1.233;
filename "gpxelinux.0";
}
echo 'dhcpd_enable="YES"' >> /etc/rc.conf
/usr/local/etc/rc.d/isc-dhcpd start

TFTP Server Configuration

nano /etc/inetd.conf

Add the following line to inetd.conf to enable FreeBSD’s built in TFTP server.

tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -p -s /usr/tftpboot

Now lets create the directories we need. The TFTP “images” directory will hold all of our images or “live” filesystem trees. Since the images can be quite large we will place it under /usr. We will also create a subdirectory pxelinux.cfg which will contain our boot menu.

mkdir -p /usr/tftpboot/images
mkdir /usr/tftpboot/pxelinux.cfg
echo 'inetd_enable="YES"' >> /etc/rc.conf
/etc/rc.d/inetd start

PXE Boot Loader Installation

For our bootloader we will be using gpxelinux.0 from the SYSLINUX Project. Download the latest version

cd /tmp
fetch http://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-4.05.tar.gz
tar -xvzf syslinux-4.05.tar.gz
cp /tmp/syslinux-4.05/memdisk/memdisk /usr/tftpboot/
cp /tmp/syslinux-4.05/gpxe/gpxelinux.0 /usr/tftpboot/
cp /tmp/syslinux-4.05/com32/modules/reboot.c32 /usr/tftpboot/
cp /tmp/syslinux-4.05/com32/modules/chain.c32 /usr/tftpboot/
cp /tmp/syslinux-4.05/com32/menu/menu.c32 /usr/tftpboot/
cp /tmp/syslinux-4.05/com32/menu/vesamenu.c32 /usr/tftpboot/

SmartOS Download & Apache install

Lets download and put our SmartOS USB Image into a new subfolder in /usr/tftpboot/images/ We will then install and configure Apache to serve our image. Although we don’t have to use Apache and could do this with the TFTP server alone, it is significantly faster and more convenient to serve this up over http.

mkdir /usr/tftpboot/images/smartos
cd /usr/tftpboot/images/smartos
fetch https://download.joyent.com/pub/iso/latest-USB.img.bz2
bunzip2 latest-USB.img.bz2
gzip latest-USB.img

Please be aware that in order to pxe-boot these images we need to extract them and then gzip them as shown above or they will not work via pxe-boot.

Lets create a menu that will be displayed when a client PXE boot’s over the network. Below you will see my sample menu that contains other bootable options. I have made the default option at the top of the list “SmartOS”. SmartOS has been configured to autoboot after 10 seconds.  It has also been configured that should someone interrupt the autoboot process by moving to another menu item, it will wait a further 90 seconds then carry on booting “SmartOS”. I have configured my menu to boot with vesamenu.c32 with a background graphic image. If a pretty boot menu does not do it for you then change “vesamenu.c32” to “menu.c32“. You will see from the sample menu that I have included various other bootable options. The configuration of the additional menu options is beyond the scope of this tutorial. I do however recommend including the “memtest” option that can be used to burn in or test the memory of a new SmartOS node before deployment.

nano /usr/tftpboot/pxelinux.cfg/default
PROMPT 0
ui vesamenu.c32
menu title SmartOS PXE Boot
menu background splash.png
MENU AUTOBOOT Starting SmartOS in # seconds
timeout 100
TOTALTIMEOUT 900

label smartos
menu label SmartOS
kernel memdisk
initrd http://10.1.1.233/images/smartos/latest-USB.img.gz
append raw

label systemrescue
menu label System Rescue
kernel http://10.1.1.233/images/systemrescue/rescuecd
append setkmap=us netboot=http://10.1.1.233/images/systemrescue/sysrcd.dat
initrd http://10.1.1.233/images/systemrescue/initram.igz

label clonezilla
menu label Clonezilla
KERNEL http://10.1.1.233/images/clonezilla/vmlinuz
append initrd=http://10.1.1.233/images/clonezilla/initrd.img boot=live live-config noswap nolocales edd=on nomodeset ocs_live_run="ocs-live-general" ocs_live_extra_param="" ocs_live_keymap="" ocs_live_batch="no" ocs_lang=""$ vga=788 nosplash fetch=http://10.1.1.233/images/clonezilla/filesystem.squashfs

label mfsbsd
menu label mfsBSD
kernel memdisk
initrd http://10.1.1.233/images/mfsbsd/mfsbsd-9.0-i386.iso
append iso raw

label memtest
menu label Memtest
kernel http://10.1.1.233/images/memtest/memtestp

Lets install and configure apache to serve content from /usr/tftpboot/images/

pkg_add -r apache22
echo 'apache22_enable="YES"' >> /etc/rc.conf
nano /usr/local/etc/apache22/httpd.conf

Edit httpd.conf

“httpd.conf”

ServerName 10.1.1.223:80

<Directory "/images">
Order Deny,Allow
Deny from all
Allow from 10.1.1.0/24
Options Indexes
</Directory>

Lets start Apache and create a symbolic link to our “images” directory.

apachectl configtest
apachectl start
ln -s /usr/tftpboot/images/ /usr/local/www/apache22/data/

Thats it! we are done, you should be able to browse to your Images directory from a web browser and see you data.

All that is left to do is boot up a machine on your network that supports PXE boot from LAN and test that its working. The below screenshot shows a SmartOS machine on my network booting off PXE and loading the SmartOS image from the PXE server. This whole PXE boot process takes about 20-30 seconds.

UPDATE: Please note that Joyent has now changed the format of their SmartOS Compressed Images from “img.gz” to “img.bz2” in order to boot these images you must download them extract them and re-compress them as “img.gz” before uploading them to your PXE Boot server or else they will not work. I have updated this article to reflect this.

UPDATE 1: Using the method in this article passes the entire iso image across to the SmartOS node, this works but the downside is that it will consume about an extra 2GB of memory on the SmartOS server. The better way of doing this is to extract the platform release into your tftp directory. This method is better as it does not permanently use up the ram. I will be posting an update to this article demonstrating how to do this.

⋅ SmartOS and friends ⋅ , , ⋅ 10 comments