One of the biggest benefits of SmartOS is that it does not require installation on a hard drive. It can be booted off CD, USB device and over the network with PXE boot.
In this article we are going to cover setting up a PXE Boot server that will boot our SmartOS Nodes over the network. Using PXE boot, there is no need for CD or USB boot media, and as long as the Node supports boot from LAN it will boot from our PXE boot server. This means if a new version of SmartOS is released, we just change the version that our PXE server distributes. Once changed all that is needed is to reboot the nodes and all of them come back up running latest version. PXE boot means you do not have to worry about USB or CDROM devices failing. The only potential downside is that should the PXE Server fail prior to your SmartOS nodes booting, you could land up with a single point of failure. In the unlikely event this ever happens you could have USB boot media available as a backup option.
In a production environment, I suggest you ensure that each SmartOS node has at least 2 network interfaces. Each NIC should be plugged into a separate Switch. NIC 1 (admin nic) is plugged into the “admin” switch and NIC 2 (external nic) is plugged into the “external” switch. We will only provide DHCP and PXE Boot on the “admin” Network.
This setup uses FreeBSD as our Host operating system, but you could of course set this up on the Operating System of your choice. You do not strictly need to run all the below services on the same server, but for my environments this is the most practical setup.
Our PXE head node will be setup with the following software:
- FreeBSD 9.0
- ISC DHCP Server – Serving as a dhcp server for our “admin” network.
- TFTP Server (built into FreeBSD) – To send PXE clients files.
- gpxelinux.0 – from the SYSLINUX project will be used as a PXE boot loader.
- Apache Web Server (optional) – much faster than TFTP to distribute large file or images over network.
The end result will be a boot menu as seen below.
Lets get started.
This guide assumes that you have a new install of FreeBSD 9 which has been setup with a static ip address. In this setup our PXE Server will have the ip address of 10.1.1.233
To start off lets fetch the Freebsd Ports tree.
portsnap fetch extract
DHCP Server Configuration
cd /usr/ports/net/isc-dhcp41-server && make install clean nano /usr/local/etc/dhcpd.conf
“dhcpd.conf”
allow booting; allow bootp; ddns-update-style none; log-facility local7; default-lease-time -1; max-lease-time 7200; authoritative; option space gpxe; option gpxe-encap-opts code 175 = encapsulate gpxe; option gpxe.bus-id code 177 = string; subnet 10.1.1.0 netmask 255.255.255.0 { use-host-decl-names on; range 10.1.1.45 10.1.1.80; option routers 10.1.1.1; option broadcast-address 10.1.1.255; option domain-name-servers 10.1.1.1, 8.8.8.8; option root-path "10.1.1.233:/usr/tftpboot/"; next-server 10.1.1.233; filename "gpxelinux.0"; }
echo 'dhcpd_enable="YES"' >> /etc/rc.conf /usr/local/etc/rc.d/isc-dhcpd start
TFTP Server Configuration
nano /etc/inetd.conf
Add the following line to inetd.conf to enable FreeBSD’s built in TFTP server.
tftp dgram udp wait root /usr/libexec/tftpd tftpd -p -s /usr/tftpboot
Now lets create the directories we need. The TFTP “images” directory will hold all of our images or “live” filesystem trees. Since the images can be quite large we will place it under /usr. We will also create a subdirectory pxelinux.cfg which will contain our boot menu.
mkdir -p /usr/tftpboot/images mkdir /usr/tftpboot/pxelinux.cfg echo 'inetd_enable="YES"' >> /etc/rc.conf /etc/rc.d/inetd start
PXE Boot Loader Installation
For our bootloader we will be using gpxelinux.0 from the SYSLINUX Project. Download the latest version
cd /tmp fetch http://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-4.05.tar.gz tar -xvzf syslinux-4.05.tar.gz cp /tmp/syslinux-4.05/memdisk/memdisk /usr/tftpboot/ cp /tmp/syslinux-4.05/gpxe/gpxelinux.0 /usr/tftpboot/ cp /tmp/syslinux-4.05/com32/modules/reboot.c32 /usr/tftpboot/ cp /tmp/syslinux-4.05/com32/modules/chain.c32 /usr/tftpboot/ cp /tmp/syslinux-4.05/com32/menu/menu.c32 /usr/tftpboot/ cp /tmp/syslinux-4.05/com32/menu/vesamenu.c32 /usr/tftpboot/
SmartOS Download & Apache install
Lets download and put our SmartOS USB Image into a new subfolder in /usr/tftpboot/images/ We will then install and configure Apache to serve our image. Although we don’t have to use Apache and could do this with the TFTP server alone, it is significantly faster and more convenient to serve this up over http.
mkdir /usr/tftpboot/images/smartos cd /usr/tftpboot/images/smartos fetch https://download.joyent.com/pub/iso/latest-USB.img.bz2 bunzip2 latest-USB.img.bz2 gzip latest-USB.img
Please be aware that in order to pxe-boot these images we need to extract them and then gzip them as shown above or they will not work via pxe-boot.
Lets create a menu that will be displayed when a client PXE boot’s over the network. Below you will see my sample menu that contains other bootable options. I have made the default option at the top of the list “SmartOS”. SmartOS has been configured to autoboot after 10 seconds. It has also been configured that should someone interrupt the autoboot process by moving to another menu item, it will wait a further 90 seconds then carry on booting “SmartOS”. I have configured my menu to boot with vesamenu.c32 with a background graphic image. If a pretty boot menu does not do it for you then change “vesamenu.c32” to “menu.c32“. You will see from the sample menu that I have included various other bootable options. The configuration of the additional menu options is beyond the scope of this tutorial. I do however recommend including the “memtest” option that can be used to burn in or test the memory of a new SmartOS node before deployment.
nano /usr/tftpboot/pxelinux.cfg/default
PROMPT 0 ui vesamenu.c32 menu title SmartOS PXE Boot menu background splash.png MENU AUTOBOOT Starting SmartOS in # seconds timeout 100 TOTALTIMEOUT 900 label smartos menu label SmartOS kernel memdisk initrd http://10.1.1.233/images/smartos/latest-USB.img.gz append raw label systemrescue menu label System Rescue kernel http://10.1.1.233/images/systemrescue/rescuecd append setkmap=us netboot=http://10.1.1.233/images/systemrescue/sysrcd.dat initrd http://10.1.1.233/images/systemrescue/initram.igz label clonezilla menu label Clonezilla KERNEL http://10.1.1.233/images/clonezilla/vmlinuz append initrd=http://10.1.1.233/images/clonezilla/initrd.img boot=live live-config noswap nolocales edd=on nomodeset ocs_live_run="ocs-live-general" ocs_live_extra_param="" ocs_live_keymap="" ocs_live_batch="no" ocs_lang=""$ vga=788 nosplash fetch=http://10.1.1.233/images/clonezilla/filesystem.squashfs label mfsbsd menu label mfsBSD kernel memdisk initrd http://10.1.1.233/images/mfsbsd/mfsbsd-9.0-i386.iso append iso raw label memtest menu label Memtest kernel http://10.1.1.233/images/memtest/memtestp
Lets install and configure apache to serve content from /usr/tftpboot/images/
pkg_add -r apache22 echo 'apache22_enable="YES"' >> /etc/rc.conf nano /usr/local/etc/apache22/httpd.conf
Edit httpd.conf
“httpd.conf”
ServerName 10.1.1.223:80 <Directory "/images"> Order Deny,Allow Deny from all Allow from 10.1.1.0/24 Options Indexes </Directory>
Lets start Apache and create a symbolic link to our “images” directory.
apachectl configtest apachectl start ln -s /usr/tftpboot/images/ /usr/local/www/apache22/data/
Thats it! we are done, you should be able to browse to your Images directory from a web browser and see you data.
All that is left to do is boot up a machine on your network that supports PXE boot from LAN and test that its working. The below screenshot shows a SmartOS machine on my network booting off PXE and loading the SmartOS image from the PXE server. This whole PXE boot process takes about 20-30 seconds.
UPDATE: Please note that Joyent has now changed the format of their SmartOS Compressed Images from “img.gz” to “img.bz2” in order to boot these images you must download them extract them and re-compress them as “img.gz” before uploading them to your PXE Boot server or else they will not work. I have updated this article to reflect this.
UPDATE 1: Using the method in this article passes the entire iso image across to the SmartOS node, this works but the downside is that it will consume about an extra 2GB of memory on the SmartOS server. The better way of doing this is to extract the platform release into your tftp directory. This method is better as it does not permanently use up the ram. I will be posting an update to this article demonstrating how to do this.
Jun 16, 2012 @ 15:30:29
Thanks for the great tutorial. I managed to use a Mac OS X machine providing PXE boot services.
First of all: be careful to copy text from this web page into your text editor (e.g. pxelinux.cfg/default). I had a couple of strange characters in the file which caused some error messages on the client that scrolled by so fast that I had to use my iPhone to capture a video and look at it frame by frame.
However I ran into a problem where the booting machine complained:
MEMDISK: Image has fractional end sector
MEMDISK: Image seems to have fractional end cylinder
And then after the following line nothing more happened an the machine got stuck:
Loading boot sector… booting…
After some fiddling around I found the problem: SmartOS is now compressed using bzip2 and memdisk seems not to support this. So after re-compressing with gzip and fixing pxelinux.cfg/default accordingly the client machine booted successfully.
Thanks again for the tutorial!
Oct 22, 2012 @ 17:13:00
Hi!
I just stumbled upon your tutorial and was wondering why you’ve chosen this setup instead of http boot with iPXE. Any reason or just because everything else was in place?
Thx
Jochen
SmartOS iPXE boot with pfSense | blog.SmartCore.net.au
Oct 07, 2013 @ 08:30:14
[…] a previous blog post we covered booting a full iso image using gPXE for etherboot. I was asked by some blog […]
Nov 14, 2013 @ 15:08:00
I tried doing a PXE boot of an install process of Linux for ages now (with using pfsense as my router) but never annoyingly being able to solve an error. Being the non defeatist person that I am tried many times to complete it. When I was reading this post I strongly think I know what I have been missing, actually it’s pretty simple to be honest don’t know why I didn’t see it before! GREAT POST!
pxeサーバ構築
Oct 01, 2014 @ 12:27:18
[…] http://blog.smartcore.net.au/smartos-network-boot-with-pxe/ […]
Jun 24, 2015 @ 23:28:20
Hi,
I have PXE server with SmartOS platform-20130629T040542Z that runs on my production environment and I want to update it to latest version release. I just wonder if I update it then reboot the server, is the server will initialize again? How about the VM’s that installed there? Is it going to run?
I’m setting up my test environment now and just curious if you can give me a hint.
Thanks in advance.
Jun 25, 2015 @ 13:27:31
Hey there Joven,
You are running quite an old platform – there is always possible issues when going from a very old version to the latest. This is despite the so-called backwards compatibility that is meant to be inherent with Platform Image releases. I would suggest caution, boot one hypervisor off the latest image and confirm all is well and there are no issues. If there are problems you could simply boot back to the earlier PI version. The one thing you may run into with such an old PI is needing to recreate the “dump device” if you run into this issue it will manifest by showing your ZPOOL as degraded. If you run into that problem, give me a holler and I have got the instructions on how to fix it.
Best of luck
Jul 30, 2015 @ 09:57:36
Hi Mark,
Thanks for your reply 🙂
Sorry for very late reply. Very busy doing some stuff on production. I already tried on my test environment, but haven’t tried to perform it to one of our production servers.
In my test environment, no problem encountered and it run smoothly. VM’s are working fine.
Let me try to my one production server and let you know. If I’m going to encounter any problem and find (or cannot find) solutions, I’ll let you know.
Thanks 🙂
Jul 30, 2015 @ 10:16:22
Good luck!
Ps ” This article is a bit dated the way I run, and recommend others run PXEboot now is via iPXE as per this post http://blog.smartcore.net.au/smartos-ipxe-boot-with-pfsense/
Jul 30, 2015 @ 12:08:57
Yes. I’m using iPXE to boot smartos.
Sorry didn’t mention it before.
Thanks.