SmartOS – My journey

Today I want to talk about SmartOS.

Before we begin, some background. For the past couple of years I have been exclusively using ZFS as my filesystem of choice primarily for data storage and as virtual machine datastores via NFS network shares for Vmware ESX. I have tested and used  mainly  Solaris based operating systems for the rock solid and reliable ZFS filesystem. I have used and in certain cases am still using OpenSolaris, Nexentastor, Freebsd, Freenas and more recently the superb fully open source OpenIndiana which has the Illumos kernel and is a replacement for openSolaris which was essentially killed when Oracle bought Sun.

My Virtual machine use case have always included either ESX or Linux based KVM nodes connecting over the network to  dedicated solaris based ZFS storage nodes. The benefits of this setup means I could leverage the snapshot’ing ability of ZFS and store the Virtual Machines on a file system that ensures end to end data integrity as well as benefit from the almost zero overhead of ZFS compression. This means with almost no storage overhead I could keep unlimited snapshots of any running virtual machine and restore to any point in time or clone a new virtual machine instance in seconds. I would create individual ZFS datasets and share them via NFS over the network and then mount them individually from ESX for VM datastores. This would allow me to have fine grained control of individual Virtual machines in regards to snapshots,  quota’s, cloning and compression. From  my perspective, one of  biggest benefit of this setup was the ability to backup running VM’s on the fly without effecting the performance of the VM or needing any dedicated backup software. I could then ZFS send the VM snapshots to remote ZFS storage servers for disaster recovery readiness.

“This was some of the many reasons I was in love with ZFS!”

However that being said, it was not without issues or complexity and I was always looking for some way to combine the virtualization and storage nodes into one server.  The main issue was troubleshooting network IO bottlenecks and the danger of brief interruptions to the network switch fabric, this would bring down all the running vm’s as the ESX nodes could not communicate with the storage node. I have had this happen a couple of times, and trust me its not fun when all your VM’s simultaneously die and have to be restarted.

Unfortunately Solaris only supported Zone / container based  OS virtualization where virtual machines would share a common Kernel and this would not work as an ESX replacement for scenarios where full hardware based virtual machines like Windows, Linux or Freebsd VM’s were needed.

“The good news!”

In August 2011 Joyent successfully ported Linux KVM to Illumos and their Open Source cloud operating system called SmartOS. Joyent had been using Smartos as the foundation for their Public Cloud  & “Smart Data Centre” product for a long time, and it has been battle tested and production proven. Porting KVM was no small feat and a major engineering challenge. Kudos to them for doing this and Open Sourcing all there efforts in the interest of sharing. Many of the highly talented Sun engineers responsible for the innovations that came out of the Sun era like ZFS, DTRACE, Crossbow now work for Joyent. According to Joyent’s measurements, the illumos KVM port performs as well as Linux KVM, at bare-metal speeds for entirely CPU-bound workloads.

This was the magic ingredient that myself and many others had been waiting for.

This meant that now I could run the VM’s on top of ZFS on the same server and get rid of the network transport. This means I could essentially double my VM node capacity by turning every Storage Node into a “VM & Storage Node” and every ESX Node into a  “VM & Storage Node” ….Wow!

Some additional benefits of moving to SmartOS would be:

• No more worrying about IO Latency & bottlenecks between storage and VM nodes.
• SmartOS does not require a harddrive for the OS to be installed on. It boots off cdrom ISO, USB thumbdrive, or via network PXE boot. This means all harddrives can be used for ZFS storage, and OS upgrades are simply achieved by booting off a new SmartOS image.
• Increased performance over Vmware ESX and Higher server utilisation.
• Increased VM security, as VM’s on SmartOS run inside a secure zone if the VM operating system is exploited attackers will essentially find themselves breaking out from one Jail cell into a bigger penitentiary where they are trapped.
• Additional security for VM network traffic. On SmartOS VM’s can not see each others network traffic, and can not spoof mac addresses. This also helps in DOS scenarios.
• Flexibility  of  using Zones or full KVM when needed.
• Access to analytics using DTRACE.
• and lots more….

SmartOS is very new and the community is still building momentum. There is not a lot of information and documentation available yet on using SmartOS, but this is slowly starting to change. There is also as far as I know, no freely available GUI based management tools and all configuration has to be done from the command line. I suspect this will change over time.

For big organisations it would probably be advisable to look at the Joyent proprietary ” Smart Data Center” product which is superb and has a ton of management and powerful analytics built in to fully manage your own private or public clouds running SmartOS Nodes. I have had a demo of Joyent SDC and it is really impressive, and the folks at Joyent are always very friendly, helpful and a pleasure to deal with.

With this Blog, I intend to post articles and screencasts on my experiences in setting up and using SmartOS to try and promote its use and uptake and to give back to the community.