I recently came to the conclusion that my home openindiana ZFS storage servers could be put to much better use being repurposed as SmartOS compute nodes. If your ZFS storage servers are anything like mine, they sit with their cpu’s idle most of the time even when they are sending or receiving ZFS IO.
With rising power costs, this is the perfect opportunity to turn multiple dedicated servers into a multi purpose single SmartOS nodes.
The conversion process was pretty simple. I simply pulled out 4 of the hard drives that has my main media collection on it, consisting of a ZFS pool called “titan”, then added in another 2TB Drive. An added benefit of going the SmartOS route is that I could free up the existing openindiana OS drive and use it and the additional NEW 2TB drive as my “zones” pool in a ZFS mirror configuration.
The setup was as follows:
- Power up the server, set it to PXE boot.
- PXE-Boot off the latest SmartOS image.
- During setup I chose the 2 x 2TB drives as the “zones” pool.
- Reboot and SmartOS is up and running. wallah!
- Put the 4 x 2TB Drives back in (Containing my “titan” zfs pool)
- Issued a zpool import with: ” zpool import -f titan “
- At this point I now have 2 pools “zones” and “titan” you gotta love ZFS for this simplicity!
- Lastly i just re-added the SSD L2 ARC device “zpool add zones cache c1t2d0”
At this stage our zpool and devices looks as follows:
[root@smartosn6 ~]# zpool status
[root@smartosn6 ~]# zpool status pool: titan state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM titan ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 c0t5000C500362FE68Ad0 ONLINE 0 0 0 c0t5000C5003EDC6901d0 ONLINE 0 0 0 c0t5000C5003EDF774Ed0 ONLINE 0 0 0 c0t5000C5003EDF8DF8d0 ONLINE 0 0 0 errors: No known data errors pool: zones state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zones ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c0t5000C5002D93C60Dd0 ONLINE 0 0 0 c0t5000C5002DA0CE37d0 ONLINE 0 0 0 cache c1t2d0 ONLINE 0 0 0 errors: No known data errors
Now that we have our server setup it is time to create a Zone Virtual Machine that can share all the existing media data in the “titan” pool with the rest of the network. The correct way to do this is to LOFS mount the existing share from the “titan” pool and make it natively accessible within the SmartOS zone. This is the JSON payload file I used to create the zone.
[root@smartosn6 ~]# vmadm create -f megastore.json
{ "brand": "joyent", "image_uuid": "60a3b1fa-0674-11e2-abf5-cb82934a8e24", "autoboot": true, "alias": "megastore", "hostname": "megastore", "max_physical_memory": 2048, "max_swap": 2048, "tmpfs": 2048, "quota": 50, "nics": [ { "nic_tag": "admin", "ip": "10.1.1.71", "netmask": "255.255.255.0", "gateway": "10.1.1.1", "primary": true } ], "filesystems": [ { "type": "lofs", "source": "/titan/markitunes", "target": "/markitunes" } ] }
Lets zlogin to our newly created zone and have a look at the disk space. I am only showing the main quota for the zones and the LOFS mount in the below snippet.
[root@smartosn6 ~]# zlogin 82730c01-deb0-428b-8780-d90944fa2466
[root@megastore ~]# df -h Filesystem Size Used Avail Use% Mounted on zones/82730c01-deb0-428b-8780-d90944fa2466 51G 1.2G 50G 3% /markitunes 3.9T 2.7T 1.2T 70% /markitunes
Next we will setup Netatalk so that the Macintosh clients can access the media via AFP (AppleShare file server) protocol.
First we make sure the zone is configured to use the correct local time so logs are easy to read. I changed my local timezone to Australia/Melbourne
[root@megastore ~]# vi /etc/default/init
TZ=Australia/Melbourne
Next we update pkg repository and install the necessary packages and dependencies. We will need a compiler and some crypto / ssl library’s in order to successfully compile netatalk.
pkgin up pkgin in gcc47 gmake libgcrypt openssl
After the necessary packages are installed we download the latest version of netatalk and extract it.
wget http://downloads.sourceforge.net/project/netatalk/netatalk/3.0.2/netatalk-3.0.2.tar.gz tar -xvzf netatalk-3.0.2.tar.gz cd netatalk-3.0.2
Now comes the tricky part getting the actual thing to compile cleanly. Here are the configure options that worked well for me.
[root@megastore ~]# ./configure --with-ssl-dir=/opt/local --with-libgcrypt --with-bdb=/opt/local --with-init-style=solaris --without-pam --prefix=/opt/local
To get it to build successfully on SmartOS we need to edit the “Makefile” and change the “servicedir” to a location that is writable under SmartOS. Lets change the following line:
[root@megastore ~]# vim distrib/initscripts/Makefile
- " servicedir = /lib/svc/manifest/network/ " + " servicedir = /var/svc/manifest/network/ "
Phew!, we are almost there! One last thing we need to do before we can build. We need to create a symbolic link or else libgcrypt will fail to compile (took me a awhile to figure this one out). Once the symbolic link is there we can finally compile it.
[root@megastore ~]# ln -s /opt/local/lib /opt/local/lib/64 [root@megastore ~]# make && make install
It should now compile cleanly without any errors. Before we we can start the services needed for AFP connectivity, we first need to edit our “afp.conf” config file with the required options and add a share. Below is a sample config that works well for me.
[root@megastore ~]# vim /opt/local/etc/afp.conf
[Global] log file = /var/log/netatalk.log uam list = uams_dhx.so,uams_dhx2.so [markdata] path = /markitunes/userdata/markdata valid users = marks rwlist = marks
Before we start the service, you need to create a user and give that user permissions to the shared folder.
Please NOTE: Do not use overly long or complicated passwords, in my testing my mac would not connect with long complicated password, but with a shorter password connected just fine. I am not sure why this is the case but would like to know the reason why if any of you know, please do tell.
[root@megastore ~]# chown marks:staff /markitunes/userdata/markdata [root@megastore ~]# svcadm enable svc:/network/dns/multicast:default [root@megastore ~]# svcadm enable svc:/network/netatalk:default
Thats it you should have a fully functioning, blazing fast AFP server for your Mac Clients. You can use this for TimeMachine as well if you like.
From your mac, mount your AFP share.
and copying a file across the network basically saturates my Gigabit NIC in my Macbook.
Enjoy your rockin! fast file transfers. For those of you who need windows file sharing, just install samba in addition to netatalk.
Jun 18, 2013 @ 11:37:00
Thanks for the great howto.
Do you know if it is relatively simple to create a package of the the compiled deamon and associated bits, so that to install on multiple zones, one does not need to install 650MB of gcc47, gmake, libgcrypt, and openssl?
I have tried following Jonathan Perkin’s piece (http://www.perkin.org.uk/posts/pkgsrc-on-smartos-creating-new-packages.html), but got a little lost, not knowing how to apply your patches etc. bmake also looped continuously trying to apply patches. Is Jonathan’s article(s) even the correct way to go about what I’m after?
It is interesting that Apple have just announced that SMB/CIFS will be their default LAN file share suite. It is a shame we can’t use the kernel SMB server in zones.
Piers
Aug 01, 2013 @ 12:46:00
You can delegate the ZFS datasets directly to the zone. Then you can use ZFS commands to manage the filesystem (snapshots, etc) just like you would from the global zone. I think “vmadm” can only set up the datasets on create. But you can edit them with the “zonecfg” command. Here’s a gist that might help: https://gist.github.com/mattconnolly/6097313
Jun 30, 2014 @ 21:25:35
This is how I do it too. I was wondering if LOFS was some new SmartOS specific way of sharing datasets between zones/vms.
Apr 14, 2014 @ 05:41:03
I found I needed to add “time machine = yes” to the share section of the netatalk config file to get Time Machine to recognise the share as a valid backup Destination.
Works great with that tweak!
R.
Sep 01, 2015 @ 23:00:20
Just tried these steps (same vm.json etc) and sadly make fails on SmartOS 20150709T171818Z with:
undefined first referenced
symbol in file
arc4random_addrandom ../.libs/libevent.so
ld: fatal: symbol referencing errors. No output written to .libs/event-test
collect2: error: ld returned 1 exit status
…
anybody had this too ?
Nov 09, 2015 @ 12:39:20
The libevent problem can be worked around with the patch found on the libevent site:
http://sourceforge.net/p/levent/bugs/351/
It seems that no other patches need to be applied at this point. The Makefile edit Mark mentioned can be replaced with the new “–with-init-dir=/var/svc/manifest/network” argument to configure.
Thanks, Mark, for the great notes that got me up and running pretty quickly! One of us really ought to get around to capturing this in a pkgsrc package. I see there is http://pkgsrc.se/wip/netatalk in the works. Hopefully that makes its way into the main line soon.