Blazingly Fast AFP on a SmartOS Zone

I recently came to the conclusion that my home openindiana ZFS storage servers could be put to much better use being repurposed as SmartOS compute nodes. If your ZFS storage servers are anything like mine, they sit with their cpu’s idle most of the time even when they are sending or receiving ZFS IO.

With rising power costs, this is the perfect opportunity to turn multiple dedicated servers into a multi purpose single SmartOS nodes.

The conversion process was pretty simple. I simply pulled out 4 of the hard drives that has my main media collection on it, consisting of a ZFS pool called “titan”, then added in another 2TB Drive. An added benefit of going the SmartOS route is that I could free up the existing openindiana OS drive and use it and the additional NEW 2TB drive as my “zones” pool in a ZFS mirror configuration.

The setup was as follows:

1. Power up the server, set it to PXE boot. 
2. PXE-Boot off the latest SmartOS image.
3. During setup I chose the 2 x 2TB drives as the “zones” pool.
4. Reboot and SmartOS is up and running. wallah!
5. Put the 4 x 2TB Drives back in (Containing my “titan” zfs pool)
6. Issued a zpool import with: ” zpool import -f titan “
7. At this point I now have 2 pools “zones” and “titan” you gotta love ZFS for this simplicity!
8. Lastly i just re-added the SSD L2 ARC device “zpool add zones cache c1t2d0”

At this stage our zpool and devices looks as follows:

[root@smartosn6 ~]# zpool status

[root@smartosn6 ~]# zpool status
pool: titan
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
titan ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c0t5000C500362FE68Ad0 ONLINE 0 0 0
c0t5000C5003EDC6901d0 ONLINE 0 0 0
c0t5000C5003EDF774Ed0 ONLINE 0 0 0
c0t5000C5003EDF8DF8d0 ONLINE 0 0 0

errors: No known data errors

pool: zones
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t5000C5002D93C60Dd0 ONLINE 0 0 0
c0t5000C5002DA0CE37d0 ONLINE 0 0 0
cache
c1t2d0 ONLINE 0 0 0

errors: No known data errors

Now that we have our server setup it is time to create a Zone Virtual Machine that can share all the existing media data in the “titan” pool with the rest of the network. The correct way to do this is to LOFS mount the existing share from the “titan” pool and make it natively accessible within the SmartOS zone. This is the JSON payload file I used to create the zone.

[root@smartosn6 ~]# vmadm create -f megastore.json

{
"brand": "joyent",
"image_uuid": "60a3b1fa-0674-11e2-abf5-cb82934a8e24",
"autoboot": true,
"alias": "megastore",
"hostname": "megastore",
"max_physical_memory": 2048,
"max_swap": 2048,
"tmpfs": 2048,
"quota": 50,
"nics": [
{
"nic_tag": "admin",
"ip": "10.1.1.71",
"netmask": "255.255.255.0",
"gateway": "10.1.1.1",
"primary": true
}
],
"filesystems": [
{
"type": "lofs",
"source": "/titan/markitunes",
"target": "/markitunes"
}
]
}

Lets zlogin to our newly created zone and have a look at the disk space. I am only showing the main quota for the zones and the LOFS mount in the below snippet.

[root@smartosn6 ~]# zlogin 82730c01-deb0-428b-8780-d90944fa2466

[root@megastore ~]# df -h
Filesystem Size Used Avail Use% Mounted on
zones/82730c01-deb0-428b-8780-d90944fa2466 51G 1.2G 50G 3% /markitunes 3.9T 2.7T 1.2T 70% /markitunes

Next we will setup Netatalk so that the Macintosh clients can access the media via AFP (AppleShare file server) protocol.

First we make sure the zone is configured to use the correct local time so logs are easy to read. I changed my local timezone to Australia/Melbourne

[root@megastore ~]# vi /etc/default/init

TZ=Australia/Melbourne

Next we update pkg repository and install the necessary packages and dependencies. We will need a compiler and some crypto / ssl library’s in order to successfully compile netatalk.

pkgin up
pkgin in gcc47 gmake libgcrypt openssl

After the necessary packages are installed we download the latest version of netatalk and extract it.

wget http://downloads.sourceforge.net/project/netatalk/netatalk/3.0.2/netatalk-3.0.2.tar.gz
tar -xvzf netatalk-3.0.2.tar.gz
cd netatalk-3.0.2

Now comes the tricky part getting the actual thing to compile cleanly. Here are the configure options that worked well for me.

[root@megastore ~]# ./configure --with-ssl-dir=/opt/local --with-libgcrypt --with-bdb=/opt/local --with-init-style=solaris --without-pam --prefix=/opt/local

To get it to build successfully on SmartOS we need to edit the “Makefile” and change the “servicedir” to a location that is writable under SmartOS. Lets change the following line:

[root@megastore ~]# vim distrib/initscripts/Makefile

- " servicedir = /lib/svc/manifest/network/ "
+ " servicedir = /var/svc/manifest/network/ "

Phew!, we are almost there! One last thing we need to do before we can build. We need to create a symbolic link or else libgcrypt will fail to compile (took me a awhile to figure this one out). Once the symbolic link is there we can finally compile it.

[root@megastore ~]# ln -s /opt/local/lib /opt/local/lib/64
[root@megastore ~]# make && make install

It should now compile cleanly without any errors. Before we we can start the services needed for AFP connectivity, we first need to edit our “afp.conf” config file with the required options and add a share. Below is a sample config that works well for me.

[root@megastore ~]# vim /opt/local/etc/afp.conf

[Global]
log file = /var/log/netatalk.log
uam list = uams_dhx.so,uams_dhx2.so

[markdata]
path = /markitunes/userdata/markdata
valid users = marks
rwlist = marks

Before we start the service, you need to create a user and give that user permissions to the shared folder.

Please NOTE: Do not use overly long or complicated passwords, in my testing my mac would not connect with long complicated password, but with a shorter password connected just fine. I am not sure why this is the case but would like to know the reason why if any of you know, please do tell.

[root@megastore ~]# chown marks:staff /markitunes/userdata/markdata
[root@megastore ~]# svcadm enable svc:/network/dns/multicast:default
[root@megastore ~]# svcadm enable svc:/network/netatalk:default

Thats it you should have a fully functioning, blazing fast AFP server for your Mac Clients. You can use this for TimeMachine as well if you like.

From your mac, mount your AFP share.

and copying a file across the network basically saturates my Gigabit NIC in my Macbook.

Enjoy your rockin! fast file transfers. For those of you who need windows file sharing, just install samba in addition to netatalk.