In part 1 of this 2 part screencast series – We cover the installation of the soon to be released Project FiFo 0.6.0 Version. This is a major release with a ton of new features and some foundational changes to the underlying storage architecture. This is a single zone configuration and not a clustered configuration. Going forward Leofs will be a required part of all FiFo installations.
We will cover:
– LeoFS the highly scalable, fault-tolerant, S3 compatible large object storage for the Web.
– LeoFS dataset import.
– Provision new fifo zone using the LeoFS dataset.
– LeoFS FiFo storage node configuration.
– FiFo packages installation.
– FiFo 0.6.0 Setup & Configuration.
– FiFo 0.6.0 UI Overview and usage Guide (Part 2)
– New Feature Overview. (Part 2)
Things to watch out for!
Important If “leofs-adm status” command gives an error state for storage or gateway the first time you reboot issue the command:
leofs-adm resume storage_0@127.0.0.1
DNS LeoFS storage needs DNS to be working and resolvable from your FiFo zone as WELL as the Global Zone of all your hypervisors in order for Chunter to work correctly. If your LeoFS endpoint is set to e.g. fifo.mycloud.net and your FiFo IP is 10.1.1.242 then the following FQDN’s need to resolve to your FiFo IP.
10.1.1.242 fifo.mycloud.net
10.1.1.242 fifo.fifo.mycloud.net
10.1.1.242 fifo-images.fifo.mycloud.net
10.1.1.242 fifo-snapshots.fifo.mycloud.net
No Central DNS – no problem. Add the FQDN’s to FiFo’s /etc/hosts file in the FiFo zone. You will also need to add them to /etc/hosts in the Global zone. Unfortunately the hosts file is lives in Ram and is reset upon SmartOS reboot. Luckily Chunter has a SMF service to ensure that /etc/hosts entries persist upon SmartOS reboots. To set it up, do the following:
Create the file and add your host entries to the file.
vi /opt/chunter/etc/hosts
Install the SMF Service
cp /opt/chunter/share/hosts.xml /opt/custom/smf/
svccfg import /opt/custom/smf/hosts.xml
After this the chunter dns entries will automatically appear in your /etc/hosts file and will survive reboots.
Commands Used In This Video :
fifo.json payload file
{ "autoboot": true, "brand": "joyent", "image_uuid": "3b6fb547-4544-4825-88c3-eafdc0e4e9b5", "max_physical_memory": 4096, "cpu_cap": 100, "alias": "myfifo", "quota": "140", "resolvers": [ "10.1.1.1", "8.8.4.4" ], "nics": [ { "interface": "net0", "nic_tag": "admin", "gateway": "10.1.1.1", "ip": "10.1.1.242", "netmask": "255.255.255.0" } ] }
Commands / Instructions
# Below DNS MUST RESOLVE # fifo.mycloud.net fifo.fifo.mycloud.net fifo-images.fifo.mycloud.net fifo-snapshots.fifo.mycloud.net svcadm enable leofs/manager0 svcadm enable leofs/manager1 svcadm enable leofs/storage leofs-adm start svcadm enable leofs/gateway leofs-adm status leofs-adm add-endpoint fifo.mycloud.net leofs-adm get-endpoints leofs-adm create-user fifo 1234 ## Generated Access Keys ## access-key-id: 486969ef0d7487488072 secret-access-key: acc07635034394aa694792620403509e8be89225 ##### leofs-adm add-bucket fifo 486969ef0d7487488072 leofs-adm add-bucket fifo-images 486969ef0d7487488072 leofs-adm add-bucket fifo-snapshots 486969ef0d7487488072 echo "http://release.project-fifo.net/pkg/rel/" >> /opt/local/etc/pkgin/repositories.conf pkgin -fy up pkgin install nginx fifo-snarl fifo-sniffle fifo-howl fifo-wiggle fifo-jingles cp /opt/local/fifo-jingles/config/nginx.conf /opt/local/etc/nginx/nginx.conf vi /opt/local/leofs/leo_gateway/etc/leo_gateway.conf vi /opt/local/fifo-sniffle/etc/sniffle.conf #change ports to 8184 and 7443 svcadm restart leofs/gateway svcadm enable epmd svcadm enable snarl svcadm enable sniffle svcadm enable howl svcadm enable wiggle svcadm enable nginx svcs epmd snarl sniffle howl wiggle nginx sniffle-admin config set storage.s3.port 7443 sniffle-admin config set storage.general.backend s3 sniffle-admin config set storage.s3.host fifo.mycloud.net sniffle-admin config set storage.s3.access_key 486969ef0d7487488072 sniffle-admin config set storage.s3.secret_key acc07635034394aa694792620403509e8be89225 sniffle-admin config set storage.s3.image_bucket fifo-images sniffle-admin config set storage.s3.general_bucket fifo sniffle-admin config set storage.s3.snapshot_bucket fifo-snapshots sniffle-admin config show fifoadm users add default admin fifoadm users grant default admin ... fifoadm users passwd default admin admin curl -O http://release.project-fifo.net/chunter/rel/chunter-latest.gz
Sample Working Setup Output :
[root@5a846f46-5aea-4e09-8f08-4999dc4c11d8 ~]# leofs-adm status | |
[System config] | |
System version : 1.1.1 | |
Cluster Id : leofs_1 | |
DC Id : dc_1 | |
Total replicas : 1 | |
# of successes of R : 1 | |
# of successes of W : 1 | |
# of successes of D : 1 | |
# of DC-awareness replicas : 0 | |
ring size : 2^128 | |
Current ring hash : a945d4e2 | |
Prev ring hash : a945d4e2 | |
[Multi DC replication settings] | |
max # of joinable DCs : 2 | |
# of replicas a DC : 1 | |
[Node(s) state] | |
-------+--------------------------+--------------+----------------+----------------+---------------------------- | |
type | node | state | current ring | prev ring | updated at | |
-------+--------------------------+--------------+----------------+----------------+---------------------------- | |
S | storage_0@127.0.0.1 | running | a945d4e2 | a945d4e2 | 2014-08-30 13:08:10 +0000 | |
G | gateway_0@127.0.0.1 | running | a945d4e2 | a945d4e2 | 2014-08-30 13:15:59 +0000 | |
[root@5a846f46-5aea-4e09-8f08-4999dc4c11d8 ~]# sniffle-admin config show | |
Storage | |
General Section | |
Key Value | |
-------------------- -------------------------------------------------- | |
storage.general.back s3 | |
S3 Section | |
Key Value | |
-------------------- -------------------------------------------------- | |
storage.s3.general_b fifo | |
storage.s3.port 7443 | |
storage.s3.access_ke 486969ef0d7487488072 | |
storage.s3.snapshot_ fifo-snapshots | |
storage.s3.host fifo.mycloud.net | |
storage.s3.secret_ke acc07635034394aa694792620403509e8be89225 | |
storage.s3.image_buc fifo-images | |
[root@5a846f46-5aea-4e09-8f08-4999dc4c11d8 ~]# leofs-adm get-endpoints | |
endpoint | created at | |
-----------------+--------------------------- | |
fifo.mycloud.net | 2014-08-30 11:59:59 +0000 | |
localhost | 2014-08-30 11:56:31 +0000 | |
s3.amazonaws.com | 2014-08-30 11:56:31 +0000 | |
[root@5a846f46-5aea-4e09-8f08-4999dc4c11d8 ~]# ping -a fifo.mycloud.net | |
fifo.mycloud.net (10.1.1.242) is alive | |
[root@5a846f46-5aea-4e09-8f08-4999dc4c11d8 ~]# ping -a fifo.fifo.mycloud.net | |
fifo.fifo.mycloud.net (10.1.1.242) is alive | |
[root@5a846f46-5aea-4e09-8f08-4999dc4c11d8 ~]# ping -a fifo-images.fifo.mycloud.net | |
fifo-images.fifo.mycloud.net (10.1.1.242) is alive | |
[root@5a846f46-5aea-4e09-8f08-4999dc4c11d8 ~]# ping -a fifo-snapshots.fifo.mycloud.net | |
fifo-snapshots.fifo.mycloud.net (10.1.1.242) is alive |
Sep 02, 2014 @ 08:08:25
Great Job, this was very easy to follow. I really appreciate you taking the time to make this screen cast. I was wondering if you could offer some insight to a problem I am having?
Either Wiggle or LeoFS_gateway will not start after a reboot. Both services complain that they are unable to ping. So I get the gateway working and then Wiggle goes offline. I then get Wiggle going again and the Gateway goes offline. Is there anything you can think of that would cause this? Both services trying to use the same port? I created another zone with LeoFS and no FIFO. This zone appears to be stable, no problems after a reboot. I am wondering, if it would make sense to use 2 different zones for LeoFS and FIFO. But after reading on the project-fifo page. It appears that LeoFS is required. But is it required to run in the same zone?
Once again, thank you for taking the time to show this great software. Looking forward to the next installment on FiFO clusters.
Sep 02, 2014 @ 12:07:06
Hi there, glad to hear you enjoyed watching. For production it’s recommended to have multiple zones 3-5 for LeoFS and Riak ring. Placed on physically separate nodes. This is to have no single point of failure and to ensure resilience. Most of the pitfalls are outlined in the blog notes under the screencast window. If those don’t help, either jump in #project-FIFO IRC channel on Freenode, or feel free to jabber/google chat me direct on “nitronap at gmail.com”
Sep 05, 2014 @ 22:22:07
When i run $pkgin -fy up
Got errors like below:
pkgin: Could not fetch http://release.project-fifo.net/pkg/rel/pkg_summary.gz
pkgin: Could not fetch http://release.project-fifo.net/pkg/dev/pkg_summary.gz
I can not found the pkg_summary.gz on your repositories (http://release.project-fifo.net/).
Any ideas?
Sep 06, 2014 @ 09:02:56
I just fixed this issue.
The root case is that in VirtualBox local zone can not access the internet.
For more detailed solution please refer LiYu’s blog -> http://liyu1981.github.io/smartos-virtualbox/
Cheers!
Sep 06, 2014 @ 07:38:50
I installed smartos in Virutalbox, when i start leofs_storage service have error : [ERROR] Storage cluster does not start, yet.
After i lose lots of hair, finally i find following link. -> https://github.com/leo-project/leo_storage/issues/9 .
Fixed this issue by switch leo_storage backend to bitcask.
If someone have the same issue will lose less hair. 🙂
Dec 08, 2014 @ 09:41:32
Hello there, Thank you so much for the guide. I’m trying to setup a fifo with leofs following your guide here. However, I’m having problem with leofs storage. I have no problem using Fifo and internal storage. I can’t import dataset after i switch to s3 storage.
I have already add the hosts ( fifo-images, fifo, fifo.fifo, fifo-snapshots) and I can ping those hosts too. Do you have any idea? Thanks!
Dec 08, 2014 @ 13:32:50
Hi there Ray – sorry to hear you are having a bit of a rough time. There are a couple of things that spring to mind.
– Which version of the leofs all in one base dataset are you using?
– Can your leofs DNS entries resolve from the Global Zone where chunter service runs? (needs to resolve from both fifo zone and all your nodes GZ’s
A good place to look is in the sniffle debug log. Also feel free to jump into the #project-fifo freenode irc channel my name there is “trentster”. I am sure we can get you sorted out. You can also get me on google chat “nitronap at gmail dot com”
May 20, 2015 @ 17:07:51
Hi. When I try to import dataset, it shows me a message: Could not import dataset.
Any solution for it?
Thank you
May 26, 2015 @ 08:49:48
Hi there, are you importing via Project-FiFo or via imgadm? If via FiFo it is probably an issue with LeoFS not running correctly or not resolving DNS properly. If via imgadm, have you changed the import repository source to datasets.at as the dataset does not exist on the Joyent dataset servers.
Aug 01, 2015 @ 08:26:31
Hi Mark, great video! I really appreciate the hard work you have done with all these! I consider it a nice treasure! 🙂
I followed along but installed the latest version which had some differences but I go through all except for howl not running complaining how ‘Node ‘howl@10…160′ not responding to pings.’ Though if I ping ‘howl’ it responds, if I ping the IP address it responds. I’ve tried using the loopback IP, though that does not make sense and did indeed not help. All the other apps are running, it’s just howl that’s down.
Oct 20, 2015 @ 13:15:21
Hi Steve, sorry for the late reply. Try hitting me up on the FiFo IRC channel my alias is “trentster” I am sure we can get you sorted.