It has always amazed me, how much can be accomplished by a single individual when they focus their time, talent and laser-honed persistence in the pursuit of a single, seemingly insurmountable objective.
This is definitely true in the case of Heinz, who upon a whim set about single-handedly building a world-class, highly available, distributed cloud orchestration platform for SmartOS. This was at a time before SDC (Smart Data Center) was open source and when no open source management for SmartOS clouds existed.
I am sure if he knew back then what he knows now, he may have questioned his sanity in undertaking this journey in the first place, but that being said, as a community member, and person who benefits from using FiFo every single day, I thought it was about time I interviewed him so that he could share his story and answer some question regarding Project FiFo.
I hope you find the questions useful, and if there are any questions I missed that you would like answered, please feel free to ask them in the comments section.
About the project
Mark: Hi there Heinz, thank you for taking the time to answer these questions. Can you tell us how the project started, did you just wake up one day and decide “hey it would be cool to write a cloud orchestration suite from scratch” ?
Heinz: It started when VMware changed their free licensing for ESXi to something that required a doctorate in maths to understand. I was running a single server for me and two friends at that time and needed an alternative.
Mark: Did you start this project as a part time side project, or was it something that you did on the side from a regular job?
Heinz: Yes, FiFo started as a hobby project, it was never planned to become as big as it is today either.
Mark: Can you give us a bit of background on who you are and how you became a fan of Solaris derived operating systems?
Heinz: I worked in the Telco industry before writing FiFo, we were a heavy user of Solaris (back then still owned by Sun) and I fell in love with it the very first time I logged into a Solaris box.
Mark: Was the choice to make the code Open Source always present from the start of the project?
Heinz: There was no plan at the start of the project, it was just some afternoon hacking, however it quickly became obvious that I was not the only one interested in something like FiFo. We all benefit a lot from OSS work, either knowingly or unknowingly. FiFo being OSS is in part a way to give back to all the people that made my life easier with their work.
Mark: What Software Licence did you release your code under and what were the reasons for your decision?
Heinz: CDDL, and while I’d like to claim it was based on GNU incompatibility, it was pure laziness, SmartOS was mostly licensed under the CDDL and it seemed like a decent enough license and by that an obvious choice.
Mark: What attracted you to SmartOS and what was your motivating factor to use it as the foundation for your software vs alternatives that already existed in the open source cloud world?
Heinz: I love Solaris or Illumus as we have to call it today, the alternatives would have all been Linux and KVM or Linux and Xen so they were no real alternatives for me really. Keep in mind that FiFo started long before SDC was open sourced.
Mark: Do you ever look back and wonder “How the heck you managed to write such a complex system almost single-handedly? and did you ever underestimate the amount of work that it would entail”?
Heinz: No, I don’t wonder that, I know exactly how I managed that, by many sleepless nights, long weekends and not taking vacations. Yes and no, I never thought it’s going to be easy or simple, however I certainly underestimated the amount of work OSS brings with it that isn’t development work.
Mark: What was the most significant “Aha moment” you have experienced thus far in the project’s journey?
Heinz: Dealing with users can sometimes be a major pain in the ass!
Mark: With so many Cloud Orchestration suites available openly today, what would you deem would be the most significant reason a potential cloud adoption customer or business should/would choose FiFo over the myriad of alternatives out there?
Heinz: To my knowledge FiFo is the only Cloud Orchestration suite that is build entirely as a cloud application. Or if you want the buzzword, FiFo unlike the others is a cloud native application. It embraces failure and scalability, you can start small and grow as you need, every component can tolerate failure and the system keeps working, much of the code is self healing. FiFo is so resilient that you have to monitor the servers it runs on as you might not notice one of them failing otherwise (and yes that is something I say from experience).
Mark: Subsequently, Joyent the company behind SmartOS later Open Sourced their own previously proprietary SmartOS cloud management software. In light of this, what effect if any did this have on your Project and community?
Heinz: I am honestly not sure. I can make a few guesses but there is no data to support them, over all I would say very little.
Mark: This question will probably be a bit close to home, but why in your opinion should/would someone choose FiFo over Smart Data Center or vica versa?
Heinz: Everyone should choose FiFo over SDC because it has cute dog related names for all it’s components. This keeps OPS happy which is key to running a good business. 😉
Mark: Are there any companies out there using Project Fifo in a commercial environment and in what types of industries?
Heinz: Yes there are, both in the public cloud business as well in supporting roles as a private cloud, the feedback has been overwhelmingly positive.
Mark: With the benefit of hindsight, knowing what you know now – is there anything you would do differently if you could do it again?
Heinz: From a code and design perspective? Very little.
Mark: Why was LeoFS chosen as the official FiFo storage mechanism?
Heinz: It has very little dependencies and compared to other systems a rather simple design. Everyone who has set up Leo will think I am crazy saying that but try to set up Ceph or something like that and you will likely agree.
LeoFS also is accessed via S3 making it a rather simple and universal interface.
Mark: What are the benefits of LeoFS over other Open Source storage backends and what does it do better?
Heinz: That is too general a question to answer without being more specific about which particular backend it is being compared to.
Mark: What are the most significant strengths of LeoFS vs other storage platforms?
Heinz: Simplicity and resilience.
Mark: What are the most significant shortfalls of LeoFS vs other storage platforms?
Heinz: Setup complexity and the fact that people just don’t read the manual.
Mark: Are third party components and libraries primarily chosen from the perspective of easy / easier Erlang integration?
Heinz: First and foremost they are chosen for quality and maintainability, see nsq for example which is not written in Erlang. I wrote wrappers around kstat and dtrace to access from erlang while they were originally C libraries, but of cause especially for libraries that is a big plus.
Mark: In what areas does Erlang excel in a cloud orchestration environment ie: what is it particularly good at?
Heinz: Erlang is a language purpose build for control plane applications, cloud orchestration is a prime example for such a task making erlang a perfect fit.
Mark: What are the shortcomings if any you have experienced in terms of using Erlang in a cloud orchestration environment?
Heinz: The biggest problem with Erlang I encountered is the misconception that it is ‘hard’ or ‘cryptic’ because it doesn’t look like C or Java.
Mark: What has been the most rewarding part of working within the bounds of an open source community and eco-system?
Heinz: At the moment, I am too frustrated with the community to give a constructive answer to this question.
Mark: What has been the most challenging part of working within an open source community and what are some of the problems you have experienced that you would like to see solved or improved upon?
Heinz: The utter ignorance of users, the lack of respect towards work and time. The unbelievable demanding attitude some users display.
Mark: How can folks help out and contribute to the project without being involved in writing code or having an Erlang background?
Heinz: It is a huge misconception that writing code is a requirement for contributing to OSS, there are tons of other things that help the project equally. To name a few, writing documentation, tutorial or blog articles. Answering questions of new users, comment on tickets or mailing lists.
Mark: I notice you are still on a pre 1.0 release of Project FiFo, what does the future hold and what can folks expect in terms of an eventual 1.0 release?
Heinz: This is a semantic versioning thing, a set of rules for versioning. Bottom line is that in 1.0.0 we should make damn sure that there aren’t any backward compatibility breaking changes.