Looking Back at SCaLE x14

For those that don’t know what SCaLE is… SCaLE stands for the Southern California Linux Expo and this marked the 14th year the conference has been held which happened to be in Pasadena, CA on Jan 21-24. This was the first time I have attended SCaLE and I have to say that it was quite refreshing going to a conference where the primary purpose wasn’t trying to sell you product but rather ideas and open source projects of interest. A lot of this is due to a very community driven focus which encompassed everything from session selection, a large volunteer staff, and etc.

There were a number of sessions and tracks ranging from Ubucon (everything Ubuntu), PostgreSQL, MySQL, Apache Bigtop, Open Source in Education, Unikernels, Robotics using Golang (pictured below) and etc. I took it upon myself to take a slice out of each pie to get a good feel for what the conference had to offer. Like everything in life, there was a range of the good, the bad and the ugly… but I would have to say there wasn’t much, if any, in the latter category. For reference, you can get a copy of the conference schedule here as a sampling of what types of sessions SCaLE provides.

Gobot - Robots Powered by Golang

The other significant difference is that audience or the make up of the attendees in this conference is significantly different than traditional conferences backed by big money which in this instance was predominately developers, DevOps peeps, and sysadmins. Gone are the sales and marketing people with the heavy focus on landing deals with private rooms off to the side where contracts are being drawn up. If you are looking to network with other developers and the actual users of a particular technology, this is a fantastic place to connect with those people.

Without further ado, here are some notes from sessions I found interesting. It should also be noted that I have included a lot of the session links in this blog as many of the slide decks are available on those pages.

IoT and Snappy Ubuntu Core

This session was titled Internet of Things gets ‘snappy’ with Ubuntu Core and its main focus was introducing people to the idea of Snappy Ubuntu Core which Ubuntu is pushing as the solution for building applications in the cloud and on embedded devices. Think of it as a minimal Ubuntu operating system built using the same libraries as traditional Ubuntu just with a much smaller footprint wrapped with a convenient package management that can orchestrate installation and upgrades. The IoT demo was an application built using snappy on a raspberry pi with an IP camera that can do facial tracking. Pretty cool! You can find more information about Snappy Ubuntu Core here.

IoT using Snappy Ubuntu Core on a Raspberry Pi

Juju Charms

This was my first exposure to Juju Charms. I had heard the name being thrown around before but didn’t have the time to take a look at the offering. The session Writing your first Juju Charm was a well presented introduction for first time users. The main takeaway is that Juju models relationships with other applications encapsulated in what is called a charm and that those charms, for example MySQL, are created, maintained, and tuned by subject experts. Pulling those vetted charms helps provide a solid foundation to build your applications on. Dependencies in charms are automatically pulled in and the likeness was compared that to the layers of an onion; you just layer in the applications you want. Then when pulling together your solution, its simply dragging and connecting these charms within the UI.

In writing your own charm, the method of hooking these applications together is done by an event driven engine. For example, your application contained within your own charm doesn’t get installed until the http server started and the MariaDB started event are received. Overall a good session. You can find more information about Juju here and here.

Docker and Unikernels

Although what I am going to write about here wasn’t a session at SCaLE per se, but speakers from both the Docker and Unikernel presentations at this special edition of the Docker LA Meetup SCaLE x14 Edition had sessions at the conference. The first speaker was Jérôme Petazzoni from Docker (pictured on the right). We discussed Swarm and the advances made in the latest release specifically around mulit-network orchestration between containers instances to satisfy the cluster use case. I did enjoy the fact that the discussion was almost completely driven by demonstration, but from the sounds of known issues even walking into this demo, it makes me think that this isn’t ready for prime time yet. I definitely appreciate the candor and honesty from the presenter about these issue; speaks to his professionalism.

The second half of the meetup quickly switched gears to talk about the Docker acquisition of Unikernel Systems who is the primary driver of the open source ecosystem of unikernel.org. Presenting a 101 type course on unikernels was Amir Chaudhry and Richard Mortier (pictured on the left). The session was quite fascinating considering I had not heard of the term unikernel until Docker made the announcement that morning. In a nutshell, the idea of a unikernel is such that you only take the various parts of a particular stack, say networking, that you need in order for your application to run. This implies that the stack is modular and has clear separation of concerns. The claim is that this type of operating system has:

  • a smaller footprint in terms of size
  • better security because they have a lower penetration profile
  • better performance because unnecessary services aren’t running
  • boots quicker than a traditional operating system which lends to cloud applications

While all this might be true, I have some reservations about dynamically pulling pieces of the operating system to build your final application image. Unikernel System via their tools claim to have solved that dependency nightmare. For the sake of argument if we accept that have been able to solve dynamic dependencies between modules, a lot of these operating systems have been rewritten to the ground up which begs the questions about stability. Additionally, these unikernels don’t have a traditional kernel or protected layer meaning that the application has full access all the way down the stack. Think about the security ramifications of this design.

It seems like unikernels are at odds with what Ubuntu Core decided to do which was just create a common minimal operating system using the same exact libraries we have been using for decades to run cloud and embedded applications. I think I prefer the Ubuntu Core design, but I have a feeling the hype and backing of Docker’s marketing machine might smash that idea.

Jérôme Petazzoni of Docker, Amir Chaudhry and Richard Mortier of Unikernel Systems

Mentoring in Open Source

I attended the Dent the Universe: Mentoring in Open Source session after a fellow coworker was interested in but was unable to attend SCaLE. I was pleasantly surprised with both the content of the discussion as well as the make up of the audience. The audience had a mix of gender, race, backgrounds, and also age… as in there were teens, toddlers, and babies in the audience (pretty sure the babies didn’t really follow was was going on).

The upshot of the session is that relationships between mentors and mentees bring forth a sense of community like what we see in the open source world. Its a platform for exchanging ideas similar to a meetup but on an individual 1 on 1 setting. There are many open source projects and organizations out there that have mentoring programs included by not limited to: Google, Apache, Fedora, and Ubuntu. The session covered what mentors are and aren’t, how you build that relationship, the dedication that mentors need to have, and even how mentees might go about getting a mentor. Of all the sessions that I did sit in on, I have to say I took the most amount of notes in this one… maybe it was because the subject matter was so different, but nonetheless this was definitely an excellent session and was well worth the time.

Nomad, a Golang Cluster Scheduler

I just want to give a quick mention about the Nomad – An introduction to Cluster Schedulers session. I found the overall design of Nomad intriguing based on the claimed feature set. Nomad is a HashiCorp project which if you don’t know who HashiCorp is, they are the ones who brought you projects like Vagrant and Consul. Nomad is written in Golang and supports scheduling for virtual infrastructure, Docker, Java applications, and etc. They have support for Global and Local cluster regions (think multi-datacenter support) and has facilities for deploying, scaling, rolling updates, load balancing, integrates with Consul and more. Although it seems like scale isn’t their strong suit as the presenter only believes the stress testing has only been done on 100 or so nodes; however, Nomad might be a good enough solution for a mid-sized business. I will be keeping on eye on this considering the success that HashiCorp has had on their other projects.

Expo Floor at SCaLE x14

Conclusion

Overall, SCaLE was a great conference with veritable buffet of topics that can interest a wide variety of people who attend. Hell for the price, under $40 if you get in on a coupon code, you can’t beat the value for the level of information in the sessions and the discussions had with the many attendees and speakers. I would definitely recommend checking out this conference next year and if you can convince your employer to fund the trip or if you can’t and have the spare time and (not much) cash to come on your own dime, its definitely a worth while experience.

twittergoogle_plusredditlinkedinmail

Difficulties with Multi-Version Mesos Support

I have been working with Apache Mesos for some time now and after a recent meeting with the Mesosphere team (the company largely driving the roadmap and development effort), I have come to learn that some Mesos users sit on the bleeding edge of releases and others that are sensitive to things like security, change management, and stability are more inclined to be a couple or several versions behind. This practice isn’t anything new, it took EMC IT a long time to adopt Windows 7… heck, Windows 10 is out now and we are still sitting at Windows 7. Not saying that is necessarily a bad thing..

Why you wait for the first Service Pack

So we have to accept the fact that we need to support multiple versions and take that into consideration when building components for Mesos. This takes us to the most recent release (0.4.0) of mesos-module-dvdi which supports multiple versions spanning from 0.23.1 to the most recent release of mesos, 0.26.0. A side note, for those not familiar with mesos-module-dvdi its a Mesos isolator module that can create and mount external storage volumes, such as ScaleIO, Amazon EBS, or etc, and provide persistent storage for Applications created on Mesos. If you are interested, you can find more information on its GitHub page.

Where’s the Beef?

The mesos-module-dvdi is a C++11 project and as such needs to implement defined Mesos interfaces so that the module can be loaded and invoked from a Mesos slave (think of it as a simple node that launches tasks). If you take a look at the Isolator interface between versions, you will immediately notice that the interface varies wildly between versions and that modules created for one version will not work on another version of Mesos. Heck, even the interface class name you need to implement has changed:

In 0.23.1:
class IsolatorProcess : public process::Process

process::Future<Nothing> recover(
const std::list<ExecutorRunState>& states,
const hashset<ContainerID>& orphans);

process::Future<Option<CommandInfo>> prepare(
const ContainerID& containerId,
const ExecutorInfo& executorInfo,
const std::string& directory,
const Option<std::string>& rootfs,
const Option<std::string>& user);

In 0.24.1:
class DockerVolumeDriverIsolator: public mesos::slave::Isolator

virtual process::Future<Nothing> recover(
const std::list<ContainerState>& states,
const hashset<ContainerID>& orphans);

virtual process::Future<Option<ContainerPrepareInfo>> prepare(
const ContainerID& containerId,
const ExecutorInfo& executorInfo,
const std::string& directory,
const Option<std::string>& user);

This is not good. This is basically Mesos’ version of Microsoft’s DLL Hell. As an experienced C++ developer, defining a C++ interface that is going to ensure backwards compatibility is very difficult. In previous projects I have worked on, the interfaces not only needed to be backwards compatible but also cross-platform (yes, Windows support) all from a single code base. That adds further complexity. These days some of these difficulties can easily be side stepped by standing up a REST endpoint. Even the worst REST APIs by in large “get the job done” even if the API is horribly designed. For example, some might argue if the API does not follow CRUD, HATEOAS, or etc they are bad designs, but I digress.

DLL Hell

A Meh Solution

So the current solution to support multiple versions of the Mesos Isolator interface from a single code base has been to use ifdefs. Yea, unfortunately the solution isn’t as elegant as it could be, but with the break in backwards compatibility in the Isolator interface, it leaves us with very little choice. This is also compounded by the fact that this isn’t the only interface that is changing between versions. There are other Mesos support libraries in which the interface is in flux and adding an abstraction layer among all these functions, classes, and structures might take you down a rabbit hole that will leave you rocking yourself in the fetal position for months crying “so many functions”.

Since we are using ifdefs, that unfortunately means our solution is a compile time solution and we therefore need to build every version that we plan on supporting. We do that in our Makefile as see below.

In the Makefile on lines 8 and 818 respectively:
ISO_VERSIONS := 0.23.1 0.24.1 0.25.0 0.26.0

$(foreach V,$(ISO_VERSIONS),$(eval $(call ISOLATOR_BUILD_RULES,$(V))))

Then we take that version and create an integer representation of the version by stripping out the periods from the version we are compiling. Represented by the MESOS_VERSION_INT preprocessor directive below. An example of that would be version 0.24.1 becomes 0241.

In the Makefile lines 790-795:
$$(ISO_MAKEFILE_$1): CXXFLAGS=-I$$(GLOG_OPT_DIR)/include
-I$$(PICOJSON_OPT_DIR)/include
-I$$(BOOST_OPT_DIR)
-I$$(PBUF_OPT_DIR)/include
-DMESOS_VERSION_INT=$$(subst .,,$1)
$$(ISO_MAKEFILE_$1): $$(ISO_CONFIGURE_$1) $$(ISO_DEPS_$1)

Then we compile the project using the MESOS_VERSION_INT and make alterations to the code to handle the ifdefs.

In the docker_volume_driver_isolator.hpp:
if MESOS_VERSION_INT != 0 && MESOS_VERSION_INT < 0240<br />
class DockerVolumeDriverIsolator: public mesos::slave::IsolatorProcess<br />
else<br />
class DockerVolumeDriverIsolator: public mesos::slave::Isolator<br />
endif

if MESOS_VERSION_INT != 0 && MESOS_VERSION_INT < 0240<br />
process::Future<Nothing> recover(
const std::list<ExecutorRunState>& states,
const hashset<ContainerID>& orphans);<br />
else<br />
virtual process::Future<Nothing> recover(
const std::list<ContainerState>& states,
const hashset<ContainerID>& orphans);<br />
endif

if MESOS_VERSION_INT != 0 && MESOS_VERSION_INT < 0240<br />
process::Future<Option<CommandInfo>> prepare(
const ContainerID& containerId,
const ExecutorInfo& executorInfo,
const std::string& directory,
const Option<std::string>& rootfs,
const Option<std::string>& user);<br />
else<br />
process::Future<Option<ContainerPrepareInfo>> prepare(
const ContainerID& containerId,
const ExecutorInfo& executorInfo,
const std::string& directory,
const Option<std::string>& user);<br />
endif

So that is it. Again, not the most elegant solution but unfortunately addressing the backwards compatibility is going to be very difficult to resolve going forward as it would require retrofitting older versions of Mesos. That could get pretty ugly to change architecture in the form of patches or even a minor release. Odds are this is not going to go away any time soon. I hope this helps other developers out there looking to create Mesos Isolators (and Mesos Frameworks because it looks like the Framework interfaces may have the same problem).

twittergoogle_plusredditlinkedinmail

Who is David vonThenen?

Hi! My name is David vonThenen. I am a Developer Advocate with EMC {code}. Yes, that EMC that makes all the storage products we have all come to know and love. I have recently been given the opportunity to join a fairly new group, the {code} team, within EMC that lives and breathes Open Source by making contributions back to the community. Part of this new journey includes a lot of outward facing collaboration with users of projects that EMC has open sourced but also a lot of collaboration with other developer and community projects out there. That leads me to my first blog post!

Who am I? I have been working in the technology industry for many years now in varying disciplines including: semiconductors, mainframes, storage area networks (FC, iSCSI, FCoE), distributed systems, backup/recovery solutions, virtualization (VMware), cloud, and etc. All these roles have been in your traditional enterprise corporate environment where you get some requirements from some product marketing monkey to tell you how they interpreted what the customer wants. Ever play a game of telephone? Now try it with a non-technical person armed with only acronyms. Yea, usually doesn’t go well but I digress.

When I heard about EMC {code}, I jump at the opportunity to take how I developed software and completely turn that world upside-down. Gone are the days of those marketing monkeys, gone are those silly 6 to 12 month development cycles, and gone are the closed proprietary software solutions. One of the biggest draws for me going Open Source was the collaboration with developers and users and the other social aspects of this change, like the idea of speaking at meetups. A lot of software engineer would say “whaaaa?” Yup, even the simple side of social media like tweeting what I have been working on without having the fear of exposing some critical feature before the release or tipping our hand to competitors.

Social Interaction What?

Another thing I am looking forward is working with all the new technology out there. Docker, Mesos, and etc. I have been predominately known as a C++ and Java Spring developer, but I am very interested in learning some of the new modern languages out there. I just recently picked up Golang by going through some online tutorials, reading a couple of books (yea, I still do that), and recreating several projects that I quite frequently need to do in C++ or Spring land by making the Golang equivalent. Looking forward to my first real project using it.

What about outside work?

Well enough of that boring crap. I got started with programming around the time I was in high school, but was self-taught programmer primarily in C++ making my own video games. One of my favorite subjects in high school was chemistry so I did start off as a chemistry major in college (hence, the Breaking Bad motif of my blog), but about 2 years in, I realized that I couldn’t leave programming, computers and technology behind. So I ended up double majoring in Computer Science as well. I love poker (particularly no limit Texas Hold’em), I actually enjoy going to the gym and have a home gym for when I can’t make it, I am the proud owner of 2 boxer dogs (Sydney and Izzy), huge fan of aged Irish Whiskey (love Glenfiddich 18 or Jameson for your casual kind of drinking), I am a foodie who will try anything once, and I am one of those car guys who drives a muscle car.

Sydney and Izzy

Now what?

Well I hope that you stick around and visit my blog every once in a while as I plan to blog on various topics relating to technology, some of the things that I am working or even things that interest me. If you happen to live in the Orange or Los Angeles County area, I have been active in the MeetUp community. You can also find me on Twitter, LinkedIn and GitHub.

twittergoogle_plusredditlinkedinmail