Great Open Source Solution for Boring HA and Scalability Problems

This is a guest post about how boring and repetitive HA and scalability problems can be solved via Open Source so you can focus on the interesting tasks. The post was written by Maarten Ectors, responsible for Cloud Strategy and Frank Mueller, a Juju Core developer, at Ubuntu/Canonical.

High-availability and scalability are exciting in general but there are certain problems that experts see over and over again. The list is long but examples are setting up MySQL clustering, sharding Mongo, adding data nodes to a Hadoop cluster, monitoring with Ganglia, building continuous deployment solutions, integrating Memcached / Varnish / Nginx,… Why are we reinventing the wheel?

At Ubuntu we made it our goal to have the community solve these repetitive and often boring tasks. How often have you had to set-up MySQL replication and scale it? What if the next time you just simply do:

1. juju deploy mysql
2. juju deploy mysql mysql-slave
3. juju add-relation mysql:master mysql-slave:slave
4. juju add-unit -n 10 mysql-slave

It’s easy to see how these four commands work. After deploying a master and a slave MySQL both are related as master and slave. After this you simply can add more slaves like it is done here with 10 more instances.

Responsible for this easy approach is one of our latest open source solutions, Juju. Juju allows any server software to be packaged inside what is called a Charm. A Charm describes how the software is deployed, integrated and scaled. Once an expert creates the Charm and uploads it to the Charm Store, anybody can use it instantly. Execution environments for Charms are abstracted via providers. The list of supported providers is growing and includes Amazon AWS, HP Cloud, Microsoft Azure, any Openstack private cloud, local LXC containers, bare metal deployments with MaaS, etc. So Juju allows any software to be instantly deployed, integrated and scaled on any cloud.

There are over a 100 Charms already and many of the most common server solutions already can be deployed, integrated and scaled instantly, e.g. Hadoop/HBase/Hive, Cassandra, MongoDB, MySQL/Postgres clusters, Memcached, Redis, Varnish, Nginx, HAProxy, Logstash, ElasticSearch, OpenTSDB, Nagios, Ganglia, Storm, Munin, Rails / Rack / Passenger, Node.JS, Tomcat clusters, PHP Lamp, Django, etc.

While the command line is the best friend for many administrators it is often very helpful to get a graphical overview and simple way of configuration for larger environments. Here Juju provides a GUI which can be installed via the Juju GUI Charm. Here new services can be deployed and integrated via drag-and-drop.

So by having an ever growing eco-system of charms, you do not have to reinvent deployment and more importantly integration and scaling any more. Additionally if you have your stack “charmed up” then you have the choice of any public/private cloud or bare metal so no more lock-in.

The first question we always get is how is Juju different from Puppet or Chef? Juju is focusing on services, Puppet and Chef focus on server provisioning. This means that you can use Puppet Manifests or Chef Recipes inside Charms to do the software install. A good example is the Rack Charm that uses Chef Recipes. But unlike Puppet or Chef when you start scaling up your architecture you scale up services not servers:

As you can see from the above example Ganglia is monitoring 100 Cassandra nodes. If you would scale to 200 you would not have to update Ganglia because Ganglia is monitoring the “Cassandra Service” and it will automatically add the 100 extra nodes.

Another question we often get is how is Juju different from PaaS? Cloud Foundry is a Charm and runs on top of Juju so if you need “traditional PaaS” you can. However often there are software solutions that can not work inside a PaaS and need another framework, e.g. Django, Rails, etc. Juju allows you to combine the power of IaaS and PaaS by creating your own PaaS where you assemble all needed solutions, sort of a DIY PaaS.

Ubuntu is eating its own dog food. Juju and MaaS are the default deployers for Openstack. So all large Openstack deployments that are currently happening on Ubuntu use Juju and MaaS. Additionally Ubuntu One and other Ubuntu cloud services are all running on top of Juju.

Except for boring scalability problems, Ubuntu is also working with startups around designing an instant continuous deployment solution hence every startup in the world can launch new features every hour. Additionally we launched a contest around building new solutions on top of Juju. There are many categories but high availability, monitoring and continuous deployment are probably of most interest to readers of High Scalability. Each category has a $10,000 price attached and there are judges like Adrian Cockcroft, Netflix’s Chief Cloud Architect.

However we don’t want this blog post to be an advertisement. Every new open source product has its limitations. We are aware that several charms are still rough around the edges. Certain charms are missing key relations. We are missing Charms and Cloud providers. Certain Cloud providers have still limitations. We should even further simplify complete stack deployments. Make different Juju environments talk to one another. Although OSX and Windows are supported on the client side, we only support Ubuntu on the server for now. This is where we would hope, the community can assist us in filing bugs, feature requests and even solving bugs and adding features. That is the beauty about open source, each addition provides synergies for the whole community. We hope you enjoy Juju and if you want to just play around with the GUI, you can go to JujuCharms.com.

Comments

[Bob]

Is it ethically ok to appropriate terms like "Juju" from other cultures, for our own ends? And, by the way, perpetuate stereotypes in the process?

Just wondering.

[Maarten Ectors]

Hi Bob,

Juju stands for magic and we use it to summaries what juju does: cloud magic. Mark Shuttleworth is South-African and as such he is proudly promoting his cultural origins. Ubuntu has an African origin as well. So we are just being consistent in our naming.

Regards,
Maarten

[Wouter]

PC Police alarm!

While I personally don't like the name Juju (nor "charms"), it's a decent name for something that has hints of magic to it. Apart from that, a word is a word, and it's sad that all too often people try to find a negative angle no matter how far-fetched or contrived. This is a software tool, and it's made by a pretty inclusive global company – no need to cry wolf. Just to make my point: depending on your definition, I speak six or seven languages, and I can almost certainly come up with a contrived bad connotation for any non-trivial word.

It's interesting Ubuntu went for the Go (golang) programming language and seems to at least initially aim more for start-ups and smaller companies – the "trendy web crowd"; this in contrast to RedHat, as an older company with a serious head start advantage, with older technologies like Java and decisively aiming for larger enterprises.

These days there are a lot of tools playing in the same sandbox though: puppet, chef, cfengine, salt and ansible to name a few, and that's not counting their cousins such as capistrano and fabric. Is it really necessary to re-invent the wheel over and over again? Most of these "enterprise" tools have companies backing their development, and the question arises whether or not it's going to harm users in the long term to build their whole infrastructure around a tool that – while open-source – has too strong a colour or tight association with one distribution or vendor.

Ubuntu tries to set itself apart from the mainstream Linux direction with Unity instead of Gnome, Mir instead of Wayland, Upstart instead of Systemd... To some extent choice is good – and I'm not saying Ubuntu is doing a bad job – but the user should retain the freedom of moving between different solutions with relative ease and not fall into the trap of vendor lock-in, so I'm feeling a bit apprehensive about the path some of the main players are taking. Even though we're talking about open-source software, one can still paint oneself into an expensive corner, especially when realising it's after all the wrong colour.

It will be interesting to see how open Ubuntu (and its tool set) remains to the user's freedom of choice, and hence whether investing time and effort into tools like Juju over alternatives is a good idea in the long run.

[eh]

Wouter, Upstart doesn't belong in that list. It predates systemd.

but the user should retain the freedom of moving between different solutions with relative ease and not fall into the trap of vendor lock-in, so I'm feeling a bit apprehensive about the path some of the main players are taking.

Does this mean systemd is out then? Once you choose it for your init system, many other choices are made for you.