Google+ is Built Using Tools You Can Use Too: Closure, Java Servlets, JavaScript, BigTable, Colossus, Quick Turnaround
Tuesday, July 12, 2011 at 8:55AM 
Joseph Smarr, former CTO of Plaxo (which explains why I recognized his picture), in I'm a technical lead on the Google+ team. Ask me anything, reveals the stack used for building Google+:
Our stack is pretty standard fare for Google apps these days: we use Java servlets for our server code and JavaScript for the browser-side of the UI, largely built with the (open-source) Closure framework, including Closure's JavaScript compiler and template system. A couple nifty tricks we do: we use the HTML5 History API to maintain pretty-looking URLs even though it's an AJAX app (falling back on hash-fragments for older browsers); and we often render our Closure templates server-side so the page renders before any JavaScript is loaded, then the JavaScript finds the right DOM nodes and hooks up event handlers, etc. to make it responsive (as a result, if you're on a slow connection and you click on stuff really fast, you may notice a lag before it does anything, but luckily most people don't run into this in practice). Our backends are built mostly on top of BigTable and Colossus/GFS, and we use a lot of other common Google technologies such as MapReduce (again, like many other Google apps do).
At first I read Clojure, which would have been a real surprise, but it's Closure, a suite of JavaScript tools consisting of a library, compiler, and templates. The compiler is a true compiler for JavaScript for making JavaScript download and run faster. The library is modular and cross-browser JavaScript library. Templates is a server-side templating system that helps you dynamically build reusable HTML and UI elements. It's all open source so you can use it too.
While you don't have Google's stack available to you, you do have some open source options. HBase is replacement for BigTable. Then there's Hadoop MapReduce. Colossus is Google's next generation file system, a replacement for GFS. Since we don't know much about Colossus, it's hard to say what a suitable replacement would be. There's the Hadoop distributed file system HDFS. And if you are looking for some of the cloud like infrastructure glue there's OpenStack (which also has storage system).
Google probably uses a custom Java servlet container, but the choice here doesn't matter that much. Most of the work will be spawned in parallel and performed on other servers implemented in C++, Java, or Python.
Whereas most communication with Google is non-existent, the Google+ development team is noticeably more responsive, turning out visible improvements quickly and consistently. Joseph tells us why: We put extra emphasis on engineering speed/agility--we try to release code updates on a daily basis while still keeping quality/stability/latency as high as you'd expect from google. This helps us move fast and respond quickly to user feedback. We try to do a full push (almost) every day, and we sometimes sneak in patch releases too if needed. But there are humans in the loop, it's not a "auto-push if all tests pass" situation or anything like that.
For Google+'s most innovative feature, video conferencing with Hangouts, GigaOM has a good article on that stack, which is based on Announcing Google+ Hangouts, written by Tech Lead Justin Uberti. Unlike Skype, which runs on a cost effective P2P model, Hangouts is completely hosted by Google. This must cost a staggering amount of money. You are on your own here. Nobody can replace the bandwidth being donated by the Google fairy.
That's Google in a box. Then again, an ex-Googler thinks you can do better using MessagePack, JSON, Hadoop, jQuery, and MongoDB. If you can do better for a world wide base of a billion users is a completely different question.
Related Articles
- Closure Library / Closure Templates / Closure Compiler
- VMware exposes its plans to be the OS for the cloud - another infrastructure option.
- Colossus of Rhodes - wonder of the world.
- Colossus Computer - electronic computing device used by British codebreakers to help read encrypted German messages during World War II. They used vacuum tubes (thermionic valves) to perform the calculations.
- Colossus: The Forbin Project - SF film about a massive American defense computer, named Colossus, becoming sentient and deciding to assume control of the world.
- This post On Hacker News
- This post On Reddit
- OpenGSE Released - Something that may be of interest to Open Source enthusiasts is the recent release of the Open Google Servlet Engine (OpenGSE). Once upon a time engineers within Google developed their own fast, lightweight servlet engine which we inside Google refer to as simply the "Google Servlet Engine" or GSE.
- On Quora: Is Google's software infrastructure obsolete? TLDR: Nope. Hector Yee has a good summary: The Google infrastructure is far more stable and well documented than the open source variants. They work together as a coherent whole. It's much easier to have a stable, massively scalable system built in the Google framework than with the equivalent open source tools. On the other hand, it's easy to chafe at the restrictions imposed by a big company with a defined way of doing things. Both view points can be true.
- nostrademons on JQuery vs Closure: Closure is used in basically every Google product. There's no comparison - in terms of pageviews, edge cases, and general real-world exposure, Closure is more widely used than probably every other JS library. I actually think they fill very different niches. JQuery is for sites that are primarily HTML but require some additional interactivity layered on via progressive enhancement. Closure is meant for JavaScript apps, where the whole client is basically written in JavaScript.
- RyanDScott on why GWT isn't used internally at Google: Closure is just JavaScript. No waiting for the GWT team to build the features you need, no having to build those features yourself. And in my experience, when I need cutting edge with GWT, I end up writing a bunch of wrappers for JavaScript functions, which makes me wonder why I'm not writing everything in JavaScript. (Like I said, the GWT library is very compelling and when planning a large project, Java often feels more comfortable for some.)
Related Articles
- Learning From The Google+ Experiment: Operating System, Platform, Apps by Stowe Boyd. But we are moving toward a world where most of the foundational elements of Google+ will be part of a next generation version of Android, and the things that feel like apps on Google+ will be, in fact, apps running on that future social OS.
Reader Comments (9)
The whole "ex-Googler thinks you can do better" thing is so bogus. I've commented on this elsewhere, but just a cursory glance at his primary critique makes it clear he only has a superficial knowledge of these technologies:
"Protocol Buffers, BigTable and MapReduce are ancient, creaking dinosaurs compared to MessagePack, JSON, and Hadoop. And new projects like GWT, Closure and MegaStore are sluggish, overengineered Leviathans compared to fast, elegant tools like jQuery and mongoDB"
Really? In fact, almost none of this is true. Protocol Buffers and Message Pack are almost identical in terms of size and efficiency, with protocol buffers being much better documented with a stronger community around it. You wouldn't necessarily know that if you just take a superficial at the MessagePack web site (as the author undoubtedly did), but their claims are completely bogus if you actually dig a little deeper and use those technologies. JSON versus Protocol Buffers? OK, weird comparison, but first of all Google uses JSON all over the place -- they're probably the biggest users of JSON in the world. Protocol Buffers are also much smaller and much faster than JSON. Hadoop versus BigTable? Really, you're going there? Hadoop's entire history is one of trying to catch BigTable in terms of performance and to copy BigTable feature for feature as best it can. Hadoop has gotten damn fast, but everything I've seen shows BigTable is still damn faster.
It goes on and on. Closure has much better compression than JQuery. Are the latest versions of JQuery faster? Probably, but this is hardly a "Leviathan" versus a new fast and elegant tool. GWT also does *browser-specific builds*, something JQuery can't touch and making it's javascript smaller and faster. MongoDB is certainly cool, but you're really going to put it up against BigTable, the only sensical comparison here? Really? That's just not even a road worth going down.
Anyway, just because someone worked at Google doesn't mean you should hang on their every word. It also doesn't mean they have deep knowledge about every single project at Google. It's a lot easier to get a job there than it used to be. There's also the problem of writing stuff that gets hits. This guy's article makes some wild claims designed to do exactly what they did: get links. It's total BS and you guys should know better at High Scalability. Seriously, you guys have some killer articles on here, but you shouldn't be linking to bogus info like that.
Note I have no dog in this fight. I don't work at Google, but I have directly used most of these technologies, including BigTable indirectly through heavy use of App Engine.
Adam, it's hard to imagine someone who delivered a significant project at Google not worthy of a link for their opinion on their experiences. We'll have to disagree on that, but I do appreciate the compliments and your comparison of the two stacks, very insightful, thanks.
Really great read, thanks for posting this!
No GWT for the front end?
Yup, I'm already working on my own google+. I will invite nobody! Muhahahah...
Todd: speaking as someone who was "early" at a top-five site and as the sole launcher of a category site that is still tops in its market, i recommend not putting too much stock in the "ex-" opinions you see from "former-"'s, i don't find that these opinions carry any noteworthy insights. there is a big difference between *creating* game-changing technologies and techniques from thin-air, and simply showing up and launching something on a very high-profile platform. the rebuttal of "ex-googler" above demonstrates that facts trump appeals to authority, and you have probably figured out by now that not every "old timer" is a genius
Proposing not to listen to someone who delivered a complex and extremely innovative product at scale because it conflicts with an accepted authority, based on an argument from authority, is a delightful tactic. Well done.
Sorry Todd, but refusing to acknowledge such carefully written critiques/comments because you've already formed an opinion is just as delightful (using your term). From your response to Adam's comment I don't think you even read it. I use GWT daily and it's laughable to compare jQuery to it...just to give one example.
I understand bashing Google generates page hits (IMO that's why this ex-Googler's post got so much coverage by bloggers), and it's always good to critically analyze strengths/weaknesses, but please be objective.
Sekhar, so you think I shouldn't even link to it? If you notice I haven't said anything on the contents of the post, except to question if those technologies would scale. The criticisms are about even linking to the post. Is that wrong your mind?