The Convergence that Changes Everything: Data Gravity + Containers + Microservices

What will change everything this time?

• Data Gravity (move compute to data) + 
• Containers (lightweight computation) +
• Microservices (composable compute modules)

That’s according to John Willis and Damon Edwards on a highly recommended recent DevOps Cafe Episode.

The high concept goes something like:

• Containers are the killer app for microservices.

• Data Gravity is the killer app for containers.

The logic goes something like:

• Data Gravity: Data is growing exponentially. Moving large chunks of data around doesn’t make sense anymore. You need to move compute to data. How?

• Containers: Containers can start modules in milliseconds, so they can perform their function in seconds, and then tear down quickly. Hundreds, even thousands of containers can easily be started. What's inside a container?

• Microservices: Microservices are small composable compute modules that can be glued together on the fly. Microservices are completely mobile. They can gravitate to wherever the data is. Containers make this happen.

A deep point is made that for this model to work you need to retool your organization. Your people need to be empowered to think in this new way. Organizations that can make this approach work will lap other more backwards organizations that are stuck in their siloed/monolithic/command and control worlds.  

It all fits together nicely. What do you think? Is this the future? Or just one possible future?

Related Articles

• Adrian Cockcroft on Microservices and DevOps

• Keynote: Microservices by Martin Fowler

• Dave McCrory on Data Gravity

• Containers: What’s New, What Isn’t, What Matters?

• [Google] Containers At Scale by Joe Beda

• Google: 'EVERYTHING at Google runs in a container' - Ad giant lifts curtain on how it uses virtualization's successor TWO BILLION TIMES A WEEK

Comments

[Yay Buzzwords]

As usual the answer is “it depends”. If the code moving around is larger than the data then it’s not solving anything.

[Robert]

I would love to see the day when SSD's come with maybe 4 gigs of ram, and a 4 or 8 core ARM chip. This would allow some filtering of data before sending it further downstream. Or compression etc. RAM is only $10/gb and ARM chips are cheap and low power.

[aditya]

Great post and def thinking in the right direction. At Coho we have already started doing this with a few large companies leveraging containers as our CTO goes into it with this blog post here: http://www.cohodata.com/blog/?p=629

[Alex]

I saw Bryan Cantrill give a talk with a very similar idea to "Data Gravity" last year at velocity. Very compelling and interesting concept.

[Anti-Gravity]

The last paragraph on that Data Gravity post shows what this is all about (so far): "There are many other things that Data Gravity might be able to be used for, so I’m looking for ideas from the community and would like participation from anyone with ideas." To me this reads similar to "Look, I got a nice solution here. The only thing I lack is a problem."

[Ali Hodroj]

Todd,

Great post. Indeed it all fits nicely together, and it's a sensible way to avoid latency bottlenecks associated with moving data around, coupled with on-demand computation offered by the container model.

I couldn't help but think though that this convergence has already been in practice (albeit at a very small scale) for quite a few years: In-memory data grids (**Disclaimer: I work for GigaSpaces, an IMDG vendor). Your modern IMDG nowadays (Coherence, GigaSpaces, or GemFire) is expected to provide all three concepts combined: Co-location of code (services) and data via a federated set of JVM heaps, while these JVM's themselves have thin containers around them to act as grid partitions or shards. Finally, gravitating code to the data happens through Map/Reduce or Scatter/Gather type code that is serialized over the network and executed within each JVM in parallel.

Do you see in-memory data grids as an enabler of this convergence?