Stuff The Internet Says On Scalability For July 15th, 2016

Hey, it's HighScalability time:

That little smudge on Jupiter is North America (size comparison). 

If you like this sort of Stuff then please support me on Patreon.

• <2%: percent of total U.S. electricity consumption used by data centers; $4.99: hourly wage of Amazon Turkers; 8,072: cores in Cassandra cluster; .5: new reward for slaving away in the Bitcoin mines; 11: source code for the original Apollo guidance computer; 10 inverse femtobarns: number of collisions recorded by the Large Hadron Collider; 34 bps: using MEMO to send molecular messages through the air; 200 MB: record for storage in DNA; 10,000+: 3D printed parts are used in a Rolls-Royce Phantom; $43.6bn: IaaS revenue to triple by 2020; 


• Quotable Quotes:
  • @PokemonGoApp: To ensure all Trainers can experience #PokemonGo, we continue to add new resources to accommodate everyone. Thank you for your patience.
  • @balajis: Pokemon Go is a classic overnight success, 10 years in the making. Ingress database, Google Maps, the Pokemon brand…
  • @avantgame: The math of Pokemon Go is pretty amazing. 21 million players in ONE week, playing 43 minutes on average a day.
  • @icecrime: Does Pokemon Go have generics?
  • @HarvardBiz: When companies start scaling, they often start seeing the future as a threat
  • Jakob Engblom: for the best performance, you want to break the design apart across cut-points with the lowest level of communication across the cut.
  • @peterpur: once again, it becomes obvious that complexity feeds itself, while simplicity needs conscious effort & hard work.
  • @jamesurquhart: Mine is already a microservice because it runs on a microcomputer. Right? Right?
  • Facebook: In our experience, every time we add a new tool, we are surprised that we managed without it.
  • @petecordell: Telling a programmer there's already a library to do X is like telling a songwriter there's already a song about love
  • @linclark: Code that my mom wrote 50 years ago just went up on GitHub
  • @danielbryantuk: "Our monolithic application was so monolithic that we gave it a name - jimmy..." Haha, awesome! @ZalandoTech at #microservices summit
  • Uri Hasson~ even across different languages, our brains show similar activity, or become “aligned,” when we hear the same idea or story.
  • @etherealmind: Bidirectional forwarding detection is most significant advance in Autonomous Routing in the last 20 years.
  • @aphyr: In particular, I'd like to note that @VoltDB has opted to preserve strong serializabilty as the default behavior, despite a latency cost.
  • @swardley: the system is based on a cycle of theft, settlers steal from pioneers forcing them to move on ...
  • Ian Adams: That the actual encoding at the CPU [for erasure coded storage] is generally not the bottleneck, but instead that the network tends to be, especially when you have really “wide” codes, e.g. 17/20 causing tons of traffic across many storage nodes for every request. 
  • Ayende: There is about 10% difference between fsync and fdatasync when using the HDD, but there is barely any difference as far as the SSD is concerned. This is because the SSD can do random updates (such as updating both the data and the metadata) much faster, since it doesn’t need to move the spindle.
  • @cpurdy: As long as flash capacities have an order-of-magnitude advantage over RAM, flash is allowed to be slower ;-)
  • @huntchr: Before you all go nuts re #serverless, #mechanicalsympathy remains important. You still need to understand what is going on under the hood.
  • Gallant: These results demonstrate that dynamic brain activity measured under naturalistic conditions can be decoded using current fMRI technology.
  • @sheeshee: trying to convince somebody to archive a really old CGI script roughly 1994 for archeological purposes.. old code is important for learning.
  • Kreps & Kleppmann: we advocate a style of application development in which each data storage and processing component focuses on “doing one thing well”. Heterogeneous systems can be built by composing such specialised tools through the simple, general-purpose interface of a log. 


• This is how you know you are Facebook. Instead of testing your new mobile software on one device you have a datacenter, with a lab, with around 60 custom made rack bristling with 2000 mobile phones, all so you can test all the different combinations and permutations. The mobile device lab at the Prineville data center.


• The challenge was made. @adrianco: Let me know when you run a 1000 node Cassandra cluster on Kubernetes :-). The challenge was met. Thousand Instances of Cassandra using Kubernetes Pet Set: We deployed 1,009 minion nodes to Google Compute Engine (GCE), spread across 4 zones, running a custom version of the Kubernetes 1.3 beta. We ran this demo on beta code since the demo was being set up before the 1.3 release date. For the minion nodes, GCE virtual machine n1-standard-8 machine size was chosen, which is vm with 8 virtual CPUs and 30GB of memory. It would allow for a single instance of Cassandra to run on one node, which is recommended for disk I/O. 


• Lures from Pokemon Go have turned out to be amazingly effective. Pokemon Go Is Driving Insane Amounts of Sales at Small Local Businesses. Here's How It Works. Building that kind of native business model driver deep into the game mechanics is the real trick of the game. Also, How the gurus behind Google Earth created 'Pokémon Go'.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Why Amazon Retail Went to a Service Oriented Architecture

When Lee Atchison arrived at Amazon, Amazon was in the process of moving from a large monolithic application to a Service Oriented Architecture.

Lee talks about this evolution in an interesting interview on Software Engineering Daily: Scalable Architecture with Lee Atchison, about Lee's new book: Architecting for Scale: High Availability for Your Growing Applications.

This is a topic Adrian Cockcroft has talked a lot about in relation to his work at Netflix, but it's a powerful experience to hear Lee talk about how Amazon made the transition with us having the understanding of what Amazon would later become. 

Amazon was running into the problems of success. Not so much from a scaling to handle the requests perspective, but they were suffering from the problem of scaling the number of engineers working in the same code base.

At the time their philosophy was based on the Two Pizza team. A small group owns a particular piece of functionality. The problem is it doesn’t work to have hundreds of pizza teams working on the same code base. It became very difficult to innovate and add new features. It even became hard to build the application, pass the test suites, and deploy the software.

The solution: move to a Service Oriented Architecture (not microservices).

Organizing around services allowed individual teams to truly own the code base, the support responsibility, and top to bottom responsibility for the functionality.

The result: a dramatic increase in innovation and the ability to grow. After a while the Amazon retail site grew with a constant stream of new capabilities. Maybe too many :-)

The process requires a culture shift, really more of an ownership shift, from being part of a larger group to be an entity of its own that has responsibilities outside of its group as well as responsibilities inside the group.

While the strategy of consciously exploiting team organization as a means of speeding up product development and encouraging innovation is not a new idea now, in the early 2000s it would have been one ballsy move.

Related Articles 

• The Secret to Amazons Success Internal APIs
• A Conversation with Werner Vogels 
• What Led Amazon to its Own Microservices Architecture
• The Story of Apollo - Amazon’s Deployment Engine

Vertical Scaling Works for Bits and Bites

This is just to delicious a parallel to pass up. 

Here we have Google building a new four story datacenter Scaling Up: Google Building Four-Story Data Centers:

And here we have a new vertical farm from AeroFarms: 

Both have racks of consumables. One is a rack of bits, the other is a rack of bites. Both used to sprawl horizontally across huge swaths of land and now are building up. Both designs are driven by economic efficiency, extracting the most value per square foot. Both are expanding to meet increased demand. It's a strange sort of convergence.

Stuff The Internet Says On Scalability For July 8th, 2016

Hey, it's HighScalability time:

Juno: 165,000mph, 1.7 billion miles, missed orbit by 10 miles. Dang buggy software. 

If you like this sort of Stuff then please support me on Patreon.

• $3B: damages awarded to HP from Oracle; 37%: when to stop looking through your search period; 70%: observed Annualized Failure Rate (AFR) in production datacenters for some models of SSDs; 


• Quotable Quotes:
  • spacerodent: After Christmas there was this huge excess capacity and that is when I first learned of the EC2 project. It was my belief EC2 came out of a need to utilize those extra Gurupa servers during the off season:)
  • bcantrill: That said, I think Sun's problem was pretty simple: we thought we were a hardware company long after it should have been clear that we were a systems company. As a result, we made overpriced, underperforming (and, it kills me to say, unreliable) hardware. And because we were hardware-fixated, we did not understand the economic disruptive force of either Intel or open source until it was too late. 
  • @cmeik: I am not convinced the blockchain and CRDTs *work.*
  • daly: Managers make decisions. Only go to management with your need for a decision and always present the options. They went to management with what was, in essence, a complaint. Worse, it was a complaint that had nothing to do with the business. Clearly they were not keeping the business uppermost in their priority queue. So management made a business decision and fixed the problem.
  • @colettecello: Architect: "we should break this down into 6 microservices" Me: "you have 6 teams who hate each other?" Architect: "how did you know that?"
  • Matt Stats: The differences between BSD and Linux all derive from basic philosophical differences. Once you understand those, everything else falls into place pretty neatly.
  • @wattersjames: "Last year, Johnson & Johnson turned off its last mainframe"
  • Allan Kelly: But in the world of software development this mindset [economies of scale] is a recipe for failure and under performance. The conflict between economies of scale thinking and diseconomies of scale working will create tension and conflict.
  • Jeff G: Today, a large part of my business is migrating companies off the monolithic Java EE containers into lightweight modular containers. Yes, even the tried and true banking and financial industries are moving away from Java EE. 
  • collyw: "Weeks of programming can save hours of planning" is a favorite quote of mine.
  • xiongchiamiov: when I see a team responsible for hundreds of microservices, it's not at all surprising when I find they're completely underwater and struggling to keep up with maintenance, much less new features.
  • Robert Plomin: We're always talking about differences. The only genetics that makes a difference is that 1 percent of the 3 billion base pairs. But that is over 10 million base pairs of DNA. We're looking at these differences and asking to what extent they cause the differences that we observe. 
  • @jmferdegue: Micro services as a cost reduction strategy for project delivery. Marco Cullen from @OpenCredo at #micromanchester
  • J.R.R. Tolkien: I like, and even dare to wear in these dull days, ornamental waistcoats.
  • @johnregehr~ HN commenter has reached enlightenment : In both cases, after about a year, we found ourselves wishing we had not rewritten the network stack.
  • @nigelbabu: OH: 9.9999% uptime is still five 9s.
  • @jessfraz: "We are going to need a floppy and a shaman" @ryanhuber
  • Peter Cohen: So, why the cloud? Because, the developer.
  • @CompSciFact: 'The fastest algorithm can frequently be replaced by one that is almost as fast and much easier to understand.' -- Douglas W. Jones
  • @igrigorik: Improved font loading in WebKit: http://bit.ly/29eaxV2  - tl;dr: 3s timeout, WOFF2, unicode-range, Font Loading API. hooray!
  • @danielbryantuk: "I've worked on teams with 200+. We had 3 people just to make JPA work" @myfear on scaling issues #micromanchester
  • AWS Origin Story: Jassy tells of an executive retreat at Jeff Bezos’ house in 2003. It was there that the executive team conducted an exercise identifying the company’s core competencies
  • @sheeshee: ".. you are charged for every 100ms your code executes and the number of times your code is triggered." the 1970ies are back. (aws lambda)
  • @KentBeck: accepting mediocrity as the price of scaling misunderstands the power law distribution of payoffs.
  • @cowtowncoder: that is: cost efficiency from AWS et al is for SMALL deployments, and at some point it always, invariably becomes cheaper to DIY
  • Exascale Computing Research priorities: Total power requirements suggest that CPUs will not be suitable commodity processors for supercomputers in the future.


• Here's how Instagram does it. Instagram + Android: Four Years Later: At the core of this principle is the idea that the Instagram app is simply a renderer of server-provided data, much like a web browser. Almost all complex business logic happens server-side, where it is easier to fix bugs and add new features. We rely on the server to be perfect, enforced through continuous integration testing, and dispense with null-checking or data-consistency checking on the client.


• Good story on how WePay is moving from a monolith to a services based architecture on top of Kubernetes. Advantages: autoscaling, rolling updates, a pure model independent of software assigned to specific machines.  WePay on Kubernetes: ‘It Changed Our Business’.


• Julia Ferraioli with a really fun explaination of Kubernetes using legos. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Machine Learning Driven Programming: A New Programming for a New World

If Google were created from scratch today, much of it would be learned, not coded. Around 10% of Google's 25,000 developers are proficient in ML; it should be 100% -- Jeff Dean

Like the weather, everybody complains about programming, but nobody does anything about it. That’s changing and like an unexpected storm the change comes from an unexpected direction: Machine Learning / Deep Learning.

I know, you are tired of hearing about Deep Learning. Who isn’t by now? But programming has been stuck in a rut for a very long time and it's time we do something about it.

Lots of silly little programming wars continue to be fought that decide nothing. Functions vs objects; this language vs that language; this public cloud vs that public cloud vs this private cloud vs that ‘fill in the blank’; REST vs unrest; this byte level encoding vs some different one; this framework vs that framework; this methodology vs that methodology; bare metal vs containers vs VMs vs unikernels; monoliths vs microservices vs nanoservices; eventually consistent vs transactional; mutable vs immutable; DevOps vs NoOps vs SysOps; scale-up vs scale-out; centralized vs decentralized; single threaded vs massively parallel; sync vs async. And so on ad infinitum.

It’s all pretty much the same shite different day. We are just creating different ways of calling functions that we humans still have to write. The real power would be in getting a machine to write the functions. And that’s what Machine Learning can do, write functions for us. Machine Learning might just might be some different kind of shite for a different day.

Machine Learning Driven Programming

Click to read more ...

Sponsored Post: Cassandra Summit, Gusto, LaunchDarkly, Awake Networks, Kinsta, Aerospike, InMemory.Net, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• IT Security Engineering. At Gusto we are on a mission to create a world where work empowers a better life. As Gusto's IT Security Engineer you'll shape the future of IT security and compliance. We're looking for a strong IT technical lead to manage security audits and write and implement controls. You'll also focus on our employee, network, and endpoint posture. As Gusto's first IT Security Engineer, you will be able to build the security organization with direct impact to protecting PII and ePHI. Read more and apply here.

• Awake Networks is an early stage network security and analytics startup that processes, analyzes, and stores billions of events at network speed. We help security teams respond to intrusions with super-human  efficiency and provide macroscopic and microscopic insight into the networks they defend. We're looking for folks that are excited about building systems that handle scale in a constrained environment. We have many open-ended problems to solve around stream-processing, distributed systems, machine learning, query processing, data modeling, and much more! Please check out our jobs page to learn more.

• Software Engineer (DevOps). You are one of those rare engineers who loves to tinker with distributed systems at high scale. You know how to build these from scratch, and how to take a system that has reached a scalability limit and break through that barrier to new heights. You are a hands on doer, a code doctor, who loves to get something done the right way. You love designing clean APIs, data models, code structures and system architectures, but retain the humility to learn from others who see things differently. Apply to AppDynamics


• Software Engineer (C++). You will be responsible for building everything from proof-of-concepts and usability prototypes to deployment- quality code. You should have at least 1+ years of experience developing C++ libraries and APIs, and be comfortable with daily code submissions, delivering projects in short time frames, multi-tasking, handling interrupts, and collaborating with team members. Apply to AppDynamics

Fun and Informative Events

• Join database experts from companies like Apple, ING, Instagram, Netflix, and many more to hear about how Apache Cassandra changes how they build, deploy, and scale at Cassandra Summit 2016. This September in San Jose, California is your chance to network, get certified, and trained on the leading NoSQL, distributed database with an exclusive 20% off with  promo code - Academy20. Learn more at CassandraSummit.org


• NoSQL Databases & Docker Containers: From Development to Deployment. What is Docker and why is it important to Developers, Admins and DevOps when they are using a NoSQL database? Find out in this on-demand webinar by Alvin Richards, VP of Product at Aerospike, the enterprise-grade NoSQL database. The video includes a demo showcasing the core Docker components (Machine, Engine, Swarm and Compose) and integration with Aerospike. See how much simpler Docker can make building and deploying multi-node, Aerospike-based applications!  

Cool Products and Services

• Dev teams are using LaunchDarkly’s Feature Flags as a Service to get unprecedented control over feature launches. LaunchDarkly allows you to cleanly separate code deployment from rollout. We make it super easy to enable functionality for whoever you want, whenever you want. See how it works.


• Kinsta provides high speed, automatically scalable managed WordPress hosting services for businesses large and small. All servers run on Google Cloud and all individual sites are completely compartmentalized using the latest LXD technology. All sites include powerful SSH access and tools like Git and WP-CLI are available out-of-the-box.


• Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.


• InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net


• VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.


• MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/


• aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com


• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.


• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below...

Click to read more ...

Stuff The Internet Says On Scalability For July 1st, 2016

Hey, it's HighScalability time:

If you can't explain it with Legos then you don't really understand it.

If you like this sort of Stuff then please support me on Patreon.

• 700 trillion: more pixels in Google's Satellite Map; 9,000km: length of new undersea internet cable from Oregon to Japan; 60 terabits per second: that undersea internet cable again; 12%: global average connection speed increase; 76%: WeChat users who spend more than 100RMB ($15) per month; 5 liters: per day pay in beer for Pyramid workers;  680: number of rubber bands it takes to explode a watermelon; 1,000: new Amazon services this year; $15 billion: amount Uber has raised; 7 million: # of feather on on each bird in Piper; 5.8 million: square-feet in Tesla Gigafactory; 2x: full-duplex chip could double phone-network data capacity; 


• Quotable Quotes:
  • @hyc_symas: A shame everyone is implementing on top of HTTP today. Contemporary "protocol design" is a sick joke.
  • @f3ew: Wehkamp lost dev and accept environments 5 days before launch. Shit happens.  48 hours to recovery. #devopsdays
  • Greg Linden: Ultimately, [serverless computing] this is a good thing, making compute more efficient by allowing more overlapping workloads and making it easier to move compute around. But it does seem like compute on demand could cannibalize renting VMs.
  • @viktorklang: What if we started doing only single-core chips with massive eDRAM on-package and PCI-E peer-writes MPI between? (Micro-blade machines?)
  • Robert Graham: Programmers fetishize editors. Real programmers, those who produce a lot of code, fetishize debuggers
  • @jasonhand: "Systems are becoming more ephemeral and we have to find a way to deal with it" (regarding monitoring) - @adrianco #monitorama
  • @aphyr: Queues *can* improve fault tolerance, but this only happens if they don't lose your messages. The only one I know of that doesn't is Kafka.
  • Tom Simon: There are, accordingly, two ways of reading books; but infinitely many ways to divide up the act of reading into two classes.
  • Puppet: High-performing IT organizations deploy 200 times more frequently than low performers, with 2,555 times faster lead times.
  • @benzobot: “The system scaled with the number of engineers - more engineers, more metrics.” #monitorama
  • fizx: You seem to have installation confused with administration. Off the top of my head you forgot security, monitoring, logging config, backups, handling common production issues such as splitbrains, write multiplication, garbage collection snafus, upgrades between versions with questionably compatible internal apis.
  • @Mark_J_Perry: This has to be one of the most remarkable achievements ever: Global Poverty Fell Below 10% for 1st Time in 2015
  • ewams: If you are a services company, he is right, you should be focusing on outcomes. But, if you can't tell me in 2-3 sentences what problem you are solving and how it benefits the customer you are doing it wrong.
  • @retrohack3r: Dance like nobody is watching. Encrypt like everyone is.
  • @GundersenMarius: ES6 + HTTP/2 + Service Workers + Bloom-filter = efficient module loading without bundlin mariusgundersen.net/module-pusher/ 
  • @timperrett: Distributed systems are about protocols, not implementations. Forget languages, protocols are everything.
  • steveblank: What’s holding large companies back?...companies bought into the false premise that they exist to maximize shareholder value – which said “keep the stock price high.” As a consequence, corporations used metrics like return on net assets (RONA), return on capital deployed, and internal rate of return (IRR) to measure efficiency. These metrics make it difficult for a company that wants to invest in long-term innovation.
  • Greg Linden: Like a lot of things at Amazon, this went through many stages of wrong before we got it right. As I remember it, this went through some unpleasant, heavyweight, and inefficient RPC (esp. CORBA) and pub-sub architectures before an unsanctioned skunkworks project built iquitos for lightweight http-based microservices 
  • @rawrthis:  "We all die." Except my legacy stack. That crap will live forever. #devopsdays
  • Jim Handy: The industry already has more than enough DRAM wafer capacity for the foreseeable future. Why is this happening?  The answer is relatively simple: the gigabytes per wafer on a DRAM wafer are growing faster than the market’s demand for gigabytes.
  • SwellJoe: A container doesn't consume complexity and emit order. The complexity is still in there; you still have to build your containers in a way that is replicable, reliable, and automatable. I'm not necessarily saying configuration management is the only way to address that complexity, but it does have to be addressed in a container-based environment.


• Deep Learning desires data, so if you want to build an AI that learns how to program this is how you would go about it, you would bring all the open source code into your giant, voracious, data crunching maw. Making open source data more available: Today, [GitHub] we're delighted to announce that, in collaboration with Google, we are releasing a collection of additional BigQuery tables to expand on the GitHub Archive...This 3TB+ dataset comprises the largest released source of GitHub activity to date. It contains activity data for more than 2.8 million open source GitHub repositories including more than 145 million unique commits, over 2 billion different file paths.


• In almost all cases, the single-threaded implementation outperforms all the others, sometimes by an order of magnitude, despite the distributed systems using 16-128 cores. Scalability! But at what COST? The paper is, at its heart, a criticism of how the performance of current research systems are evaluated. The authors focus on the field of graph processing, but their arguments extend to most distributed computation research where performance is a key factor. They observe that most systems are currently evaluated in terms of their scalability, how their performance changes as more compute resources are used, but that this metric is often both useless and misleading.


• hinkley: The older I get the more I feel like we're an accelerate version of the fashion industry. At least in fashion you can make an excuse that the design is thirty years old and most people don't remember the last time we did this. With software it's every six or seven. It's hard not to judge my peers for having such short memories. We were in the midst of one of these upheavals when I first started, and so I learned programming in that environment. It also means I have one more cycle than most people near my age. Now it all looks the same to me, and I understand those people who wanted to be more conservative. In fact I probably owe some people an apology.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Scaling Hotjar's Architecture: 9 Lessons Learned

Hotjar offers free website analytics so they have a challenging mission: handle hundreds of millions of requests per day from mostly free users. Marc von Brockdorff, Co-Founder & Director of Engineering at Hotjar, summarized the lessons they've learned in: 9 Lessons Learned Scaling Hotjar's Tech Architecture To Handle 21,875,000 Requests Per Hour.

In response to the criticism their architecture looks like a hot mess, Erik Näslund, Chief Architect at Hotjar, gives the highlights of their architecture:

• We use nginx + lua for the really hot code paths where python doesn't quite cut it. No language is perfect and you might have to break out of your comfort zone and use something different every now and then.
• Redis, Memcached, Postgres, Elasticsearch and S3 are all suitable for different kinds of data storage and we eventually needed them all to be able to query and store data in a cost effective way. We didn't start out using 5 different data-stores though...it's something that we "grew into".
• Each application server is a (majestic) monolith. Micro-services are one way of architecting things, monoliths are another - I'm still waiting to be convinced that one way is superior to the other when it comes to a smaller team of developers.

What have they learned?

Click to read more ...

How Facebook Live Streams to 800,000 Simultaneous Viewers

Fewer companies know how to build world spanning distributed services than there are countries with nuclear weapons. Facebook is one of those companies and Facebook Live, Facebook’s new live video streaming product, is one one of those services.

Facebook CEO Mark Zuckerberg: 

The big decision we made was to shift a lot of our video efforts to focus on Live, because it is this emerging new format; not the kind of videos that have been online for the past five or ten years...We’re entering this new golden age of video. I wouldn’t be surprised if you fast-forward five years and most of the content that people see on Facebook and are sharing on a day-to-day basis is video.

If you are in the advertising business what could better than a supply of advertising ready content that is never ending, always expanding, and freely generated? It’s the same economics Google exploited when it started slapping ads on an exponentially growing web.

An example of Facebook’s streaming prowess is a 45 minute video of two people exploding a watermelon with rubber bands. It reached a peak of over 800,000 simultaneous viewers who also racked up over 300,000 comments. That’s the kind of viral scale you can generate with a social network of 1.5 billion users.

As a comparison The 2015 Super Bowl was watched by 114 million viewers with an average 2.36 million on the live stream. On Twitch there was a peak of 840,000 viewers at E3 2015. The September 16th Republican debate peaked at 921,000 simultaneous live streams.

So Facebook is right up there with the state of the art. Keep in mind Facebook would have a large number of other streams going on at the same time as well.

A Wired article quotes Chris Cox, Facebook’s chief product officer, who said Facebook:

• Has more than a hundred people working on Live. (it started with ~12 and now there are more than 150 engineers on the project)

• Needs to be able to serve up millions of simultaneous streams without crashing.

• Need to be able to support millions of simultaneous viewers on a stream, as well as seamless streams across different devices and service providers around the world.

Cox said that “It turns out it’s a really hard infrastructure problem.”

Wouldn't it be interesting if we had some details about how that infrastructure problem was solved? Woe is we. But wait, we do!

 Federico Larumbe from Facebook’s Traffic Team, which works on the caching software powering Facebook’s CDN and the Global Load Balancing system, gave an excellent talk: Scaling Facebook Live, where he shares some details about how Live works.

Here’s my gloss on the talk. It’s impressive.

Origin Story

Click to read more ...

Stuff The Internet Says On Scalability For June 24th, 2016

Hey, it's HighScalability time:

A complete and accurate demonstration of the internals of a software system.

If you like this sort of Stuff then please support me on Patreon.

• 79: podcasts for developers; 100 million: daily voice calls made on WhatsApp; 2,000; cars Tesla builds each week; 2078 lbs: weight it takes to burst an exercise ball; 500 million: Instagram users; > 100M: hours watched per day on Netflix; 400 PPM: Antarctica’s CO2 Level; 2.5 PB: New Relic SSD storage; 


• Quotable Quotes:
  • Alan Kay: The Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free? The Web, in comparison, is a joke. The Web was done by amateurs.
  • @jaykreps: Actually, yes: distributed systems are hard, but getting 100+ engineers to work productively on one app is harder.
  • @adrianco: All in 2016: Serverless Architecture: AWS Lambda, Codeless Architecture: Mendix, Architectureless Architecture: SaaS
  • @AhmetAlpBalkan: "That's MS SQL Server running on Ubuntu on Docker Swarm on Docker Datacenter on @Azure" @markrussinovich #dockercon
  • Erik Darling: EVERYTHING’S EASY WHEN IT’S WORKING
  • @blueben: Bold claim by @brianl: Most of the best tech of the last 10 years has come out of work at Google. #VelocityConf
  • Joe: there is no such thing as a silver bullet … no magic pixie dust, or magical card, or superfantastic software you can add to a system to make it incredibly faster. Faster, better performing systems require better architecture (physical, algorithmic, etc.). You really cannot hope to throw a metric-ton of machines at a problem and hope that scaling is simple and linear. Because it really never works like that.
  • Eran Hammer: The web is the present and it’s a f*cking mess. Deal with it.
  • @etherealmind: If you believe in DevOps/NetOps you have to believe that leaving Europe is a difficult but better course of action. Small, fast & fluid
  • Sanfilippo: Redis is currently not good for data problems where write safety is very important. One of the main design sacrifices Redis makes in order to provide many good things is data retention. It has best effort consistency and it has a configurable level of write safety, but it’s optimized for use cases where most of the time, you have your data, but in cases of large incidents you can lose a little bit of it. 
  • David Smith: The best time you are ever going to have to make a new app is when there's a new iOS update. Go through the diffs. Go through the What's New? Find something that couldn't be possible before and make an app around that. 
  • @DanielEricLee: There was a timezone bug so I wrote a test and then the test failed because the CI runs in a different timezone and then I became a farmer
  • @jasongorman: Reminds me of someone I know who led a dev team who built something that won an award. None of team invited to awards bash. Only snr mgmt.
  • David Robinson: My advice to graduate students: create public artifacts
  • @cdixon: Because distributed clusters of commodity machines are more powerful.
  • @wmf: 128-port TORs have outrun the compute capacity of most racks, so putting two mini-TORs in 1U is a great idea.
  • msravi: I moved from AWS EC2 to Google Cloud a few days ago. Google really seems to have beaten AWS, at least in pricing and flexibility. On AWS (Singapore region) a 2-vCPU, 7.5G RAM instance costs $143/month (not including IOPS and bandwidth costs), while a similar one on GC works out to about $56/month. That's a massive difference. In addition, GC allows me to customize cores and RAM flexibly to a point, which is important for me.
  • Mobile Breakfast: What is clear that we will get rid of the classic circuit-switched technology and move to all IP networks fairly soon in the US.
  • Douglas Rushkoff: I think they all come down to how you can optimize your business or the economy for flow over growth; for the circulation of money rather than the extraction of money.
  • Alan Hastings~ [Species] that go into synchrony may be more subject to extinction because a single driver can trigger a collapse
  • @TechCrunch: More cars than phones were connected to cell service in Q1 http://tcrn.ch/28MLtmt  by @kristenhg
  • @docker: "Nobody cares about #containers, it's the application that matters!" - @solomonstre, CTO @Docker #DockerCon
  • @cmeik: The most commercially successful NoSQL database: Lotus Notes.
  • Brittany Fleit: behavior-based push results in open rates 800 percent higher than immediate blasts. Personalizing a message based on individual actions garners much more engagement.


• Dilbert, of course, nails it on AI.


• How will hackers be stopped from using Return-oriented programming (ROP) to execute privilege escalation attacks? ROP was created when "clever hackers realized that they could successively invoke snippets of code at the end of existing subroutines as 'proxies' to accomplish the work they needed done." Randomizing memory locations didn't stop them so Intel created new hardware magic called Control-flow Enforcement Technology. Intel added a new "ENDBRANCH" instruction and created a new "shadow stack". It turns out the immense power and beauty of the stack in von neumann architectures is also a great weakness. The story of life. Steve Gibson with an inspiring deep dive on CET in Security Now 565. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...