Stuff The Internet Says On Scalability For June 14, 2013

(Steve Gibson on Security Now with a plausible analysis of the tech behind PRISM)

• 27 billion: WhatsApp messages per day
• Quotable Quotes:
  • Richard Feinman: If Bill Gates walks into a bar, on average, everybody in the bar is a millionaire.
  • @giltene: Financial Programmers get paid by the CPU cycle. Web developers get paid by the developer cycle.
  • @johndmitchell: “It’s the I/O, stupid.” 
  • @PatrickMcFadin: More people registering at #cassandra13 No worries. Adding more nodes at the reg desk. 
• Google does it with science. Here's a list of Excellent Papers for 2012 from Googlers and friends. Most relevant for HS readers is a wildly inspiring Spanner: Google's Globally-Distributed Database. But you'll also see the influence of extracting knowledge from data to do subtle and interesting things. On that theme is Improving Photo Search: A Step Across the Semantic Gap. Google is doing hard hard things with seemingly small paybacks, but taken as whole it's clear what is being created is a generative ecosystem built around creating and applying knowledge. 
• When applications move to the browser it turns out we get the same problems as we had on servers. Effectively Managing Memory at Gmail scale describes how Gmail suffered from 100% CPU usage. As usual the problem was memory: learn how the Gmail team used Chrome DevTools to identify, isolate, and fix their memory problems.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Busting 4 Modern Hardware Myths - Are Memory, HDDs, and SSDs Really Random Access?

"It’s all a numbers game – the dirty little secret of scalable systems"

Martin Thompson is a High Performance Computing Specialist with a real mission to teach programmers how to understand the innards of modern computing systems. He has many talks and classes (listed below) on caches, buffers, memory controllers, processor architectures, cache lines, etc.

His thought is programmers do not put a proper value on understanding how the underpinnings of our systems work. We gravitate to the shiny and trendy. His approach is not to teach people specific programming strategies, but to teach programmers to fish so they can feed themselves. Without a real understanding strategies are easy to apply wrongly.  It's strange how programmers will put a lot of effort into understanding complicated frameworks like Hibernate, but little effort into understanding the underlying hardware on which their programs run.

A major tenant of Martin's approach is to "lead by experimental observation rather than what folks just blindly say," so it's no surprise he chose a MythBuster's theme in his talk Mythbusting Modern Hardware to Gain "Mechanical Sympathy." Mechanical Sympathy is term coined by Jackie Stewart, the race car driver, to say you get the best out of a racing car when you have a good understanding of how a car works. A driver must work in harmony with the machine to get the most of out of it. Martin extends this notion to say we need to know how the hardware works to get the most out of our computers. And he thinks normal developers can understand the hardware they are using. If you can understand Hibernate, you can understand just about anything.

The structure of the talk is to take a few commonly held myths and go all MythBusters on them by seeing if they are really true. Along the way there's incredible detail on how different systems work, far too much detail to gloss here, but it's an absolute fascinating talk. Martin really knows what he is talking about and he is a good teacher as well.

The most surprising part of the talk is the counter intuitive idea that many of the devices we think of as random access, like RAM, HDDs, and SSDs, effectively become serial devices in certain circumstances. A disk, for example, is really just a big tape that's fast. It's not true random access. Keep on reading to see why that is...

Click to read more ...

Sponsored Post: Apple, Two Sigma, Cendea, RAMP, Blurocket, Incapsula, Dow Jones, Surge, Rackspace, aiCache, Aerospike, Percona, ScaleOut, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• An exciting opportunity for a Software Engineer to join Apple's Messaging Services team. We build the cloud systems that power some of the busiest applications in the world. You'll have the opportunity to explore a wide range of technologies, developing the server software that is driving the future of messaging and mobile services. To apply please visit this URL. 
• Two Sigma is building our next generation research environment, and we're looking for a functional programmer with a passion for distributed computing. We're scaling machine learning and operations research to tens of thousands of CPUs. Please send qualifications to buildstuff@twosigma.com.
• Have strong LAMP skills and the ability to make systems really scale?  Our Client – a multibillion dollar global entity, has Retained us to find someone who can lead the Architecture of a portfolio of LAMP oriented companies. Interested applicants should email WAllen@Cendea.com.
• Blurocket is looking for smart and fun people to build its next generation ecommerce platform.  If creating scalable services is in your DNA, let us know! (Salary $250k+). Apply over at StackOverflow.
• Amazing things are happening at Dow Jones – help build the next generation News and Media platforms that serve the best journalism in the world. High-impact, passionate, and driven technologists thrive in our environment, building platforms that deliver trusted content that enlightens and inspires millions around the world.  Please apply online
• LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online. 
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• Surge - The Scalability & Performance Conference, presented by OmniTI is happening on Sept. 12th-13th. Special, High Scalability Reader Rate: $50 off registration--now through September 10! Book hotel and get $50 off, from OmniTI. FREE Puppet training w/registration, September 11.
• It's back! Join the MySQL Community at the annual Percona Live MySQL Conference and Expo in Santa Clara, April 22-25. This year's conference features an outstanding lineup of 92 speakers delivering 112 breakout sessions over three days! 
• Visit Budapest this July and learn from those who have been in the trenches of building highly scalable systems at RAMP.

Cool Products and Services

• With a 5-minute setup, Incapsula provides any website with protection against all known and emerging threats (Including spam, SQL Injections, DDoS and more), and acceleration through a global CDN, caching and optimization. See the power of Incapsula on your website with our Free trial.
• Install Aerospike Community Edition in 10 Minutes! To help you get up and running with our Community Edition as quickly as possible, we've created a tutorial video on AerospikeTV that provides guidance on installing your free database. Download our FREE Database and see how easy it is to install!
• The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.
• aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required . http://aicache.com/
• New Benchmark shows Aerospike nearly 10x Faster than the Competition. Thumbtack Technology YCSB Benchmark shows Aerospike nearly 10x faster than Cassandra, Couchbase and Mongodb. Read it now!
• ScaleOut Software. In-Memory Data Grids for the Enterprise. Download a Free Trial.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

The 10 Deadly Sins Against Scalability

In the moral realm there may be 7 deadly sins, but scalability maven Sean Hull has come up with Five More Things Deadly to Scalability that when added to his earlier 5 Things That are Toxic to Scalability, make for a numerologically satisfying 10 sins again scalability:

1. Slow Disk I/O – RAID 5 – Multi-tenant EBS. Use RAID 10, it provides  good protection along with good read and write performance. The design of RAID 5 means poor performance and long repair times on failure. On AWS consider Provisioned IOPS as a way around IO bottlenecks.
2. Using the database for Queuing. The database may seem like the perfect place to keep work queues, but under load locking and scanning overhead kills performance. Use specialized products like RabbitMQ and SQS to remove this bottleneck.
3. Using Database for full-text searching. Search seems like another perfect database feature. At scale search doesn't perform well. Use specialized technologies like Solr or Sphinx.
4. Insufficient Caching at all layers. Use memcache between your application and the database. Use a page like cache like Varnish between users and your webserver. Select proper caching options for your html assets.
5. Too much technical debt. Rewrite problem code instead of continually paying a implementation tax for poorly written code. In the long run it pays off.
6. Object Relational Mappers. Create complex queries that hard to optimize and tweak.
7. Synchronous, Serial, Coupled or Locking Processes. Locks are like stop signs, traffic circles keep the traffic flowing. Row level locking is better than table level locking. Use async replication. Use eventual consistency for clusters.
8. One Copy of Your Database. A single database server is a choke point. Create parallel databases and let a driver select between them.
9. Having No Metrics. Visualize what's happening to your system using one of the many monitoring packages.
10. Lack of Feature Flags. Be able to turn off features via a flag so when a spike hits features can be turned off to reduce load.

Stuff The Internet Says On Scalability For June 7, 2013

Hey, it's HighScalability time:

(Ever feel like everyone has already climbed your Everest?)

• Trillion Particles, 120,000 cores, and 350 TBs: Lessons Learned From a Hero I/O Run on Hopper
• Quotable Quotes:
  • @PenLlawen: @spolsky In my time as a scalability engineer, I’ve seen plenty of cases where optimisation was left too late. Even harder to fix.
  • @davidlubar: Whoever said you can't fold a piece of paper in half more than 7 times probably forget to unfold it each time. I'm up to 6,000.
  • deno:  A quick comparison of App Engine vs. Compute Engine prices shows that App Engine is at best 10x more expensive per unit of RAM.
  • Fred Wilson: strategy is figuring out what part of the market the company wants to play in, how it goes to market, and how it differentiates itself in the market it is about what you are going to do and importantly what you are not going to do
  • Elon Musk: SpaceX was able to achieve orders of magnitude savings in rockets. Instead of looking at what other rockets cost, they looked at the material cost (which is only 1-2% of total cost) which was small; clearly people were doing silly things on how the rockets were put together; for SpaceX, the savings came from efficiently putting the rocket together!


• Switched away from App Engine, couldn't be happier. Shows how GCE may start to cannibalize GAE even though they seem to be targeted at different market segments. When costs go up with usage on GAE there's a natural force pushing programmers to drop down an abstraction level and lower costs and get more power. In this case GCE cost about a third of what GAE cost with much more performance headroom and much better latency. What you give up is safety, GAE replicates and you have to do it by hand on GCE. It's always about tradeoffs.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Paper: Memory Barriers: a Hardware View for Software Hackers

It's not often you get so enthusiastic a recommendation for a paper as Sergio Bossa gives Memory Barriers: a Hardware View for Software Hackers: If you only want to read one piece about CPUs architecture, cache coherency and memory barriers, make it this one.

It is a clear and well written article. It even has a quiz. What's it about?

So what possessed CPU designers to cause them to inflict memory barriers on poor unsuspecting SMP software designers?

In short, because reordering memory references allows much better performance, and so memory barriers are needed to force ordering in things like synchronization primitives whose correct operation depends on ordered memory references.

Getting a more detailed answer to this question requires a good understanding of how CPU caches work, and especially what is required to make caches really work well. The following sections:

1. present the structure of a cache,
2. describe how cache-coherency protocols ensure that CPUs agree on the value of each location in memory, and, finally,
3. outline how store buffers and invalidate queues help caches and cache-coherency protocols achieve high performance.

We will see that memory barriers are a necessary evil that is required to enable good performance and scalability, an evil that stems from the fact that CPUs are orders of magnitude faster than are both the interconnects between them and the memory they are attempting to access.

A Simple 6 Step Transition Guide for Moving Away from X to AWS

If you just want to visit Rome and not go full on Cloud Native like Netflix, then Soundslice's Adrian Holovaty in Why I left Heroku, and notes on my new AWS setup provides a simple guide for helping make your first trip a good one.

First, let's dispose of why Soundslice left Heroku. The essence is because of various issues "Heroku lost my trust." YMMV, but once a fact, what do you do?

After a consultation with Scott VanDenPlas, former director of dev ops for the Obama reelection tech team, they came up a simple transition guide that I think is quite good and generally useful (full details in the original post): 

Click to read more ...

GOV.UK - Not Your Father's Stack

I'm not sure what I was expecting the stack GOV.UK used at launch to look like. Maybe some messenger owls and lots of cobwebs? But not so at all. So much not so I thought any organization looking at their own stack for ideas could learn something from the considered choices of others.

The diversity of technologies used was surprising. They use "at least five different programming languages, three separate database types, two versions of an operating system." Some may think of this as a weakness, but they think it a strength:

The reason we operate such a diverse ecosystem is that we are focused on solving real problems. Our first task is to understand the problem or need we are solving and then to choose the best tool for the job. If we restrict ourselves to moulding the need to the tools we already have, then we risk not solving the initial problem in the best way possible for the user. By restricting software diversity or enforcing rigid organisational standards on a project, there is a possibility of descending into a cargo cult, where we simply repeat the same patterns and mistakes in everything we make.

This "use the best tool no matter what" policy is outlined in a blog post Benefits of diversity. The only choice that wouldn't be found in a modern startup is the use of Skyscape as their cloud provider. I'm assuming this has to do with legal issues around data sovereignty as this is government site, but otherwise it's all straight out of standard modern web practice: monitoring, dashboards, continuous release, polyglot persistence, distributed source code control, etc. Good to see a government getting it.

What stack are they using? (it's a direct copy so feel free to read the original)

Click to read more ...

Stuff The Internet Says On Scalability For May 31, 2013

Hey, it's HighScalability time:

(resource consumption scales sub-linearly with population; economic output scales super-linearly)

• 4PB: Ancestry.com's House of Us
• Quotable Quotes:
  • @kellabyte: XBOX Live scaling up from 15,000 servers to 300,000 servers. That's some scale right there.
  • @giuseppegurgone: [...] that could work for small projects. "why do we worry about scalability on day 1?" because tomorrow it could be too late :)
  • Max Boot: American troops found their tactics and technology, still designed to defeat an opponent like the now defunct Red Army, woefully inadequate to deal with these new threats. In this sort of war, there were no flanks to turn, few bastions to storm, no capitals to seize. 
• Steve Jobs on the power of having 120 million credit cards on file: You need us more than we need you.
• Netflix: Dystopia as a Service. Adrian Cockcroft describes an out of Eden world where hardware and software by nature sin against man and how you can fortify yourself against them with righteous coding.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Google Finds NUMA Up to 20% Slower for Gmail and Websearch

When you have a large population of servers you have both the opportunity and the incentive to perform interesting studies. Authors from Google and the University of California in Optimizing Google’s Warehouse Scale Computers: The NUMA Experience conducted such a study, taking a look at how jobs run on clusters of machines using a NUMA architecture. Since NUMA is common on server class machines it's a topic of general interest for those looking to maximize machine utilization across clusters.

Some of the results are surprising:

Click to read more ...