Stuff The Internet Says On Scalability For December 5th, 2014

Hey, it's HighScalability time:

InfoSec Taylor Swift is wise...haters gonna hate.

• 6 billion+: Foursquare checkins; 25000: allocs for every keystroke in Chrome's Omnibox
• Quotable Quotes:
  • @wattersjames: Pretty convinced that more value is created by networks of products in today's world than 'stacks'--'stack' model seems outdated.
  • @ChrisLove: WOW 70% of http://WalMart.com  traffic over the holidays was from a 'mobile' device! #webperf #webdevelopment #html5
  • @Nick_Craver: No compelling reason - we can run all of #stackexchange on one R610 server (4yr old) @ 5% CPU. #redis is incredibly efficient.
  • @jehiah: The ticker on http://bitly.com  rolled past 20 BILLION Bitlinks today. Made possible by reliable mysql clusters + NSQ.
  • @tonydenyer: micro services how to turn your ball of mud into a distributed ball of mud" #xpdaylon
  • @moonpolysoft: containers are the new nosql b/c all are dimly aware of a payoff somewhere, and are more than willing to slice each other up to get there.
  • @shipilev: Shipilev's First Law of Performance Issues: "It is almost always something very simple and embarrassing to admit once you found it"
  • Gérard Berry: We discovered that this whole hypothesis of being infinitely fast both simplified programming and helped us to be faster than others, which was fun.
  • @randybias: OH: “The religion of technology is featurism.”  [ brilliant observation ]
  • @rolandkuhn: ACID 2.0: associative commutative idempotent distributed #reactconf @PatHelland
  • @techmilind: @adrianco @wattersjames @cloudpundit Intra-region latency of <2ms is the killer feature. That's what makes Aurora possible.
  • @timreid: async involves a higher level of initial essential complexity, but a greatly reduced level of continual accidental complexity #reactconf
  • @capotribu: Docker Machine + Swarm + (proposed) Compose = multi-containers apps on clusters in 1 command #DockerCon
  • @dthume: "Some people say playing with thread affinity is for wimps. Those people don't make money" - @mjpt777 at #reactconf
  • @jamesurquhart: Reading a bunch of apparently smart people remain blind to the lessons of complexity. #rootcauseismerelyaclue
  • Facebook: the rate of our machine-to-machine traffic growth remains exponential, and the volume has been doubling at an interval of less than a year.


• In the US we tend to be practical mobile users instead of personal and social fun users. Innovation is happening elsewhere as is clearly shown in Dan Grover's epic exploration of Chinese Mobile App UI Trends: using voice instead of text for messaging; QR codes for everything; indeterminate badges to indicate there's something interesting to look at; a discover menu item that contains "changing menagerie of fun"; lots of app stores; using phone numbers for login, even on websites; QR code logins; chat as the universal UI; more damn stickers; each app has a wallet; use of location in ways those in the US might find creepy; tight integration with offline consumption; common usage of the assistive touch feature on the iPhone; cutesy mascots in loading and error screens; pollution widgets; full ad splash screen when an app starts; theming of apps. 


• Awesome analysis. A really deep dive with great graphics on Facebook's new network architecture. Facebook Fabric Networking Deconstructed: the new Facebook Network Fabric is in fact a Fat-Tree with 3-levels.


• Just a tiny thing. AMD, Numascale, and Supermicro Deploy Large Shared Memory System: The Numascale system, installed over the past two weeks, consists of 5184 CPU cores and 20.7 TBytes of shared memory which is housed in 108 Supermicro 1U servers connected in a 3D torus with NumaConnect, using three cabinets with 36 servers apiece in a 6x6x3 topology. Each server has 48 cores three AMD Opteron 6386 CPUs and 192GB memory, providing a single system image and 20.7 TB to all 5184 cores.


• Quit asking why something happened. The question that must be answered is how. The Infinite Hows (or, the Dangers Of The Five Whys). 


• What do customers want? Answers. Greg Ferro talks about how a company that hired engineers to answer sales inquiries doubled their sales. All people wanted were their questions answered. Once answered they would place an order. No complex time wasting sales cycle required. Technology has replaced the information gather part of the sales cycle. Customers already know everything about a product before making contact. Now what they need are answers.


• Docker Networking is as simple as a reverse 4 and a half somersault piked from a 3 metre board into a black hole.


• How is that Google Cloud Platform thingy working out? Pretty well says Aerospike. 1M Aerospike reads/sec for just $11.44/hour, on 50 nodes, with  linear scalability for 100% reads and 100% writes.


• Your bright human mind can solve a maze. So what? Fatty acid chemistry can too: This study demonstrates that the Marangoni flow in a channel network can solve maze problems such as exploring and visualizing the shortest path and finding all possible solutions in a parallel fashion. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

All employees should be limited only by their ability rather than an absence of resources.

James Hamilton hid a pearl of wisdom inside Why Renewable Energy (Alone) Won't Full Solve the Problem that I think is well worth prying out:

I’ve long advocated the use of economic incentives to drive innovative uses of computing resources inside the company while preventing costs from spiraling out of control.  Most IT departments control costs by having computing resources in short supply and only buying more resources slowly and with considerable care. Effectively computing is a scarce resource so it needs to get used carefully. This effectively limits IT cost growth and controls wastage but it also limits overall corporate innovation and the gains driven by the experiments that need these additional resources.

I’m a big believer in making effectively infinite computing resources available internally and billing them back precisely to the team that used them. Of course, each internal group needs to show the customer value of their resource consumption. Asking every group to effectively be a standalone profit center is, in some ways, complex in that the “product” from some groups is hard to quantitatively measure. Giving teams the resources they need to experiment and then allowing successful experiments to progress rapidly into production encourages innovation, makes for a more exciting place to work, and the improvements brought by successful experiments help the company be more competitive and better serve its customers.

I argue that all employees should be limited only by their ability rather than an absence of resources or an inability to argue convincingly for more. This is one of the most important yet least discussed advantages of cloud computing: taking away artificial resource limitations in support light-weight experimentation and rapid innovation. Making individual engineers and teams responsible to deliver more value for more resources consumed makes it possible encourage experimentation without fear that costs will rise without sufficient value being produced.

Auth0 Architecture - Running in Multiple Cloud Providers and Regions

This is a guest post by Jose Romaniello, Head of Engineering, at Auth0. There is a new version of this post! Check out Auth0 Architecture - Running In Multiple Cloud Providers And Regions (2018).

Auth0 provides authentication, authorization and single sign on services for apps of any type: mobile, web, native; on any stack.

Authentication is critical for the vast majority of apps. We designed Auth0 from the beginning with multipe levels of redundancy. One of this levels is hosting. Auth0 can run anywhere: our cloud, your cloud, or even your own servers. And when we run Auth0 we run it on multiple-cloud providers and in multiple regions simultaneously.

This article is a brief introduction of the infrastructure behind app.auth0.com and the strategies we use to keep it up and running with high availability.

Core Service Architecture

The core service is relatively simple:

• Front-end servers: these consist of several x-large VMs, running Ubuntu on Microsoft Azure.

• Store: mongodb, running on dedicated memory optimized X-large VMs.

• Intra-node service routing: nginx

All components of Auth0 (e.g. Dashboard, transaction server, docs) run on all nodes. All identical.

Multi-cloud / High AvailabilityMulti cloud architecture

Last week, Azure suffered a global outage that lasted for hours. During that time our HA plan activated and we switched over to AWS

• The services runs primarily on Microsoft Azure (IaaS). Secondary nodes on stand-by always ready on AWS.

• We use Route53 with a failover routing policy. TTL at 60 secs. The Route53 health check detects using a probe against primary DC, if it fails (3 times, 10 seconds interval) it changes the DNS entry to point to secondary DC. So max downtime in case of primary failure is ~2 minutes.

• We use puppet to deploy on every "push to master". Using puppet allows us to be cloud independent on the configuration/deployment process. Puppet Master runs on our build server (TeamCity currently).

• MongoDB is replicated often to secondary DC and secondary DC is configured as read-only.

• While running on the secondary DC, only runtime logins are allowed and the dashboard is set to "read-only mode".

• We replicate all the configuration needed for a login to succeed (application info, secrets, connections, users, etc). We don’t replicate transactional data (tokens, logs).

• In case of failover, there might might some logging records that are lost. We are planning to improve that by having a real-time replica across Azure and AWS.

• We use our own version of chaos monkey to test the resiliency of our infrastructure https://github.com/auth0/chaos-mona

Automated Testing

• We have 1000+ unit and integration tests.

• We use saucelabs to run cross-browser (desktop/mobile) integration tests for Lock, our JavaScript login widget.

• We use phantomjs/casper for integration tests. We test, for instance, that a full flow login with Google and other providers works fine.

• All these run before every push to production.

CDN

Our use case is simple, we need to serve our JS library and its configuration (which providers are enabled, etc.). Assets and configuration data is uploaded to S3. It has to support TLS on our own custom domain (https://cdn.auth0.com). We ended up building our own CDN.

• We tried 3 reputable CDN providers, but run into a whole variety of issues: The first one we tried when we didn't have our own domain for cdn. At some point we decided we needed our own domain over SSL/TLS. This cdn was too expensive if you want SSL and customer domain at that point (600/mo). We also had issues configuring it to work with gzip and S3. Since S3 cannot serve both version (zipped and not) of the same file and this CDN doesn't have content negotiation, some browsers (cough IE) don't play well with this. So we moved to another CDN which was much cheaper.

• The second CDN, we had a handful of issues and we couldn't understand the root cause of them. Their support was on chat and it took time to get answers. Sometimes it seemed to be S3 issues, sometimes they had issues on routing, etc.

• We decided to spend more money and we moved to a third CDN. Given that this CDN is being used by high load services like GitHub we thought it was going to be fine. However, our requirements were different from GitHub. If the CDN doesn't work for GitHub, you won't see an image on the README.md. In our case, our customers depends on the CDN to serve the Login Widget, which means that if it doesn't work, then their customers can't login.

• We ended up building our own CDN using nginx, varnish and S3. It's hosted on every region on AWS and so far it has been working great (no downtime). We use Route53 latency based routing.

Sandbox (Used to run untrusted code)

One of the features we provide is the ability to run custom code as part of the login transaction. Customers can write these rules and we have a public repository for commonly used rules.

• The sandbox is built on CoreOS, Docker and etcd.

• There is a pool of Docker instances that gets assigned to a tenant on-demand.

• Each tenant gets its own docker instance and there is a recycling policy based on idle time.

• There is a controller doing the recycling policy and a proxy that routes the request to the right container.

More information about the sandbox is in this JSConf presentation Nov 2014: https://www.youtube.com/watch?feature=player_detailpage&v=I4VkZ5H9PE8#t=7015 and slides: http://tjanczuk.github.io/about/sandbox.html

Monitoring

Initially we used pingdom (we still use it), but we decided to develop our own health check system that can run arbitrary health checks based on node.js scripts. These run from all AWS regions.

• It uses the same sandbox we developed for our service. We call the sandbox via an http API and send the node.js script to run as an HTTP POST.

• We monitor all the components and we also do synthetic transactions against the service (e.g. a login transaction).

If a health check fails we get notified through Slack. We have two Slack channels #p1 and #p2. If the failure happens 1 time, it gets posted to #p2. If it happens 2 times in a row it gets posted to #p1 and all members of devops get an SMS (via Twilio).

For detailed performance counters and response times we use statsd and we send all the metrics to Librato. This is an example of a chart you can create.

We also setup alerts based on derivative metrics (i.e. how much something grows or shrinks in a time period). For instance, we have one based on logins: if Derivate(logins) > X => Send an alert to Slack.

Finally, we have alerts coming from NewRelic for infrastructure components.

For logging we use ElasticSearch, Logstash and Kibana. We are storing logs from nginx and mongodb at this point. We are also parsing mongo logs using logstash in order to identify slow queries (anything with a high number of collscans).

Website

• All related web properties: the auth0.com site, our blog, etc. run completely separate from the app and runtime, on their own Ubuntu + Docker VMs.

Future

This is where we are going:

• We are moving to CoreOS and Docker. We want to move to a model where we manage clusters as a whole instead of doing configuration management over individual nodes. Docker helps also by removing some moving parts by doing image-based deployment (and be able to rollback at that level as well).

• MongoDB will be geo-replicated across DCs between AWS and Azure. We are testing latency.

• For all the search related features we are moving to ElasticSearch to provide search based on any criteria. MongoDB didn't work out well in this scenario (given our multi-tenancy).

Make Any Framework Suck Less With These 10 Insightful Lessons

Alexey Migutsky in 2 years with Angular has a lot to say about Angular, which I can't comment on at all, not being an Angular user. But burried in his article are some lessons for building better frameworks that obviously come from deep experience. Frameworks will always suck, but if you follow these lessons will your frameworks suck less? Yes, I think they will.

Here are Alexey's Lessons for framework (and metaframework) developers:

1. You should have as small as possible number on abstractions.
2. You should name things consistent with your "thought domain".
3. Do not mix several responsibilities in your components. Make fine-grained abstractions with well-defined roles.
4. Always describe the intention for your decisions and tradeoffs in your documentation.
5. Have a currated and updated reference project/examples.
6. You abstractions should scale "from bottom up". Start with small items and then fit them to a Composite pattern. Do not start with the question "How do we override it globally?".
7. Global state is pure evil. It's like darkness in the horror films - you never know what problems you will have when you tread into it...
8. The dataflow and data changes should be granular and localized to a single component.
9. Do not make things easy to use, make your components and abstractions simple to understand. People should learn how to do stuff in a new and effective way, do not ADAPT to their comfort zone.
10. Do not encode all good things you know in the framework.

Sponsored Post: Apple, Asana, Hypertable, Sprout Social, Scalyr, FoundationDB, AiScaler, Aerospike, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here. 
  • Sr. Software Engineer-iOS Systems. Do you love building highly scalable, distributed web applications? Does the idea of performance tuning Java applications make your heart leap? Would you like to work in a fast-paced environment where your technical abilities will be challenged on a day to day basis? Do you want your work to make a difference in the lives of millions of people? Please apply here.
  • Apple Pay - Site Reliability Engineer. You already know this… every issue counts. A single ticket can be the key to discovering an issue impacting thousands of people. And now that you’ve found it, you can’t wait to fix it. You also know that owning the quality of an application is about separating the signal from the noise. Finding that signal is what motives you. Now that you’ve found it, you’re next step is to role up the sleeves and start coding. As a member of the Apple Pay SRE team, you’re expected to not just find the issues, but to write code and fix them. Please apply here.
  • Senior Software Engineer -iOS Systems. This role demands the best and brightest engineers. The ideal candidate will be well rounded and offer a diverse skill set that aligns with key qualifications. Practical experience integrating with a diverse set of third-party APIs will also serve to distinguish you from other candidates. This is a highly cross functional role, and the typical team member's day to day responsibilities on the Carrier Services team. Please apply here. 

• Aerospike is hiring! Join the innovative team behind the world's fastest flash-optimized in-memory NoSQL database. Currently hiring for positions in our Mountain View, Calif., and Bangalore offices. Apply now! http://www.aerospike.com/careers


• As a production-focused infrastructure engineer at Asana, you’ll be the person who takes the lead on setting and achieving our stability and uptime goals, architecting the production stack, defining the on-call experience, the build process, cluster management, monitoring and alerting. Please apply here.


• Performance and Scale Engineer, Sprout Social, will be like a physical trainer for the Sprout social media management platform: you will evaluate and make improvements to keep our large, diverse tech stack happy, healthy, and, most importantly, fast. You'll work up and down our back-end stack - from our RESTful API through to our myriad data systems and into the Java services and Hadoop clusters that feed them - searching for SPOFs, performance issues, and places where we can shore things up. Apply here.


• UI Engineer– AppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.


• Software Engineer - Infrastructure & Big Data – AppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.

Fun and Informative Events

• Sign Up for New Aerospike Training Courses.  Aerospike now offers two certified training courses; Aerospike for Developers and Aerospike for Administrators & Operators, to help you get the most out of your deployment.  Find a training course near you. http://www.aerospike.com/aerospike-training/

Cool Products and Services

• Hypertable Inc. Announces New UpTime Support Subscription Packages. The developer of Hypertable, an open-source, high-performance, massively scalable database, announces three new UpTime support subscription packages – Premium 24/7, Enterprise 24/7 and Basic. 24/7/365 support packages start at just $1995 per month for a ten node cluster -- $49.95 per machine, per month thereafter. For more information visit us on the Web at http://www.hypertable.com/. Connect with Hypertable: @hypertable--Blog.
  


• FoundationDB launches SQL Layer. SQL Layer is an ANSI SQL engine that stores its data in the FoundationDB Key-Value Store, inheriting its exceptional properties like automatic fault tolerance and scalability. It is best suited for operational (OLTP) applications with high concurrency. Users of the Key Value store will have free access to SQL Layer. SQL Layer is also open source, you can get started with it on GitHub as well.


• Diagnose server issues from a single tab. The Scalyr log management tool replaces all your monitoring and analysis services with one, so you can pinpoint and resolve issues without juggling multiple tools and tabs. It's a universal tool for visibility into your production systems. Log aggregation, server metrics, monitoring, alerting, dashboards, and more. Not just “hosted grep” or “hosted graphs,” but enterprise-grade functionality with sane pricing and insane performance. Trusted by in-the-know companies like Codecademy – try it free!


• aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/


• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.


• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

A Flock of Tasty Sources on How to Start Learning High Scalability

This is a guest repost by Leandro Moreira.

When we usually are interested about scalability we look for links, explanations, books, and references. This mini article links to the references I think might help you in this journey.

DISCLAIMER:

You don’t need to have N machines to build/test a cluster/high scalable system, currently you can use Vagrant and up N machines easily.

THE REFERENCES:

Now that you know you can empower yourself with virtual servers, I challenge you to not only read these links but put them into practice.

• First of all, motivate yourself by watching this tutorial using nodejs + nginx + applying static caching + load balancing + testing, all this in 7 minutes.
• Add these words and their meaning to your vocabulary: scalability, failover, single point of failure (SPOF), sharding, replication and load balancing; even if you don’t understand them completely.
• In order to have a general overview and the reasons/whys about scalable systems, I strongly recommend you to read Scalable Web Architecture and Distributed Systems. This is a great introduction.
• After you get the general idea you can move on to understand how to use a load balancer and what decisions and problems you will face. And then you can try to run a haproxy and make it not a single point of failure too.
• Dare yourself to serve 3 million requests per second but for this task you’ll need to generate 3 million requests, fine tune your web server and finally scale and test it.
• Your application is already scalable, now you need to scale your databases. They are very important part of your application, here I recommend you to read at least how MongoDB scales with sharding and replication and Cassandra with its almost linear scalability and the ease of adding nodes to the cluster.
• Since your application and database are scalable and fault tolerant, it’s good to save your servers unnecessary workload and also make the responses to the user faster. Learn that a good request is the one that never reached the “real server”.
• Let’s assume we’re deploying the whole infrastructure within a single data center, now we have another SPOF. Since all servers are in the same space, some natural disaster might happen or even the simple power outages. Good news is that Cassandra have support to multiple data center out of the box and you can see how google face this issue. If your user is on Brazil, don’t make him travel longer than he needs and remember even with the best situation we still have latency.

Good questions to test your knowledge:

Click to read more ...

Stuff The Internet Says On Scalability For November 21st, 2014

Hey, it's HighScalability time:

Sweet dreams brave Philae. May you awaken to a bright-throned dawn for one last challenge.

•  80 million: bacteria xferred in a juicy kiss;
• Quotable Quotes:
  • James Hamilton: Every day, AWS adds enough new server capacity to support all of Amazon's global infrastrucrture when it was a $7B annual revenue enterprise.
  • @iglazer: What is the test that could most destroy your business model?  Test for that. @adrianco #defragcon
  • @zhilvis: Prefer decoupling over duplication. Coupling will kill you before duplication does by @ICooper #buildstufflt
  • @jmbroad: "Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful." ~George Box
  • @RichardWarburto: Optimisation maybe premature but measurement isn't.
  • @joeerl: Hell hath no version numbers - the great ones saw no need for version numbers - they used port numbers instead. See, for example, RFC 821,
  • JustCallMeBen: tldr: queues only help to flatten out burst load. Make sure your maintained throughput is high enough.
  • @rolandkuhn: «the event log is a database of the past, not just of the present» — @jboner at #reactconf
  • @ChiefScientist: CRUD is dead. -- @jboner #reactconf 
  • @fdmts: 30T of flash disks cabled up and online.  Thanks @scalableinfo!
  • monocasa: Immutable, statically linked, minimal system images running micro services on top of a hypervisor is a very old concept too. This is basically the direction IBM went in the 60's with their hypervisors and they haven't looked back.
  • Kiril Savino: Scaling is the process of decoupling load from latency.


• Perhaps they were controlled by a master AI? Google and Stanford Built Similar Neural Networks Without Knowing It: Neural networks can be plugged into one another in a very natural way. So we simply take a convolutional neural network, which understands the content of images, and then we take a recurrent neural network, which is very good at processing language, and we plug one into the other. They speak to each other—they can take an image and describe it in a sentence.


• You know how you never really believed the view in MVC was ever really separate? Now this is MVC. WatchKit apps run on the iPhone as an extension, only the UI component runs on the watch. XWindows would be so proud.


• Shopify shows how they Build an Internal Cloud with Docker and CoreOS: Shopify is a large Ruby on Rails application that has undergone massive scaling in recent years. Our production servers are able to scale to over 8,000 requests per second by spreading the load across 1700 cores and 6 TB RAM.


• Machine learning isn't just about creating humavoire AIs. It's a technology, like electricity, that will transform everything it affixes with its cyclops gaze. Here's a seemingly mundane example from Google, as discussed on the Green (Low Carbon) Data Center Blog. Google has turned inward, applying Machine Learning to its data center fleet. The result:  Google achieved from 8% to 25% reduction in its energy used to cool the data center with an average of 15%.  Who wouldn’t be excited to save an average of 15% on their cooling energy costs by providing new settings to run the mechanical plant? < And this is how the world will keep those productivity increases reaching skyward.


• Does anyone say "I love my water service"? Or "I love my garbage service"? Then why would anyone say "I love Facebook"? That's when you've arrived. When you are so much a part of the way things are that people don't even think of loving them or not. They just are. The Fall of Facebook. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

We are leaving 3x-4x performance on the table just because of configuration.

Performance guru Martin Thompson gave a great talk at Strangeloop: Aeron: Open-source high-performance messaging, and one of the many interesting points he made was how much performance is being lost because were aren't configuring machines properly.

This point comes on the observation that "Loss, throughput, and buffer size are all strongly related."

Here's a gloss of Martin's reasoning. It's a problem that keeps happening and people aren't aware that it's happening because most people are not aware of how to tune network parameters in the OS.

The separation of programmers and system admins has become an anti-pattern. Developers don’t talk to the people who have root access on machines who don’t talk to the people that have network access. Which means machines are never configured right, which leads to a lot of loss. We are leaving 3x-4x performance on the table just because of configuration.

We need to workout how to bridge that gap, know what the parameters are, and how to fix them.

So know your OS network parameters and how to tune them.

Related Articles

• Aeron: Do We Really Need Another Messaging System?
• Strategy: Exploit Processor Affinity For High And Predictable Performance
• 12 Ways To Increase Throughput By 32X And Reduce Latency By  20X
• Busting 4 Modern Hardware Myths - Are Memory, HDDs, And SSDs Really Random Access?
• Paper: Network Stack Specialization For Performance 
• Strategy: Use Linux Taskset To Pin Processes Or Let The OS Schedule It?
• Big List Of 20 Common Bottlenecks
• The Secret To 10 Million Concurrent Connections -The Kernel Is The Problem, Not The Solution
• Russ’ 10 Ingredient Recipe For Making 1 Million TPS On $5K Hardware
• 42 Monster Problems That Attack As Loads Increase
• 9 Principles Of High Performance Programs

Aeron: Do we really need another messaging system?

Do we really need another messaging system? We might if it promises to move millions of messages a second, at small microsecond latencies between machines, with consistent response times, to large numbers of clients, using an innovative design.  

And that’s the promise of Aeron (the Celtic god of battle, not the chair, though tell that to the search engines), a new high-performance open source message transport library from the team of Todd Montgomery, a multicast and reliable protocol expert, Richard Warburton, an expert on compiler optimizations, and Martin Thompson, the pasty faced performance gangster.

The claims are Aeron is already beating the best products out there on throughput and latency matches the best commercial products up to the 90th percentile. Aeron can push small 40 byte messages at 6 million messages a second, which is a very difficult case.

Here’s a talk Martin gave on Aeron at Strangeloop: Aeron: Open-source high-performance messaging. I’ll give a gloss of his talk as well as integrating in sources of information listed at the end of this article.

Martin and his team were in the enviable position of having a client that required a product like Aeron and was willing to both finance its development while also making it open source. So go git Aeron on GitHub. Note, it’s early days for Aeron and they are still in the heavy optimization phase.

The world has changed therefore endpoints need to scale as never before. This is why Martin says we need a new messaging system. It’s now a multi-everything world. We have multi-core, multi-socket, multi-cloud, multi-billion user computing, where communication is happening all the time. Huge numbers of consumers regularly pound a channel to read from same publisher, which causes lock contention, queueing effects, which causes throughput to drop and latency to spike. 

What’s needed is a new messaging library to make the most of this new world. The move to microservices only heightens the need:

As we move to a world of micro services then we need very low and predictable latency from our communications otherwise the coherence component of USL will come to rain fire and brimstone on our designs.

With Aeron the goal is to keep things pure and focused. The benchmarking we have done so far suggests a step forward in throughput and latency. What is quite unique is that you do not have to choose between throughput and latency. With other high-end messaging transports this is a distinct choice. The algorithms employed by Aeron give maximum throughput while minimising latency up until saturation.

“Many messaging products are a Swiss Army knife; Aeron is a scalpel,” says Martin, which is a good way to understand Aeron. It’s not a full featured messaging product in the way you may be used to, like Kafka. Aeron does not persist messages, it doesn’t support guaranteed delivery, nor clustering, nor does it support topics. Aeron won’t know if a client has crashed and be able to sync it back up from history or initialize a new client from history. 

The best way to place Aeron in your mental matrix might be as a message oriented replacement for TCP, with higher level services written on top. Todd Montgomery expands on this idea:

Aeron being an ISO layer 4 protocol provides a number of things that messaging systems can't and also doesn't provide several things that some messaging systems do.... if that makes any sense. Let me explain slightly more wrt all typical messaging systems (not just Kafka and 0MQ). 

One way to think more about where Aeron fits is TCP, but with the option of reliable multicast delivery. However, that is a little limited in that Aeron also, by design, has a number of possible uses that go well beyond what TCP can do. Here are a few things to consider: 

Todd continues on with more detail, so please keep reading the article to see more on the subject.

At its core Aeron is a replicated persistent log of messages. And through a very conscious design process messages are wait-free and zero-copy along the entire path from publication to reception. This means latency is very good and very predictable.

That sums up Aeron is nutshell. It was created by an experienced team, using solid design principles sharpened on many previous projects, backed by techniques not everyone has in their tool chest. Every aspect has been well thought out to be clean, simple, highly performant, and highly concurrent.

If simplicity is indistinguishable from cleverness, then there’s a lot of cleverness going on in Aeron. Let’s see how they did it...

Click to read more ...

Stuff The Internet Says On Scalability For November 14th, 2014

Hey, it's HighScalability time:

Spectacular rendering of the solar system to scale. (Roberto Ziche)

• 700: number of low-orbit satellites in a sidecar cheap internet; 130 terabytes: AdRoll ad data processed daily; 15 billion: daily Weather Channel forecasts; 1 million: AWS customers
• Quotable Quotes:
  • @benkepes: Each AWS data center has typically 50k to 80k physical servers. Up to 102Tbps provisioned networking capacity #reinvent
  • @scottvdp: AWS just got rid of infrastructure behind any application tier. Lambda for async distributed events, container engine for everything else.
  • @wif: AWS is handling 7 trillion DynamoDB requests per month in a single region. 4x over last year. same jitter. #reinvent
  • Philae: If my path was off by even half a degree the humans would have had to abort the mission.
  • Al Aho: Well, you can get a stack of stacks, basically. And the nested stack automaton has sort of an efficient way of implementing the stack of stacks, and you can think of it as sort of almost like a cactus. That's why some people are calling it cactus automata, at the time.
  • Gilt: Someone spent $30K on an Acura & LA travel package on their iPhone.
  • @cloudpundit: Gist of Jassy's #reinvent remarks: Are you an enterprise vendor? Do you have a high-margin product/service? AWS is probably coming for you.
  • @mappingbabel: Things coming out from the AWS #reinvent analyst summit - Amazon has minimum 3 million servers & lights up own globe-spanning fibre.
  • @cloudpundit: James Hamilton says mobile hardware design patterns are future of servers. Single-chip offerings, semiconductor-level innovation. #reinvent
  • @rightscale: RT @owenrog: AWS builds its own electricity substations simply because the power companies can't build fast enough to meet demand #reInvent
  • @timanderson: New C4 instances #reinvent up to 36 cores up to 16TB SSD
  • @holly_cummins: L1 cache is a beer in hand, L3 is fridge, main memory is walking to the store, disk access is flying to another country for beer. 
  • @ericlaw: Sample HTTP compression ratios observed on @Facebook: -1300%, -34.5%, -14.7%, -25.4%. ProTip: Don't compress two byte responses. #webperf
  • @JefClaes: It's not the concept that defines the invariants but the invariants that define the concept.


• It's hard to imagine just a few short years ago AWS did not exist. Now it has 1 million customers, runs 70 million hours of software per month, and their AWS re:Invent conference has a robust 13,500 attendees. Re:Invent shows if Amazon is going to be disrupted, a lack of innovation will not be the cause. The key talking point is that AWS is not just IaaS anymore, AWS is a Platform. The underlying subtext is lock-in. Minecraft-like, Amazon is building out their platform brick by brick. Along with GCE, AWS announced a Docker based container service. Intel designed a special new cloud processor for AWS, which will be available in a new C4 instance type. There's Aurora, a bigger, badder MySQL. To the joy of many EBS is getting bigger and faster. The world is getting more reactive, S3 is emitting events. With less fan fare are an impressive suite of code deployment and management tools. There's also a key management service, a configuration manager, and a service catalog. Most provocative is Lambda, or PaaS++, which as the name suggests is the ability to call a function in response to events. Big deal? It could be, though quite limited currently, only supporting Node.js and a few event types. You can't, for example, terminate a REST request. What it could grow in to is promising, a complete abstraction layer over the cloud so any sense of machines and locations are removed. They aren't the first, but that hardly matters.


• It's not a history of the Civil War. Or WWW I. Or the Dust Bowl. But An Oral History of Unix. Yes, that much time has passed. Interviewed are many names you'll recognize and some you've probably never heard of. A fascinating window into the deep deep past.


• No surprise at all. Plants talk to each other using an internet of fungus: We suggest that tomato plants can 'eavesdrop' on defense responses and increase their disease resistance against potential pathogen...the phantom orchid, get the carbon they need from nearby trees, via the mycelia of fungi that both are connected to...Other orchids only steal when it suits them. These "mixotrophs" can carry out photosynthesis, but they also "steal" carbon from other plants...The fungal internet exemplifies one of the great lessons of ecology: seemingly separate organisms are often connected, and may depend on each other. 


• How do you persist a 200 thousand messages/second data stream while guaranteeing data availability and redundancy? Tadas Vilkeliskis shows you how with Apache Kafka. It excels at high write rates, compession saves lots on network traffic, and a custom C++ http-to-kafka accommodates performance.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...