F1 and Spanner Holistically Compared

This aricle, F1: A Distributed SQL Database That Scales by Srihari Srinivasan, is republished with permission from a blog you really should follow: Systems We Make - Curating Complex Distributed Systems.

With both the F1 and Spanner papers out its now possible to understand their interplay a bit holistically. So lets start by revisiting the key goals of both systems.

Key Goals of F1′s design

• System must be able to scale up by adding resources
• Ability to re-shard and rebalance data without application changes
• ACID consistency for transactions
• Full SQL support, support for indexes

Spanner’s objectives

Click to read more ...

Ask HS: Is Microsoft the Right Technology for a Scalable Web-based System?

This question was asked over email and I thought a larger audience might want to take a whack at it.

I have a problem I’d like to have your view on. I’ve looked around a lot, and I haven’t found a definite answer. The question is this:

Is it true that for a scalable web-based system targeting millions of users (hopefully), using Microsoft technology(.Net/SQL Server) over open source technologies like python/ruby/php and mysql (mariadb) / postgresql will cost you more? Is there any justification for paying up for Microsoft licenses(OS, SQL Server, …)?

I am in charge of selecting the technology toolbox for a startup which is going to build a scalable public web platform. I’ve worked as a developer and database developer/admin (mainly as a DBA) using different platforms and technologies, but my main focus is on Microsoft technology. I’ve considered all other important factors for this decision, and at the end, I always come back to the question of money. When I finish developing the first stage of the system, and present it to possible investors to raise money and expand it, will it be a negative point(or even a deal breaker) to have a system developed on top of Microsoft technology stack?

Every time I decide to go with Microsoft, I ask myself “why no other major web-based system (other than stackoverflow) is built on Microsoft technology?”, and I’m back to square one.

Thanks for your time.

On HackerNews / On Reddit

Stuff The Internet Says On Scalability For October 4th, 2013

Hey, it's HighScalability time:

(Consumption as a Service: Amazon has 80 1.2 million square foot warehouses.)

• 100 megapixel cameras, taking 40 million pictures a second, creating of 1 petabyte of data every second:  Large Hadron Collider in Higgs search


• Quotable Quotes:
  • Tim Bell: You can't do transformation in transitional stages
  • @lleung: Also, we could really use a “high scalability” like blog for Enterprise IT to see the interesting stuff
  • xkcd: Functional programming combines the flexibility and power of abstract mathematics with the intuitive clarity of abstract mathematics.
  • @pbailis: That said, just because you can now build distributed transactions doesn't mean they'll be fast or work well during failures.


• Ever wonder how you can get those shoes and that tube of miracle cream from Amazon at the same time in the same box? Here's a fascinating look at The Amazon Warehouses chaotic storage system: Chaotic storage is like organized confusion. It’s an organic shelving system without permanent areas or sections. That means there is no area just for books, or a place just for televisions (like you might expect in a retail store layout).

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

RFC 1925 - The Twelve (Timeless) Networking Truths

The Twelve Networking Truths is one of a long series of timeless truths documented in sacred April Fools' RFCs. Though issued in 1996, it's no less true today.

It's hard to pick a favorite because they are all good. But if I had to pick, it would be:

Some things in life can never be fully appreciated nor understood unless experienced firsthand.

As we grow comfortable behind garden walls, clutching gadgets like lifelines and ideologies like shields, empathy is the true social network.

Network Working Group                                  R. Callon, Editor
Request for Comments: 1925                                          IOOF
Category: Informational                                     1 April 1996


                      The Twelve Networking Truths
  
        
Click to read more ...
  
      

Sponsored Post: Apple, Intechnica, Couchbase, MongoDB, Stackdriver, BlueStripe, Surge, Booking, Rackspace, AiCache, Aerospike, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• Intechnica is looking for Performance Architects, Performance Engineers, a Lead Automation Engineer, and a Solution Assurance Analyst. If making super-fast systems is your forte, send your CV with covering letter to careers@intechnica.co.uk.


• Apple is hiring a Senior Storage Engineer. Software Engineering Operations (SEO) is seeking an experienced storage engineer to join our team. This role will focus on designing, deploying and maintaining critical SAN and NAS storage solutions. Please apply here. 


• Stackdriver is looking for systems + cloud + dev + ops guru to serve as our liaison within the DevOps community. If you are passionate about monitoring and automation, enjoy working on open source, and are excited by the prospect of sharing your expertise with your peers, get in touch with us today! http://bit.ly/143ARmy


• We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.


• Apple Applications Architect. Apple's Customer Systems group, within the global Information Systems and Technology (IS&T) organization, helps design and implement all of the critical infrastructure that allows Apple to provide the best support in the world. Imagine what you could do here. Please apply here.


• Apple Software Engineer, Researcher. The OS X Analytics group is looking for a skilled software engineer. In this position you will develop tools to collect, process, and analyze data from millions of systems. Imagine what you could do here. Please apply here.


• LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online. 


• New Relic is looking for a Java Instrumentation Engineer, Java Scalability Engineer,  Distributed Systems Engineer and Android app engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter? 

Fun and Informative Events

• Surge - The Scalability & Performance Conference, presented by OmniTI, Sept. 12th-13th, features speakers from Joyent, Fastly, Dyn, Netflix, Linkedin and Amazon. Special, High Scalability Reader Rate: $50 off registration--through Sept. 10! Book hotel and get $50 off, from OmniTI. 

Cool Products and Services

• The leading technology companies use Couchbase as their NoSQL database. Download the free open-source version of Couchbase Server and make something awesome today.


• MongoDB Management Service (MMS) is a cloud-based suite of services for managing MongoDB deployments. In addition to monitoring and alerting, now you can seamlessly back up your MongoDB deployment to the cloud using using MMS. To get started with monitoring and backup, visit mms.mongodb.com.


• BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.


• AppDynamics is an easy-to-use application performance management solution that offers code-level insight into Java, .NET and PHP applications. Get the free trial.


• NEW! Aerospike 3 - Download FREE. Introducing the new Aerospike 3 database that builds off of Aerospike's legacy of speed, scale, and reliability, adding an extensible data model that supports complex data types, large data types, queries using secondary indexes, user defined functions (UDFs) and distributed aggregations using Stream UDFs for real-time data.


• The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.


• aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required . http://aicache.com/


• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.


• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit  http://www.appdynamics.com/freetrial.


• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.


• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

Expandability: Steve Wozniak's Biggest Success and Nolan Bushnell's Biggest Regret

When two titans of the early computing industry, Apple co-founder Steve Wozniak and Atari founder Nolan Bushnell, had a chance to sit down for a chat at the C2SV conference last Friday, we learned how a little foresight and luck can decide futures.  

Steve Wozniak's Success - Expandability 

Click to read more ...

Stuff The Internet Says On Scalability For September 27, 2013

Hey, it's HighScalability time:

(The WINLAB at Rutgers, with software defined radios tied into GENI.)

• 384 cores & 32TB of RAM: Oracle's SPARC M6 
• Quotable Quotes:
  • @jennyinc: 2003: "I replaced you with a set of very small shell scripts." 2013: "I replaced your scripts with a six-figure enterprise DevOps platform."
  • @tomdale: OH: “Redis is so fast, why don’t we replace RAM with Redis?”
  • @petrillic: OH "Promises/futures are the one-night stands of architectural constructs" nice #strangeloop
  • @TwitterEng: "Java and Scala let Twitter readily share and modify its enormous codebase across a team of hundreds of developers." 


• Lots of juicy numbers revealed at Structure:Europe: Netflix streams 114,000 years of video every month; Custom build Netflix boxes for its content-delivery network that contain between 100 and 150 terabytes of storage apiece; Netflix accounts for 35 percent of all web traffic; An unnamed social network recently purchased 85 petabytes of storage from EMC; Blogging platform WordPress now underpins nearly 20 percent of the world’s blogs; Felix Baumgartner’s record-breaking jump last year generated 8 million concurrent views on YouTube and consumed almost a quarter of the world’s bandwidth.


• 6 hostage negotiation techniques that will get you what you want. Useful for any project. After all, aren't meetings hostage situations? The steps: active listening, empathy, rapport, influence, behavioural change. Be aware, if other hostages are also using these techniques then the world will explode.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Great Open Source Solution for Boring HA and Scalability Problems

This is a guest post about how boring and repetitive HA and scalability problems can be solved via Open Source so you can focus on the interesting tasks. The post was written by Maarten Ectors, responsible for Cloud Strategy and Frank Mueller, a Juju Core developer, at Ubuntu/Canonical.

High-availability and scalability are exciting in general but there are certain problems that experts see over and over again. The list is long but examples are setting up MySQL clustering, sharding Mongo, adding data nodes to a Hadoop cluster, monitoring with Ganglia, building continuous deployment solutions, integrating Memcached / Varnish / Nginx,… Why are we reinventing the wheel?

At Ubuntu we made it our goal to have the community solve these repetitive and often boring tasks. How often have you had to set-up MySQL replication and scale it? What if the next time you just simply do:

Click to read more ...

Salesforce Architecture - How they Handle 1.3 Billion Transactions a Day

This is a guest post written by Claude Johnson, a Lead Site Reliability Engineer at salesforce.com.

The following is an architectural overview of salesforce.com’s core platform and applications. Other systems such as Heroku's Dyno architecture or the subsystems of other products such as work.com and do.com are specifically not covered by this material, although database.com is. The idea is to share with the technology community some insight about how salesforce.com does what it does. Any mistakes or omissions are mine.

This is by no means comprehensive but if there is interest, the author would be happy to tackle other areas of how salesforce.com works. Salesforce.com is interested in being more open with the technology communities that we have not previously interacted with. Here’s to the start of “Opening the Kimono” about how we work.

Since 1999, salesforce.com has been singularly focused on building technologies for business that are delivered over the Internet, displacing traditional enterprise software. Our customers pay via monthly subscription to access our services anywhere, anytime through a web browser. We hope this exploration of the core salesforce.com architecture will be the first of many contributions to the community.

Definitions

Let’s start with some basic salesforce.com terminology:

Click to read more ...

Stuff The Internet Says On Scalability For September 20, 2013

Hey, it's HighScalability time:

• $21 Billion: Google spend on datacenters
• Quotable Quotes:
  • @lintool: Jeff Dean at XLDB: Largest Google Bigtable cluster: ~100s PB data; sustained: 30M ops/sec; 100+ GB/s I/O (gmail?)
  • @neil_conway: On a quick skim, the most surprising thing about the F1 paper is how conventional of an MPP database it is.
  • @Kufat: PSA: 1024 bytes = 1 KB. If someone says it's "1 KiB," they are a Cylon, replicant, or shapeshifter, and must be destroyed forthwith.
  • @mikeleeorg: "If you only do things where you know the answer in advance, your company goes away." - Jeff Bezos
  • @andybritcliffe: Keynotes are always good for stats. Amazon are deloying software updates every 16 seconds #awssummit #impressive
  • @adrianco: last March numbers on US fixed access traffic was Netflix 32.25% and Youtube 17.11%, only takes two to get to 49.36%
  • @cpurdy: OH: "Objective-C is the lack of a type system of C smushed together with the s****y performance of Smalltalk."


• A Focus on Efficiency. A 70 page paper on how Internet.org will accomplish its goal of providing Internet access to the 5 billion people who don't currently have it. The paper is in two parts. The first part describes how Facebook will essentially become the platform supporting these new users. The idea is to create a 10x improvement reduction in the underlying costs of delivering data, and a 10x reduction of data usage by apps. The strategies are basically taken out of Facebook's playbook, so the paper is an excellent guide to all Facebook has developed over the years. The second part of the paper is Qualcomm presenting an overview of their plan to expand global wireless capacity by 1000 times. It's an audacious gambit to be sure.


• Another step in the Google vs Oracle divorce proceedings. Google Waves Goodbye To MySQL In Favor Of MariaDB. Using Golang over Java was another important step. It takes time for a supertanker to change direction, but it can be done.


• If you like ideas made pictures then you'll like Do you know Cassandra? For a comic the humor is so subtle I didn't get it, but it did talk about quorums and stuff.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...