Stuff The Internet Says On Scalability For October 28, 2011

You deserve a HighScalability today:

• S3: 566 Billion Objects, 370K requests/sec; Titan: 38,400-processor, 20-petaflop
• 1,000,000 daily users and no cache. Wooga flash game with 50K DB updates/second, Ruby backend. They hit an IO wall with MySQL at 1000 DB updates/sec. They needed more so they went with Redis. Not quite honest to say no cache was used as everything is RAM, but maybe that's the point. Use a lot of automation. Inactive users are archived. Moved away from EBS. 
• Scaling Twilio. Twilio has scaled traffic by more 100x over the past year, and expanded their server infrastructure from a few servers to 100′s running in the cloud. Core technologies: PHP, Python, Twisted/gevent, Java, Asterisk/FreeSwitch/JSR289, MySQL, and Redis. Core principles: Simplicity, Automation, Shipping, Empiricism, and Humbleness.

Want more of what the Internet has to say? Then please read more below...

Click to read more ...

Stuff The Internet Says On Scalability For September 30, 2011

You deserve a HighScalability today:

• Tumblr > Wikipedia
• Potent quotables:
  • @tokutek : Yelp generates close to 400 GB of compressed logs per day according to @petersirota of Amazon #Strataconf #BigData. More at From Under the Desk to the Cloud
  • @LHK_ITRG : Massive scalability: 80,000 users on a single AppSense server. I think that should do...
  • @solarce : OH: "Automation is a great way to distribute failure across the system" #surgecon
  • palominodb : #surgeconf - DataDog presenting on their "Data Mullet" All SQL in front, NoSQL party in the back. Classic.
  • Ryan Dahl : I hate almost all software
• Software Design Glossary. Apparently Kent Beck didn't get the memo, only algorithms matter now, software engineering is dead. In case you don't feel that way, Kent wrote a short glossary of important software design concepts. Also, Screaming Architecture by Bob Martin.

If you deserve even more Stuff the Internet has to Say on Scalability, please click below...

Click to read more ...

Stuff The Internet Says On Scalability For September 23, 2011

I'd walk a mile for HighScalability:

• 1/12th the World Population on Facebook in One Day; 1.8 ZettaBytes of data in 2011; 1 Billion Foursquare Checkins;  2 million on Spotify; 1 Million on GitHub; $1,279-per-hour, 30,000-core cluster built on EC2; Patent trolls cost .5 trillion dollars; 235 terabytes of data collected by the U.S. Library of Congress in April.
• Potent quotables:
  • @jstogdill : Corporations over protect low value info assets (which screws up collaboration) and under protects high value assets. #strataconf
  • @sbtourist : I think BigMemory-like approaches based on large put-and-forget memory cans, are rarely a solution to performance/scalability problems.
• 1 Million TCP Connections. Remember when 10K was a real limit and you had to build out boxes just to handle the load? Amazing. We don't know how much processing can be attached to these connections, how much memory the apps use, or what the response latency is to requests over these connections, but still, that's cool. Good discussion on Hacker News. Also, A Million-user Comet Application with Mochiweb, Part 3, Optimising TCP/IP connectivity: An exploratory study in network-intensive Erlang systems, Linux Kernel Tuning for C500k.

To read more of the amazing, surprising, and illuminating Stuff the Internet says on Scalability, please click below...

Click to read more ...

Stuff The Internet Says On Scalability For September 16, 2011

Between love and madness lies HighScalability:

• Google now 10x better: MapReduce sorts 1 petabyte of data using 8000 computers in 33 minutes; 1 Billion on Social Networks; Tumblr at 10 Billion Posts; Twitter at 100 Million Users; Testing at Google Scale: 1800 builds, 120 million test suites, 60 million tests run daily.
• From the Dash Memo on Google's Plan: Go is a very promising systems-programming language in the vein of C++. We fully hope and expect that Go becomes the standard back-end language at Google over the next few years. On GAE Go can load from a cold start in 100ms and the typical instance size is 4MB. Is it any wonder Go is a go? Should we expect to see Java and Python deprecated because Go is so much cheaper to run at scale?
• Walmart uses Muppet labor to power their real-time social shopping systems: You can’t do MapReduce computing every time (with every Tweet). You’ll die. How do you do it in real-time? We built MapUpdate, or what we call Muppet. We could map a huge amount of data and handle a huge firehose with little latency across millions of entities… We can monitor 100 million (items) at scale. That could be products, stores, anything. It’s the equivalent of MapReduce for fast data.
• Like humans, this AI software is always seeking relations. TextRunner produces facts by digesting 500 million web pages and billions of lines of text. Peter Norvig, director of research at Google: "The significance of TextRunner is that it is scalable because it is unsupervised. It can discover and learn millions of relations, not just one at a time. With TextRunner, there is no human in the loop: it just finds relations on its own."

For many more bon mots the internet has to say on scalability, please click and enter the down below...

Click to read more ...

Stuff The Internet Says On Scalability For September 9, 2011

Scale the modern way / No brush / No lather / No rub-in / Big tube 35 cents - Drug stores / HighScalability:

• GAE Serves 1.5 Billion Pages a Day
• Potent quotables:
  • @kendallmiller : The code changes I'm most proud of are the ones few people will ever see - like I just tripled the scalability of our session analysis.
  • @Kellblog : Heard: "Cassandra is more a system on which you build a DBMS than a DBMS itself."
  • @DDevine_au :  Ah dammit. I'm thinking of using a #NoSQL database. Down the rabbit hole I go.
• A comprehensive guide to parallel video decoding. Emeric Grange with a sweet explanation of the decoding process. 
• Node.js vs. Scala - "Scaling in the large". tedsuo tldrs it: in node, there is only one concurrency model.  A number of other platforms offer multiple concurrency models.  If you want access to one of those other models down the line, you will have to carve off that part of your application and rewrite it in another language. 

Please click below to drink a fine merlot:

Click to read more ...

Stuff The Internet Says On Scalability For September 2, 2011

Scale the modern way / No brush / No lather / No rub-in / Big tube 35 cents - Drug stores / HighScalability:

• 8868 Tweets per second during VMAs; Facebook: 250 million photos uploaded each day; Earth: 7 Billion People Strong
• Potent quotables:
  • @kevinweil : Wow, 8868 Tweets per second last night during the #VMAs. And that's just the writes -- imagine how many reads we were doing!
  • @tristanbergh : #NoSQL isn't cool, it's a working kludge of existing architectures, bowing to the current tech limits, not transcending them
• After a particularly difficult Jeopardy match, Watson asked IBM to make him a new cognitive chip so he could continue to kick human butt. The result, a newish chip design collocates data and computation. RAM and CPU are interconnected together. IBM explains. "One core contains 262,144 programmable synapses and the other contains 65,536 learning synapses." The win: lower power usage and better pattern recognition. On HackerNews. Watson is now said to happy, petting a virtual kitten that never leaves his virtual lap.
• What do we have here? Spotify, surreptitiously, is P2P? Crack investigation by Frank Catalano in Practical Nerd: The hidden price of “free”, says it's so. [I was] less pleasantly surprised to see that when Spotify wasn’t playing audio, it was using my network connection. A lot.

For more to know click below...

Click to read more ...

Stuff The Internet Says On Scalability For August 26, 2011

You may not scale often, but when you scale, please drink HighScalability:

• Facebook Hits 1 Trillion Pageviews; IBM builds 120 petabyte cluster out of 200,000 hard drives; Codecademy Surges To 200,000 Users, 2.1 Million Lessons Completed In 72 Hours; 
• Potent quotables:
  • @scankp : Turn up the bitrate to the data furnace honey. We’re expecting a frost tonight. We can use the extra cash, too
  • @gazzar_rj : The need for scale-out, automated, one storage pool +PB, linear performance scalability, the same admins" Sujal Patel 
  • @tweetimages :  We served 551,296,643 @twitter avatars for the month of July. Not bad!
• You thought a memcache service would be a great product idea for Amazon? So did Amazon.  They've released ElastiCache - a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud. Aren't you glad you waited on the idea? Sebastian Kreutzberger warns Amazon is making the same mistakes by hosting the service in only one Availability Zone. Also take a look at Sebastian's very well done Scalability for Dummies series.

For many more things the internet has to say on scalability, please read on...

Click to read more ...

Stuff The Internet Says On Scalability For August 19, 2011

You may not scale often, but when you scale, please drink HighScalability:

• Akamai: - 95,811 Servers, 1,000 Networks, 70 Countries.
• Quotably quotable quotes:
  • @segphault : Linus talking about the kernel's scalability. Beneficial to have one kernel used from embedded to high-end bc improvements span use cases.
  • @russferriday : Just completed a proposal for a rare bird data gathering system using #CouchDB *and* #Cassandra. Nice project. #NoSQL
  • @drelu : Oracle - everything is very convenient until it fails. #nosql
• How do you model Google+ circles with MongoDB? Some ideas in this Google Groups thread. More on MongoDB with Mat Wall explaining Why I Chose MongoDB for guardian.co.uk . 

Click to read more ...

Stuff The Internet Says On Scalability For August 12, 2011

Submitted for your scaling pleasure, you may not scale often, but when you scale, please drink us:

• Quotably quotable quotes:
  • @mardix : There is no single point of truth in #NoSQL . #Consistency is no longer global, it's relative to the one accessing it. #Scalability
  • @kekline : RT @CurtMonash: "...from industry figures, Basho/Riak is our third-biggest competitor." How often do you encounter them? "Never have" #nosql
  • @dave_jacobs : Love being in a city where I can overhear a convo about Heroku scalability while doing deadlifts. #ahsanfrancisco
  • @sufw : How can it be possible that Tagged has 80m users and I have *never* heard of it!?!
  • @EventCloudPro : One of my vacation realizations? Whole #bigdata thing has turned into a lotta #bighype - many distinct issues & nothing to do w/ #bigdata
• NoSQL as dynamic duos. NoSQL combinations - what works best? A common pattern seems to be Redis as a cache and Riak as the distributed backend.

Click to read more ...

Stuff The Internet Says On Scalability For August 5, 2011

Submitted for your beginning of the end of summer scaling pleasure: 

• Google Uses About 900,000 Servers; eBay deploys 100TB of flash storage
• The cloud isn't for closers. Another gaming startup pulls back from the cloud by Derrick Harris. Digital Chocolate is following the Zynga strategy of moving games into higher performing datacenter infrastructure once it becomes popular enough in the cloud to justify the primo stuff. We talked about this strategy in Zynga's Z Cloud - Scale Fast Or Fail Fast By Merging Private And Public Clouds. An architectural approach made all the more sensible with Amazon's new AWS Direct Connect service, which enables lower latency and higher bandwidth services by skipping the Internet and connecting directly to the AWS network. AWS Direct Connect FAQs. Amazon Virtual Private Cloud. 
• Quotes that are quotable:
  • @Werner : "If You Are Slow, You Can't Grow" - Peecho Architecture - scalability on a shoestring http://wv.ly/n4fpPC #aws #highscalability #peecho
  • @tcolar : Using neo4j .... After so many years of RDMS it's almost feels wrong having all that freedom to have an intuitive model. - almost !
  • @robertmclaws : Am I wrong in my opinion that #nosql is just an excuse for script kiddies not to learn how to build scalable systems? Or to not pay 4 MSSQL?
  • @b_erb : Quote of the day: "All computers wait at the same speed" – Dr. Thomas E. Bell

For much more Stuff the Internet has to say, please keep on reading on...

Click to read more ...