Stuff The Internet Says On Scalability For July 29, 2011

Submitted for your end of July scaling pleasure: 

• YouTube: 3 billion videos viewed a day; 48 hours of footage uploaded every minute. 64 core Tilera chip.
• Google wants to be your CDN. They figure the only way to make the web faster...is to host it. Page Speed Service - Web Performance, Delivered. An eventually for pay service that caches your website and distributes it around the world. No cost information. Your speed may vary. See the longish list of limitations.
• Nobody said anything interesting on scalability this week! A disaster of non-quotable proportions. If I missed something, now is your chance. 
• Moving an Elephant: Large Scale Hadoop Data Migration at Facebook. Paul Yang describes the greatest westward expansion since the land bridge across the Bering Strait. It's a story of moving a 30PB Hadoop cluster from an over populated datacenter to the wide open spaces of a new continent. Unlike the early settlers, Facebook did not move the boxes over, that would disrupt service, they instead mirrored the data to their new datacenter. 

For a lot more Stuff the Internet said on scalability, please keep on reading below...

Click to read more ...

Stuff The Internet Says On Scalability For July 22, 2011

Submitted for your scaling pleasure: 

• Google's PageRank involves 500 million variables and 2 billion terms. SeaMicro Packs 768 Cores Into its Atom Server. Twitter: 1 Billion Items Delivered A Day Is Nice, Google+. We Do 350 Billion.
• Potent quotables:
  • @merv - Does elastic scalability matter? Ask Apple about 1 million e-transactions in a day. $30M worth of multi-gig downloads that must work.
  • @lmacvittie - Cloud has shifted the focus of scalability from applications to architecture.
  • @bartbohn - Love the phrase "anarchic scalability" by Roy Fielding
  • @iAjayMe - Programming: I have come to the conclusion, if you want to do something right the first time (scalability,performance) write it in C++

For a lot more Stuff the Internet says, please read more below...

Click to read more ...

Stuff The Internet Says On Scalability For July 8, 2011

Submitted for your scaling pleasure: 

• Facebook confirms 750 million users, sharing 4 billion items daily; Yahoo: 42,000 Hadoop nodes storing 180-200 petabytes; Formspring hits 25 million users.
• Zynga's Cadir Lee: It’s not the amount of hardware that matters. It’s the architecture of the application. You have to work at making your app architecture so that it takes advantage of Amazon. You have to have complete fluidity with the storage tier, the web tier. We are running our own data centers. We are looking more at doing our own data centers with more of a private cloud.
• Love the sensing making described by Hunch’s Infographic on their Taste Graph. 500 million people, 200 million items, 30 billion edges. 48 processors. 1 TB RAM. 
• Is MongoDB is the New MySQL? Stephen O'Grady thinks so using a worse is better argument: wide adoption by applications, enterprise inroads, simple feature set, and the number complainers. Who plays PostgreSQL in this movie?

For even more Stuff the Internet says, please read more below...

Click to read more ...

Stuff The Internet Says On Scalability For July 1, 2011

Submitted for your scaling pleasure: 

• Twitterers tweet 200 million tweets a day. Popular topics are eclectic, ranging from Swine Flu to Rebecca Black. Twitter has a really cool video on the global flow of tweets in the world. Worth watching. It looks like a rainbow arcing across the northern hemisphere.
• Amazon Cloud Now Stores 339 Billion Objects, more than doubling last years volume. 
• Quotable quotes for independence Alex:
  • n8foo: My fav part about the new #AWS pricing announcement - 500TB is the level where they say 'contact us'.
  • stevedekorte: Are the folks advocating FP for scalability unaware of the Von Neumann bottleneck?
  • lhazlewood: I don't think I've ever had a Love/Hate relationship like I've had with NoSQL. It's all awesome. And it all sucks.
• Free is good says Amazon, No Inbound Data Transfer Fees and slightly lower outbound fees. Will Amazon make up for this with voume? Maybe S3 sales (which are not cheaper)? it's true, most sites upload little data compared to their outbound traffic, but for some use cases it's a real win, email, scraper bots, chat, video, and backup.

For lots more Stuff the Internet said, please read below...

Click to read more ...

Stuff The Internet Says On Scalability For June 24, 2011

Submitted for your scaling pleasure: 

• Achievements:
  • Watson uses 10,000's of watts, the computer between the ears uses 20. With only 200 million pages and 2TB of data, Watson is BigInsights, not BigData.
  • That Google is pretty big: 1 billion unique monthly visitors
  • tweetimages: We peaked at 22m avatars yesterday. Bandwidth peaked at 9GB of @twitter avatars in a single hour.
  • Foursquare Surpasses 10 Million Users 
  • Reddit Hits 1.2B Monthly Pageviews, More Than Doubles Its Engineering Staff
  • Twitter: 185 million tweets are posted daily;  1.6 billion search queries daily; indexing latency is less than 10 seconds. 
• Quotable quotes:
  • skr: OH: "people wait their whole lives for a situation where they can use bloom filters"
  • joeweinman: @Werner at #structureconf : as of Nov 10, 2010, all Amazon.com traffic was served from AWS. <-- The child surpasses the parent.
  • bbatsov: A compiled language does not scalability make -- Yoda
  • swredman: If i read the marketing buzzwords 'scalability' or leverage your data' one more time, gonna lose my sh*t.
  • ipeirotis: Most tasks are too arbitrary to even decompose in atomic steps, while handling quality, cost, scalability and interactions.
  • ArmonDadgar: Some people. You had to use NoSQL for 2 TPS and a DB that can fit on my iPhone? This is some serious #BigData.
  • aphethean: For a guy who started Dev life programming applications on hierarchical databases #nosql looks remarkably familiar
  • "Software is an entropic system whose arrow of time flows in the direction of failure, aided and abetted by human bullsh*t"
  • bstg: Wonder if the best way to improve insights from #bigdata isn't better analytics, but a fundamental change in the way we capture it? #in
  For much more on what the Internet said this week, please read more below...

  Click to read more ...

Stuff The Internet Says On Scalability For June 10, 2011

Submitted for your scaling pleasure: 

• Achievements:
  • Every day, Amazon Web Services adds enough new capacity to support all of Amazon.com’s global infrastructure through the company’s first 5 years, when it was $2.7 billion annual revenue. From Cloud Computing Is Driving  Infrastructure Innovation by James Hamilton. Where's is all that money be spent? Facilities, servers, power, and popcorn.
  • Evernote hits 10 million users. StackExchange hits 1 million users. No lawsuits expected in either case. 
  • Neural waves of brain. The brain's waves drive computation, sort of, in a 5 million core, 9 Hz computer.
• Quotes of a quotable essence:
  • robinduckett: FACT: You are not a web developer if you need third party services which handle scalability so you can "focus on the programming".
  • Twitter’s Bain: Facebook May Have More Scale, We Have More Engagement
  • shervin: Fallibility without malleability sheds scalability.
  • uisdans: Fat client/server is over. We're moving from #apps #social web #iaas to a #nui #richapp #bigdata #paas spanning the private/public cloud
• Ex-Google Engineer Says the Company's Software Infrastructure is Obsolete. Arguments don't follow IMHO. Creating a global infrastructure in the large is a very different goal that following the latest trends for personal projects. Though it is no doubt limiting to have to use this infrastructure for everything.
• Datacenters are becoming their own technological niche. Why use Internet tech in datacenter? TCP becomes Data Center TCP (DCTCP) - an enhancement to the TCP congestion control algorithm for data center networks.

To see much more of Stuff the Internet says, please read more below...

Click to read more ...

Stuff The Internet Says On Scalability For June 3, 2011

Submitted for your scaling pleasure: 

• Twitter indexes an average of 2,200 TPS (peek is 4x that) while serving 18,000 QPS (1.6B queries per day). eBay serves 2 billion page views every day requiring more than 75 billion database requests.
• Quotable Quotes:
  • Infrastructure is adaptation --Kenneth Wright, referencing reservoir building by the Anasazi
  • MattTGrant: You say: "Infinite scalability" - I say: "fractal infrastructure"
• Like the rich, More is different, says Zillionics. Large quantities of something can transform the nature of those somethings. Zillionics is a new realm, and our new home. The scale of so many moving parts require new tools, new mathematics, new mind shifts.  Amen.
• Data mine yourself says the Quantified Self. All that jazz about monitoring and measuring services to continually improve them-- that works for you too! You may not be a number, but self-numbers are a path towards being all you can be. Motivated by this same spirit, some time ago I published an empirical process control method for weight control centered on creating and using a feed back system. More at: The 10 Designer Principles for Controlling Your Weight and the The Designer Way. 

To see much more of Stuff the Internet says, please read more below...

Click to read more ...

Stuff The Internet Says On Scalability For May 27, 2011

Submitted for your scaling pleasure: 

• Good idea: Open The Index And Speed Up The Internet. SmugMug estimates 50% of their CPU is spent serving crawler robots. Having a common meta-data repository wouldn't prevent search engines from having their own special sauce. Then the problem becomes one of syncing data between repositories and processing change events. A generous soul could even offer a shared MapReduce service over the data. Now that would speed up the internet.
• Scaling Achievements: YouTube Sees 3 Billion Views per Day; Twitter produces a sustained feed of 35 Mb per second; 
• Quotes that are Quotably Quotable:
  • eekygeeky: When cloud computing news is slow? Switch to "big data"-100% of the vaguery, none of the used-up, mushy marketing feel!
  • singhns: an API is like diamonds, a huge range of value based upon four c's - clarity, consistency, convenience and channel
  • dotmanish: "The sex appeal of company comes from the scalability" - Rajul at #opengurukul
• Twitter improves the Ruby runtime 2.7x the speed of standard Ruby 1.8. 

Read much more of Stuff the Internet Says by clicking the down below...

Click to read more ...

Stuff The Internet Says On Scalability For May 20, 2011

Submitted for your reading pleasure on this beautiful morning: 

• Group Decision Making in Honey Bee Swarms. In distributed computing systems nodes reach a quorum when deciding what to do as a group. It turns out bees also use quorum logic when deciding on where to nest! Bees do it a bit differently of course:  A scout bee votes for a site by spending time at it, somehow the scouts act and interact so that their numbers rise faster at superior sites, and somehow the bees at each site monitor their numbers there so that they know whether they've reached the threshold number (quorum) and can proceed to initiating the swarm's move to this site. Ants use similar mechanisms to control foraging. Distributed systems may share common mechanisms based on their nature as being a distributed system,  the components may not matter that much.
• Fire! Fire!  Brent Chapman shows how to put that IT fire out in Incident Command for IT: What We Can Learn from the Fire Department. 
• Scale Fail (part 1). Josh Berkus warns against hopping on the trendy train: Scaling an application is all about management of resources and administrative repeatability. Use data so that you work on real unknowns instead of unknown unknowns. And blocking processes, just don't do it.
• Quotable quotes:
  • @Sri_few_words: Every 600 phones, means a new server in data center" - Cloud Computing being driven strongly by smartphones; tablets

There's a lot more Stuff the Internet says, don't be left out, read more below...

Click to read more ...

Stuff The Internet Says On Scalability For May 6th, 2011

Submitted for your reading pleasure...Hi Mom!...

• We don't need no stinking servers says the W3C. This Could be Big: Decentralized Web Standard Under Development by W3C by Marshall Kirkpatrick. Browsers talking to directly to other browsers. Marshall is right, this could be very big.
• Quotable Quotes for Pi Alex:
  • @eric_brewer The Amazon outage & CAP theorem: http://aws.amazon.com/message/65648/ (partition is the root cause)
  • @kylecordes A problem with cloud hosting (EC2) is that it brings the problems of scalability to systems that *don't* need scalability.
  • @virtualpete Last month everyone was a nuclear physicist. Today everyone is a web scalability architect
  • @jfelipe We cannot overlook migration/federation issues (scalability) in cloud tech: open standards are a plus compared 2 closed (Amazon)
  • @lapsu Stored procedures aren't so bad if you write them in Javascript & they do MapReduce. That makes them cool. #nosql
• Adapteva wants your tablet and phone to have 64 processors. What can you do with all that power? Process the world around you in real-time. Analyzing sound, video, making sense of it, embedding you in a data enchanted world. That's one option anyway.

For a lot more Stuff the Internet Says please read below...

Click to read more ...