Stuff The Internet Says On Scalability For September 7th, 2018

Hey, it's HighScalability time:

Get antsy waiting 60 seconds for a shot? Imagine taking over 300,000 photos over 14 years, waiting for Mount Colima to erupt. Sergio Tapiro studied, waited, and snapped.

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• 3.5 Pflop/s: fully synchronous tensorflow data-parallel training; 3.3 million: new image/caption training set; 32,408,715: queries sent to Pwned Passwords; 53%: Memory ICs Total 2018 Semi Capex; 11: story Facebook datacenter prison in Singapore; $740,357: ave cost of network downtime; 


• Quotable Quotes:
  • @BenedictEvans: Recorded music: $18 billion. Cars: $1 trillion. Retail: $20 trillion.
  • @JoeEmison: Lies that developers tell (themselves): (1) This is the best stack/IaaS for us to use [reality: I know it and want to start now] (2) DevOps doesn’t matter until scaling [you’ll spend 30% of your time dealing with ops then] (3) We’ll just rebuild it if we get traction [hahahaha]
  • @sapessi: Lambda simplifies concurrency at the frontend, enforcing one event per function at a time. This makes it easy to reason about complex distributed system. Once inside the function, there's nothing wrong with multi-threading to do the work as efficiently as possible
  • @JonErlichman~ Comparing valuations: Amazon: $1 trillion; Combined $960 billion: Best Buy Macy’s Target Costco Nike  Sears  Home Depot Starbucks McDonald’s Barnes & Noble J.C. Penney Dollar Tree Office Depot Nordstrom Kroger Kohls
  • Kevin Kelly: The biggest invention in Silicon Valley was not the transistor but the start-up model, the culture of the entrepreneurial start-up.
  • Dare Obasanjo: Amazon made $2.2B from search ads last quarter. This is twice as much as Snapchat ($262M) and Twitter ($711M) combined. However still far from Google ($28B) and Facebook ($13.2B). Expect next step is for Amazon ads to start show scale  
  • Matthew Dillon: This is *very* impressive efficiency.  Whod a thought that one would be
    able to run an 8-core/16-thread CPU at full load at only 85W and still reap
    most of the benefit in a memory-heavy workload! This is *very* impressive efficiency.  Whod a thought that one would beable to run an 8-core/16-thread CPU at full load at only 85W and still reapmost of the benefit in a memory-heavy workload!
  • @laurencetratt: When we set out to look at how long VMs take to warm up, we didn’t expect to discover that they often don’t warm up. But, alas, the evidence that they frequently don’t warm up is hard to argue with. Of course, in some cases the difference to performance is small enough that one can live with it, but it’s often bad enough to be a problem. We fairly frequently see performance get 5% or more worse over time in a single process execution. 5% might not sound like much, but it’s a huge figure when you consider that many VM optimisations aim to speed things up by 1% at most. It means that many optimisations that VM developers have slaved away on may have been incorrectly judged to speed things up or slow things down, because the optimisation is well within the variance that VMs exhibit. 
  • Eli Bendersky: Just for fun, I rewrote the same benchmark in Go; two goroutines ping-ponging short message between themselves over a channel. The throughput this achieves is dramatically higher - around 2.8 million iterations per second, which leads to an estimate of ~170 ns switching between goroutines [3]. Since switching between goroutines doesn't require an actual kernel context switch (or even a system call), this isn't too surprising. For comparison, Google's fibers use a new Linux system call that can switch between two tasks in about the same time, including the kernel time.
  • @ben11kehoe: So instead of a processor [for @iRobot] that is maxed out by the features it supports at launch (which is the route to the lowest build cost), we've got headroom to keep pace even with software features we're planning for the generation of robots after this.
  • It was a quiet week this week. Just a few more quotes...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Sponsored Post: NationBuilder, Twitch, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr, MemSQL

Who's Hiring? 

• NationBuilder — if you’re a systems engineer, SRE or DevOps focused developer and have been looking for a place where you can help other people while still working in tech? We can give that opportunity. Please apply here. 


• Twitch's commerce team in San Francisco is looking to hire senior developers to keep up with rapidly increasing demand for our Subscriptions and Payment platform. Engineers will be tasked with building new products and features to solve business and ecommerce challenges as we're dealing with engaging problems at a massive scale and will create solutions that impact millions of people around the world. Apply here. 


• Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

• For heads of IT/Engineering responsible for building an analytics infrastructure, Etleap is an ETL solution for creating perfect data pipelines from day one. Unlike older enterprise solutions, Etleap doesn’t require extensive engineering work to set up, maintain, and scale. It automates most ETL setup and maintenance work, and simplifies the rest into 10-minute tasks that analysts can own. Read stories from customers like Okta and PagerDuty, or try Etleap yourself.

• Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

• Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.


• MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.


• Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

---

Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to see if they can help you scale your career faster.

---

The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today. 

---

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Stuff The Internet Says On Scalability For August 31st, 2018

Hey, it's HighScalability time:

This mind blowing creation is from John Williamson. It's the first million integers, represented as binary vectors indicating their prime factors, laid out with UMAP. No, I really have no idea what that means either, but it did make me consider that our universe could be created by an algorithm. What are the wiggly cycles on the periphery? Groups of numbers that share a minimum amount of prime factors, further out groups are numbers that have increasing amounts of shared prime factors. So the primes are at the core, ungrouped as they have no prime factors to use to join groups. Primorials should be furthest out.

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• 800 million: internet users in China; 19,000: hours of audio from historic Apollo 11 mission; 2: mph Frodo walking pace to Mordor; 67%: daily trade volume faked; 60 TFLOPS: Data's theoretical peak, which is 8 NVIDIA Voltas, which you can rent on ec2 for $25 an hour; 7nm: an expense too far for GlobalFoundries; 98%: atoms replaced every year in human body; 150 billion: hours of Fitbit heart data; 4: times per second human brain oscillates in and out of focus; 27.9%: DNS requests over UDP from China to Google Public DNS are intercepted; 


• Quotable Quotes:
  • Chris Hadfield: the best antidote to fear is competence.
  • @kellabyte: I’m making fun of GraphQL but I like it so far. I’m just having fun with it because it’s literally re-learning a new way to do the same thing a group of other people solved some of the same problems almost 20 years ago. But like I said, not a hater. I like GraphQL so far.
  • Corey Quinn: Kubernetes is Named After the Greek God of Spending Money on Cloud Services
  • @QuinnyPig: #SREcon @aaronblohowiak "Once upon a time when the stars were young we had an ELB outage" ...because ELBs are incredibly durable right up until "Oh shit! Load!" at which point TCP terminates on the floor.
  • @GossiTheDog: This is wild - the White House are rolling back cybersecurity baseline standards for government (saying they will issue new ones in the future), and ditching deployment of DNSSEC and IPv6 (as mandated under Obama admin) saying they are already in place (they aren’t).
  • @nicolefv: Outsourcing is a bad idea. (shocker) Low performers are almost 4x as likely to use functional outsourcing. Read about it and some of the consequences (search for “Misguided performers”) in the 2018 Accelerate State of DevOps Report 
  • Bernard Golden: I see Snowball Edge with EC2 as an AWS initiative designed to reduce the friction of migrating data into AWS with the ultimate goal of enabling applications to migrate to AWS, not a way to run AWS applications on-prem. In other words, this new Snowball variant isn’t about processing data on-prem, it’s about making Snowball a better on-ramp to AWS.
  • @allspaw: In "managing workload there are only four coping strategies: (1) shed load, (2) do all components but do each less thoroughly, thereby, consuming fewer resources, (3) shift work in time to lower workload periods, (4) recruit more resources." (Woods and Hollnagel, 2006)
  • Stacy Horn: In those days journalists wrote that I started Echo to provide a safe place for women on the Net. Bite me. I wanted to get more women on Echo to make it better.
  • @michael_beh: there are so many price comparisons between #serverless and non-serverless deployments missing costs for multi-AZ, multi-region, operational OS mgmt, load balancer, pre-prod envs, etc. All of them are part of GBsec price in serverless. It's too often an apples/oranges comparison.
  • Andy Hertzfeld: Steve got mad. He goes, “The Apple II was going to be dead. Your OS is going to be obsolete before it’s finished!” And then he said something like, “The Macintosh is the future of Apple and you are going to start now.” I just wanted one day, really. It was Thursday afternoon. I said “Monday” because Monday seems to be a good time to make a new start. But he goes, “No! You’re going to start on it now!” And he went and he pulled the plug on my Apple II, which had my code not saved. I was right in the middle of working on it. He just yanks the plug out, and then without pausing he picks up the Apple II and starts walking away.
  • More more more...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Auth0 Architecture: Running In Multiple Cloud Providers And Regions

This is article was written by Dirceu Pereira Tiegs, Site Reliability Engineer at Auth0, and originally was originally published in Auth0.

Auth0 provides authentication, authorization, and single sign-on services for apps of any type (mobile, web, native) on any stack. Authentication is critical for the vast majority of apps. We designed Auth0 from the beginning so that it could run anywhere: on our cloud, on your cloud, or even on your own private infrastructure.

In this post, we'll talk more about our public SaaS deployments and provide a brief introduction to the infrastructure behind auth0.com and the strategies we use to keep it up and running with high availability. 

A lot has changed since then in Auth0. These are some of the highlights:

• We went from processing a couple of million logins per month to 1.5+ billion logins per month, serving thousands of customers, including FuboTV, Mozilla, JetPrivilege, and more.

• We implemented new features like custom domains, scaled bcrypt operations, vastly improved user search, and much more.

• The number of services that compose our product in order to scale our organization and handle the increases in traffic went from under 10 to over 30 services.

• The number of cloud resources grew immensely as well; we used to have a couple dozen nodes in one environment (US), now we have more than a thousand over four environments (US, US-2, EU, AU).

• We doubled-down decided to use a single cloud provider for each of our environments and moved all our public cloud infrastructure to AWS.

Core service architecture

Click to read more ...

Stuff The Internet Says On Scalability For August 24th, 2018

Hey, it's HighScalability time:

Images from a far flung galaxy? Nope. It's the mind blowing swirling beauty of ink in motion. 

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• $10 billion: damages in worlds largest cyberattack; .5%: bitcoins use of all the electricity on earth; 1/200th: Verizon throttling California firefighters for leverage; 4.6%: YC companies reaching $100M+ valuation; 45: ave age of successful startup founder; $250,000: monthly take from browser-based Monero mining; 300+: 3D digitizatized Greek and Roman sculptures; 80: employees are chipped at a company; 100k: bike graveyard from failed startups; 70%: executives think they are block chain experts; $7 billion: Slack valuation; 120: AWS instance types; 27.6 petabytes: Microsoft’s undersea data center webcam of swimming fish; 42%: product is the reason startups fail; $334bn: tech and telecom M&A market; 6x: Nvidia's new GPU; 50%: Cisco's revenue is now subscription based; 


• Quotable Quotes:
  • @hichaelmart: The biggest problem I have with all the "best practices" assertions recently are that almost none of them seem to be accompanied with any data. We do *many* of these "non-best-practices" at large scale at @bustle The "connections take time" one especially irks me. HTTP anyone?
  • @JoeEmison: The history of web-scale companies says: at scale you build all your own stuff. Not at scale, you should leverage the best of what you can pay-by-use for.
  • @RealSexyCyborg: You don't give lessons on the internet in how to avoid Chinese law enforcement.
  • @rsingel: The fight against net neutrality is a fight to monetize scarcity. ISPs have no interest in a world of fast, cheap, ubiquitous connections. That road leads to being a commodity
  • chx: Is Amazon AWS the new reality distortion field? Gosh, just rent two dedicated boxes, one master, one slave, switch over to slave manually in the extreme rare case if the master fails and be done. This entire article screams "right tool for the job and this is not the right tool".
  • Liz Pelly: The result of this experiment: I found Spotify’s most popular and visible playlists to be staggeringly male-dominated. Not only this, I approached the project by listening from a brand new account in order to confirm that gender bias would be reproduced by way of algorithmic recommendations—that when a user listens to mostly male-dominated playlists, what is produced are yet more male-dominated playlists.
  • @dhh: "Wherever per-person Facebook use rose to one standard deviation above the national average, attacks on refugees increased by about 50 percent", Facebook usage is directly correlated with hate crimes in bombshell study from German. 
  • So many more quotes. Keep reading to become the person you always knew you could be.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

What do you believe now that you didn't five years ago? Centralized wins. Decentralized loses.

Decentralized systems will continue to lose to centralized systems until there's a driver requiring decentralization to deliver a clearly superior consumer experience. Unfortunately, that may not happen for quite some time.

I say unfortunately because ten years ago, even five years ago, I still believed decentralization would win. Why? For all the idealistic technical reasons I laid out long ago in Building Super Scalable Systems: Blade Runner Meets Autonomic Computing In The Ambient Cloud.

While the internet and the web are inherently decentralized, mainstream applications built on top do not have to be. Typically, applications today—Facebook, Salesforce, Google, Spotify, etc.—are all centralized.

That wasn't always the case. In the early days of the internet the internet was protocol driven, decentralized, and often distributed—FTP (1971), Telnet (<1973), FINGER (1971/1977),  TCP/IP (1974), UUCP (late 1970s) NNTP (1986), DNS (1983), SMTP (1982), IRC(1988), HTTP(1990), Tor (mid-1990s), Napster(1999), and XMPP(1999).

We do have new decentalized services: Bitcoin(2009), Minecraft(2009), Ethereum(2104), IPFS(2015), Mastadon(2016), and PeerTube(2018). We're still waiting on Pied Piper to deliver the decentralized internet. 

On an evolutionary timeline decentralized systems are neanderthals; centralized systems are the humans. Neanderthals came first. Humans may have interbred with neanderthals, humans may have even killed off the neanderthals, but there's no doubt humans outlasted the neanderthals.

The reason why decentralization came first is clear from a picture of the very first ARPA (Advanced Research Projects Agency) network, which later evolved into the internet we know and sometimes love today:

Click to read more ...

Sponsored Post: NationBuilder, Twitch, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr, MemSQL

Who's Hiring? 

• NationBuilder — if you’re a systems engineer, SRE or DevOps focused developer and have been looking for a place where you can help other people while still working in tech? We can give that opportunity. Please apply here. 


• Twitch's commerce team in San Francisco is looking to hire senior developers to keep up with rapidly increasing demand for our Subscriptions and Payment platform. Engineers will be tasked with building new products and features to solve business and ecommerce challenges as we're dealing with engaging problems at a massive scale and will create solutions that impact millions of people around the world. Apply here. 


• Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

• For heads of IT/Engineering responsible for building an analytics infrastructure, Etleap is an ETL solution for creating perfect data pipelines from day one. Unlike older enterprise solutions, Etleap doesn’t require extensive engineering work to set up, maintain, and scale. It automates most ETL setup and maintenance work, and simplifies the rest into 10-minute tasks that analysts can own. Read stories from customers like Okta and PagerDuty, or try Etleap yourself.

• Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

• Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.


• MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.


• Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

---

Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to see if they can help you scale your career faster.

---

The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today. 

---

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Stuff The Internet Says On Scalability For August 17th, 2018

Hey, it's HighScalability time:

The amazing Zoomable Universe from 10^27 meters—about 93 billion light-years—down to the subatomic realm, at 10^-35 meters.

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• 2.24x10^32T: joules needed by the Death Star to obliterate Alderaan, which would liquify everyone in the Death Star; 13 of 25: highest paying jobs are in tech; 70,000+: paid Slack workspaces; 13: hours ave american sits; $13.5 million: lost in ATM malware hack; $1.5 billion: cryptocurrency gambling ring busted in China; $8.5B: Auto, IoT, Security startups; 10x: infosec M&A; 1,000: horsepower needed to fly a jet suit; 30% Google's energy savings from AI control of datacenters;


• Quotable Quotes:
  • The Jury Is In: From the security point of view, the monolithic OS design is flawed and a root cause of the majority of compromises. It is time for the world to move to an OS structure appropriate for 21st century security requirements.
  • @coryodaniel: Rewrote an #AWS APIGateway & #lambda service that was costing us about $16000 / month in #elixir. Its running in 3 nodes that cost us about $150 / month. 12 million requests / hour with sub-second latency, ~300GB of throughput / day. #myelixirstatus !#Serverless...No it’s not Serverless anymore it’s running in a few containers on a kubernetes cluster
  • @cablelounger: OH: To use AWS offerings, you really need in-house dev-ops expertise vs GCP, they make dev ops transparent to you     I've a lot of experience with AWS and mostly agree with the first point. I haven't really used GCP in earnest. I'd love to hear experiences from people who have?
  • @allspaw: engineer: “Unless you’re familiar with Lamport, Brewer, Fox, Armstrong, Stonebraker, Parker, Shapiro...(and others) you don’t know distributed systems.” also engineer: “I read ‘Thinking Fast and Slow’ therefore I know cognitive psychology and decision-making theory.”
  • alankay1: To summarize here, I said I love "Rocky's Boots", and I love the basic idea of "Robot Odyssey", but for end-users, using simple logic gates to program multiple robots in a cooperative strategy game blows up too much complexity for very little utility. A much better way to do this would be to make a "next Logo" that would allow game players to make the AI brains needed by the robots. So what I actually said, is that doing it the way you are doing it will wind up with a game that is nxot successful or very playable. Just why they misunderstood what I said is a bit of a mystery, because I spelled out what could be really good for the game (and way ahead of what other games were doing). And of course it would work on an Apple II and other 8 bit micros (Logo ran nicely on them, etc.)
  • Michael Malone: Nolan was the first guy to look at Moore’s law and say to himself: You know what? When logic and memory chips get to be under ten bucks I can take these big games and shove them into a pinball machine.
  • @hichaelmart: To be honest, I think the main lesson from this is that API Gateway is expensive – 100% agree. We have a GAE app doing a very similar thing, billions of impressions/mth – and *much* cheaper than if it were on API Gateway.
  • Keep on reading for many more quotes hot off the internet. You'll be a better person.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

What do you believe now that you didn't five years ago?

Decentralized systems will continue to lose to centralized systems until there's a driver requiring decentralization to deliver a clearly superior consumer experience. Unfortunately, that may not happen for quite some time.

I say unfortunately because ten years ago, even five years ago, I still believed decentralization would win. Why? For all the idealistic technical reasons I laid out long ago in Building Super Scalable Systems: Blade Runner Meets Autonomic Computing In The Ambient Cloud.

While the internet and the web are inherently decentralized, mainstream applications built on top do not have to be. Typically, applications today—Facebook, Salesforce, Google, Spotify, etc.—are all centralized.

That wasn't always the case. In the early days of the internet the internet was protocol driven, decentralized, and often distributed—FTP (1971), Telnet (<1973), FINGER (1971/1977),  TCP/IP (1974), UUCP (late 1970s) NNTP (1986), DNS (1983), SMTP (1982), IRC(1988), HTTP(1990), Tor (mid-1990s), Napster(1999), and XMPP(1999).

We do have new decentalized services: Bitcoin(2009), Minecraft(2009), Ethereum(2104), IPFS(2015), Mastadon(2016), and PeerTube(2018). We're still waiting on Pied Piper to deliver the decentralized internet. 

On an evolutionary timeline decentralized systems are neanderthals; centralized systems are the humans. Neanderthals came first. Humans may have interbred with neanderthals, humans may have even killed off the neanderthals, but there's no doubt humans outlasted the neanderthals.

The reason why decentralization came first is clear from a picture of the very first ARPA (Advanced Research Projects Agency) network, which later evolved into the internet we know and sometimes love today:

Click to read more ...

Stuff The Internet Says On Scalability For August 10th, 2018

Hey, it's HighScalability time (out Thur-Fri, so we're going early):

London Maker Faire 1851—The Great Exhibition—100,000 objects, displayed along more than 10 miles, by over 15,000 contributors.

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• 90%: accuracy predicting gender from retinal image; $1 billion: Ebay sales per quarter from AI; $78 billion: global AI software market by 2025; $75m: penalty for botched SAP upgrade; 35 million: m^3 of mud dredged out of the Dutch waterways; 138 terabytes: memory per square inch; 500 million: Uber metrics per second; 22x: new faster JSON Sparser; 


• Quotable Quotes:
  • @IanColdwater: The JIRA tickets will continue until morale improves
  • @david_perell: Three crazy stats from @mikedariano’s newsletter. 1. People watch more Minecraft hours than the NBA, NHL, NFL, and MLB combined.  2. Only 26 countries have more people than PewDiePie has subscribers.  3. Only 20% of YouTube’s traffic is from the United States. 
  • Charlie Demerjian: Why does SemiAccurate say that Intel knows? We have seen their internal documents that show exactly how frightened the company is. The documents go into specifics we don’t feel are appropriate to discuss publicly but there is one thing we can say, Intel knows their position. One of the documents says in no uncertain terms that the company understands they will not be competitive in the server market until AFTER Sapphire Rapids, the 2022 server part. AMD has a clear run in Intel’s core market for at least 4 years.
  • @ScottMcGready: Can we just take a moment to remember that one company I worked for backed up their stuff on tapes religiously- all tapes sent to a warehouse nightly. Years later someone tested a tape... turns out nothing had been written... ever. We had a (paid) warehouse full of empty tapes
  • Uber: Since 2016, Uber has added several new lines of business to its platform, including Uber Eats, Uber Freight, and Jump Bikes. Now, we complete over 15 million trips a day, with over 75 million monthly active riders. In the last eight years, the company has grown from a small startup to 18,000 employees across the globe.
  • @mims: Here is a super important thing that we don't talk about enough: Almost all of the increase in income inequality from 1978 to the present can be accounted for by the difference in wages between top performing firms and everyone else. And now we have some idea what's driving unequal growth in productivity of top-performing firms -- it's how they build and use *their own, proprietary software and other IT/technology
  • @taotetek: Distributed systems tip: Write your system without any queues first. You might find you don't need queues. If you end up needing queues, the retry and reliability code you wrote in order to function without queues will still make your system more reliable.
  • @theburningmonk: I think the visual flow is sometimes under-appreciated - our app support team can easily look at it and figure out what went wrong without knowing ins & outs of implementation details. I can also show the diagram to a product person and he/she would get it as well
  • John Mark: It’s time to understand something about open source software development: it is not going to save us. Using or developing more open source software is not going to improve anyone’s lives. Developing open source software is not a public good. It’s not going to result in a fairer or more equitable society. In fact, as currently structured, open source development is part of the problem. 
  • There are a few more quotes. Don't miss them.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...