Stuff The Internet Says On Scalability For April 19th, 2019

Wake up! It's HighScalability time:

Spirit? Smoke? Lightning? Nope. It's a gorgeous LIDAR image showing 1500 years of Willamette River movement (@Blacky_Himself)

Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon. I wrote Explain the Cloud Like I'm 10 for people who need to understand the cloud. And who doesn't these days? On Amazon it has 44 mostly 5 star reviews (102 on Goodreads). They'll learn a lot and hold you in awe.

• 536: IRS tax return submissions per second; 400,000: drone planted trees in a day; 200 million: smart speaker observers installed by year end; 54 million: GoT pirated in first 24 hours; 123,052: kg of crashed human spacecraft on the surface of the moon; 610 pounds: 128 kilobytes of  IBM S/360 core memory; 33%: per account month over month Lambda function growth; $100,000: Netflix bug bounty payout; $1 million: Shopify bug bount payout; ~$2300: cost to transfer 23TB from S3 to Backblaze B2 in 7 hours; 14%: Netflix users share passwords; 88%: believe P != NP; 30-90: minutes saved by StackOverflow per week; 83%: US teens have an iPhone; $2m: Microsoft bug bounty payout; $1 million: made by Colin Cowherd on 331 million Facebook page views, moving more to Instagram; 0: lines of code in Pong; ~95%: redis is slower when GDPR complient;


• Quotable Quotes:
  • melodysheep: The universe has only just begun.
  • @matthew_d_green: I spent the year before Heartbleed visiting important people in DC trying to convince them OpenSSL was a mess, and they should fund it as “critical infrastructure”. They laughed and told me that term referred to dams and power plants.
  • Tim Cook: No
  • @asymco: Among 8,000 U.S. high school students surveyed, 83% have an iPhone, 9% Android. 86% plan their next phone to be an iPhone. -Piper Jaffray Taking Stock With Teens survey
  • @dialtone: Do you know S3 throughput is higher than a SATA 3 controller and almost as much as PCIe 4.0 2x? Depending on the usecase, mounting S3 as a filesystem, not only makes sense, but it saves a LOT of money (and time) as well. You just need to use it a special kind of filesystem.
  • Steven Melendez: By 2022, automation will displace about 75 million jobs worldwide. On the other hand, they will create an estimated 133 million new jobs. The predictions come from extrapolating from surveys sent to more than 300 major employers around the world. 
  • Charlie Demerjian: Then Apple suddenly caved. And paid Qualcomm handsomely for their troubles. And agreed to buy 5G modems from Qualcomm for the next 6-8 years. Qualcomm beat Apple like a drum because they were in the right and Apple was in the wrong. But why did Apple fold now? Why didn’t they drag things out for another few decades in court through appeals, delays, and the spurious tactics they deployed against Samsung? There is a really good reason for that, Apple was screwed and would have lost the iPhone market if they had waited any longer. Qualcomm had Apple over a barrel and just had to wait. The longer Apple postured and threatened, the stronger Qualcomm’s position got, and likely the bigger the check Apple had to write was.
  • @jolson88: I think the half-page of code in the LISP 1.5 manual about its metacircular evaluator would have to be included. I agree with Alan Kay that it's basically as close as we'll ever get to a "Maxwell's Equations for Software".
  • HBR: Thales Teixeira, associate professor at Harvard Business School, believes many startups fail precisely because they try to emulate successful disruptive businesses. He says by focusing too early on technology and scale, entrepreneurs lose out on the learning that comes from serving initial customers with an imperfect product. He shares how Airbnb, Uber, Etsy, and Netflix approached their first 1,000 customers very differently, helping to explain why they have millions of customers today.
  • Google: We were happy to find no difference in the effectiveness, performance ratings,  or promotions for individuals and teams whose work requires collaboration with colleagues around the world versus Googlers who spend most of their day to day working with colleagues in the same office. 
  • Bob Sutton: People make better decisions when they look into the future and they imagine that they already failed, and they tell a story about what happened. With better planning, it won’t be a story that has to be bleeped out.
  • ecnahc515: This is my experience with most systems that send webhooks, in particular payments, and subscription management systems. As you've elaborated on in other comments, queuing and periodic retries are generally the best way to handle interacting with what is effectively, an eventually consistent API/system.
  • Twitter: This was a pretty big investigation in the end that included a few engineers and multiple teams but a 25% reduction is [Redis] cluster size is a pretty nice result 
  • @colmmacc: I think right around this minute is just about exactly 5 years since the Heartbleed vulnerability in OpenSSL became public. I remember the day vividly, and if you're interested, allow me to tell you about how the day, and the subsequent months, and years unfolded ...
  • kitsunesoba: IMO, every attempt thus far has approached cross platform entirely the wrong way. They’re all way too focused on providing deeply custom UI. I believe that it may better to instead abstract app navigation as its own separate thing, making it a setting in a config file (e.g. tabbed, hamburger, split-pane tabbed, etc) and restricting UI coding to individual screens, which would have style hinting abilities and could be built via any number of platform agnostic (and perhaps language agnostic) ways. All this would then compile down to native UIKit, Android SDK, UWP, AppKit, etc.
  • @QuinnyPig: Confluent, Datastax, Neo4j, MongoDB, InfluxData, Elastic. It's a who's-who of who AWS has stomped on recently. Partnering with GCP seems a lot less perilous...#GoogleNext19
  • Erik Bernhardsson~ Why software tasks always take longer than you think? While the median blowup factor imputed from this fit is 1x (as before), the 99% percentile blowup factor is 32x, but if you go to 99.99% percentile, it’s a whopping 55 million! One (hand wavy) interpretation is that some tasks end up being essentially impossible to do. In fact, these extreme edge cases have such an outsize impact on the mean, that the mean blowup factor of any task ends up being infinite. This is pretty bad news for people trying to hit deadlines!
  • @MayaKaczorowski: "When Google researchers discovered the Spectre vulnerability, we used live migration to patch every single GCP server with zero downtime for our users. So our customers were protected even before they knew they needed to be" - Brad Calder #GoogleNext19
  • Ted Kaminski: We’re composing things together when we can reason compositionally about the result, and we’re extending when we need non-compositional reasoning.
  • Gregory Travis: I believe the relative ease—not to mention the lack of tangible cost—of software updates has created a cultural laziness within the software engineering community. Moreover, because more and more of the hardware that we create is monitored and controlled by software, that cultural laziness is now creeping into hardware engineering—like building airliners. Less thought is now given to getting a design correct and simple up front because it’s so easy to fix what you didn’t get right later.
  • DSHR: The peer-to-peer architecture of the LOCKSS system is unusual among digital preservation systems for a specific reason. The goal of the system was to preserve published information, which one has to assume is covered by copyright. One hour of a good copyright lawyer will buy, at [2014] prices, about 12TB of disk, so the design is oriented to making efficient use of lawyers, not making efficient use of disk.
  • NASA: Given that the majority of the biological and human health variables remained stable, or returned to baseline, after a 340-day space mission, these data suggest that human health can be mostly sustained over this duration of spaceflight.
  • royjacobs: What I find unfortunate about infrastructure-as-code tooling is that a lot of the tooling isn't actually using code, but instead uses esoteric configuration languages. Indeed, the article refers to Terraform with its custom syntax. Imho tools should use actual code (whether it's TypeScript or Kotlin or whatever) instead of reinventing constructs like loops and string interpolation. Thankfully these tools are getting more popular, because frankly I can't stand configuring another Kubernetes or GCP resource using a huge block of copy/pasted YAML.
  • Mathew Cherukara: The beauty is that this molecular model has no right to be as accurate as the atomistic models, but still ends up being so.
  • Geoff Huston:  By pushing client-side DNS queries into HTTPS the Internet itself has effectively lost control of the client end of DNS, and each and every application, including the vast array of malware, can use DOH [DNS over HTTPS] and the DNS as a command and control channel in a way that is undetectable by the client or client's network operator. Much of today's malware containment frameworks, including DNS firewalling, are rendered useless by DOH. Whether or not the browser has DOH enabled by default, applications can generate DOH requests for DNS resolution in a manner that bypasses today's DNS-based malware containment mechanisms. As has been recently observed on a DOH-related mailing list: "Pandora's box is now open and DOH has escaped, and there seems to be little we can do about it now. The times they are a changing."
  • @AssaadRazzouk: Incredible Shrinking Battery Costs: Electric vehicles crossover point - when EVs  are cheaper than their polluting equivalents — was 2026 in 2017, 9 years out; then 2024 in 2018; and now 2022, just 3 years out. On current trends, expect parity next year
  • @obra: 6 hours later, I have replaced my 15 line perl script with a 500 meg Docker image and a few hundred lines of python. It works almost as well as what I had before, too!
  • @srhtcn: #Serverless 5 takeaways: - Still need great engineers. - LessOps, not NoOps - Cheaper, even at scale - Can do many use-cases - Vendor lock-in is often a myth #Serverless 5 takeaways:- Still need great engineers.- LessOps, not NoOps- Cheaper, even at scale- Can do many use-cases- Vendor lock-in is often a myth
  • Kieren McCarthy: So what does Polystream do instead? It streams game data and leaves the graphics processing to your device's GPU to sort out. The result? A fraction of the cost per user.
  • @mcmillen: One of the worst management red-flags that I ever saw at Google was the time a large team was forcibly re-orged into Nest & were told that they weren't allowed to transfer out for N months. Not the best way to motivate engineers & forever tarnished my opinion of Nest leadership.
  • Mike Titus: Don’t tell me that Amazon Cloud and Google Cloud, they wouldn’t love to have our [Black-Hole Picture] data and store it for us. Too much data and too much money—that’s why we don’t do it that way. Nothing beats the bandwidth of a 747 filled with hard disks.
  • @abbyfuller: someone just non-ironically suggested sharding database volumes across containers and i have heartburn now
  • @mipsytipsy: If you're scared of pushing to production on Fridays, I recommend reassigning all your developer cycles off of feature development and onto your CI/CD process and observability tooling for as long as it takes to ✨fix that✨.
  • @kelseyhightower: The OS abstracts away the machine while leaking hardware faults. Docker abstracts away the OS while leaking software faults. Kubernetes abstracts away multiple machines while leaking the distributed system faults. We are the plumbers.
  • Jeffrey Zeldman: And internet investors don’t want a modest return on their investment. They want an obscene profit right away, or a brutal loss, which they can write off their taxes. Making them a hundred million for the ten million they lent you is good. Losing their ten million is also good—they pay a lower tax bill that way, or they use the loss to fold a company, or they make a profit on the furniture while writing off the business as a loss…whatever rich people can legally do under our tax system, which is quite a lot.
  • Gill Lee: Emerging memories like MRAM, PCRAM and ReRAM are leading candidates to complement and, in some cases, replace today’s mainstream technologies—promising higher performance, lower power and lower cost.
  • @davidbrunelle: The Starbucks web team deploys to production on a daily basis. We made a conscious decision to optimize for the ability to catch and resolve issues quickly. Most new features are either toggled off, or kept out of the production build until we want them exposed to customers. Shipping more frequently has a few benefits for us: 1. Each deployment is limited in size. Less risk and easier to isolate issues. 2. Deployments become routine and low stress. Almost non-events. 3. Much faster feedback cycles. 4. Everyone on the team becomes familiar with prod 6 replies 32 retweets 139 likes
  • Ahmed Kabil: After the French Revolution, when large parts of the cathedral were desecrated and damaged, oak trees from Versailles were used to rebuild it. The oaks that were planted thereafter were intended to be used to help rebuild Notre Dame, should it become necessary in the future.
  • Paul Johnston: A serverlss Application is one that provides maximum business value over it's application lifecycle and that costs you nothing to run if nobody is using it, excluding data storage costs. 
  • David Baker: Humans have only been able to harness the power of proteins by making very small changes to the amino acid sequences of the proteins we’ve found in nature. This is similar to the process that our Stone Age ancestors used to make tools and other implements from the sticks and stones they found around them.
  • Erika Hamden: I work on FIREBall because what I want to take our view of the universe from one of mostly darkness, with just light from stars, to one where we can see and measure nearly every atom that exists.
  • BRIAN GALLAGHER: Another person’s mind comes through their mouth. 
  • Geoff Tate: In the edge, you’ve may have one camera, which could be a surveillance camera, a camera in your robot or your set-top box, and it’s processing one image. Any architecture that uses large batch sizes to get high throughput is disqualified in the edge. You should be able to do a good job with processing one image at a time, which is also known as batch size equals one. To be in an edge device, you’ve got to be single-digit watts. You’re not going to put an Nvidia Tesla T4 card at $2,000 and 75 watts into your surveillance camera because it’s too much power. But the people at the edge want to do real-time. Lots of detection and recognition, and processing bigger images on the tougher models, is what gives them better prediction accuracy. They want to get as much throughput as they can or fewer watts for their dollar budget.
  • Rob Matheson: MIT researchers have designed a novel flash-storage system that could cut in half the energy and physical space required for one of the most expensive components of data centers: data storage. In a paper being presented at the ACM International Conference on Architectural Support for Programming Languages and Operating Systems, the researchers describe a new system called LightStore that modifies SSDs to connect directly to a data center’s network — without needing any other components — and to support computationally simpler and more efficient data-storage operations.
  • Matt Hayes: Using what they call DASH (DNA-based Assembly and Synthesis of Hierarchical) materials, Cornell engineers constructed a DNA material with capabilities of metabolism, in addition to self-assembly and organization – three key traits of life.
  • Paul McLellan: It turns out that a similar approach seems to work for photonic designs, although it goes under the name inverse design: you say what you want the photonic device to do, and then the system experiments until it hones in on an approach that works. It also turns out that, like the chair example I pictured above, you end up with designs no human would come up with. They are literally superhuman designs.
  • @fcosta_oliveira: just finished plotting the # io-threads vs RPS on @antirez Redis threaded-io branch single instance benchmark results, using TCP loopback and pipelining, on @GCPcloud n1-highcpu-96. Near 300K OPS on GETs. If we use pipelining we surpass 1.2M OPS. (1/3)
  • @Carnage4Life: There's a book by Eric Schmidt where he mentions Sundar had the idea to recoup acquisition cost of Google Earth by having it install Google toolbar & hijack search defaults. They made hundreds of millions. This stuff is literally my job to know. 
  • Jennifer Valentino-DeVries: Law enforcement officials across the country have been seeking information from a Google database called Sensorvault — a trove of detailed location records involving at least hundreds of millions of devices worldwide
  • @kelseyhightower: FaaS Monolith: a collection of functions disguised as nano services, behind a single API gateway, leveraging the same database.
  • Maggie Koerth-Baker: To get a picture of the black hole, itself, the EHT project used a network of 10 Earthbound radio telescopes, linked together to function as a single system. The telescopes collected high-frequency radio waves from space, and four independent teams of scientists used algorithms to convert the radio signals into visual images.
  • @cgervais: One of our [Kyrus] Engineering teams started with #serverless last summer with @awscloud Lambda services and it turned out super-successful. So much so that two more teams are migrating their services as well. If you’re interested,we’re doing more with #serverless
  • @IamStan: Google Cloud Run: For those people that want to go Serverless, but really can't let go of their Dockerfiles. Playing with it this morning... Initial thoughts... To use it, you need to create a Dockerfile, and you need to handle  HTTP traffic (ie run Express or some sort of http server)... These are two things you don't need with Lambda or other Cloud Function platforms. This will make transition easier for developers who are used to running their own http server, and are comfortable with Docker
  • @davidgerard: "I used to work at Tumblr, the entirety of their user content is stored in a single multi-petabyte AWS S3 bucket, in a single AWS account, no backup, no MFA delete, no object versioning. It is all one fat finger away from oblivion."
  • @newstodayohboy: If YouTube or Facebook are distributed computer systems where the CPUs are people, then a bad actor thinks of finding exploits in the people... cognitive biases become as important as programming errors.
  • Brent Ozar: 15 Reasons Your Query Was Fast Yesterday, But Slow Today. There are different workloads running on the server (like a backup is running right now.) You have a different query plan due to parameter sniffing. The query changed, like someone did a deployment or added a field to the select list...
  • R Danes: At the Cloud Next conference in San Francisco yesterday, Google signaled it’s ready to meet customers where they are with ready-to-wear products. It is tackling a main enterprise concern — hybrid and multicloud — with new offerings like the Anthos platform for managing applications across environments.
  • lget: I am not surprised. I was working in an open-plan office for a few years, and I also preferred talking to colleagues sitting only a few meters away via mail. It's incredibly stressful to talk to a person face-to-face knowing that everyone on the team listens to the conversation. Not only do you have to constantly weight and evaluate your sentences, you also have the constant feeling that you are disturbing other team members just by talking. On the other hand, you also don't want to drag a person into another room for privacy if you just want some quick update on something. 
  • @tmclaughbos: If you're building custom software to be given to and run by a client, and you want to go serverless, I'm going to suggest you think long and hard about the operability of that software by your client. Hint: You should probably just stick code into a container and hand them a docker image. It'll be a lot easier for you and the customer.
  • @CJHandmer: Do you ever lie awake and night and wonder how much cargo a SpaceX Starship could deliver to the moon, and then from the moon back to the Earth, without refueling on the Moon? Lots!! Amidst all the recent NASA Moon chat it's easy to overlook just how transformative Starship is.
  • @antirez: Because who will replace us needs to find something inside herself/himself. Role models are not really sufficient or even needed, we need a way to understand how to put inside our children that thing that will make them adults that want to *do something*. The rest will follow.
  • C4ADS: found that GNSS spoofing activities in the Russian Federation, its occupied territories, and its overseas military facilities are larger in scope, more geographically diverse, and started earlier than any public reporting has suggested to date. Reports by CNN and the RNT Foundation identify fewer than 450 vessels affected since late 2016. Using Automatic Identification System (AIS) ship location data collected at scale, C4ADS identified 9,883 instances of GNSS spoofing that affected 1,311 commercial vessels beginning in February 2016. The disruptions appear to have originated from ten or more locations in Russia and Russian-controlled areas in Crimea and Syria.
  • @johnath: Over and over. Oops. Another accident. We’ll fix it soon. We want the same things. We’re on the same team. There were dozens of oopses. Hundreds maybe? I’m all for “don’t attribute to malice what can be explained by incompetence” but I don’t believe google is that incompetent.
  • @hcoyote: You have to admit … watching AWS Red Wedding is simultaneously horrible and amazing. Horrible in that  … all those poor people seeing their hopes and dreams crushed. Amazing in that … I can’t believe they get away with it. I can’t bring myself to look away.
  • IRS: We have about 60 different applications. I think we have about 12,000 or 13,000 servers on 12 mainframes. It’s difficult to continually patch. At some point, we need to replace and we’re definitely at that point. We’re as well posed as I think we’ve ever been to — and I was somewhat familiar with this when I was on the outside — but ours is as well posed as it’s ever been to be able to modernize both the infrastructure as well as the language so we’re moving forward with the ability to be agile and flexible as newer technologies come along.
  • MICHAEL FORTE: On top of this idiosyncratic population bias due to uneven population growth rates, there is a more persistent early adopter bias. These early adopters tend to be much more tech-savvy than the general population, trying out new products to be on the cutting edge of technology. This tech-savvy population desires features that can be detrimental to the target population. In our music example, tech-savvy users will want to select the specific bit-rate and sampling frequency of the song they are buying, but forcing our target population through this flow would lead to confusion and decreased conversion rates.
  • rchaud: FB's public comments about these remind me a lot of the "5 Standard Excuses" scene in the '80s BBC sitcom Yes Minister, where a civil servant lists the best CYA mea culpas for politicians to use when something goes wrong. 1. It occurred before certain important facts were known, and couldn’t happen again 2. It was an unfortunate lapse by an individual, which has now been dealt with under internal disciplinary procedures. 3. There is a perfectly satisfactory explanation for everything, but security forbids its disclosure. 4. It has only gone wrong because of heavy cuts in staff and budget which have stretched supervisory resources beyond their limits. 5. it was a worthwhile experiment, now abandoned, but not before it had provided much valuable data and considerable employment

• What can a single person do these days? So much. Designing a modern serverless application with AWS Lambda and AWS Fargate: Recently I built and open sourced a sample application called changelogs.md. The application watches for open source packages on NPM, RubyGems, and PyPI. When a package is added or updated it crawls any changelog found in the package source code.
  • This is a crawler built on in serverless environment and makes full use of AWS services: CloudFront, ALB, S3, Fargate, ElastiCache, Redis Pub/Sub, SNS, API Gateway, Lambda, DynamoDB. Redis Pub/Sub messages get published back to your browser via a Docker container running Socket.io on AWS Fargate.
  • Why Fargate? There's no reason to make an all-in choice on only one type of compute. Run code where it runs best. In changelogs.md short lived compute jobs are all being run in AWS Lambda. Long compute jobs that have no designated end run on Fargate. A design point you might not have considered is isolation: But it’s easy to imagine that someone may try to attack this component by creating a massive changelog that is gigabytes in size, or take advantage of an edge case in the parser to consume large amounts of resources and DDOS the parsing code. So the Lambda model of execution is perfect for these jobs because it keeps the invokes isolated from each other. If someone triggers a crawl on a malicious changelog that single Lambda execution might timeout but the rest of the system won’t experience any impact, because the individual Lambda invokes are separated from each other and the rest of the stack.
  • As a comparion here's The Anatomy of a Large-Scale Hypertextual Web Search Engine by Sergey Brin and Lawrence Page. You might recognize those names. Obviously a search engine is different, but there's an essential complexity related to crawling that both share.
  • Another interesting aspect is the use of AWS Cloud Development Kit (CDK) instead of CloudFormation: I find that CDK is at the right level of abstraction for my applications: it doesn’t stop me from using the core cloud native services that I want to use, and it doesn’t require me to write a ton of boilerplate to use those services.
  • Did it work? Depnds on the goals. The goals were minimal ops and minimal costs. Those were met: On average the price to run the service is about $3.80 a day. For about the price of my daily cup of coffee I can have an application crawling more than 80k open source repos a day. Here are some more fun stats: every month the application uses 40 million DynamoDB read units and 10 million write units, serves several 100k website visits using S3 and CloudFront, does about one million Lambda invokes and delivers over 500k notifications via SNS. It has two Docker containers and a small Redis instance running 24/7 all month. And remember all of this costs less than $4 a day!


• Only star football players have the leverage necessary to demand a new contract. Not everyone has the leverage to port their social network over to a new platform, but it's time content producers realize distribution is not the key part of the value chain anymore—they are. Distribution is free, global, and direct to customers over the internet. Unbundle yourself. You are the only differentiated source of you. You are the app. Disaggregate the aggregators! Use social networks like Instagram as on ramps to your own differentiated social network of you. Influencers are flocking to a surprising new kind of social media.
  • 350+ influencers with a collective audience of 3.5 billion people are flocking to a platform called Escapex, which gives them their own apps. It’s part of the next wave of social media focused on smaller, more private groups
  • For Abigail Ratchford, a glam model who posts sexy images of herself on Instagram to an audience of 8.9 million, this control has freed her up to share as much content as she wants with her biggest fans, who pay $9.99 a month for the privilege of accessing it, without living in fear of the Instagram algorithm
  • These personal apps aren’t just for influencers. Osric Chau, an actor who is best known for his role in the cult TV show Supernatural [a very minor character], has 624,000 followers on Instagram and launched his app in January 2018. He uses it for a very different purpose: to connect with his fans more intimately. While he charges $4.99 for a subscription
  • One key feature of Escapex apps is that fans like Enna earn points based on how much they engage with the app, including the fan feed (points are also available for purchase). Then, they can use those points to boost their comment on one of Chau’s posts so that he’s guaranteed to see it. According to Shapira, that’s the primary way that celebs like Jeremy Renner monetize their apps–all the content is free, but people pay to be seen
  • Being a superfan by yourself is a diminished experience. Imagine a huge sports fan who lives in a country where no one follows the sport. It’s part of your identity and connecting with others who share that identity is really important. It’s the major driver of fandom, which is why you see the same processes across all these different celebrities
  • With the future of major social networks unclear, going independent on social media is a savvy financial bet. “I would not be surprised at all if there’s a handful of people who are making crazy livings on apps in the future"


• Instagram promises to tell you how they hold back the thundering herd. Thundering Herds & Promises: At Instagram, when turning up a new cluster we would run into a thundering herd problem as the cluster’s cache was empty. We then used promises to help solve this: instead of caching the actual value, we cached a Promise that will eventually provide the value. When we use our cache atomically and get a miss, instead of going immediately to the backend we create a Promise and insert it into the cache. This new Promise then starts the work against the backend. The benefit this provides is other concurrent requests will not miss as they’ll find the existing Promise — and all these simultaneous workers will wait on the single backend request. The net-effect is you’re able to maintain your assumptions around caching of requests. Assuming your request distribution has the property that 90% are cache-able; then you’d maintain that ratio to your backend even when something new happens or your service is restarted.


• 3DXPoint’s most intriguing application is as a byte-addressable persistent memory that user space applications map into their address space (with the mmap() system call) and then access directly with loads and stores. Early Measurements of Intel’s 3DXPoint Persistent Memory DIMMs: The most critical difference between 3DXPoint and DRAM is that 3DXPoint has longer latency and lower bandwidth.  Load and store performance is also asymmetric. On average, random loads take 305 ns compared to 81 ns for DRAM.  For sequential loads, latencies are 169 ns, suggesting some buffering or caching inside the 3DXPoint DIMM. [Write] latency is 94 ns for 3DXPoint compared to 86 ns for DRAM.  For reads (at left), bandwidth peaks at 39.4 GB/s.  For writes (at right), it takes just four threads to reach saturation at 13.9 GB/s. Performance for read and write rises quickly until access size reaches 256 B and slowly climbs to a peak of 1.5 GB/s for stores and 2.8 GB/s for loads. 256 B is 3DXPoint’s internal block size. It represents the smallest efficient access granularity for 3DXPoint. Loads and stores that are smaller than this granularity waste bandwidth as they have the same latency as a 256 B access. Stores that are smaller also result in write amplification since 3DXPoint writes at least 256 B for every update, incurring wear and consuming energy. The figure also shows the benefits of building native, memory-mapped, persistent data structures for Redis and RocksDB.  The impact varies widely: performance for RocksDB increases by 3.5×, while Redis 3.2 gains just 20%. 


• Taking programming up a notch. Facebook build a recommendation engine to how other programmers solved similar problems. Aroma: Using machine learning for code recommendation. Something like this for GitHub would be great. Though it would be interesting to know how well it works in practice. Do Facebook developers actually find it useful?


• We now have more ways than ever of packaging up and running bits of serverless functionality. Lucet from Fastly is the most recent: "With Lucet, Fastly’s edge cloud can execute tens of thousands of WebAssembly programs simultaneously, in the same process, without compromising security. The Lucet compiler and runtime work together to ensure each WebAssembly program is allowed access to only its own resource." It doesn't use a VM, container, or isolate, it: @jedisct1: "Not a V8 sandbox. This is a new runtime, written from scratch in Rust, specifically designed for concurrency and low latency. It’s open source, and included in Lucet." @nathankpeck~ "I think the future of serverless is the exact opposite direction. Right now the most interesting platform I've seen is Fastly Lucet …Long story short it compiles Rust, TypeScript, C, and C++ to WASM which executes in a V8 sandbox, only 50micros of overhead." It's still early days so we don't know much about how it works in practice, but it's good to see competition. Also, Running Unikernels in 2019 with OPS. Also also, Wasmer is taking WebAssembly beyond the browser.


• Basic but good intro to the Google Network. The High-Performance Network. 


• How Airbnb Avoids Double Payments in a Distributed Payments System: There are three different common techniques used in distributed systems to achieve eventual consistency: read repair, write repair, and asynchronous repair. Our solution in this particular post utilizes write repair, where every write call from the client to the server attempts to repair an inconsistent, broken state...Write repair requires clients to be smarter (we’ll expand on this later), and allows them to repeatedly fire the same request and never have to maintain state (aside from retries). Clients can thus request eventual consistency on-demand, giving them control over the user experience. Idempotency is an extremely important property when implementing write repair...For an API request to be idempotent, clients can make the same call repeatedly and the result will be the same...We implemented and utilized “Orpheus”, a general-purpose idempotency library, across multiple payments services...An idempotency key is passed into the framework, representing a single idempotent request. Tables of idempotency information, always read and written from a sharded master database (for consistency). Database transactions are combined in different parts of the codebase to ensure atomicity, using Java lambdas. Error responses are classified as “retryable” or “non-retryable”.


• Looks like Murat has made good use of his sabbatical because this is an awesome description of Azure Cosmos DB: Microsoft's Cloud-Born Globally Distributed Database: To realize [Global distribution, Elastic scalability of throughput and storage worldwide, Fine grained multi-tenancy] Cosmos DB uses a novel nested distributed replication protocol, robust scalability techniques, and well-designed resource governance abstractions, and I will try to introduce these next. Good discussion on HN.


• What is Taking Serverless to the Next Level? It involves lots and lots of configuration files. We have all these UIs that do so much checking and cross linking and the best practice is to type text into configuration files. Configuration Driven Programming requires programmers to build a runtime model of the entire AWS ecosystem in their head to understand how their system executes. This is just like programming, only the runtime environment is far more complex and hidden. Is this really progress? Capital One's move to serverless is given as an example. 70% performance gain. 90% cost saving by moving from EC2, ELB, and RDS to DynamoDB, S3, SNS, Lambda. 30% increase in velocity by adopting CI/CD pipeline. 


• AWS Hero Ben Keho takes a look at The Good and the Bad of Google Cloud Run.
  • "So what’s bad about Cloud Run? Inside your container is a fully-featured web server doing all the work! The point of serverless is to focus on business value, and the best way to do that is to use managed services for everything you can, ideally only resorting to custom code for your business logic. If we try to compare Cloud Run to what’s possible inside a Lambda function execution environment, we’re missing the point. The point is that the code you put inside Lambda, the code that you are liable for, can be smaller and more focused because so much of the logic can be moved into the services themselves." 
  • Ben deftly highlights a major philosophical difference between AWS and GCP. GCP tends to promote a world view where programmers generate value through well understood code rather than spending hours if not days coercing tools like API Gateway into doing their bidding. Web servers are simple and well understood. Dispatching a function is easy. Many lambda functions even dispatch internally using similar dispatch code to Express. API Gateway is not simple. It can do a lot, but it's complex, confusing, and expensive. How many hours do you have to waste with velocity templates and a horrible local testing infrastructure to get to the point that running a web server in a container is not the path to a lack of value, but is the path to good local testing, clearer code, better control, and less frustration? Let's not make a virtue out a technology stack choice based on such an undefinable quality like "value." There are many paths to value. It doesn't matter if Cloud Run is FaaS, is like or not like Lambda, or uses Kubernetes—what matters is can you get the job done?
  • How Google Cloud Run Combines Serverless with Containers: My initial observation is that Cloud Run delivers the same promise as the original PaaS. The fundamental difference between PaaS and Cloud Run lies in transparency. Since the underlying layer is based on Knative, every step can be easily mapped to the functionality of Istio and Kubernetes. When Cloud Run becomes available on GKE On-Prem and Anthos, customers will be able to target a consistent platform with a repeatable workflow to deploy modern applications.


• Indeed it is. Nicely done. A Detailed Overview of AWS API Gateway


• Satellites Are Reshaping How Traders Track Earthly Commodities. Using satellites and AI you can track all supply chains in real-time to make your bets using information others do not have. Put your surveillance capitalism hat on. One wonders is all those Google map cars are doing more than making maps? Could they be gathering intel? It's impossible to know their true purpose. We've seen this before in WWII. It's incredible how much effort was put into disinformation. Entire armies were faked to feed misinformation to Axis powers. And it worked. It's all deliciously described in The Deceivers: Allied Military Deception in the Second World War. A blast to read. In the future we're likely to see the same sort of deception techniques used to confuse those who spy from above.


• Good Recap: LinkedIn at EmberConf 2019. 


• The storage wars continue they do. Google now offers ice cold storage. AWS offers Glacier Deep. There's more to consider than just price. tpetry: The interesting part it's cheaper than AWS Glacier ($4 per TB per month) and slightly more expansive than AWS Glacier Deep Archive ($0.99 per TB per month) but the data is available immediately and not in hours like glacier where you have to pay a hefty premium for faster access to the data. Youden: The place where this won't be as cheap as Backblaze is retrieval. Unless Google makes a big change, you'll still have to pay for network egress, which is obscenely priced. Good discussion on HN.


• Because we all know systems are used just for their stated purpose. Tools are always used to suit the purpose of their wielder, not the best hopes of their creators. The Messy Truth About Social Credit: But the social credit system as it currently exists is not aimed at Orwellian social control. Rather, the cluster of policy initiatives designated by the term are intended to promote greater trust—namely, trust between companies and their customers, and between citizens and the government. 


• When you hit a scaling inflection point—both organizationally and customer load related—do you continue refactoring the old system or do you build a new one? Here's everything you should not do. Microservices Gone Wrong. 
  • They chose to build a second new green field product. They chose microservices using containers on Mesos to run Spark, Hadoop, Storm, and apps. Do not treat microservices as objects. Have a couple of services will well defined boundaries. Start as a service as big as possible because it can always be split later. It's hard to stich services together later. Have a dedicated team dedicated to prod and dev infrastructure. Automate everything deployment, migration, backups, state restoration, everything. Gracefully handle failures first, not the happy path. Define service level objectives early in terms of what users care about. Manage complexity is the number one job. You can never eliminate complexity, you can only move it around or add it. It's too easy to create a system so complicated you can't understand it and if you can't understand it how can you hope to run it?
  • David Moore: This talk is dangerous. He's giving a talk advising people to not choose microservices model because he incorrectly applied that pattern to his product and unsurprisingly had issues. This does not warrant advising others to not choose a very successful pattern that works for the biggest tech companies because you didn't understand correctly.  1) The biggest red flag - he decomposed the services by database entity. All you've done at this point is create an extremely coupled ball of mud. Of course "simple refactoring became major coordinated surgery" because each service is a representation of an entity. 2) The second major issue is his primary issues were lack of proper CI/CD infrastructure. Circuit breakers, service mesh, service resilience, scaling the containers, monitoring etc. All of this is needed for a fault tolerant system. You can't just build out some services and expect it to solve your problems because you checked a few boxes. 


• Maybe too complicated. A simple linear call chain works too. Chaining Serverless Functions for Stateful Workflows: AWS Step Functions using Adapter Pattern. Also, Building an AWS Serverless ML Pipeline with Step Functions


• Integrating AI directly into data stores makes a lot of sense All 29 AI announcements from Google. 


• Microsoft/BosqueLanguage: The Bosque language derives from a combination of TypeScript inspired syntax and types plus ML and Node/JavaScript inspired semantics. This document provides an overview of the syntax, operations, and semantics in the Bosque language with an emphasis on the distinctive or unusual features in the language.


• dgryski/go-perfbook: This document outlines best practices for writing high-performance Go code.


• Microsoft/BlingFire: we are a team at Microsoft called Bling (Beyond Language Understanding), we help Bing be smarter. Here we wanted to share with all of you our FInite State machine and REgular expression manipulation library (FIRE). We use Fire for many linguistic operations inside Bing such as Tokenization, Multi-word expression matching, Unknown word-guessing, Stemming / Lemmatization just to mention a few. sergeio76: There are deterministic finite state machines underneath implemented in lean C++. These automata allow to implement operations with strings optimally or close to that. In the readme file there is a link to how to recompile linguistic resources, if you look inside the makefile that is used for resource compilation you will the steps.


• emichael/dslabs (article): a new framework for creating, testing, model checking, visualizing, and debugging distributed systems lab assignments. paper: Students often march though test cases incrementally, fixing problems only once they occur. A particular student tried this for the primary-backup assignment and got stuck: the fix for a problem found by one test would often break the solution for previous tests. The student found he could find a version to pass each of the tests, just not the same version. After we encouraged him to start over with a clean design that met all of the criteria simultaneously, he was able to quickly converge on a solution.


• OverSketched Newton: Fast Convex Optimization for Serverless Systems: Motivated by recent developments in serverless systems for large-scale machine learning as well as improvements in scalable randomized matrix algorithms, we develop OverSketched Newton, a randomized Hessian-based optimization algorithm to solve large-scale smooth and strongly-convex problems in serverless systems. OverSketched Newton leverages matrix sketching ideas from Randomized Numerical Linear Algebra to compute the Hessian approximately. These sketching methods lead to inbuilt resiliency against stragglers that are a characteristic of serverless architectures. We establish that OverSketched Newton has a linear-quadratic convergence rate, and we empirically validate our results by solving large-scale supervised learning problems on real-world datasets. Experiments demonstrate a reduction of ~50% in total running time on AWS Lambda, compared to state-of-the-art distributed optimization schemes.


• Distributed consensus revised: In this thesis, we re-examine the foundations of how Paxos solves distributed consensus. Our hypothesis is that these limitations are not inherent to the problem of consensus but instead specific to the approach of Paxos. The surprising result of our analysis is a substantial weakening to the requirements of this widely studied algorithm. Building on this insight, we are able to prove an extensive generalisation over the Paxos algorithm.


• Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies. Uber on Keeping Master Green at Scale: This paper presents the design and implementation of SubmitQueue. It guarantees an always green master branch at scale: all build steps (e.g., compilation, unit tests, UI tests) successfully execute for every commit point. SubmitQueue has been in production for over a year, and can scale to thousands of daily commits to giant monolithic repositories.


• Analyzing the Impact of GDPR on Storage Systems: We show that despite needing to introduce a small set of new features, a strict real-time compliance (e.g., logging every user request synchronously) lowers Redis’ throughput by ∼95%. Our work reveals how GDPR allows compliance to be a spectrum, and what its implications are for system designers. We discuss the technical challenges that need to be solved before strict compliance can be eciently achieved.


• Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems: Fail-slow hardware is an under-studied failure mode. We present a study of 101 reports of fail-slow hardware incidents, collected from large-scale cluster deployments in 12 institutions. We show that all hardware types such as disk, SSD, CPU, memory and network components can exhibit performance faults. We made several important observations such as faults convert from one form to another, the cascading root causes and impacts can be long, and fail-slow faults can have varying symptoms. From this study, we make suggestions to vendors, operators, and systems designers

Comments

[Privacy lover]

I noticed the site isn’t served through HTTPS. Please make the transition.

[HighScalability Team]

I’d love to but Squarespace does a hard redirect to http on their backend so it causes a redirect loop when I use https through cloudflare.

[Franz Wong]

The hyperlink is mistyped (it includes an extra "l" at the end) :)
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/improving-key-expiration-in-redis.html

[me]

Squarespace seems to support SSL for free, maybe there is just some configuration missing: https://support.squarespace.com/hc/en-us/articles/205815898-Squarespace-und-SSL

[HighScalability Team]

Not for SS5. Believe me, I've been through this.